Retrieval-augmented generation, or RAG, lets a language model answer from documents fetched at question time, such as policies, guidelines or a patient's own notes, instead of from what it absorbed in training. In healthcare, the retrieval step is where most of the privacy and safety work sits.
This guide follows a single question through a RAG pipeline with PHI controls: which model endpoints may receive PHI, how retrieval enforces access rights, how answers cite their sources, how the system is evaluated, and where builds go wrong.
What is RAG in healthcare?
Deciding how to build a RAG assistant?
Agnotic's fractional CTO engagement guides the choice of model endpoint, index design and evaluation plan before any RAG code is written.
RAG in healthcare is an architecture that pairs a search step over approved clinical or administrative content with a language model that writes an answer from the passages found. Done well, every answer cites its sources, retrieval respects the user's access rights, and PHI only reaches model endpoints covered by a business associate agreement.
The term comes from research that combined a retriever over a document index with a generator model. Healthcare teams adopted the pattern because answers can be checked against named documents, and content can be updated by changing the index rather than retraining a model.
Uses fall into two groups. Knowledge assistants answer from shared content such as formularies, payer policies or clinical guidelines, with no PHI in the index. Chart assistants answer from one patient's records, which puts PHI in the index, the prompt and the logs.
Which model endpoints can receive PHI?
Only an endpoint whose provider has signed a business associate agreement with the covered entity, or with the vendor acting for it, should receive PHI. Consumer chat products are not that endpoint.
- OpenAI API: OpenAI signs BAAs for eligible API services configured for zero data retention.
- ChatGPT Enterprise and Edu: OpenAI offers BAAs for these through its sales team. Neither consumer ChatGPT nor ChatGPT Business falls under these agreements.
- Cloud model services: Major cloud providers list which AI services fall under their BAA. Confirm the specific model service and region, not only the provider.
- Embedding endpoints: The embedding call sends the same text that later appears in prompts, so it needs the same coverage whenever chunks contain PHI.
- Vector database hosts: The HHS cloud guidance makes a host that stores ePHI a business associate even when all it holds is encrypted data, and a hosted vector index is such a host.
A knowledge assistant with no PHI in its index or prompts has more hosting freedom. Keep it that way by screening free-text patient details at the input, or by routing those questions to the covered path.
How does a retrieval pipeline with PHI controls work?
A RAG pipeline with PHI controls runs in two phases: ingestion, which prepares the index, and query, which answers a user. Each of the ten stages below carries its own control.
- 1. Source intake: Approved documents arrive from a document management system, a policy library or, for chart assistants, the EHR through FHIR DocumentReference and related resources. Each source has a named owner and a version.
- 2. Chunking and tagging: Documents are split into passages and tagged with metadata: source ID, version, effective date, patient ID where present, and the access groups allowed to read it.
- 3. Embedding: Each passage is converted to a vector through an embedding endpoint covered by a BAA whenever the passage contains PHI.
- 4. Vector index: Vectors and metadata are stored encrypted in an index hosted under a BAA, with correction and deletion jobs that follow the source system.
- 5. User authentication and context: The user signs in through the organization's identity provider, and a chart assistant receives patient context through a SMART on FHIR launch.
- 6. Permission-filtered retrieval: The question is embedded and searched with filters for the user's access groups and the patient in context applied inside the search, before ranking.
- 7. Prompt assembly: Only the passages needed for the answer go into the prompt, with retrieved text marked as content rather than instructions.
- 8. Generation: The prompt goes to a BAA-covered model endpoint under the retention terms agreed in the contract.
- 9. Citation check and response: Each claim in the draft must map to a retrieved passage. Unsupported drafts are replaced with a refusal, and the user sees the answer with links to its sources.
- 10. Audit log: The user, patient, question, retrieved passage IDs, model version and response are recorded for review.
Stages 2, 6 and 10 are where PHI controls usually fail, because they depend on metadata and logging that a prototype often skips. Build them first rather than adding them after a demo succeeds.
Access control at retrieval time
Access control has to run inside retrieval, because anything retrieved can end up in the answer. Filtering the model's output afterwards is too late.
Two designs are common. A shared index stores every passage with access metadata and applies a filter on each query. A partitioned index keeps a separate namespace per patient or department, so a query cannot reach other data at all. Partitioning is simpler to reason about, while a shared index is usually cheaper to run at scale.
- Pre-filter, not post-filter: Apply permission filters in the vector search itself. Retrieving the top results and then removing forbidden ones leaves too few passages and can leak information through ranking.
- Mirror the source: Derive access groups from the EHR or document system, not from a separate list that drifts out of date.
- Follow corrections: When a note is amended or marked entered in error, the matching chunks must be updated or deleted.
- Emergency access: If the EHR allows break-the-glass access, decide whether the assistant honors it, and flag every such retrieval for privacy review.
- Minimum necessary: Retrieve only the record sections the use case requires. The HIPAA minimum necessary standard points the same way.
How should answers cite their sources?
Every answer should point to the exact passage, document and version it came from, and the system should decline when it finds no support. Citations are what let a clinician check the answer instead of trusting it.
- Passage-level links: Link to the chunk and its location in the source document, not only the document title.
- Version and date: Show the document version and effective date, so a superseded policy is obvious.
- Supported or declined: When no retrieved passage supports an answer, say plainly that the sources do not cover the question.
- Visible conflicts: When sources disagree, show both rather than letting the model pick one silently.
Citations also change how the tool is used. A clinician who can open the passage behind an answer can judge it on its merits, and a reviewer auditing an answer weeks later can see exactly what the model was given.
How do you know the system answers correctly?
You know by testing retrieval and generation separately against a fixed set of real questions with known answers, and rerunning that set on every change. A demo with a few good answers proves little.
- Question set: Collect real questions from the intended users, each with the correct answer and its source passage, reviewed by a clinician or policy owner.
- Retrieval recall: Measure how often the correct passage appears in the retrieved set. If retrieval misses it, the model cannot answer correctly.
- Groundedness: Score whether every statement in the answer is supported by a retrieved passage, which is the direct check on confabulation.
- Citation accuracy: Check that each citation points to the passage that actually supports the claim.
- Refusal behavior: Include questions the sources do not cover, and count how often the system correctly declines.
- Adversarial cases: Add documents and questions with embedded instructions to test prompt injection, plus queries that try to reach another patient's data.
Group the test cases by risk: confabulation, data privacy and information security each need their own questions and their own pass mark, and a release should not ship if any group regresses.
Where healthcare RAG systems go wrong
Most RAG failures come from the index and its permissions, not from the model. Several of these failure points match entries in the OWASP Top 10 for LLM Applications 2025.
- Index leaks: A shared index without enforced filters returns another patient's note. OWASP lists vector and embedding weaknesses as LLM08:2025.
- Injected instructions: A scanned letter or patient message contains text telling the model to ignore its rules, which OWASP lists as LLM01:2025 Prompt Injection.
- Stale content: A withdrawn policy stays in the index and keeps being cited.
- Uncovered embeddings: The team signs a BAA for the chat model but sends PHI chunks to an embedding service outside it.
- Logs full of PHI: Prompts and responses land in a general observability tool that is neither covered nor access-controlled.
- Chunks that split meaning: A dosing table or a contraindication is cut in half, so retrieval returns the passage without its condition.
- No audit trail: Nobody can show which passages produced an answer a clinician questioned, although every system that holds ePHI needs audit controls under the HIPAA Security Rule.
How Agnotic can help
Most of the risk in a healthcare RAG project is decided in the architecture: which endpoint, which index design and which evaluation set. Agnotic Technologies helps teams make those decisions through its fractional CTO service, then builds the ingestion pipeline, permission-filtered retrieval, citation display and audit logging for the client.
For a chart assistant, Agnotic also builds the SMART on FHIR launch and the correction and deletion jobs that keep the index in step with the EHR.
Plan your RAG architecture with Agnotic
Explore fractional CTO support for healthcare, or read more articles in the Healthcare AI hub.
Related reading on Agnotic
Continue with generative AI in healthcare, ChatGPT use cases, HIPAA-ready AI apps, human review and SMART on FHIR scopes.
