Generative AI

What is generative AI?

Generative AI is a class of machine learning models, most often large language models, that generate new text, code, images or structured output in response to a prompt and supplied context. Generative AI in healthcare drafts clinical documentation from visit audio, summarizes records, writes prior authorization and appeal letters and drafts patient messages in plain language, with a person reviewing the output before it is used.

How it differs from earlier AI

• Output: Earlier models classified or predicted. Generative models produce new content that reads as if a person wrote it.
• Steering: Behavior is shaped by prompts and retrieved context at run time, not only by training.
• Failure mode: Confident, fluent errors, often called confabulation or hallucination, rather than a low confidence score.
• Data exposure: Prompts and context may contain PHI, so where the model runs and what it retains matters.
• Evaluation: Quality is judged by reviewers and rubrics, not a single accuracy number.

Why does generative AI in healthcare matter now?

• A profile exists: The NIST AI Risk Management Framework (AI 100-1, January 2023) organizes AI risk work into govern, map, measure and manage, and its Generative AI Profile (AI 600-1, July 2024) names twelve risks specific to or amplified by generative AI, including confabulation, and suggests actions for each.
• FDA addressed documentation and decision support: FDA's Clinical Decision Support Software guidance, dated January 2026, explains when software is not a device under section 520(o)(1)(E), adds examples on clinical documentation tools, and treats time-critical and patient-facing recommendations as devices.
• Transparency is certified: ONC certification criterion 170.315(b)(11) requires source attribute information for predictive decision support interventions in certified EHRs.
• Discrimination must be checked: Under 45 CFR 92.210, from May 1, 2025 covered entities must make reasonable efforts to identify patient care decision support tools that use protected characteristics and mitigate the risk of discrimination.
• Vendors are business associates: Under 45 CFR 164.502(e), a model or platform vendor that receives PHI in prompts, audio or documents needs a Business Associate Agreement.
• Application security has a list: The OWASP Top 10 for Large Language Model Applications names risks such as prompt injection and sensitive information disclosure that healthcare deployments must design against.

Where does generative AI fit in healthcare?

The diagram sorts common generative AI uses by how close they sit to a clinical or payment decision, and shows the controls each level needs. The further from the decision, the more latitude; the closer, the more review.

• Engineering and operations: Code assistance, test generation, documentation and internal knowledge search. No PHI, light review.
• Administrative drafting: Prior authorization and appeal letters, referral summaries, patient instructions and inbox replies. PHI under a BAA, staff review before sending.
• Clinical documentation: Ambient notes and visit summaries. Clinician edits and signs before the record changes; edit rates measured.
• Clinical question answering: Guideline and policy answers with citations the clinician can check. Non-device when independently reviewable.
• Decision support: Suggestions about diagnosis or treatment. Subject to the FDA criteria, ONC transparency and Section 1557 checks, with a human in the loop.
• Patient-facing advice: Diagnosis or treatment recommendations straight to patients are device territory and need a regulatory plan.

The risks specific to generative AI

Three risks dominate healthcare deployments. Confabulation: the model states something false in fluent prose, so outputs need grounding in retrieved source documents and review by someone who can check them. Data exposure: PHI in prompts, context and logs, so minimize what is sent, control retention and keep processing under a BAA. Prompt injection: instructions hidden in documents or messages that redirect the model, so treat all retrieved content as untrusted and keep the model's permissions narrow.

Grounding and citations

Retrieval-augmented generation, where the model answers from documents you supply rather than from memory, is the standard pattern for healthcare question answering. Show the retrieved passages beside the answer, let reviewers click through to the source, and log which documents were used. An answer without a checkable source is a draft, not an answer.

Choosing where the model runs

The hosting decision sets most of the compliance work: a vendor API under a Business Associate Agreement with retention and training terms you have read, a model hosted in your own cloud account, or a smaller model run on your infrastructure. Each changes who can see PHI, how long it persists and what you can prove to an auditor.

Evaluating before anyone sees it

A demo proves nothing. Build a test set of real cases with identifying details removed, write a rubric for what a good output contains and what a dangerous one contains, have reviewers score outputs blind, and include adversarial prompts and documents. Re-run the set whenever the model, the prompt or the retrieval index changes.

How do teams build a generative AI feature step by step?

• 1. Pick the task and its level: Where it sits on the decision spectrum decides the controls.
• 2. Classify it: Device or not under the FDA criteria, patient care decision support under 45 CFR 92.210 or administrative.
• 3. Choose the model and the hosting: BAA, data retention, training on your data, where processing happens.
• 4. Ground it: Retrieval over your documents with citations, and PHI minimized in every prompt.
• 5. Design the review: Who sees the output, what evidence they see and how accept, edit and reject are recorded.
• 6. Evaluate before launch: A test set with rubrics, scored by reviewers, including adversarial prompts.
• 7. Launch in shadow mode: Outputs logged and compared before anyone sees them.
• 8. Monitor and govern: Edit rates, errors, fairness and cost, reviewed by a committee that can stop the feature.

What are generative AI best practices?

• Review before use: Every output that touches a record, a claim or a patient passes a person first.
• Ground and cite: Answers come from supplied documents with visible sources.
• Minimize PHI: Send the fields the task needs, strip the rest, control retention.
• Treat retrieved content as untrusted: Defend against prompt injection in documents and messages.
• Evaluate with rubrics: Reviewer-scored test sets, re-run when the model or prompt changes.
• Log everything: Prompts, context, outputs and decisions, for audit and improvement.
• Keep permissions narrow: The model drafts; it does not act on systems without a reviewed step.

The practices above are what separates a useful generative AI feature from an incident. The mistakes below are the incidents we are asked to prevent or clean up.

What are the common mistakes with generative AI in healthcare?

• Consumer tools with PHI: Staff pasting records into public chat tools with no BAA.
• Unreviewed outputs: Drafts sent to patients or written to records without a person reading them.
• No grounding: Answers from model memory presented as policy or guideline.
• Prompt injection ignored: Documents and messages trusted as instructions.
• Evaluation by demo: A few impressive examples instead of a scored test set.
• Scope creep into device territory: A summarization tool that starts recommending treatment.

What does generative AI mean for your build plan?

• Use cases and levels: Each sits somewhere on the decision spectrum, with matching controls.
• Model and hosting: BAA, retention, training terms and processing location.
• Grounding: Document pipelines, retrieval and citation display.
• Review interfaces: Screens and logging for accept, edit and reject.
• Evaluation and monitoring: Test sets, rubrics, fairness checks and cost tracking.
• Governance: A committee, a policy and the regulatory classification work.

For the review design, our human in the loop AI guide goes deeper, and our HIPAA compliant AI app guide covers the data handling around any AI feature.

How Agnotic can help

Agnotic Technologies is a software development services company. We build generative AI features for healthcare products and operations: grounded question answering over your documents, drafting for letters, notes and messages, summarization of records, and the PHI controls, review screens, evaluation sets and audit logging around them, with the FDA and Section 1557 classification settled first. Your reviewers stay in charge. The model does the first draft.

Sources

• NIST: AI Risk Management Framework (AI 100-1)
• NIST AI 600-1: Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profile
• FDA: Clinical Decision Support Software guidance (January 2026)
• ASTP/ONC: Decision support interventions, 170.315(b)(11)
• 45 CFR 92.210: Nondiscrimination in the use of patient care decision support tools
• 45 CFR 164.502: Uses and disclosures of PHI, including business associates
• OWASP: Top 10 for Large Language Model Applications
• 45 CFR 164.312: HIPAA Security Rule technical safeguards