AI agent frameworks for healthcare: how to choose the layer your agent runs on
Engineering leads building a healthcare AI agent choose between open-source orchestration frameworks, a cloud provider's managed agent service, or a packaged agent product. Feature lists look alike. What separates them for PHI work is state, approvals, audit logs, where traces go and whether the runtime sits inside your BAA boundary.
How do you choose an AI agent framework for healthcare?
Quick answer
Choose an AI agent framework for healthcare by testing what PHI work needs, not by feature lists: durable state with human approval steps, a log of every tool call, control over PHI in traces, and a runtime inside your BAA boundary. Agnotic Technologies runs these tests in a short proof of concept before committing a client's build.
| Criterion | Open-source framework | Managed agent service |
|---|---|---|
| Durable state and resume | Graph or state-machine designs usually checkpoint state; you host the store and set retention | Usually built in; check where session state is stored, for how long, and who can read it |
| Human approval steps | Built as an interrupt in the graph that waits for a person before a tool call | Check whether the service can pause a session for approval and resume it later |
| Tool-call audit log | You log each tool call, its inputs and its result to your own store | Check whether the provider's logs record tool inputs and outputs, and where they are kept |
| PHI in traces and telemetry | You choose the tracing backend; default integrations may send prompts to a third-party service | Traces stay in the provider's console; confirm that feature is covered by your BAA |
| Running inside the BAA boundary | Runs in your own cloud account, under the BAA you already have | Only if the specific agent service is on the provider's HIPAA eligible list |
| Model portability | Swap providers behind your own interface | Often tied to the provider's model catalog |
| Evaluation and injection tests | Add your own evaluation suite and red-team cases | Use the provider's evaluation tools and add your own cases |
| Licence and support | Check the licence of the framework and every plugin you install | Covered by the cloud agreement and its support terms |
What they are
What is Open-source framework
A library you run in your own environment that handles the agent loop: calling the model, choosing tools, keeping state and coordinating agents. You choose the model provider, the state store and the tracing backend, so the compliance boundary is the one you already operate.
What is Managed agent service
A cloud provider's hosted runtime for agents, with sessions, tool connections, memory and monitoring run as a service. It removes infrastructure work, but only the services the provider lists as HIPAA eligible may handle PHI, and its tracing and memory features follow the provider's design.
Key differences
Which categories of agent framework are there?
Four categories cover most choices. Graph or state-machine orchestration: LangGraph's documentation describes it as a low-level orchestration framework and runtime for long-running, stateful agents. Role-based multi-agent: CrewAI's documentation describes crews of role-playing agents with goals and tools. Conversation-based multi-agent: AutoGen's AgentChat is described as a framework for conversational single and multi-agent applications. The fourth is a cloud provider's managed agent service. For PHI work, the category matters more than the brand.
Why do PHI workloads need durable state and approval steps?
A healthcare agent often waits: for a prior authorization answer, a nurse's approval or a patient's reply. If state lives only in memory, a restart loses the case or repeats an action. Choose a framework that saves state at each step and can pause before any write to an EHR, a claim or a message, then resume with the person's decision recorded.
What should a tool-call audit log record?
45 CFR 164.312(b) requires mechanisms that record and examine activity in information systems that contain or use ePHI. For an agent, that means each tool call with the user or case it served, the inputs, the result, the model version and the time. Test that the framework exposes those events in a form you can store, rather than only in a vendor dashboard.
How do you test a framework against prompt injection?
OWASP's LLM01:2025 describes indirect prompt injection, where content the model reads from websites or files changes its behaviour. Agents read referral letters, faxes and portal messages, so test with injected instructions inside those documents and check that the framework lets you restrict which tools a step may call. Our build vs buy guide covers the separate choice of hosted or self-hosted models.
When should you buy a packaged agent product instead?
Buy when the task is a defined product category, such as ambient documentation or appointment reminders, and the vendor's workflow fits yours. Then the framework question becomes the vendor's problem and your job is the contract and the integration. The framework choice matters when the agent is your own workflow, which is where Agnotic Technologies builds.
When to use each
When to use Open-source agent framework
- You need full control over state storage, logging and where every request goes.
- Your agent runs inside an existing cloud account that already holds PHI under a BAA.
- You expect to switch model providers or run open-weight models later.
- Your team can own upgrades, plugin reviews and licence checks.
When to use Managed cloud agent service
- You already run on that cloud and the agent service is on its HIPAA eligible list.
- You want hosted sessions, scaling and monitoring rather than running them yourself.
- The agent's tools are mostly services in the same cloud.
- You accept the provider's choices on state retention and trace storage after checking them.
What should an AI agent framework proof of concept test?
Run the same small healthcare task on two candidates, with synthetic data only. Check each item and record pass or fail:
1. Kill the process mid-task. Does the agent resume from saved state without repeating a tool call?
2. Insert an approval step before a write. Does the session pause, wait and resume with the approver recorded?
3. Export the tool-call log. Does it hold inputs, outputs, model version and timestamps in your own store?
4. Trace a run. Does any prompt or output leave your environment through a default tracing or analytics integration?
5. Swap the model. How much code changes when you point the agent at a second model provider?
6. Inject an instruction inside a test document. Can you limit the tools that step may call?
7. Read the licences of the framework and each plugin, and list who supports them.