Healthcare Software Testing

What is healthcare software testing?

Healthcare software testing is the verification of clinical, patient-facing and administrative health software for calculation accuracy, HL7 FHIR, HL7 v2 and X12 data exchange, HIPAA access and audit behavior, accessibility and failure handling. It runs on synthetic or de-identified patient data, so the test environment never becomes a second store of real PHI.

What does flipping the rule mean for a QA team?

Flipping the rule means moving most verification from manual screen clicking at the end of a sprint into automated checks that run on every change. Human testers then spend their time on usability, assistive technology and hazard review.

• The old pattern: A tester walks through a few happy-path demo patients in the browser before release, and data contracts, unit boundaries and permission rules get whatever time is left.
• The flipped pattern: Every pull request runs thousands of synthetic patients through dosing logic, FHIR profile validation, HL7 v2 parsing, permission matrices and audit-event assertions before anyone opens a screen.
• Where people still lead: Clinician walkthroughs of real tasks, screen reader and keyboard sessions, and structured review of what could go wrong for a patient.

The payoff is speed with less risk. A schema regression fails the build in minutes instead of surfacing at a partner hospital a week after release.

Clinician acceptance testing also gets better. Because the basic logic is already proven, a nurse or physician tester can spend a session on whether the task makes sense in a real shift rather than on finding broken buttons.

Which rules and standards shape the test plan?

Rules decide both what you must show and what data you may test with. These five carry the most weight for a health software team.

• HIPAA Security Rule: Any environment holding real ePHI needs the safeguards of 45 CFR Part 164 Subpart C, and any outside tool that receives it needs a business associate agreement under 164.502(e). De-identified data under 164.514 or synthetic data avoids both.
• ONC Inferno: The ASTP/ONC Inferno test kit is the tool certified health IT developers use to show conformance with the 45 CFR 170.315(g)(10) standardized API criterion. Other teams can run it as a useful check, but it is not a legal test for their product.
• IEC 62304: Device software teams often follow IEC 62304, a voluntary standard FDA recognizes. It asks for every software requirement to be verified, and the safety class decides how deep unit and integration testing goes.
• FDA Computer Software Assurance: For software that runs device production or the quality system, FDA's CSA guidance (February 2026 version) now applies in place of section 6 of the old software validation guidance.
• Section 504 accessibility: Patient portals built for HHS-funded organizations have to conform to WCAG 2.1 Level AA, the bar 45 CFR 84.84 sets for their websites and mobile apps, so the test plan needs AA criteria.

For AI features, the NIST AI Risk Management Framework (NIST AI 100-1) and its generative AI profile (NIST AI 600-1) give a voluntary structure for testing validity, reliability and bias.

How does the five-layer testing pyramid work?

The pyramid stacks five layers of automated checks, with synthetic data at the base and accessibility and device behavior at the top. Each layer depends on the one below it.

• Layer 1, synthetic patients: Generate longitudinal records with a tool such as Synthea, then add hand-built hard cases: infants, pregnant patients, twins with similar names, duplicate MRNs and missing birth times. Add fuzzed HL7 v2 messages with missing segments and bad delimiters.
• Layer 2, interface contracts: Run the HL7 FHIR Validator against US Core profiles, replay HL7 v2 ACK and NAK sequences over MLLP, and parse malformed X12 270/271 and 835 files from partners.
• Layer 3, clinical logic and security: Write boundary tests for weight-based dosing, unit conversion, renal adjustment and time zones. Assert that user A can never read patient B's record and that each PHI access writes an audit event.
• Layer 4, AI evaluation: Score any AI feature against a locked, clinician-graded case set on every prompt or model change.
• Layer 5, accessibility and devices: Run automated axe-core scans in Playwright, then manual VoiceOver and TalkBack sessions, Bluetooth drop simulations and offline outbox replay.

Fields worth seeding on purpose

Generators produce plausible records, but defects hide in specific fields. Seed these deliberately so every run exercises them.

• Identity: Two patients with the same name and birth date, a merged MRN, a name with an apostrophe or hyphen, and a missing middle name.
• Dates and times: A birth time near midnight, an encounter that spans a daylight-saving change, and an order placed in one time zone and given in another.
• Units and codes: Weight in pounds where kilograms are expected, a local lab code with no LOINC mapping, and a result flagged with a nonstandard abnormal indicator.
• Consent and sensitivity: A record tagged as 42 CFR Part 2 data, a minor with confidential services, and a patient who has restricted a disclosure.

AI on both sides of the test suite

AI shows up in healthcare software testing in two different roles, and they need different controls. One uses AI to help write tests; the other tests AI features themselves.

As a test author, an LLM or agent can draft Playwright scripts, propose edge-case HL7 v2 and FHIR payloads and repair selectors after a UI change. The control is simple: the model sees synthetic schemas and synthetic patients only, unless it runs inside a cloud tenant covered by a business associate agreement.

As the thing under test, an ambient scribe, prior authorization agent or retrieval assistant cannot be checked with pass or fail unit tests alone. Keep a versioned evaluation set of cases graded by clinicians, and score each change for grounding, missed allergies or medications, invented citations and prompt injection.

Bias checks belong in the same set. Customers that are covered entities may not use patient care decision support tools that discriminate (Section 1557, 45 CFR 92.210), and they will ask how subgroup performance was tested.

A six-step pipeline from first commit

• 1. Remove real PHI from non-production: Replace production clones with a seeded synthetic cohort that builds fresh for each pull request.
• 2. List the hazards: Before UI tests, write down every calculation, conversion, allergy check, triage threshold, routing rule and consent boundary where a defect could harm a patient or expose data.
• 3. Gate merges on safety and access tests: Require dosing boundary tests, the role permission matrix and audit-event assertions to pass before merge.
• 4. Add contract validation to CI: Load the FHIR Validator with the US Core version your partners use, and add an HL7 v2 and X12 replay harness.
• 5. Stand up AI evaluation: Run the locked case set against each candidate prompt or model and record sensitivity, specificity, citation accuracy and override triggers.
• 6. Generate traceability: Tag requirements, hazard controls and test cases in Git so each release produces its own evidence for hospital questionnaires or IEC 62304 files.

Where healthcare test suites miss defects

These failure points come from the shape of healthcare data and integrations. Check your suite against each one.

• Clean fixtures: Real HL7 v2 feeds carry trailing spaces, Z-segments, local lab codes and duplicate MRNs. Fixtures without them pass parsers that fail in production.
• Mock-only FHIR testing: A home-built mock server forgives missing headers and unsupported search parameters that Epic, Oracle Health (Cerner) or athenahealth sandboxes reject.
• Untested clocks: Medication schedules, RPM device-day counts and the 72-hour expedited prior authorization timeframe under CMS-0057-F break across time zones and daylight-saving changes.
• Scanner-only accessibility: Automated tools catch only part of WCAG issues. Focus order, keyboard traps and screen reader announcements need a person.
• No chaos on partner links: Simulate HTTP 429 responses, expired OAuth tokens, dropped MLLP sockets and slow EHR replies to prove retry queues and dead-letter alerts work.
• PHI pasted into public tools: Pasting a live HL7 message into a consumer chatbot to debug it discloses PHI to a vendor with no agreement.
• Eyeballed AI changes: Reading five sample notes after a prompt change misses rare dosage errors that a graded case set would catch.

How Agnotic can help

A test suite is only as good as its data and its gates. Agnotic Technologies sets both up for health software teams: synthetic cohorts with the hard cases built in, FHIR and HL7 v2 contract checks in CI, permission and audit-event tests, and evaluation sets for clinical AI.

Through a healthcare fractional CTO engagement, Agnotic can also design the test strategy, choose the tools and set the release gates your QA team runs after handover.

Sources

• 45 CFR Part 164 Subpart C: HIPAA Security Rule safeguards
• 45 CFR 164.514: De-identification of protected health information
• ASTP/ONC: Inferno test kits for certification testing
• FDA: Computer Software Assurance for Production and Quality Management System Software
• IEC 62304: Medical device software life cycle processes
• Federal Register: Section 504 web and mobile accessibility compliance date extension (May 2026)
• NIST: AI Risk Management Framework (NIST AI 100-1)