Compliance testing is the work of proving that the security and privacy controls a product claims to have are in place and doing their job. A policy that says sessions time out is a statement; a test that leaves a session idle and checks that it ends is proof.
For health software sold to hospitals and clinics, those proofs decide security reviews and audits. This guide covers how a control test differs from a functional test, how to write one, which safeguards to test first, how often to rerun them and where control tests give a false sense of safety.
What is compliance testing in software?
Need control tests that hold up in a review?
Agnotic writes the control test cases, automates the ones a pipeline can run and sets up the evidence store for the client's product.
Compliance testing in software is the structured checking of each required control, such as access limits, audit logging, session timeout or backup restore, to show that it exists, works as written and keeps working after changes. Each test has a stated control, a method, an expected result and saved evidence a reviewer can inspect.
The output is not a pass mark for the product. It is a set of results tied to named controls, so a buyer, an auditor or the team itself can see which promises are proven, which failed and which have not been tested yet.
Most of the work sits with engineering. Controls live in code, cloud settings and identity configuration, so the people who build those layers are the ones who can design tests that exercise them.
How is a control test different from a functional test?
A functional test asks whether a feature does what the user wanted. A control test asks whether the system refuses, records or recovers the way a rule or contract says it must, which usually means testing what should not happen.
Take a chart view. The functional test logs in as a nurse and confirms the chart opens. The control tests log in as a nurse from another clinic and confirm the chart does not open, then confirm the denied attempt produced an audit event with the user, patient, time and outcome.
Control tests also care about state over time. A functional suite runs against a fresh build. A control check also has to run against production configuration, because a setting changed in the cloud console last week can undo a control that passed in CI.
The two suites share tooling but not owners. Product QA owns functional tests. Control tests are owned by whoever is accountable for the control, often a security or platform engineer, so a failing result reaches the person who can fix it.
Which assessment methods should a test use?
A control test should use whichever method gives real assurance for that control, and many controls need more than one. NIST SP 800-53A Rev. 5, published in January 2022, defines three assessment methods: examine, interview and test.
- Examine: Review a document, a configuration or a mechanism, such as an infrastructure-as-code file that sets encryption on every database volume, or the identity provider's password and session policy.
- Interview: Ask the people who run the control how they do it, such as how an engineer gets emergency production access and who approves it.
- Test: Exercise the control under set conditions and compare the result with the expected behavior, such as calling an API with an expired token and confirming a 401 response.
Test is the strongest of the three for technical controls, because it shows behavior rather than intent. Examine suits settings that are hard to trigger safely in production. Interview is weakest on its own and is best used to find out what to examine or test next.
Write the chosen method into the test case. A reviewer reading "examined Terraform module and tested restore" knows more than one reading "verified".
How do you write a test case for a control?
Start from a control statement a tester can check, then add the method, the steps, the expected result and the evidence. The template below keeps every control test in the same shape, which makes gaps easy to spot.
- 1. Control ID and statement: One sentence that can be true or false, for example "Idle clinician sessions end after 15 minutes of inactivity." The number is the product's own policy choice, not a legal limit.
- 2. Source: The rule, contract clause or customer requirement the control answers, so a reviewer can trace why it exists.
- 3. Scope: The systems, environments and user roles the test covers, and those it does not.
- 4. Method: Examine, interview, test or a combination, with the reason for the choice.
- 5. Preconditions: Test accounts, synthetic patients, feature flags and any production read access the test needs.
- 6. Steps: Numbered actions another engineer could repeat without asking questions.
- 7. Expected result: The exact behavior that counts as a pass, including the log or event that must appear.
- 8. Evidence: The artifact saved from the run, such as a CI job log, a screenshot with a timestamp, an exported configuration or a sample audit record.
- 9. Frequency and owner: How often the test runs, what triggers an extra run and who acts on a failure.
Write negative steps on purpose. For an access control, the test that matters is the one where a user with the wrong role, the wrong clinic or a revoked account tries the action and is refused.
Keep the test cases in source control next to the code they check. A pull request that changes an authorization rule should also show the control test that proves the new rule.
Control test cases for common HIPAA safeguards
The HIPAA Security Rule's technical safeguards at 45 CFR 164.312 are a sensible first set for a product that handles ePHI. Unique user identification, emergency access and audit controls are required specifications, while automatic logoff and encryption are addressable, which means they are implemented or an equivalent is documented.
- Unique user identification: Create two users with the same display name and confirm every audit event records a distinct, permanent user ID, and that shared or generic accounts cannot sign in.
- Emergency access: Run the break-glass procedure in a test environment, confirm it grants access only for the stated time, and confirm it raises an alert and a review task.
- Audit controls: For each sensitive action (view, create, update, export, print, delete), perform it and query the audit store for a matching event with actor, patient, action, time, source and outcome.
- Automatic logoff: Leave a web session and a mobile session idle past the policy limit, then attempt an action and confirm the user must authenticate again.
- Person or entity authentication: Call each API with no token, an expired token, a token for another tenant and a token with too few scopes, and confirm each is rejected.
- Transmission security: Scan public endpoints for old protocol versions and weak cipher suites, and confirm internal service calls that carry ePHI are encrypted too.
- Backup restore: Restore last night's backup into an isolated account, open a known synthetic record and compare it with the source.
Each of these produces evidence as a side effect. The audit-control test, for example, saves the query and the returned event, which is exactly what a hospital reviewer asks to see.
Administrative controls need tests too, even though they look less technical. Offboarding can be tested by comparing the HR leaver list with active accounts in the identity provider, the cloud console, source control and support tools on a schedule.
How often should controls be retested?
Retest on a schedule and on change. The HIPAA Security Rule's evaluation standard at 45 CFR 164.308(a)(8) calls for a periodic technical and nontechnical evaluation, and another one in response to environmental or operational changes that affect the security of ePHI.
In engineering terms, that gives three triggers. Automated control tests run on every merge and on a nightly schedule against production configuration. Manual and interview-based tests run on a fixed calendar, such as each quarter. Any significant change, such as a new integration, a new cloud region, a new identity provider or an AI feature that reads patient data, triggers a targeted retest of the controls it touches.
Track the age of each result. A control last tested before the most recent major release is effectively untested, even if its last result was a pass.
Cloud tools help with the nightly part. AWS Config, Azure Policy and the security posture tools in Google Cloud can flag configuration drift, such as an unencrypted volume or a public bucket, and those findings can open tickets against the owning control.
Where compliance tests give false comfort
Control tests fail quietly when they check the wrong thing. These failure points are worth reviewing before results go into a buyer's security packet.
- Testing the happy path only: The suite confirms that an admin can export data but never tries the export as a front-desk user.
- Testing staging, trusting production: Controls pass in a test account whose settings differ from the production account they are meant to describe.
- Mocked identity: Tests use a fake token service that accepts any signature, so expired or forged tokens never get rejected in the test run.
- Log written, never read: The audit event exists, but nobody tested that it reaches the separate audit store or that engineers cannot edit it there.
- Screenshots without context: Evidence images with no timestamp, environment name or test case ID cannot be tied to a specific control run.
- Exceptions left open: A failed test is marked accepted for a release and never retested, so the exception becomes permanent.
- Vendor controls assumed: The team relies on a cloud provider's controls without testing its own configuration of the services it chose.
A simple check catches most of these. For each control, ask whether a reviewer could rerun the test from the written steps and get the same result in production. If not, the test needs work before its evidence is shared.
How Agnotic can help
Control tests are only useful when they exercise the real system and leave evidence nobody has to rebuild by hand. Agnotic Technologies writes control statements and test cases for the client's product, automates the access, audit, session and encryption checks in CI and against production configuration, and sets up a write-once evidence store.
For teams facing a first hospital security review, Agnotic can also run the failure-point review above and close the gaps before the packet goes out.
Prove your controls with Agnotic
Start with compliance engineering, or browse more articles in the software engineering hub.
Related reading on Agnotic
Continue with domain testing, healthcare QA, control mapping, healthcare IT compliance and quality software practice.
