Diagnosis software accuracy: from study figures to real review load
Sensitivity and specificity describe the software; predictive value describes what your clinicians will see. Agnotic Technologies builds diagnosis software, local validation and reviewer workflows to your requirements, sized for the review load in your setting.
Last reviewed by the Agnotic Technologies engineering team
How accurate will diagnosis software be in your setting?
Diagnosis software accuracy in a given setting depends on how common the condition is there, not only on sensitivity and specificity. Low prevalence lowers positive predictive value, so false positives can outnumber true finds. Agnotic Technologies builds diagnosis software with local validation and outcome capture, so predictive value is measured on your own population.
Which problems sit behind accuracy claims?
A launch figure from a study is a starting point. These are the gaps that open once the software meets real patients.
Figures from a different population
A study population can differ from yours in age, sex, disease stage and referral pattern. Its accuracy figures may not carry over.
Rare conditions, heavy review queues
When a condition is rare, a small false positive rate applies to a large negative group. False alarms can then outnumber true finds.
Pooled results that hide subgroups
Results reported only in aggregate can mask weaker performance in one subgroup, one site or on one type of device.
No reference standard
FDA's statistical guidance says PPV and NPV cannot be computed when a test is compared only with a non-reference standard.
Alerts clinicians learn to ignore
A flag that fires too often stops changing care. A tool that looks accurate on paper can still fail in the workflow.
No measurement after launch
Without confirmed outcomes flowing back, nobody knows whether real predictive value matches the launch figure.
What do we build for diagnosis software?
Agnotic Technologies builds to your intended use, care setting and requirements, whether the model is your own or a vendor's. These are the parts we typically build.
Local validation pipeline
Tooling that assembles a validation set from your own records, with the confirmed outcome for each case, and reports sensitivity, specificity, PPV and NPV with confidence intervals.
Subgroup and site reporting
Performance broken down by age band, sex, site and device, so weak spots show before go-live.
Prioritized worklist
Flags routed to a ranked worklist rather than an inbox, as in our diagnostic imaging AI agent for worklist triage.
Reviewer screen
The input, the score and the reason in one place, with agree or disagree recorded in one action.
Threshold settings per site
The operating point as a setting with a documented change history, so a low-prevalence site can trade sensitivity for fewer false alarms after review.
Outcome capture from the EHR
Confirmed diagnoses flowing back from the EHR, so the team measures real PPV each month instead of trusting the launch figure.
Monitoring after launch
Flag rate, reviewer agreement and measured PPV per site on one dashboard, with alerts when they drift.
Version records for the regulatory path
Which model version and threshold produced each flag. Intended use decides whether the software is a device; see FDA SaMD compliant software for the engineering side.
Why does prevalence change the result?
Sensitivity is measured among people who have the condition, and specificity among people who do not. Neither changes with prevalence, which is why vendors report them.
Predictive value does change. When a condition is rare, most cases the software sees are negative, so a small false positive rate applies to a large group and its false alarms can outnumber the true finds.
The same software can therefore look excellent in a specialist referral clinic and generate a heavy review queue in a screening program. Measure predictive value separately for each setting where you plan to deploy it.
What should you ask before trusting the figures?
Use this checklist with a vendor, a research partner or your own data science team.
- What reference standard confirmed each case, and who read it?
- Does the study population match yours in age, sex, disease stage and referral pattern?
- Are results reported per subgroup and per site, or only pooled?
- Which devices, scanners or sample types produced the inputs?
- Are confidence intervals given for sensitivity and specificity?
- Was the comparison against a reference standard, or only agreement with another test?
- Is the model locked, or does it change after release?
- What operating threshold produced the figures, and can it be set per site?
What do accuracy figures leave out?
Validating a model on your own population is part of AI and ML development for healthcare.
- Spectrum bias: a study of clear cases and clear controls overstates accuracy on borderline cases.
- Subgroup performance, which can differ from the pooled figure.
- Site differences in devices, protocols and case mix.
- Whether clinicians act on a flag, or learn to ignore one that fires too often.
How does Agnotic validate diagnosis software?
For a full platform, see our AI diagnostics platform development service.
Agree the intended use, the care setting and the reference standard on a scoping call.
Assemble a local validation set from your own records, with the confirmed outcome for each case.
Measure sensitivity and specificity on that set, then use your prevalence to compute PPV, NPV and the expected monthly review queue.
Set the threshold with the clinical lead, build the reviewer workflow and launch with outcome capture switched on.
What does FDA say about diagnostic accuracy and AI software?
Each fact links to the FDA page it comes from.
FDA's statistical guidance on diagnostic test studies names the predictive value of a positive result (PPV) and of a negative result (NPV) as measures that help interpret test results, alongside sensitivity and specificity.
Source: FDA, Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests (March 2007)The same guidance states that when a new test is compared with a non-reference standard, PPV, NPV and likelihood ratios cannot be computed, and recommends reporting the 2x2 table with positive and negative percent agreement instead.
Source: FDA, Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests (March 2007)FDA's AI-Enabled Medical Device List identifies AI-enabled medical devices authorized for marketing in the United States; FDA notes that the list is not comprehensive and is updated periodically.
Source: FDA, Artificial Intelligence-Enabled Medical DevicesFDA's Clinical Decision Support Software guidance, issued in final form in January 2026, explains which decision support functions the 21st Century Cures Act excluded from the device definition in section 201(h) of the FD&C Act.
Source: FDA, Clinical Decision Support Software guidance (January 2026)
Trusted by global innovators
Which standards does Agnotic build healthcare software against?
HIPAA
Health Insurance Portability and Accountability Act
Protect PHI with privacy-first architecture, encrypted storage and transmission, strict access controls, and traceable audit logs.
GDPR
General Data Protection Regulation
Implement lawful consent flows, data minimization, retention controls, and secure processing for sensitive health data.
FHIR
Fast Healthcare Interoperability Resources
Enable standardized health data exchange across apps, care teams, and systems through standards-based FHIR APIs.
HL7
Health Level Seven International
Support enterprise-grade interoperability with HL7-based integrations for records, events, and clinical messaging workflows.
HITRUST
Health Information Trust Alliance
Align security programs to healthcare-specific control and risk management practices trusted by providers and ecosystem partners.
HITECH
Health Information Technology for Economic and Clinical Health Act
Design with breach notification readiness, digital record safeguards, and operational controls that support regulated care programs.
SaMD
FDA Software as a Medical Device
Plan software quality, traceability, and documentation pathways for products that may require SaMD review and submission.
MDR (EU)
Medical Device Regulation (European Union)
Prepare EU market-ready processes for risk classification, evidence tracking, and lifecycle governance under MDR expectations.
SAMHSA
Substance Abuse and Mental Health Services Administration
Apply confidentiality controls and consent-aware sharing models for behavioral and mental health data experiences.
Where should you go next?
- AI diagnostics platform developmentDiagnosis and precision medicine software built for clinical review.
- Clinical decision supportWhat CDS is and how decision support fits into clinical workflows.
- Diagnostic imaging AI agentWorklist triage for imaging studies, with radiologist review.
- FDA SaMD compliant softwareIEC 62304 and ISO 14971 engineering for software as a medical device.
- AI and ML development for healthcareModel evaluation, local validation and monitoring after launch.