Lab report data extraction: how do you choose and test a model on your own reports?
Teams pulling results out of lab report PDFs usually compare three approaches: OCR with templates and rules, a hosted general-purpose model, and a self-hosted open-weight model. Published benchmarks rarely match your report layouts, scans and units, so the useful comparison is a test on a labeled set of your own reports, scored field by field.
How do OCR with rules and model-based extraction compare on lab reports?
Quick answer
Choosing a model for lab report data extraction should rest on a field-level evaluation of your own reports: score patient identifier, test name, value, unit, reference range, flag and dates for OCR with rules, a hosted model and a self-hosted model. Agnotic Technologies builds that evaluation set and the review routing before any approach goes live.
| Criterion | OCR and rules | Model-based |
|---|---|---|
| New report layouts | Each new layout needs a template or rule set | Often handles unseen layouts, but test each source lab |
| Poor scans and faxes | OCR errors pass straight into fields | Text models inherit OCR errors; vision models read the image but can misread digits |
| Multi-page result tables | Reliable when the table structure is fixed | Check rows split across pages and repeated headers |
| Units and reference ranges | Extracted as printed; normalization is a separate step | May normalize on its own; check it never invents a unit |
| How it fails | Visibly: empty or misaligned fields | Quietly: a plausible value in the wrong field |
| Confidence signal | OCR confidence per word or character | Needs explicit checks: field rules, cross-field consistency or agreement between runs |
| Effort to start | Template work per lab format | Prompt or fine-tuning work plus an evaluation set |
| Change control | Rules change only when you change them | Pin the model version; a hosted model update needs a rerun of the evaluation |
| PHI hosting | Runs where you host it | Hosted model covered by a BAA, or self-hosted in your environment |
| Output | Validate, then map to DiagnosticReport and Observation | Same: validate before writing FHIR, never trust the model's FHIR as is |
What they are
What is OCR and rules
OCR with templates turns the page into text with positions, then rules find each field by layout, label or pattern. It is predictable and easy to explain, but every new report layout needs a template, and changes at a source lab break extraction until someone updates it.
What is Model-based
Model-based extraction uses a language or vision-language model to read the report and return structured fields. A hosted model runs as a provider API; a self-hosted open-weight model runs in your environment. Both handle varied layouts better than templates but can fail quietly, so they need field checks and review.
Key differences
Why not rely on a published benchmark?
A benchmark measures someone else's reports. Your mix of labs, layouts, languages, scan quality and units decides how an approach performs for you, and a model that tops a public list can still put a reference range in the result field on your most common report. Treat published results as a way to pick candidates, then test them yourself.
How do you build an evaluation set from your own reports?
Sample reports from every source lab and format you receive, including faxes, phone photos, multi-page panels and reports in every language you handle. Label each field by hand, ideally twice by different people, and keep a held-out set you never use while tuning prompts or rules. Store the set inside your PHI controls, or de-identify it first.
Which report elements must survive extraction?
Under CLIA, 42 CFR 493.1291(c), a test report must show positive patient identification, the name and address of the laboratory location, the report date, the test performed, the specimen source when appropriate, the result with units or interpretation, and information about specimens that did not meet acceptability criteria. Those elements are the minimum your extraction must carry, and reference intervals must be available too.
Why normalize units and codes before scoring?
The same potassium result can print as 4.1 mmol/L or 4.1 mEq/L. UCUM, the Unified Code for Units of Measure, is a code system intended to include all units of measure in contemporary use, so normalize extracted units to UCUM before you compare them with the label. Map test names to LOINC codes the same way, so you score meaning rather than spelling.
Hosted or self-hosted: what changes for PHI and licences?
A hosted model must be covered by a business associate agreement before it receives reports, and its version can change under you unless you pin it. A self-hosted open-weight model keeps data in your environment, but you run the serving stack and must read the model's licence terms for clinical and commercial use. Our build vs buy AI agents guide covers the hosting trade in more depth.
Where should a reviewer step in?
Route a report to a person when a required field is missing, a unit cannot be normalized, a value falls outside a plausible range, a critical result is detected, or checks disagree. NIST's AI Risk Management Framework describes a MEASURE function that uses quantitative, qualitative or mixed methods to analyze, assess, benchmark and monitor AI risk; field-level scoring and review rates are how that looks for extraction.
When to use each
When to use OCR with templates and rules
- Most reports come from a few labs with stable layouts.
- You need fully explainable extraction for each field.
- You cannot send reports to a model service and do not want to host one.
- Fields are printed in fixed tables with clear labels.
When to use Model-based extraction (hosted or self-hosted)
- Reports arrive from many labs with layouts you do not control.
- Scans, faxes and photos vary in quality and format.
- You can host a model or use a hosted one covered by a BAA.
- You will build validation and human review around the output.
What should a field-level evaluation sheet score?
1. Patient identifier: exact match after removing spaces and punctuation. Any error routes to review, never to auto-filing.
2. Test name: match after normalizing to your test dictionary.
3. LOINC code: exact match against your mapping for that lab and test.
4. Result value: exact match after numeric normalization, such as decimal separators and leading zeros.
5. Unit: match after normalizing to UCUM; an unconvertible unit counts as an error.
6. Reference range: match on low and high bounds after unit conversion.
7. Abnormal flag: exact match on high, low, critical or normal.
8. Specimen collection date and report date: exact date match, with time where printed.
9. Performing laboratory: match on name and address after normalization.
10. Review threshold: for each field, set the error rate your team will accept for auto-filing, and route everything else to a reviewer.