Healthcare data validation for analytics pipelines
The checks a clinical or claims data pipeline should run before anyone builds a dashboard on it: FHIR profiles, terminology, units, identity, dates and completeness. For each one, see where it runs, whether a rule or an anomaly model fits, and what happens to the record that fails. Agnotic Technologies builds these checks into your pipeline, to your requirements.
For data engineering leads, analytics product owners and the CTO or CIO who has to stand behind the numbers.
Where the checks run
- Ingest4 checks
Records arrive from EHRs, claims files and devices. Reject or quarantine what cannot be trusted at all.
- Transform6 checks
Records are mapped, merged and linked. Catch identity, unit and code problems before they spread.
- Serve2 checks
Data feeds a measure, report or model. Check it is complete enough for that use.
What is healthcare data validation?
Healthcare data validation is the set of checks clinical and claims data must pass on its way into an analytics platform, from FHIR profile and terminology checks at ingest to completeness checks before a measure is served. Agnotic Technologies builds these checks into client pipelines, using rules where a standard defines valid and anomaly models where it does not.
Last reviewed by the Agnotic Technologies engineering team
What goes wrong in healthcare data pipelines?
Most analytics problems start upstream, in records that were accepted without being checked. These are the ones teams meet most often.
Profiles claimed but never checked
A feed says it sends US Core resources, but nothing validates them. A missing required element surfaces months later as a gap in a dashboard.
Local codes with no mapping
Lab and order codes arrive as local codes instead of LOINC or SNOMED CT. Without a terminology check they fall out of every measure that filters on standard codes.
Units that do not agree
The same test arrives in different units from two sources. Averages, trends and thresholds built on the mix are wrong, and nothing on the report says so.
The same patient counted twice
One person appears in the EHR feed and the claims feed with slightly different details. Population counts, risk scores and outreach lists all inherit the duplicate.
Failing records that vanish
A record fails a check and is dropped with no queue, owner or count. The data goes missing quietly, which is harder to find than a loud error.
Source changes nobody announced
A source system upgrade shifts code mix, volume or missing fields. Reports change overnight and the first person to notice is a reader asking why.
What do we build for it?
Agnotic Technologies builds validation into your pipeline to your requirements: the sources you have, the measures you publish and the thresholds your clinicians agree. Typical features include:
FHIR profile validation in the pipeline
Each resource checked at ingest against HL7 FHIR R4 and the US Core profiles you claim, using the HL7 FHIR validator or the $validate operation, with the validator message kept on any quarantined record.
Terminology binding checks
Codes checked against the bound code system and value set for the element and date, with unknown local codes routed to a mapping queue instead of dropped.
UCUM unit normalisation
Quantity units validated as UCUM and converted to the unit each measure expects. Values that cannot be converted are flagged and held out of unit-sensitive measures.
Duplicate patient detection
Matching on identifiers, name and date of birth across sources, with candidate links sent to an identity review queue. Records are never merged without a person deciding.
Quarantine and review queues
Reject, quarantine and flag destinations decided per check, each with an owner, an age limit and escalation, so no failing record disappears.
Completeness reports per measure
Every published measure carries the completeness of the fields its definition needs, so readers see how much of the population it really counts.
Data quality dashboards and alerts
Reject, quarantine and flag counts per source per day, plus anomaly models on volume, arrival time and distribution that alert the feed owner.
Validation rules as code
Checks versioned and reviewed like any other code, run in observe mode before enforcement, and re-run when a code system or value set updates.
Validation checks by pipeline layer
The example rules are patterns, not thresholds: set your own limits from your data. The method column says whether a fixed rule is enough or an anomaly model earns its place.
| Check | Example rule | Where it runs | Rule or model | Failing record |
|---|---|---|---|---|
| FHIR structure and cardinality | Resource parses, required elements present, no element repeats beyond its maximum | Ingest | Rule | Reject and return an error to the sender |
| Profile conformance | Patient, Observation and Condition validate against the US Core profiles you claim to support | Ingest | Rule | Quarantine with the validator's message |
| Terminology binding | Codes exist in the bound code system and value set for that element and date | Ingest | Rule | Quarantine; send unknown local codes to a mapping queue |
| Units of measure | Quantity units are valid UCUM and convertible to the unit the measure expects | Transform | Rule | Flag and hold out of unit-sensitive measures |
| Value plausibility | A lab value or vital sign inside the range your clinicians agreed is physically possible | Transform | Rule or model | Flag for review; never auto-correct a clinical value |
| Referential integrity | Every Observation and Claim points to a Patient and Encounter that exist | Transform | Rule | Quarantine until the parent record arrives |
| Duplicate patients | Two records with matching identifiers, name and date of birth across sources | Transform | Rule or model | Flag as a candidate link for identity review |
| Duplicate records | The same claim line or result received twice from one source | Transform | Rule | Keep one, log the duplicate |
| Date plausibility | Service dates after birth, before death and not in the future; discharge after admission | Transform | Rule | Flag and exclude from date-based measures |
| Volume and arrival | A feed's daily record count or arrival time departs from its usual pattern | Ingest | Model | Alert the feed owner; do not drop records |
| Distribution drift | The mix of codes, values or missing fields shifts after a source system change | Serve | Model | Alert the data owner; hold the affected report for review |
| Completeness for the measure | Every field the measure's definition needs is present for the population it counts | Serve | Rule | Report the measure with its completeness, or withhold it |
Where does AI-driven data validation help?
AI-driven data validation is useful where the rule is hard to write down: patterns in volume, timing and distribution. It is the wrong tool where a standard already defines what valid means.
Use a rule when
A specification says what valid is: FHIR structure, a bound value set, a UCUM unit, a date that cannot be in the future. Rules are cheaper, explainable and give the same answer every time.
Use an anomaly model when
Valid depends on what is normal for this source: a feed that usually sends far more records, a lab whose values shifted after an analyzer change, a sudden jump in one diagnosis code.
Keep people on the decision
A model flags; a person decides. Models should never correct clinical values or merge patient records on their own, and every flag should carry the reason it was raised.
Reject, quarantine or flag?
Every failing record needs a destination decided in advance, or it quietly disappears.
Reject
When: The record cannot be read or trusted at all, and the sender can fix and resend it.
Watch: Return a clear error; count rejects per source.
Quarantine
When: The record is probably right but incomplete or unmapped, and will be usable once something arrives or is fixed.
Watch: Give the quarantine an owner and an age limit.
Flag
When: The record is usable for most purposes but suspect for some measures.
Watch: Carry the flag downstream so reports can include or exclude it.
Trusted by global innovators
Is validation run like part of the product?
The checks are only half of validation. These operating practices keep them working after the first release.
Ownership
- Every source feed has a named owner who receives its validation alerts.
- The quarantine has an owner and a maximum age before escalation.
- Clinicians agreed the plausibility ranges, and the agreement is recorded.
Change control
- Validation rules are versioned and reviewed like code.
- Code system and value set updates trigger a re-run of terminology checks.
- A source system upgrade triggers a baseline check before its data reaches reports.
Visibility
- Reject, quarantine and flag counts are reported per source and per day.
- Each published measure shows its completeness alongside its value.
- Anomaly model flags record the reason and the person who resolved them.
How do you add validation without stopping the feed?
Run every check in observe mode
Log what would fail without rejecting anything, so you learn your real failure rates first.
Baseline each source
Record normal volumes, arrival times and failure counts per feed. Anomaly models need this history.
Agree the destinations
Decide reject, quarantine or flag for each check with the data owners and the report owners.
Turn on enforcement layer by layer
Start at ingest with structural checks, then transform, then serve, watching the counts after each step.
Review weekly, then monthly
Retire checks that never fire, tighten ones that miss problems and add a check for every incident.
Which specifications define the checks?
The deterministic checks come from published standards. Each statement links to its source.
The HL7 FHIR R4 specification describes validating a resource as checking its structure, cardinality, value domains, coding and CodeableConcept bindings, invariants, profiles, questionnaires and business rules, and names duplicate checks and resolving references as business rules made outside the specification.
Source: HL7 FHIR R4, Validating ResourcesUS Core is the HL7 FHIR implementation guide that profiles FHIR resources for the USCDI data classes used in the United States.
Source: HL7 International (US Core)The Unified Code for Units of Measure (UCUM) is a code system intended to include all units of measure in contemporary use in science, engineering and business, so that quantities and their units can be communicated without ambiguity.
Source: UCUM specification (ucum.org)The HL7 FHIR R4 vital signs profile expects a numeric result value with a standard UCUM unit taken from the unit codes it lists for each vital sign, and a reason when there is no numeric result.
Source: HL7 FHIR R4, Vital Signs profile
Which standards does Agnotic build data platforms to?
HIPAA
Health Insurance Portability and Accountability Act
Protect PHI with privacy-first architecture, encrypted storage and transmission, strict access controls, and traceable audit logs.
GDPR
General Data Protection Regulation
Implement lawful consent flows, data minimization, retention controls, and secure processing for sensitive health data.
FHIR
Fast Healthcare Interoperability Resources
Enable standardized health data exchange across apps, care teams, and systems through standards-based FHIR APIs.
HL7
Health Level Seven International
Support enterprise-grade interoperability with HL7-based integrations for records, events, and clinical messaging workflows.
HITRUST
Health Information Trust Alliance
Align security programs to healthcare-specific control and risk management practices trusted by providers and ecosystem partners.
HITECH
Health Information Technology for Economic and Clinical Health Act
Design with breach notification readiness, digital record safeguards, and operational controls that support regulated care programs.
SaMD
FDA Software as a Medical Device
Plan software quality, traceability, and documentation pathways for products that may require SaMD review and submission.
MDR (EU)
Medical Device Regulation (European Union)
Prepare EU market-ready processes for risk classification, evidence tracking, and lifecycle governance under MDR expectations.
SAMHSA
Substance Abuse and Mental Health Services Administration
Apply confidentiality controls and consent-aware sharing models for behavioral and mental health data experiences.
What can validation not prove?
That the data is clinically right
A valid record can still be wrong. Validation catches what breaks rules and patterns, not every documentation error.
That a measure is defined well
Completeness checks test data against a definition. Whether the definition is right is a clinical and quality decision.
Your thresholds
Plausibility ranges, anomaly sensitivity and quarantine age limits come from your data and your clinicians.
Related data services
Agnotic Technologies builds these pipelines and the checks inside them for clients.
- Healthcare data analyticsFHIR data lakehouse builds with terminology services and measure reporting.
- Data engineering servicesPHI-safe pipelines, deduplication and terminology mapping.
- FHIR integration servicesFHIR R4 feeds, profiles and Bulk Data exports that land in your pipeline.
- Data science servicesPredictive models, including the anomaly models that watch your feeds.
- ETL vs ELT for healthcare pipelinesWhere identified PHI lands, and whether healthcare formats are parsed before or after load.
- FHIR data stores comparedHow HAPI, HealthLake, Azure and Google FHIR stores validate what they ingest.
- Mirth Connect assessmentCheck the interface engine that feeds your pipeline, including its FHIR output.