Model Drift
Model drift is the drop in a machine learning model's accuracy after it goes live, because the data it sees or the outcome it predicts has changed since training. In healthcare, a new intake form, a new patient population or a coding change can all cause it. Drift is found by monitoring, not by waiting for complaints.
Key takeaways
- Drift is a loss of accuracy after go-live, not a software bug
- Data drift: the inputs change. Concept drift: what the inputs mean for the outcome changes
- Common causes are form changes, new populations, coding updates and new clinical practice
- Watch input mix, output rates and measured accuracy once outcomes are known
- A retrained model is a new release and goes through change control
What Model Drift means
A model learns patterns from past data. When the present stops looking like that past, the model keeps giving answers with the same confidence, but they are right less often. Nothing crashes and no error appears, which is why drift is easy to miss unless someone is looking.
There are two kinds. Data drift means the inputs change: a clinic adds a field to its intake form, a hospital starts serving an older population or a lab switches analyzers. Concept drift means the relationship changes: a new treatment changes who gets readmitted, or a payer rule changes which claims are denied. The inputs can look the same while the right answer moves.
Agnotic Technologies builds drift monitoring into the AI systems it delivers, with a named owner on the client side. The pipeline and tooling behind it are explained under MLOps.
Who should watch for model drift?
Two people, at least. A technical owner watches the signals and investigates when they move. A clinical or operational owner decides what the change means for patients or work, and whether to pause, adjust or keep the model. Without the second person, alerts get acknowledged and nothing changes.
Agree before go-live what triggers a review: a set change in input mix, output rate or accuracy, a known upstream change such as a new form or code set, or a rise in users overriding the model. Write it down with the owner's name.
How is drift handled in regulated software?
For a predictive decision support tool in certified health IT, the developer has to describe how and how often its validity and fairness are monitored, and how often it is updated. For an FDA-regulated device, a retrained model may follow a predetermined change control plan. In both cases the fix is a controlled release, not a quiet swap.
What does each type of drift look like in healthcare?
| Drift type | Healthcare example | Signal to watch | Who acts |
|---|---|---|---|
| Data drift: patient mix | A hospital opens a new unit and the model starts seeing older, sicker patients | Shift in age, diagnosis or unit mix compared with the training data | Technical owner checks; clinical owner decides whether results still hold |
| Data drift: input format | An intake form adds a field or changes a pick list | Jump in missing or unexpected values for one input | Technical owner fixes the mapping before any retraining |
| Data drift: coding change | A new code set version or local code mapping goes live | Codes the model has never seen, or a known code suddenly disappearing | Data owner updates mappings; technical owner retests the model |
| Concept drift | A new treatment pathway changes which patients are readmitted | Accuracy falls once outcomes arrive, while inputs look stable | Clinical owner leads the review; retraining goes through change control |
| User behavior signal | Clinicians start dismissing the model's suggestions more often | Rise in the override or dismissal rate | Clinical owner talks to users; technical owner looks for the cause |
Facts and primary sources
The NIST AI Risk Management Framework calls for the functionality and behavior of an AI system and its components to be monitored when in production (MEASURE 2.4).
Source: NIST AI RMF core45 CFR 170.102 defines a predictive decision support intervention as technology that supports decisions using algorithms or models that derive relationships from training data and produce a prediction, classification, recommendation, evaluation or analysis.
Source: eCFR, 45 CFR 170.102For predictive decision support interventions, 45 CFR 170.315(b)(11) requires source attributes describing the process and frequency for monitoring validity and fairness over time, validity in local data, and the schedule for updates.
Source: eCFR, 45 CFR 170.315
Explore this on Agnotic
Where Model Drift shows up in how we build.