
MLOps (Machine Learning Operations)
MLOps, short for machine learning operations, is the practice of versioning, deploying, monitoring and retraining machine learning models once they run in production. It treats each model as a release with recorded data, code, metrics and approvals. In healthcare it adds controls for PHI in training data and for FDA change control.
Key takeaways
- Covers data versioning, training, evaluation, model registry, deployment, monitoring and retraining
- Each model version is pinned to its data, code, metrics and approval
- PHI in training sets is de-identified under 45 CFR 164.514 or kept under HIPAA safeguards
- Changes to device software can follow an FDA predetermined change control plan
- Monitoring after deployment is part of the system, not a later project
What MLOps means
MLOps covers what happens after a model works in a notebook: recording the data and code behind it, storing the approved version, deploying, monitoring and releasing retrained versions safely. Agnotic Technologies sets up these pipelines in the machine learning systems it builds for clients.
Healthcare adds three MLOps needs: PHI in training data is de-identified or kept under HIPAA safeguards, a regulated device model change may need FDA review unless an authorized predetermined change control plan covers it, and clinicians must see which model version produced an output.
How is MLOps different from DevOps?
DevOps releases code. MLOps releases code plus a trained artifact whose behavior depends on data. The same code can produce a different model from different data, and a model can get worse with no code change at all because the data it sees has shifted.
That is why MLOps adds data versioning, a model registry and performance monitoring on top of the usual build and release pipeline.
What does each MLOps stage keep, and what does healthcare add?
| Stage | Artifact kept | Healthcare control |
|---|---|---|
| Data versioning | Snapshot or hash of each training, validation and test set | De-identification record, or HIPAA access controls and logs for PHI |
| Training | Code version, parameters, environment and the training run log | Training data access limited to approved people and environments |
| Evaluation | Metrics overall and by patient subgroup, with the test set used | Clinical review sign-off and a check for performance gaps across groups |
| Model registry | Approved model version with its lineage and model card | Version pinning, so each clinical output traces to one model |
| Deployment | Release record, configuration and rollback plan | Change checked against the device's PCCP or quality system before release |
| Monitoring | Input distributions, output rates and performance once labels arrive | A named clinical owner and a threshold that triggers review |
| Retraining | New data snapshot and a comparison with the live model | Release only through the same evaluation and approval path |
Facts and primary sources
FDA's final guidance on predetermined change control plans for AI-enabled device software functions recommends that a PCCP describe the planned modifications, the method to develop, validate and implement them, and an assessment of their impact.
Source: FDA guidance on PCCPs for AI-enabled device software functionsUnder that guidance, modifications described in an authorized PCCP can be implemented without a new marketing submission for each one.
Source: FDA guidance on PCCPs for AI-enabled device software functionsUnder 45 CFR 164.514(b), health information is de-identified either through a documented expert determination that the risk of identifying an individual is very small, or by removing the listed identifiers.
Source: eCFR, 45 CFR 164.514The NIST AI Risk Management Framework organizes AI risk work into four functions, govern, map, measure and manage, and calls for post-deployment monitoring plans that include change management (MANAGE 4.1).
Source: NIST AI RMF core
Explore this on Agnotic
Where MLOps shows up in how we build.