GPU vs CPU for healthcare ML inference: when do you need a GPU?
Most healthcare models do not need a GPU to serve predictions. Imaging models and large language models usually do, while tabular risk scores, many clinical NLP classifiers and signal models often run well on CPU once they are batched or quantized. The workload, the latency it must meet and where PHI may be processed decide the hardware.
When does a healthcare model need GPU inference?
Quick answer
GPU vs CPU for healthcare ML inference comes down to the workload: imaging models and LLM serving usually need GPUs, while tabular risk scores, many clinical NLP classifiers and waveform models often meet their latency on CPU after batching or quantization. Agnotic Technologies profiles each model on your data before you commit to GPU capacity.
| Criterion | GPU | CPU |
|---|---|---|
| Medical imaging models | Usually needed: 3D volumes, image series and whole-slide tiles carry heavy computation per study, often with a reader waiting | Possible for small 2D models in overnight batch, if the time per study fits the reading workflow |
| LLM serving | Needed for self-hosted models beyond small sizes; generation speed decides how long the user waits | Only small or heavily quantized models at low concurrency; otherwise use a hosted model API covered by a BAA |
| Clinical NLP | Large encoders on a high-volume real-time feed | Often enough for smaller encoders, backfills and per-note calls that can wait a second or two |
| Tabular risk scores | Rarely needed | The usual choice: tree ensembles and linear models score quickly on CPU |
| Waveform and device signals | Deep models over long multichannel windows or many concurrent streams | Feature-based models, single streams and on-device scoring |
| Latency target | Interactive use where a clinician waits on the result | Queued jobs, batch windows and small models that already meet the target |
| Traffic shape | Steady, high volume that keeps the card busy; batching raises throughput | Spiky or low volume, where an idle GPU costs more than scaling CPU instances |
| PHI hosting check | Confirm the GPU instance type or managed endpoint is covered by your cloud BAA in your region | Same check; CPU options exist in more regions and services |
| After optimization | Mixed precision or compiled engines change numerics: rerun the evaluation set | 8-bit quantization changes numerics too: rerun the evaluation set |
What they are
What is GPU
GPU inference runs the model on graphics processors built for large parallel matrix work. It pays off when each request carries a lot of computation, such as a 3D imaging study or token generation in a language model, or when traffic is steady enough to keep the card busy.
What is CPU
CPU inference runs the model on general-purpose processors, the same instances that run the rest of the application. It suits models with small per-request cost, batch jobs that have hours to finish and spiky traffic that autoscaling handles better than a reserved GPU.
Key differences
Why does the workload decide, not the framework?
The same framework can serve a model on either processor. What changes is the computation per request and how many requests arrive at once. A chest CT segmentation model and a readmission score may sit in the same product, yet one needs a GPU to return in seconds and the other finishes on CPU before the page renders. Profile each model on realistic inputs from your own data, not on vendor demo inputs.
Real-time or batch: which does your workflow need?
Ask who waits for the answer. A radiologist reading a study or a nurse at a triage screen needs a result in the workflow, so latency drives the hardware. A nightly population risk run, a coding backlog or a research extract only needs to finish inside its window. Many teams buy GPU capacity for jobs that would finish on CPU overnight, and run interactive models on CPU where users then wait.
What should you optimize before buying GPU capacity?
Start with the model. ONNX Runtime documents quantization as 8-bit linear quantization of an ONNX model, mapping floating point values to an 8-bit integer space, and provides APIs to convert a 32-bit float model to an 8-bit integer model. Distillation into a smaller model, exporting to ONNX, trimming input size and batching requests often bring a model inside its CPU latency target. Each of these changes the model, so it needs revalidation.
Can one serving stack run both CPU and GPU models?
Yes. NVIDIA Triton Inference Server lets a model configuration place execution instances on the CPU with an instance group of kind KIND_CPU, even when a GPU is present, or on GPUs with KIND_GPU. Its dynamic batcher combines individual inference requests into a batch on the server, which typically raises throughput. One stack serving both keeps monitoring, versioning and access control in one place.
Where may the model process PHI?
Inference on clinical data is PHI processing. The instance family, managed endpoint or model API must be covered by your cloud provider's business associate agreement in the region you deploy to, and GPU types are not offered in every region. Check this before sizing, because the answer can rule out a hardware option. Our healthcare cloud selection guide covers the BAA check per service.
What changes if the model is part of a medical device?
If the model is an AI-enabled device software function, a change such as quantizing it or moving it to new hardware is a modification to the device. FDA's final guidance on predetermined change control plans for AI-enabled devices recommends that a plan describe the planned modifications, the method to develop, validate and implement them, and an assessment of their impact. Plan hardware changes inside that process, not after it.
When does on-device or edge inference make sense?
When the device or phone must work without a connection, when sending raw signals to the cloud is too slow or too costly, or when keeping data on the device reduces what leaves it. Edge inference runs on CPU or a small neural processor, so it forces the same optimization work early: a smaller model, quantized weights and a fixed input size.
When to use each
When to use GPU inference
- Imaging models over CT or MRI volumes, X-ray series or whole-slide pathology tiles.
- Self-hosted language models where a clinician or patient waits for the generated text.
- Large transformer encoders running clinical NLP on a high-volume real-time stream.
- Deep models over long multichannel waveforms from many devices at once.
When to use CPU inference
- Tabular risk scores such as readmission, no-show or deterioration screens on structured data.
- Smaller clinical NLP classifiers, coding suggestions or note routing that tolerate some latency.
- Overnight or queued batch scoring across a population.
- Feature-based signal models and single-stream scoring at the edge.
Where does healthcare inference fail after launch?
1. Cold starts. A scaled-to-zero GPU endpoint takes too long to load the model for the first request of the morning. Fix: keep a warm minimum for interactive models, or move them to CPU if they fit.
2. GPU quota or regional capacity. The account cannot get the instances it planned for in the BAA-covered region. Fix: confirm quota and availability before launch, and keep a tested CPU fallback for degraded service.
3. A quantized model that was never revalidated. Throughput improves, but sensitivity on a subgroup drops. Fix: rerun the full evaluation set, including subgroups, on every optimized build.
4. PHI sent to a non-covered endpoint. A test endpoint or a new model API ends up in production traffic. Fix: allow-list approved endpoints at the network layer and alert on any other destination.
5. Batch windows that quietly overrun. A nightly score finishes after the morning huddle starts. Fix: alert on job duration, not just job failure.
6. Different numerics in each environment. Staging runs on CPU and production on GPU, so results differ slightly. Fix: validate on the hardware and runtime you will serve on.
7. Unpinned model versions. A container rebuild pulls a newer model file. Fix: version models like code and record the version with every prediction.
Which sources does this comparison rely on?
- FDA final guidance: predetermined change control plans for AI-enabled device software functions (fda.gov)
- ONNX Runtime documentation: quantize ONNX models (onnxruntime.ai)
- NVIDIA Triton Inference Server: model configuration and instance groups (docs.nvidia.com)
- NVIDIA Triton Inference Server: dynamic batcher (docs.nvidia.com)