What happens to an HL7 message when something goes wrong?
In a demo, every HL7 message is well-formed and every endpoint is up. In production, messages arrive malformed, downstream systems go down, and data violates assumptions your mapping made. The interfaces that survive aren't the ones that never fail — they're the ones where failure is caught, queued, and recoverable instead of silently dropped. This guide covers the error-handling architecture that makes an HL7 interface reliable: acknowledgments, queuing, retries, dead-lettering, and reprocessing.
Two kinds of failure
Distinguish transient failures from permanent ones, because they demand opposite responses. A transient failure — the downstream system is momentarily down, the network blipped — will succeed if you try again later, so the right response is queue and retry. A permanent failure — the message is malformed, references a patient that doesn't exist, violates a required mapping — will never succeed on retry, so retrying forever just clogs the queue. The right response there is to dead-letter it for human attention. Conflating the two is a common design flaw.
Never drop a message silently
The cardinal rule: no message should ever disappear without a trace. Whether it succeeds, is retried, or is dead-lettered, there must be a durable record. That starts with honest acknowledgments — don't return AA if you couldn't process the message — and continues with persistent storage of every message and its outcome. Silent loss is the failure mode that erodes clinical trust, because it surfaces as 'the result never arrived' with no evidence of what happened.
Queue on the outbound side
A robust interface decouples receiving from delivering with a durable queue. When you accept a message, persist it before attempting delivery; then deliver from the queue with retries. This way a downstream outage doesn't reject or lose inbound messages — they safely back up in the queue and drain when the endpoint recovers. Interface engines provide this (Mirth's destination queue, for example); the key is to actually enable and monitor it rather than delivering synchronously and failing hard.
Retry with backoff and limits
- Retry transient failures with exponential backoff so you don't hammer a struggling endpoint.
- Cap retries or retry duration — infinite retries on a permanent error hide the problem.
- After the cap, move the message to a dead-letter store rather than dropping it or retrying forever.
- Make delivery idempotent so a retried message that actually did arrive doesn't create a duplicate.
Dead-letter queues for the unrecoverable
A dead-letter queue (DLQ) is where messages go when they can't be processed automatically — a malformed message, an unmappable value, an exhausted retry. The DLQ isn't a graveyard; it's a work queue. Each dead-lettered message should carry the reason it failed and enough context to diagnose it. Someone (or an alert) needs to watch the DLQ, because a growing DLQ means real data isn't flowing. A DLQ nobody looks at is just silent loss with extra steps.
Reprocessing: fix and replay
The payoff of storing everything is reprocessing. When you fix a mapping bug or a downstream comes back online, you replay the affected messages instead of asking the source to resend (which it often can't). Good interface engines let you select historical or dead-lettered messages and reprocess them through updated channel logic. This turns a mapping defect from a data-loss event into a fix-and-replay operation — provided you kept the messages in the first place.
Monitoring and alerting
- Alert on queue depth growth — it's the earliest sign a downstream is failing.
- Alert on errored and dead-lettered messages, not just on total outages.
- Track ACK codes: a spike in AE/AR means the receiver is rejecting valid-looking traffic.
- Watch connection health so a silently dead MLLP link is caught before messages pile up.
The reliability mindset
Reliable HL7 integration assumes failure as normal and designs for recovery: honest acknowledgments, persist-before-deliver, backed-off retries for transient errors, dead-lettering for permanent ones, reprocessing to fix and replay, and monitoring that catches trouble early. None of it is exotic — it's ordinary distributed-systems reliability applied to healthcare messaging. What makes it non-negotiable here is the stakes: a dropped message can be a missing lab result, and that's not a bug ticket, it's a patient-safety issue.
