A message arrives that your consumer cannot process. Maybe the payload is malformed, maybe it references a record that no longer exists, maybe there is a bug in the handler. The consumer throws, the message is not acknowledged, and the broker redelivers it. The consumer throws again. This repeats forever, and in an ordered partition everything behind that message waits behind it.
A dead letter queue is the escape valve: after a defined number of failures, the message is moved aside so processing continues.
What changed in 2026
- Managed brokers made it a default. More platforms shipped dead letter configuration as standard rather than something to build, which raised the baseline.
- Failure classification improved. Distinguishing retriable from non-retriable errors in consumer code, rather than retrying uniformly, became recognized as the meaningful design decision.
- Replay tooling matured. Purpose-built interfaces for inspecting, editing, and selectively replaying dead-lettered messages reduced the amount of ad hoc scripting involved.
- Monitoring gaps got attention. Enough incidents traced back to unmonitored dead letter queues that alerting on depth became standard advice rather than an afterthought.
Transient versus permanent failure
| Failure |
Type |
Right response |
| Downstream service timeout |
Transient |
Retry with backoff |
| Rate limited by an API |
Transient |
Retry after the indicated delay |
| Database connection lost |
Transient |
Retry |
| Malformed payload |
Permanent |
Dead letter immediately; retry is pointless |
| Schema version unsupported |
Permanent |
Dead letter; needs a code change |
| Referenced entity does not exist |
Ambiguous |
Retry briefly; it may be an ordering issue |
| Business rule rejection |
Permanent |
Dead letter or route to a review flow |
Retrying a permanently failing message wastes capacity and delays the dead-lettering that should have happened immediately. Classifying failures in the handler — retriable exception versus terminal exception — is a small amount of code that meaningfully improves both throughput and diagnosis.
The ambiguous row is worth handling deliberately. A message referencing an entity that does not exist yet may just have arrived before its creation event. A short bounded retry handles the ordering race; indefinite retry hides a real problem.
What to store and how to replay
Store enough context to diagnose without the original system state. The message body, the failure reason, the exception, the number of attempts, the consumer version, and a timestamp. A dead letter queue containing only the original payload leaves you guessing why it failed.
Replay needs discipline. Fix the underlying cause first — replaying into the same bug produces the same failure and doubles the queue. Then verify with a single message. Then replay in batches small enough that you can observe the result before committing to the whole backlog.
Consumers must be idempotent for replay to be safe. A message may have partially succeeded before failing, and replaying it will re-execute whatever completed. That is the same requirement discussed in idempotency keys explained, applied to message consumers rather than HTTP endpoints.
Alert on queue depth. A dead letter queue nobody watches is a place messages go to be forgotten, and teams routinely discover thousands of dropped events months after the fact. Any non-zero depth deserves a look; sustained growth deserves a page.
Common mistakes
- No dead letter queue at all. One poison message stalls a partition indefinitely.
- No monitoring on it. Silent data loss that surfaces during an audit.
- Retrying permanent failures. Wastes capacity and delays diagnosis.
- Replaying before fixing. Same failure, more messages, more confusion.
- Non-idempotent consumers. Makes replay unsafe, which makes the whole mechanism unusable.
- Discarding failure context. Diagnosis becomes archaeology.
FAQ
How many retries before dead-lettering?
For transient failures, a small number with exponential backoff — three to five is common. For permanent failures, zero; go straight to the dead letter queue.
Should the dead letter queue have its own dead letter queue?
No. It is a terminal destination. If writing to it fails, that is an alert-worthy infrastructure problem, not a case for another layer.
How long should messages stay there?
Long enough to diagnose and replay, which usually means weeks rather than days. Set a retention period rather than letting it grow forever.
Can I edit a message before replaying?
Sometimes appropriate for a malformed payload where the intent is clear. Log that you did it, because a replayed message that differs from the original is a fact investigators need.
Where to go next
For overload handling upstream, read backpressure explained. For safe reprocessing, idempotency keys explained and outbox pattern explained.