A sender delivers a message and waits for acknowledgement. The acknowledgement is lost in transit. The sender does not know whether the message arrived, so it either resends — risking a duplicate — or does not — risking loss.
There is no third option. Any protocol has this problem at its last step, and adding more acknowledgements just moves it. Exactly-once delivery is not achievable across an unreliable channel, and no amount of engineering makes it so.
What changed in 2026
- The framing shifted to exactly-once effects. Vendor messaging moved toward describing outcomes rather than delivery, which is more honest.
- Idempotent consumers became the standard pattern. Building for duplicates rather than trying to prevent them became the default assumption.
- Broker-level deduplication improved within boundaries. Systems offering it within their own transactional scope got better, with the boundary limitation better understood.
- Retention windows got documented. The finite nature of deduplication state became a design consideration rather than a surprise.
Delivery versus effects
The useful distinction.
Exactly-once delivery means the message is transmitted and received precisely once. Impossible, for the reason above.
Exactly-once effects means the result of processing occurs once, regardless of how many times the message arrives. Entirely achievable, and it is what you actually wanted.
| Guarantee |
Achievable |
How |
| At-most-once |
Yes |
Do not retry; accept loss |
| At-least-once |
Yes |
Retry until acknowledged |
| Exactly-once delivery |
No |
— |
| Exactly-once effects |
Yes |
At-least-once plus idempotency |
So the design is: deliver at least once, and make processing idempotent so duplicates are harmless. The messaging layer guarantees arrival; the consumer guarantees the effect happens once.
The deduplication key
Idempotent processing needs a way to recognise a message it has already handled, and where that key comes from matters.
The producer must supply it. A key generated by the receiver on arrival is different for each delivery of the same message, which makes it useless for deduplication.
It must be stable across retries. If the producer generates a new identifier when retrying, the retry looks like a new message.
It should be meaningful where possible. An order identifier, a transaction reference, or a natural business key is more useful than a random value, because it deduplicates across producers and retries alike.
Recording processed keys and skipping anything already seen turns duplicate delivery into a no-op — the same mechanism as idempotency explained.
The retention problem
Deduplication state cannot grow forever. You must eventually forget processed identifiers, and the moment you forget one, a very delayed duplicate would be processed again.
So deduplication is bounded by a retention window. Choose one longer than any plausible delivery delay — including the broker's maximum retry period and any dead-letter reprocessing — and accept that a duplicate arriving after it is theoretically possible.
For most systems a window of days is comfortably sufficient. Where it is not, the alternative is making the operation naturally idempotent rather than relying on deduplication state: an upsert keyed on a business identifier, or a state transition checking current state, needs no memory of past messages at all.
That is the stronger design where it applies — see upsert patterns.
Common mistakes
- Trusting an exactly-once claim across a system boundary. Broker guarantees stop at the broker.
- Receiver-generated deduplication keys. Cannot recognise duplicates.
- Unstable producer keys. Retries look like new messages.
- Unbounded deduplication state. Grows forever.
- Ignoring ordering. Duplicates and reordering are separate problems.
- Idempotency checks not atomic with the effect. A crash between check and effect reintroduces the duplicate.
- At-most-once by accident. Acknowledging before processing loses messages on failure.
FAQ
What about brokers advertising exactly-once?
They typically provide it within their own transactional scope — read, process, and write back inside one transaction the broker controls. That is genuine and it stops at the boundary. A side effect in an external system is outside the guarantee.
Should I acknowledge before or after processing?
After, for at-least-once. Acknowledging first gives at-most-once, where a crash after acknowledgement loses the message. At-least-once with idempotency is nearly always the right choice.
How do I handle ordering?
Separately — deduplication does not address it. Partition by a key so related messages are ordered, and include a version or timestamp so a late-arriving older message can be recognised and ignored.
What about messages that fail repeatedly?
A dead-letter queue after a retry limit, so a poison message does not block the stream. Reprocessing from there must respect the same deduplication window.
Where to go next
For the idempotency mechanism, read idempotency explained and upsert patterns. For the webhook case of the same problem, webhook signing.