Message queues are what you reach for when a synchronous API call is the wrong shape for the problem — when the sender doesn't need to wait, the receiver might be slow, or the work needs to survive a service restart. They're also one of the most over-engineered infrastructure choices in the industry. Here is the clarity you need in 2026.
What changed in 2026
- Redpanda 4.x reached full Kafka API compatibility with no JVM dependency and significantly better tail latency — many teams are migrating from self-hosted Kafka to Redpanda for operational simplicity.
- AWS SQS added FIFO deduplication windows up to 24 hours and improved its exactly-once semantics for Lambda-triggered consumers.
- RabbitMQ 4.0 shipped a rewritten quorum queue implementation that is now the default over classic mirrored queues for all new deployments.
- NATS JetStream matured into a serious Kafka alternative for teams that want persistence + stream replay without the Kafka operational surface.
Queue vs topic vs stream
| Model |
Delivery |
Replay |
Best for |
| Queue (SQS, RabbitMQ) |
One consumer per message |
No |
Task distribution, work queues |
| Topic/fan-out (SNS, RabbitMQ exchange) |
All subscribers get a copy |
No |
Notifications, fan-out events |
| Stream (Kafka, Redpanda, NATS JetStream) |
Consumer groups, offset-based |
Yes, indefinitely |
Event sourcing, audit logs, replay |
The distinction matters: queues distribute work; streams broadcast events for any consumer to replay from any point.
Delivery guarantees
| Guarantee |
Meaning |
Hard to achieve because |
| At-most-once |
May be lost, never duplicated |
Network ack before persistence |
| At-least-once |
Never lost, may duplicate |
Retry on no-ack can resend |
| Exactly-once |
Never lost, never duplicated |
Requires 2PC or idempotency key |
At-least-once is the practical standard. Design every consumer to be idempotent — processing the same message twice must produce the same result as processing it once.
def process_payment(message: dict):
payment_id = message["payment_id"]
# Idempotency check: skip if already processed
if db.exists("SELECT 1 FROM payments WHERE id = %s AND status = 'processed'",
payment_id):
return # Safe to acknowledge and discard
db.execute("INSERT INTO payments ... ON CONFLICT (id) DO NOTHING", ...)
ack(message)
Dead-letter queues
A dead-letter queue (DLQ) receives messages that failed processing after N retries. Without one, a poison message blocks the queue indefinitely (or is silently dropped).
Set up a DLQ for every queue. Set a maximum receive count (3–5 retries is typical). Monitor DLQ depth — rising DLQ messages are an alert-worthy event.
// SQS RedrivePolicy
{
"maxReceiveCount": 5,
"deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789:my-queue-dlq"
}
Broker comparison
| Broker |
Throughput |
Replay |
Managed option |
Operational cost |
| AWS SQS |
Medium (3k msg/s/queue) |
No |
Yes (fully) |
Very low |
| RabbitMQ |
High |
No |
CloudAMQP, AmazonMQ |
Medium |
| Kafka |
Very high (millions/s) |
Yes |
Confluent, MSK |
High |
| Redpanda |
Very high |
Yes |
Redpanda Cloud |
Medium |
| NATS JetStream |
High |
Yes |
Synadia Cloud |
Low |
For most product teams: start with SQS (simple, cheap, managed, good enough). Graduate to Kafka/Redpanda when you need stream replay or >10k messages/second sustained.
How to pick
- Do you need replay or event sourcing? → Kafka, Redpanda, or NATS JetStream.
- Is simplicity and cost your priority? → SQS (or Azure Service Bus / Google Pub/Sub).
- Do you need priority queues or complex routing? → RabbitMQ.
- Are you already on Kafka and need less ops overhead? → Migrate to Redpanda or Confluent Cloud.
- Do you need sub-millisecond latency? → NATS (core, without JetStream).
Common mistakes
Not making consumers idempotent. At-least-once delivery is the contract; duplicate messages are not a bug in the broker, they are guaranteed behaviour. Handle them.
Using a RDBMS as a queue. Polling SELECT + DELETE at scale creates lock contention, table bloat, and competes with your primary workloads. Fine for <100 msg/min; a problem above that.
Unbounded retry without backoff. Retrying a failing consumer immediately at full speed amplifies the downstream problem. Implement exponential backoff with jitter.
No DLQ. A poison message with no DLQ will either block the queue forever or be silently dropped — neither is acceptable.
Large message payloads. Message brokers are for metadata and pointers, not binary blobs. Store large payloads in S3/GCS and pass the reference in the message.
What to skip
- Exactly-once semantics for high-throughput paths — the 2PC cost usually isn't worth it; idempotency + at-least-once is cheaper and good enough.
- Kafka for simple task queues — the operational overhead of Kafka is significant; SQS is the right tool for basic async task dispatch.
- Synchronous message processing — if you're waiting for the consumer before responding to the caller, you've built a slower RPC call, not a queue.
FAQ
Queue or API call for inter-service communication?
Queue when you don't need the result immediately, when the consumer may be slow or temporarily unavailable, or when you want automatic retry. API call when you need a synchronous response.
How do I handle message ordering?
SQS FIFO queues and Kafka partition keys both provide ordering within a partition/group. Cross-partition ordering requires application-level sequence numbers — plan for this upfront.
What is a competing consumer pattern?
Multiple consumer instances pulling from the same queue, each processing a different message in parallel. This is the standard horizontal scaling pattern for queue-based workers.
How large should messages be?
Keep messages under 64 KB as a rule of thumb. SQS hard limit is 256 KB; Kafka default is 1 MB. For anything larger, use the claim-check pattern (store payload externally, pass reference).
Where to go next
See Event-driven architecture in 2026 for how message queues fit into a larger async system, and Caching strategies in 2026 for reducing the volume of work that needs to be queued in the first place.