A message broker is a piece of infrastructure that accepts messages from producers and delivers them to consumers — asynchronously, reliably, and with decoupling between the two sides. Producers don't need to know who consumes their messages, and consumers don't need to be running when the message is published. This makes message brokers a foundational pattern for microservices, event-driven systems, and data pipelines. In 2026, the two dominant open-source options remain Apache Kafka and RabbitMQ, and the choice between them is consequential.
What changed in 2026
- KRaft mode is now Kafka's default. Apache Kafka 3.7+ fully replaced ZooKeeper with KRaft (Kafka Raft), simplifying deployment significantly — fewer components, faster metadata operations.
- RabbitMQ 4.x with Khepri replaced the Mnesia metadata store with Raft-based Khepri, improving clustering stability and recovery.
- Serverless brokers matured. AWS EventBridge, Google Pub/Sub, and Upstash Kafka offer consumption-based pricing with zero ops overhead — relevant for teams that don't need Kafka-level throughput.
- WASM consumers run broker consumers at the edge (Cloudflare Workers with Confluent Cloud) without spinning up full VMs.
Kafka vs RabbitMQ: the core difference
Kafka is a distributed, ordered, immutable log. Messages are retained for a configurable period (days to forever). Multiple consumer groups can independently read the same stream from any offset. Kafka is a pub/sub system optimized for high throughput and replay.
RabbitMQ is a message broker that routes messages to queues. Messages are consumed once and deleted (by default). It supports complex routing (topic, fanout, direct, headers exchanges) and acknowledgment-based delivery. RabbitMQ is a queue system optimized for task distribution and flexible routing.
Comparison table
| Dimension |
Kafka |
RabbitMQ |
| Retention |
Time/size-based log |
Consumed messages deleted |
| Consumer model |
Consumer groups read independently |
Competing consumers share a queue |
| Throughput |
Very high (millions msg/s per cluster) |
High (hundreds of thousands msg/s) |
| Ordering |
Per-partition ordering |
Per-queue ordering |
| Replay |
Yes — seek to any offset |
No — consumed messages gone |
| Routing |
By topic/partition key |
Rich exchange routing |
| Latency |
~5–15 ms typical |
~1–5 ms typical |
| Operational complexity |
High (but KRaft helps) |
Medium |
When to use Kafka
- Event streaming / CDC — capturing every change to a system as an ordered stream.
- Data pipelines — feeding multiple downstream consumers (analytics, search, ML) from one event stream.
- Event sourcing — Kafka is the event store; consumers derive state by replaying.
- High throughput ingestion — IoT telemetry, clickstreams, log aggregation.
- Audit trails — immutable log with configurable retention.
# Kafka producer with confluent-kafka-python
from confluent_kafka import Producer
p = Producer({"bootstrap.servers": "kafka:9092"})
def delivery_report(err, msg):
if err:
print(f"Delivery failed: {err}")
p.produce("order-events", key=str(order_id),
value=order_json, callback=delivery_report)
p.flush()
When to use RabbitMQ
- Task queues — background jobs, email sending, video encoding.
- Work distribution — multiple workers compete for tasks; each task handled once.
- RPC patterns — request-reply over a broker.
- Complex routing — messages fanned out to some queues but not others based on routing keys.
- Low-latency delivery — when per-message latency matters more than throughput.
# RabbitMQ task queue with pika
import pika, json
connection = pika.BlockingConnection(pika.ConnectionParameters("rabbitmq"))
channel = connection.channel()
channel.queue_declare(queue="email_tasks", durable=True)
channel.basic_publish(
exchange="",
routing_key="email_tasks",
body=json.dumps({"to": "user@example.com", "template": "welcome"}),
properties=pika.BasicProperties(delivery_mode=2), # persistent
)
Managed options compared
| Service |
Backend |
Pricing model |
Best for |
| Confluent Cloud |
Kafka |
Per CKU / storage |
Enterprise Kafka |
| AWS MSK |
Kafka |
Per broker-hour |
AWS-native apps |
| Upstash Kafka |
Kafka |
Per message |
Serverless / edge |
| CloudAMQP |
RabbitMQ |
Per month/tier |
Simple task queues |
| AWS SQS |
Custom (queue) |
Per request |
Simple queues on AWS |
| Google Pub/Sub |
Custom (topic) |
Per message |
GCP-native streaming |
How to pick
- Need replay or multiple consumers reading independently? → Kafka.
- Task queue — each job handled by exactly one worker? → RabbitMQ or SQS.
- Complex routing logic between producers and consumers? → RabbitMQ exchanges.
- Millions of events per second? → Kafka.
- Minimal ops overhead? → Upstash (Kafka), SQS, or Pub/Sub managed services.
- Already on AWS? → SQS for simple queues, MSK or Confluent for Kafka.
Common mistakes
Not handling consumer failures. If a consumer crashes after reading but before acknowledging, the message must be redelivered. Always use explicit ACK/NACK; never auto-ACK in production workers.
Kafka consumer group offset management. Committing offsets before processing completes causes message loss on crash. Commit after successful processing.
# Kafka consumer — commit after processing
for message in consumer:
process(message) # do the work first
consumer.commit() # then commit the offset
No dead-letter queue (DLQ). Messages that fail processing repeatedly should move to a DLQ for inspection, not loop forever in the main queue.
Kafka topic proliferation. Creating a new Kafka topic for every event type without governance leads to hundreds of topics, no schema registry, and consumer confusion.
What to skip
- Self-managed Kafka for small teams — Kafka cluster operations (ZooKeeper/KRaft, replication, disk management) require dedicated expertise. Use Confluent or MSK.
- Message brokers for synchronous request/response — if the caller blocks waiting for a reply, an HTTP API is simpler. Brokers shine for fire-and-forget.
- RabbitMQ for event sourcing — without replay, RabbitMQ cannot serve as an event store. Use Kafka or a dedicated event store (EventStoreDB).
FAQ
Can Kafka and RabbitMQ coexist in the same system?
Yes, and it is common. Kafka handles the high-throughput event stream; RabbitMQ handles task distribution for specific workloads.
What is a consumer group in Kafka?
A consumer group is a set of consumers that collectively read a topic. Each partition is assigned to one consumer in the group, enabling parallel processing. Multiple groups can read the same topic independently.
How do I guarantee exactly-once delivery?
Kafka supports exactly-once semantics (EOS) with idempotent producers and transactions (enable.idempotence=true, transactional.id). For RabbitMQ, implement idempotency in the consumer since at-least-once is the native guarantee.
What is a schema registry and do I need one?
A schema registry (Confluent Schema Registry, AWS Glue) enforces Avro/Protobuf/JSON Schema for messages. Highly recommended for Kafka in production — prevents schema drift from breaking consumers.
Where to go next