Message queues and event streams
RabbitMQ, Kafka, SQS, Google Pub/Sub, Azure Service Bus: they all "move messages", and they are not interchangeable. Some are queues that hand each task to one worker and forget it; some are logs that keep every event for many readers to replay. This guide explains how each product works, how they differ, and which to choose for a given job, with a chooser you can try and code you can run.
- Sam RabbitMQ, Kafka, SQS, Pub/Sub. Aren't they all just queues?
- Priya They all move messages, but they aren't interchangeable. Some are queues: each task goes to one worker and is deleted when it's done. Others are logs: every event is kept, and many readers can read it, each at its own pace, and even rewind.
- Sam How do I tell which shape I need?
- Priya Ask what a message is. A task to do once wants a queue. An event several systems care about, or one you might need to re-read, wants a log.
- Sam Walk me through it.
- Priya Four orders. In a queue, each goes to one worker and is hidden while it's processed. If the worker crashes, the message comes back after the timeout, behind the others, so order is lost, and after a few failures it goes to a dead-letter queue. In a log, billing and analytics each read all four, and analytics can rewind to the start and replay.
- Sam What are the main products?
- Priya RabbitMQ, a broker that routes messages to queues, with quorum queues and streams. Kafka, a partitioned log; since version four point two, its share groups add queue-style consumption. Pulsar and NATS. And the managed ones: Amazon SQS, a queue with messages up to one mebibyte; Kinesis, a sharded log; Google Pub/Sub, with retention and seek for replay; and on Azure, Service Bus for queues and Event Hubs for streams.
- Sam Which keep order?
- Priya Per key, not globally. Kafka and Kinesis order per partition or shard, Pub/Sub per ordering key, SQS FIFO per message group, Service Bus per session. Almost nothing needs global order.
- Sam So how do I choose?
- Priya Task or event first. Then, on a cloud, start with its managed service, and only run your own cluster for a reason: portability, Kafka's ecosystem, complex routing, very low latency, or cost at huge volume.
- Priya Your turn. You're on Google Cloud, and three services need every order event. What fits?
- Priya Google Pub/Sub, with one subscription per service. Turn on topic retention if you need replay.
- Sam What does the code show?
- Priya The same orders through a queue and a log. The queue redelivers a failed message and dead-letters the poison one, and the retried order finishes last. The log lets two groups read everything, and one replays from the start.
- Sam Common mistakes?
- Priya A queue when several systems need the event. Kafka for a few hundred messages a minute. Assuming exactly-once end to end. Asking for global order. And putting large payloads in messages instead of object storage.
1. What it is
A message system sits between the services that produce work or events and the services that consume them. It lets the producer carry on without waiting, absorbs bursts, and survives a consumer being down. Module 18 teaches the concepts; this guide is about the products.
Every product is one of two shapes, or a hybrid:
| Queue | Log (event stream) | |
|---|---|---|
| What a message is | A task to do once | An event that happened, which many systems may care about |
| After it's processed | Deleted when the consumer acknowledges it | Kept for the retention period; each reader tracks its own position (offset) |
| Many readers | Competing workers share the messages | Each consumer group reads every event independently |
| Replay | No: once acknowledged, it's gone | Yes: rewind the offset and read again |
| Examples | RabbitMQ classic and quorum queues, Amazon SQS, Azure Service Bus | Kafka, Kinesis, Azure Event Hubs, RabbitMQ streams, Pulsar, Google Pub/Sub with retention and seek |
2. How it works
The same four orders, through a queue and through a log.
3. The products
RabbitMQ
Producers publish to an exchange, which routes messages to queues by key, pattern or headers; the broker pushes them to consumers with a prefetch limit. Three queue types: classic (not replicated), quorum (replicated with Raft) and streams (an append-only log). Classic queue replication was removed in 4.x.
Best for: task queues with flexible routing, request-reply, moderate volumes.
Apache Kafka
A topic is split into partitions, each an ordered, replicated log on the brokers. Consumers pull and commit offsets; retention is by time or size, with compaction to keep the latest value per key. Runs on KRaft, with no ZooKeeper. Kafka 4.2 (February 2026) made share groups, queue-style consumption, production-ready.
Best for: high-throughput event streams, many independent consumers, replay, stream processing.
Apache Pulsar
Stateless brokers serve topics stored in Apache BookKeeper, so storage and serving scale separately. Subscriptions can be exclusive, shared, failover or key-shared; geo-replication and tiered storage are built in.
Best for: multi-tenant platforms, geo-replication, mixing queue and stream use.
NATS
Core NATS is very low-latency publish-subscribe with at-most-once delivery and wildcard subjects. JetStream adds persistent streams with at-least-once delivery and a deduplication window.
Best for: low-latency service messaging, edge and IoT, simple operations.
Redis Streams
An append-only stream held in memory, with consumer groups and a pending list per consumer; XAUTOCLAIM lets another consumer take over stuck messages.
Best for: modest volumes when you already run Redis or Valkey.
Amazon SQS
A fully managed queue. Standard queues are at-least-once with best-effort order and nearly unlimited throughput; FIFO queues order messages within a message group and deduplicate. Messages up to 1 MiB, kept 1 minute to 14 days (default 4 days); a visibility timeout (default 30 seconds, up to 12 hours) hides a message while it's processed; failures redrive to a dead-letter queue.
Best for: task queues on AWS with no operations work. Pair with SNS to fan out.
Amazon Kinesis Data Streams
A managed stream of shards. In provisioned mode each shard takes up to 1 MB/s or 1,000 records/s in and 2 MB/s out; on-demand mode scales shards for you. Records up to 10 MiB, retained up to 365 days; enhanced fan-out gives registered consumers their own read capacity.
Best for: event streams and analytics pipelines on AWS.
Google Pub/Sub
Global topics with subscriptions that pull or push. Messages up to 10 MB; unacknowledged messages kept 7 days by default, and topics can retain messages up to 31 days for replay with seek. Ordering keys order messages per key (up to 1 MBps per key); exactly-once delivery is an option.
Best for: almost any messaging on Google Cloud, both task fan-out and event streams.
Azure Service Bus
Managed queues and topics with subscriptions, filters, sessions for ordered processing, duplicate detection and dead-lettering. Messages up to 256 KB on Standard, and up to 100 MB on Premium over AMQP; queues and topics up to 80 GB.
Best for: enterprise task queues and pub/sub on Azure.
Azure Event Hubs
A managed, partitioned event stream with a Kafka-compatible endpoint. Retention up to 7 days on Standard and 90 days on Premium and Dedicated; events up to 1 MB (20 MB on Dedicated); each throughput unit takes 1 MB/s in and 2 MB/s out.
Best for: event streams on Azure, and running Kafka clients without running Kafka.
4. Differences at a glance
| Product | Shape | Ordering | Delivery | Replay | Run by |
|---|---|---|---|---|---|
| RabbitMQ | Broker with queues; streams as a log | Per queue (lost on redelivery) | At-least-once with acks | Streams only | You |
| Kafka | Partitioned log | Per partition (per key) | At-least-once; exactly-once within Kafka with transactions | Yes, within retention | You, or a managed Kafka service |
| Pulsar | Log over BookKeeper | Per partition or per key | At-least-once; deduplication available | Yes | You, or a managed service |
| NATS | Pub/sub; JetStream streams | Per subject stream | At-most-once (core); at-least-once (JetStream) | JetStream only | You, or a managed service |
| Redis Streams | In-memory log | Per stream | At-least-once with a pending list | Yes, while kept | You, or managed Redis/Valkey |
| SQS | Queue | None (Standard); per message group (FIFO) | At-least-once; FIFO deduplicates | No | AWS |
| Kinesis | Sharded log | Per shard | At-least-once | Yes, up to 365 days | AWS |
| Pub/Sub | Pub/sub with retention | Per ordering key | At-least-once; exactly-once option | Yes, with topic retention and seek | Google Cloud |
| Service Bus | Broker: queues and topics | Per session | At-least-once; duplicate detection | No | Azure |
| Event Hubs | Partitioned log | Per partition | At-least-once | Yes, within retention | Azure |
5. When to choose which, and why
- Task or event? A task done once by one worker wants a queue. An event several systems react to, or one you may need to re-read, wants a log.
- Where do you run? On a cloud, start with its managed service: SQS (+ SNS), Kinesis, Pub/Sub, Service Bus or Event Hubs. Running your own cluster is a job; take it on only for a reason.
- Reasons to run your own: portability across clouds, Kafka's ecosystem (Connect, Streams, the clients), complex routing (RabbitMQ), sub-millisecond messaging (NATS), or costs at very high volume.
- Ordering: almost nothing needs global order. Order per key (a user, an order) is enough, and every product above offers it.
- Throughput: tens of thousands of messages a second and beyond points to a partitioned log; plan partitions (Module 18 has the arithmetic).
Try it: choose for your requirements
6. Queue vs log, in code
The same orders through a queue and a log. In the queue, a message that fails is redelivered and, after three failures, goes to the dead-letter queue; notice that the retried order finishes last, so a plain queue doesn't keep order across retries. In the log, two consumer groups read every event independently, and one rewinds to replay. The program was run in all three languages; each prints the output below it.
class Queue:
"""A work queue: each message goes to one worker; unacked messages come back; repeated failures go to a DLQ."""
def __init__(self, max_tries=3):
self.ready, self.dlq, self.tries, self.max_tries = [], [], {}, max_tries
def send(self, msg):
self.ready.append(msg)
def receive(self):
return self.ready.pop(0) if self.ready else None
def nack(self, msg): # crash or error: the visibility timeout expires and it comes back
self.tries[msg] = self.tries.get(msg, 0) + 1
(self.dlq if self.tries[msg] >= self.max_tries else self.ready).append(msg)
class Log:
"""An event log: messages stay; each consumer group keeps its own offset and can rewind."""
def __init__(self):
self.entries, self.offsets = [], {}
def append(self, msg):
self.entries.append(msg)
def poll(self, group):
start = self.offsets.get(group, 0)
self.offsets[group] = len(self.entries)
return self.entries[start:]
def seek(self, group, offset):
self.offsets[group] = offset
ORDERS = ["order-1", "order-2", "poison", "order-4"]
q = Queue()
for o in ORDERS:
q.send(o)
done, crashed_once = [], False
while (msg := q.receive()) is not None:
if msg == "poison" or (msg == "order-2" and not crashed_once):
crashed_once = crashed_once or msg == "order-2"
q.nack(msg) # order-2 fails once (a worker crash); poison fails every time
else:
done.append(msg)
print("queue: processed", ", ".join(done), "| dead-letter:", ", ".join(q.dlq), f"| poison tried {q.tries['poison']} times")
log = Log()
for o in ORDERS:
log.append(o)
print("log: billing reads", ", ".join(log.poll("billing")))
print("log: analytics reads", ", ".join(log.poll("analytics")), "(independently, same events)")
log.append("order-5")
print("log: billing then reads", ", ".join(log.poll("billing")))
log.seek("analytics", 0)
print("log: analytics replays", len(log.poll("analytics")), "events from offset 0")
class Queue {
// A work queue: each message goes to one worker; unacked messages come back; repeated failures go to a DLQ.
ready: string[] = []; dlq: string[] = []; tries = new Map<string, number>();
constructor(private maxTries = 3) {}
send(msg: string) { this.ready.push(msg); }
receive(): string | undefined { return this.ready.shift(); }
nack(msg: string) { // crash or error: the visibility timeout expires and it comes back
const n = (this.tries.get(msg) ?? 0) + 1;
this.tries.set(msg, n);
(n >= this.maxTries ? this.dlq : this.ready).push(msg);
}
}
class Log {
// An event log: messages stay; each consumer group keeps its own offset and can rewind.
entries: string[] = []; offsets = new Map<string, number>();
append(msg: string) { this.entries.push(msg); }
poll(group: string): string[] {
const start = this.offsets.get(group) ?? 0;
this.offsets.set(group, this.entries.length);
return this.entries.slice(start);
}
seek(group: string, offset: number) { this.offsets.set(group, offset); }
}
const ORDERS = ["order-1", "order-2", "poison", "order-4"];
const q = new Queue();
for (const o of ORDERS) q.send(o);
const done: string[] = [];
let crashedOnce = false, msg: string | undefined;
while ((msg = q.receive()) !== undefined) {
if (msg === "poison" || (msg === "order-2" && !crashedOnce)) {
crashedOnce = crashedOnce || msg === "order-2";
q.nack(msg); // order-2 fails once (a worker crash); poison fails every time
} else done.push(msg);
}
console.log(`queue: processed ${done.join(", ")} | dead-letter: ${q.dlq.join(", ")} | poison tried ${q.tries.get("poison")} times`);
const log = new Log();
for (const o of ORDERS) log.append(o);
console.log("log: billing reads", log.poll("billing").join(", "));
console.log("log: analytics reads", log.poll("analytics").join(", "), "(independently, same events)");
log.append("order-5");
console.log("log: billing then reads", log.poll("billing").join(", "));
log.seek("analytics", 0);
console.log("log: analytics replays", log.poll("analytics").length, "events from offset 0");import java.util.*;
public class Main {
static class Queue {
// A work queue: each message goes to one worker; unacked messages come back; repeated failures go to a DLQ.
final Deque<String> ready = new ArrayDeque<>();
final List<String> dlq = new ArrayList<>();
final Map<String, Integer> tries = new HashMap<>();
final int maxTries;
Queue(int maxTries) { this.maxTries = maxTries; }
void send(String msg) { ready.addLast(msg); }
String receive() { return ready.pollFirst(); }
void nack(String msg) { // crash or error: the visibility timeout expires and it comes back
int n = tries.merge(msg, 1, Integer::sum);
if (n >= maxTries) dlq.add(msg); else ready.addLast(msg);
}
}
static class Log {
// An event log: messages stay; each consumer group keeps its own offset and can rewind.
final List<String> entries = new ArrayList<>();
final Map<String, Integer> offsets = new HashMap<>();
void append(String msg) { entries.add(msg); }
List<String> poll(String group) {
int start = offsets.getOrDefault(group, 0);
offsets.put(group, entries.size());
return new ArrayList<>(entries.subList(start, entries.size()));
}
void seek(String group, int offset) { offsets.put(group, offset); }
}
public static void main(String[] args) {
List<String> orders = List.of("order-1", "order-2", "poison", "order-4");
Queue q = new Queue(3);
for (String o : orders) q.send(o);
List<String> done = new ArrayList<>();
boolean crashedOnce = false;
String msg;
while ((msg = q.receive()) != null) {
if (msg.equals("poison") || (msg.equals("order-2") && !crashedOnce)) {
crashedOnce = crashedOnce || msg.equals("order-2");
q.nack(msg); // order-2 fails once (a worker crash); poison fails every time
} else done.add(msg);
}
System.out.println("queue: processed " + String.join(", ", done) + " | dead-letter: " + String.join(", ", q.dlq) + " | poison tried " + q.tries.get("poison") + " times");
Log log = new Log();
for (String o : orders) log.append(o);
System.out.println("log: billing reads " + String.join(", ", log.poll("billing")));
System.out.println("log: analytics reads " + String.join(", ", log.poll("analytics")) + " (independently, same events)");
log.append("order-5");
System.out.println("log: billing then reads " + String.join(", ", log.poll("billing")));
log.seek("analytics", 0);
System.out.println("log: analytics replays " + log.poll("analytics").size() + " events from offset 0");
}
}queue: processed order-1, order-4, order-2 | dead-letter: poison | poison tried 3 times
log: billing reads order-1, order-2, poison, order-4
log: analytics reads order-1, order-2, poison, order-4 (independently, same events)
log: billing then reads order-5
log: analytics replays 5 events from offset 0
7. Say it in the interview
- "Kafka or RabbitMQ?" Kafka when events are shared by many consumers, need replay, or run at high throughput; RabbitMQ for task queues with flexible routing at moderate volume. On a cloud, I'd first ask whether the managed service (Pub/Sub, SQS, Service Bus) does the job.
- "How do you keep order?" Per key, not globally: partition by order ID (Kafka, Kinesis, Event Hubs), use an ordering key (Pub/Sub), a message group (SQS FIFO) or a session (Service Bus).
- "What happens when a consumer fails?" At-least-once delivery: the message comes back after the visibility timeout or ack deadline, so consumers are idempotent, and repeated failures go to a dead-letter queue we alert on.
8. Common mistakes
9. Check yourself
Several services need every order event, and analytics wants to re-read last week. Queue or log?
How do Kafka, Pub/Sub and SQS FIFO keep order?
You're on Azure and need a task queue with sessions and dead-lettering. Which fits?
A message keeps failing. What should happen?
10. Sources and further reading
Product limits checked on 4 November 2026 against each vendor's documentation. They change; check before relying on a number.
| Source | Used for |
|---|---|
| Apache Kafka 4.2.0 release announcement | Share groups production-ready |
| RabbitMQ: queues | Queue types; classic replication removed in 4.x |
| Amazon SQS message quotas | Size, retention, visibility timeout, FIFO throughput |
| Kinesis Data Streams quotas | Shard throughput, record size, retention, fan-out |
| Google Pub/Sub quotas and limits | Message size, retention, ordering-key throughput, exactly-once |
| Azure Service Bus quotas | Message and entity sizes by tier |
| Azure Event Hubs quotas | Retention, event size, throughput units, Kafka support |
| Module 18: Queues and event streams | Delivery guarantees, ordering, consumer groups, partitions |