InterviewKata
System design · Technology guide T01 · Builds on Module 18

Message queues and event streams

RabbitMQ, Kafka, SQS, Google Pub/Sub, Azure Service Bus: they all "move messages", and they are not interchangeable. Some are queues that hand each task to one worker and forget it; some are logs that keep every event for many readers to replay. This guide explains how each product works, how they differ, and which to choose for a given job, with a chooser you can try and code you can run.

A 3-minute conversation: Sam asks, Priya explains
0:00 / 2:42

1. What it is

A message system sits between the services that produce work or events and the services that consume them. It lets the producer carry on without waiting, absorbs bursts, and survives a consumer being down. Module 18 teaches the concepts; this guide is about the products.

Every product is one of two shapes, or a hybrid:

QueueLog (event stream)
What a message isA task to do onceAn event that happened, which many systems may care about
After it's processedDeleted when the consumer acknowledges itKept for the retention period; each reader tracks its own position (offset)
Many readersCompeting workers share the messagesEach consumer group reads every event independently
ReplayNo: once acknowledged, it's goneYes: rewind the offset and read again
ExamplesRabbitMQ classic and quorum queues, Amazon SQS, Azure Service BusKafka, Kinesis, Azure Event Hubs, RabbitMQ streams, Pulsar, Google Pub/Sub with retention and seek

2. How it works

The same four orders, through a queue and through a log.

3. The products

Broker · open source

RabbitMQ

Producers publish to an exchange, which routes messages to queues by key, pattern or headers; the broker pushes them to consumers with a prefetch limit. Three queue types: classic (not replicated), quorum (replicated with Raft) and streams (an append-only log). Classic queue replication was removed in 4.x.

Best for: task queues with flexible routing, request-reply, moderate volumes.

Log · open source

Apache Kafka

A topic is split into partitions, each an ordered, replicated log on the brokers. Consumers pull and commit offsets; retention is by time or size, with compaction to keep the latest value per key. Runs on KRaft, with no ZooKeeper. Kafka 4.2 (February 2026) made share groups, queue-style consumption, production-ready.

Best for: high-throughput event streams, many independent consumers, replay, stream processing.

Log · open source

Apache Pulsar

Stateless brokers serve topics stored in Apache BookKeeper, so storage and serving scale separately. Subscriptions can be exclusive, shared, failover or key-shared; geo-replication and tiered storage are built in.

Best for: multi-tenant platforms, geo-replication, mixing queue and stream use.

Messaging · open source

NATS

Core NATS is very low-latency publish-subscribe with at-most-once delivery and wildcard subjects. JetStream adds persistent streams with at-least-once delivery and a deduplication window.

Best for: low-latency service messaging, edge and IoT, simple operations.

Stream · in Redis

Redis Streams

An append-only stream held in memory, with consumer groups and a pending list per consumer; XAUTOCLAIM lets another consumer take over stuck messages.

Best for: modest volumes when you already run Redis or Valkey.

Queue · AWS managed

Amazon SQS

A fully managed queue. Standard queues are at-least-once with best-effort order and nearly unlimited throughput; FIFO queues order messages within a message group and deduplicate. Messages up to 1 MiB, kept 1 minute to 14 days (default 4 days); a visibility timeout (default 30 seconds, up to 12 hours) hides a message while it's processed; failures redrive to a dead-letter queue.

Best for: task queues on AWS with no operations work. Pair with SNS to fan out.

Log · AWS managed

Amazon Kinesis Data Streams

A managed stream of shards. In provisioned mode each shard takes up to 1 MB/s or 1,000 records/s in and 2 MB/s out; on-demand mode scales shards for you. Records up to 10 MiB, retained up to 365 days; enhanced fan-out gives registered consumers their own read capacity.

Best for: event streams and analytics pipelines on AWS.

Pub/sub · Google Cloud managed

Google Pub/Sub

Global topics with subscriptions that pull or push. Messages up to 10 MB; unacknowledged messages kept 7 days by default, and topics can retain messages up to 31 days for replay with seek. Ordering keys order messages per key (up to 1 MBps per key); exactly-once delivery is an option.

Best for: almost any messaging on Google Cloud, both task fan-out and event streams.

Broker · Azure managed

Azure Service Bus

Managed queues and topics with subscriptions, filters, sessions for ordered processing, duplicate detection and dead-lettering. Messages up to 256 KB on Standard, and up to 100 MB on Premium over AMQP; queues and topics up to 80 GB.

Best for: enterprise task queues and pub/sub on Azure.

Log · Azure managed

Azure Event Hubs

A managed, partitioned event stream with a Kafka-compatible endpoint. Retention up to 7 days on Standard and 90 days on Premium and Dedicated; events up to 1 MB (20 MB on Dedicated); each throughput unit takes 1 MB/s in and 2 MB/s out.

Best for: event streams on Azure, and running Kafka clients without running Kafka.

4. Differences at a glance

ProductShapeOrderingDeliveryReplayRun by
RabbitMQBroker with queues; streams as a logPer queue (lost on redelivery)At-least-once with acksStreams onlyYou
KafkaPartitioned logPer partition (per key)At-least-once; exactly-once within Kafka with transactionsYes, within retentionYou, or a managed Kafka service
PulsarLog over BookKeeperPer partition or per keyAt-least-once; deduplication availableYesYou, or a managed service
NATSPub/sub; JetStream streamsPer subject streamAt-most-once (core); at-least-once (JetStream)JetStream onlyYou, or a managed service
Redis StreamsIn-memory logPer streamAt-least-once with a pending listYes, while keptYou, or managed Redis/Valkey
SQSQueueNone (Standard); per message group (FIFO)At-least-once; FIFO deduplicatesNoAWS
KinesisSharded logPer shardAt-least-onceYes, up to 365 daysAWS
Pub/SubPub/sub with retentionPer ordering keyAt-least-once; exactly-once optionYes, with topic retention and seekGoogle Cloud
Service BusBroker: queues and topicsPer sessionAt-least-once; duplicate detectionNoAzure
Event HubsPartitioned logPer partitionAt-least-onceYes, within retentionAzure

5. When to choose which, and why

  1. Task or event? A task done once by one worker wants a queue. An event several systems react to, or one you may need to re-read, wants a log.
  2. Where do you run? On a cloud, start with its managed service: SQS (+ SNS), Kinesis, Pub/Sub, Service Bus or Event Hubs. Running your own cluster is a job; take it on only for a reason.
  3. Reasons to run your own: portability across clouds, Kafka's ecosystem (Connect, Streams, the clients), complex routing (RabbitMQ), sub-millisecond messaging (NATS), or costs at very high volume.
  4. Ordering: almost nothing needs global order. Order per key (a user, an order) is enough, and every product above offers it.
  5. Throughput: tens of thousands of messages a second and beyond points to a partitioned log; plan partitions (Module 18 has the arithmetic).

Try it: choose for your requirements

6. Queue vs log, in code

The same orders through a queue and a log. In the queue, a message that fails is redelivered and, after three failures, goes to the dead-letter queue; notice that the retried order finishes last, so a plain queue doesn't keep order across retries. In the log, two consumer groups read every event independently, and one rewinds to replay. The program was run in all three languages; each prints the output below it.

class Queue:
    """A work queue: each message goes to one worker; unacked messages come back; repeated failures go to a DLQ."""

    def __init__(self, max_tries=3):
        self.ready, self.dlq, self.tries, self.max_tries = [], [], {}, max_tries

    def send(self, msg):
        self.ready.append(msg)

    def receive(self):
        return self.ready.pop(0) if self.ready else None

    def nack(self, msg):                   # crash or error: the visibility timeout expires and it comes back
        self.tries[msg] = self.tries.get(msg, 0) + 1
        (self.dlq if self.tries[msg] >= self.max_tries else self.ready).append(msg)


class Log:
    """An event log: messages stay; each consumer group keeps its own offset and can rewind."""

    def __init__(self):
        self.entries, self.offsets = [], {}

    def append(self, msg):
        self.entries.append(msg)

    def poll(self, group):
        start = self.offsets.get(group, 0)
        self.offsets[group] = len(self.entries)
        return self.entries[start:]

    def seek(self, group, offset):
        self.offsets[group] = offset


ORDERS = ["order-1", "order-2", "poison", "order-4"]

q = Queue()
for o in ORDERS:
    q.send(o)
done, crashed_once = [], False
while (msg := q.receive()) is not None:
    if msg == "poison" or (msg == "order-2" and not crashed_once):
        crashed_once = crashed_once or msg == "order-2"
        q.nack(msg)                         # order-2 fails once (a worker crash); poison fails every time
    else:
        done.append(msg)
print("queue: processed", ", ".join(done), "| dead-letter:", ", ".join(q.dlq), f"| poison tried {q.tries['poison']} times")

log = Log()
for o in ORDERS:
    log.append(o)
print("log: billing reads", ", ".join(log.poll("billing")))
print("log: analytics reads", ", ".join(log.poll("analytics")), "(independently, same events)")
log.append("order-5")
print("log: billing then reads", ", ".join(log.poll("billing")))
log.seek("analytics", 0)
print("log: analytics replays", len(log.poll("analytics")), "events from offset 0")
class Queue {
  // A work queue: each message goes to one worker; unacked messages come back; repeated failures go to a DLQ.
  ready: string[] = []; dlq: string[] = []; tries = new Map<string, number>();
  constructor(private maxTries = 3) {}
  send(msg: string) { this.ready.push(msg); }
  receive(): string | undefined { return this.ready.shift(); }
  nack(msg: string) {                      // crash or error: the visibility timeout expires and it comes back
    const n = (this.tries.get(msg) ?? 0) + 1;
    this.tries.set(msg, n);
    (n >= this.maxTries ? this.dlq : this.ready).push(msg);
  }
}

class Log {
  // An event log: messages stay; each consumer group keeps its own offset and can rewind.
  entries: string[] = []; offsets = new Map<string, number>();
  append(msg: string) { this.entries.push(msg); }
  poll(group: string): string[] {
    const start = this.offsets.get(group) ?? 0;
    this.offsets.set(group, this.entries.length);
    return this.entries.slice(start);
  }
  seek(group: string, offset: number) { this.offsets.set(group, offset); }
}

const ORDERS = ["order-1", "order-2", "poison", "order-4"];

const q = new Queue();
for (const o of ORDERS) q.send(o);
const done: string[] = [];
let crashedOnce = false, msg: string | undefined;
while ((msg = q.receive()) !== undefined) {
  if (msg === "poison" || (msg === "order-2" && !crashedOnce)) {
    crashedOnce = crashedOnce || msg === "order-2";
    q.nack(msg);                           // order-2 fails once (a worker crash); poison fails every time
  } else done.push(msg);
}
console.log(`queue: processed ${done.join(", ")} | dead-letter: ${q.dlq.join(", ")} | poison tried ${q.tries.get("poison")} times`);

const log = new Log();
for (const o of ORDERS) log.append(o);
console.log("log: billing reads", log.poll("billing").join(", "));
console.log("log: analytics reads", log.poll("analytics").join(", "), "(independently, same events)");
log.append("order-5");
console.log("log: billing then reads", log.poll("billing").join(", "));
log.seek("analytics", 0);
console.log("log: analytics replays", log.poll("analytics").length, "events from offset 0");
import java.util.*;

public class Main {
    static class Queue {
        // A work queue: each message goes to one worker; unacked messages come back; repeated failures go to a DLQ.
        final Deque<String> ready = new ArrayDeque<>();
        final List<String> dlq = new ArrayList<>();
        final Map<String, Integer> tries = new HashMap<>();
        final int maxTries;
        Queue(int maxTries) { this.maxTries = maxTries; }
        void send(String msg) { ready.addLast(msg); }
        String receive() { return ready.pollFirst(); }
        void nack(String msg) {                // crash or error: the visibility timeout expires and it comes back
            int n = tries.merge(msg, 1, Integer::sum);
            if (n >= maxTries) dlq.add(msg); else ready.addLast(msg);
        }
    }

    static class Log {
        // An event log: messages stay; each consumer group keeps its own offset and can rewind.
        final List<String> entries = new ArrayList<>();
        final Map<String, Integer> offsets = new HashMap<>();
        void append(String msg) { entries.add(msg); }
        List<String> poll(String group) {
            int start = offsets.getOrDefault(group, 0);
            offsets.put(group, entries.size());
            return new ArrayList<>(entries.subList(start, entries.size()));
        }
        void seek(String group, int offset) { offsets.put(group, offset); }
    }

    public static void main(String[] args) {
        List<String> orders = List.of("order-1", "order-2", "poison", "order-4");
        Queue q = new Queue(3);
        for (String o : orders) q.send(o);
        List<String> done = new ArrayList<>();
        boolean crashedOnce = false;
        String msg;
        while ((msg = q.receive()) != null) {
            if (msg.equals("poison") || (msg.equals("order-2") && !crashedOnce)) {
                crashedOnce = crashedOnce || msg.equals("order-2");
                q.nack(msg);                       // order-2 fails once (a worker crash); poison fails every time
            } else done.add(msg);
        }
        System.out.println("queue: processed " + String.join(", ", done) + " | dead-letter: " + String.join(", ", q.dlq) + " | poison tried " + q.tries.get("poison") + " times");

        Log log = new Log();
        for (String o : orders) log.append(o);
        System.out.println("log: billing reads " + String.join(", ", log.poll("billing")));
        System.out.println("log: analytics reads " + String.join(", ", log.poll("analytics")) + " (independently, same events)");
        log.append("order-5");
        System.out.println("log: billing then reads " + String.join(", ", log.poll("billing")));
        log.seek("analytics", 0);
        System.out.println("log: analytics replays " + log.poll("analytics").size() + " events from offset 0");
    }
}
queue: processed order-1, order-4, order-2 | dead-letter: poison | poison tried 3 times
log: billing reads order-1, order-2, poison, order-4
log: analytics reads order-1, order-2, poison, order-4 (independently, same events)
log: billing then reads order-5
log: analytics replays 5 events from offset 0

7. Say it in the interview

Two-sentence answers
  • "Kafka or RabbitMQ?" Kafka when events are shared by many consumers, need replay, or run at high throughput; RabbitMQ for task queues with flexible routing at moderate volume. On a cloud, I'd first ask whether the managed service (Pub/Sub, SQS, Service Bus) does the job.
  • "How do you keep order?" Per key, not globally: partition by order ID (Kafka, Kinesis, Event Hubs), use an ordering key (Pub/Sub), a message group (SQS FIFO) or a session (Service Bus).
  • "What happens when a consumer fails?" At-least-once delivery: the message comes back after the visibility timeout or ack deadline, so consumers are idempotent, and repeated failures go to a dead-letter queue we alert on.

8. Common mistakes

Using a queue when several systems need the eventEach system ends up with its own copy of the publishing code. Publish once to a log or a topic with several subscriptions.
Running Kafka for a few hundred messages a minuteA cluster to operate, for a job a managed queue does with no servers.
Assuming exactly-once end to end"Exactly-once" options cover the broker; your database write still needs idempotency.
Asking for global orderIt limits you to one partition or one consumer. Order per key almost always suffices.
Ignoring size limitsPayloads over the limit (256 KB on Service Bus Standard, 1 MiB on SQS) belong in object storage, with a reference in the message.

9. Check yourself

Several services need every order event, and analytics wants to re-read last week. Queue or log?

Queues delete acknowledged messages; logs keep them for every reader.

How do Kafka, Pub/Sub and SQS FIFO keep order?

Per-key order keeps parallelism; global order would need a single partition.

You're on Azure and need a task queue with sessions and dead-lettering. Which fits?

Service Bus is Azure's broker for queues and topics; Event Hubs is its event stream.

A message keeps failing. What should happen?

A poison message shouldn't block the queue or vanish.

10. Sources and further reading

Product limits checked on 4 November 2026 against each vendor's documentation. They change; check before relying on a number.

SourceUsed for
Apache Kafka 4.2.0 release announcementShare groups production-ready
RabbitMQ: queuesQueue types; classic replication removed in 4.x
Amazon SQS message quotasSize, retention, visibility timeout, FIFO throughput
Kinesis Data Streams quotasShard throughput, record size, retention, fan-out
Google Pub/Sub quotas and limitsMessage size, retention, ordering-key throughput, exactly-once
Azure Service Bus quotasMessage and entity sizes by tier
Azure Event Hubs quotasRetention, event size, throughput units, Kafka support
Module 18: Queues and event streamsDelivery guarantees, ordering, consumer groups, partitions