Queues and event streams

Not every piece of work has to happen while the user waits. Put it on a queue and let another service do it when it can: the user gets a fast answer, a spike becomes a backlog instead of an outage, and services stop depending on each other being up at the same moment. This module covers the three shapes of messaging, what "delivered" really means, ordering, and how to handle messages that keep failing.

1. The idea

A producer sends messages; a broker stores them; consumers receive and process them. Messaging buys three things:

• Decoupling in time: the producer doesn't wait for, or even need, the consumer to be running.
• Absorbing bursts: 10× traffic for five minutes becomes a backlog that drains, instead of overloading the consumer.
• Fan-out: one event ("order placed") can feed many independent consumers (email, analytics, warehouse) without the producer knowing about them.

Three shapes of messaging

2. How it works

What "delivered" means

Networks and consumers fail mid-message, so every system picks a delivery guarantee:

• At-most-once: the message is removed before it's processed. A crash loses it. Acceptable for metrics you can afford to drop.
• At-least-once: the message is removed only after the consumer acknowledges it. A crash after processing but before the ack means it's delivered again. The usual default.
• Exactly-once delivery over a network isn't possible in general. What systems offer is effectively once: at-least-once delivery plus idempotent processing (a duplicate has no extra effect), or transactions that commit the result and the consumer's position together (Kafka's exactly-once within Kafka).

In practice: assume duplicates, and make consumers idempotent. Record processed message IDs, use upserts, or use the idempotency keys from Module 07.

Acknowledgements and the visibility timeout

In SQS, receiving a message hides it from other consumers for a visibility timeout (30 seconds by default). If the consumer deletes it in time, it's done; if the consumer crashes, the message reappears and another worker takes it. Set the timeout longer than the slowest normal processing time, or a slow worker and a second worker will process the same message at once.

Partitions, ordering and consumer groups

Kafka splits a topic into partitions. Messages with the same key go to the same partition and stay in order; there's no order across partitions. In a consumer group, each partition is read by exactly one consumer, so the number of partitions caps how many consumers can work in parallel. Each group tracks its offset (position) per partition. Press Play.

Messages that keep failing

A malformed message (a poison message) fails every time and, with naive retries, blocks or burns workers forever. After a few attempts (3–5 is typical), move it to a dead-letter queue (DLQ), alert on it, fix the bug, and replay it. Retries themselves should back off with jitter (Module 24), possibly through separate retry topics with increasing delays.

Back-pressure: when the queue keeps growing

A queue only absorbs bursts. If messages arrive faster than consumers process them on average, the backlog grows without limit, and latency with it. Watch queue depth and consumer lag; scale consumers on them, not on CPU; and if you can't keep up, push back on producers (reject, slow down) or shed low-value work. Try it in section 3.

Scaling writes: the strategies in order

The core idea is to reduce the write throughput each component handles. Do the math first, then go in order:

1. Vertical scaling and the right database. Check you're actually hitting disk, CPU or network limits; modern machines are huge. For append-heavy data, a write-optimised store (Cassandra, a time-series database) trades read speed for write speed, because it appends instead of updating pages (Module 12).
2. Sharding and vertical partitioning (Module 15): ten shards ≈ ten times the writes, if the key spreads load; split hot counters from content.
3. Queues for bursts, load shedding for overload. A queue absorbs a spike and lets the database drain it steadily, but if input stays above capacity on average, the queue grows forever. Load shedding drops the least valuable writes when overloaded: a location ping (a fresher one arrives in seconds), impressions before clicks.
4. Batching and hierarchical aggregation. Combine many small writes into a few: a "like batcher" sums likes per post per second, turning 100 writes into 1. For live events with millions of viewers, aggregate in layers (below).

State the trade-offs: queues add delay and eventual consistency; partitioning can make some reads harder; batching adds latency and moving parts, and loses the batch on a crash unless the source (a Kafka topic) can be re-read.

3. Try it yourself

Messages arrive at a steady rate with a burst in the middle. Workers each handle a fixed number per second. Find the worker count that drains the burst within your target, and see what happens when the average rate exceeds capacity.

4. Trade-offs

5. In real systems

• Amazon SQS: fully managed. Messages up to 1 MiB (larger payloads go to S3, with a pointer in the message), retained for 4 days by default and up to 14 days, a 30-second default visibility timeout, DLQs via a redrive policy. Standard queues give near-unlimited throughput with at-least-once delivery and best-effort order; FIFO queues give order per message group and deduplication.
• Apache Kafka: the most widely used event log. Since Kafka 4.0 (March 2025) it runs without ZooKeeper, using its own Raft-based KRaft controllers. Kafka 4.2 (February 2026) added share groups (KIP-932, "Queues for Kafka"), letting several consumers share one partition with per-message acknowledgement, after early access in 4.0. The current release in October 2026 is 4.3.
• Kafka durability: replicate partitions (replication factor 3), produce with acks=all and a minimum in-sync replica count, and use idempotent producers to avoid duplicates on retry.
• RabbitMQ offers flexible routing (exchanges), quorum queues for durability, and streams for log-style use. Google Cloud Pub/Sub and Amazon SNS are managed pub/sub; SNS commonly fans out into several SQS queues. Redis Streams suit modest queues where Redis is already running.
• Keep messages small: pass IDs and pointers ("video 123 uploaded to s3://…"), not the video. Brokers are built for many small messages.

The technologies up close

6. Remember it

7. Common mistakes

8. Design exercise (25 minutes)

1. Pick the shape (queue, log or pub/sub) for: sending password-reset emails, streaming every order to analytics and search, telling all app servers to drop a cached setting, processing video uploads.
2. Design an idempotent consumer for "charge the customer for order X". What do you store, and what happens on a duplicate?
3. Choose keys: a bank streams account transactions through Kafka. What key guarantees per-account order? How many partitions for 50,000 events a second if one consumer handles 2,000?
4. Size the workers in the simulator for 300 messages a second normally and a 10× burst, drained within 10 minutes.
5. Write it down: the three delivery guarantees, and "assume duplicates".

9. Self-check quiz

10. Go deeper

11. Under the hood (optional)

How queues and logs really deliver, order, retry and store messages. Not needed for most interviews; open it when you want to know how the products really do it. All the deep dives are listed on the Under the hood page.