Software architecture · Level 4: Communication and integration · Module 20 of 46

Messaging patterns

Choosing asynchronous messaging is the start, not the end. Messages can be lost in three different places, processed out of order, delivered twice, or pile up faster than consumers can handle them. A small set of well-known patterns deals with each of these. This module covers queues versus logs, preventing message loss, keeping order while processing in parallel, and controlling a queue that's filling up.

See it builtThe Architecture styles track builds this style for one bike-share app, with code you can run: 12 Broker.

1. The idea

A post room. Letters can be lost when they're handed in, while they sit in the sorting room, or when they're delivered to a desk where nobody reads them. Letters for the same customer must be read in the order they were written. And when the post room floods on a Monday, someone has to either call in more sorters or ask senders to slow down. Messaging patterns are the post room's standard procedures.

Queues and logs

	Queue (RabbitMQ, ActiveMQ, SQS)	Log (Kafka, Pulsar, Kinesis)
After reading	The message is removed once acknowledged	The message stays for the retention period
Many consumers	Compete for messages (each gets some)	Each consumer group reads everything, at its own position
Replay	Not possible once consumed	Rewind and reprocess from any offset
Order	Per queue, until consumers compete	Per partition, by key
Best for	Work distribution, commands, tasks	Event streams, many independent readers, replay and audit

The patterns in this module

Pattern	Problem it solves
Guaranteed delivery	Messages lost between producer, broker and consumer (scene below)
Transactional outbox	Saving to the database and publishing a message must both happen or neither: write the message to an outbox table in the same transaction; a relay publishes it
Thread delegate	Processing in parallel without breaking per-key order (scene below)
Supervisor consumer	A supervisor watches queue depth and starts or stops consumers (lab below)
Producer control flow	Producers slow down, or are told to, when a queue is filling (lab below)
Multi-broker	One broker as a throughput limit or single point of failure: producers spread messages across several brokers; consumers read from all
Priority ("ambulance") channel	Urgent messages stuck behind a backlog: give them their own channel and consumers
Channel monitoring	Problems noticed too late: watch depth, age of oldest message, consumer count and dead letters
Dead-letter channel	A message that keeps failing blocks or loops: move it aside after N attempts for inspection

2. How it works

Three places a message gets lost

Thread delegate: parallel, but in order per key

Order events must be processed in sequence for each order (created, then paid, then cancelled), but there are thousands of orders. Watch naive parallelism break the sequence, then a delegate fix it.

3. Try it yourself

A queue under a burst

Messages arrive at 200 a second, then 1,200 a second for three minutes. Each consumer handles 150 a second. Choose how the system responds and watch the queue depth and how long messages wait.

Simulation in one-second steps over ten minutes. The supervisor checks every 10 seconds, starts consumers to drain the backlog within about a minute, and each new consumer takes 5 seconds to start. Producer control flow halves the accepted rate while more than 20,000 messages are waiting (producers buffer or retry the rest later).

4. Trade-offs

Durability vs speed

Persistent messages, synchronous sends and acknowledgements after processing prevent loss, and cost throughput and latency. Use them for orders and payments; skip them for metrics you can afford to lose.

Order vs parallelism

Strict global order means one consumer at a time. Order per key (per order, per account) allows parallelism across keys. Ask which order actually matters.

Scale consumers vs slow producers

Adding consumers drains a backlog but needs capacity downstream (databases, APIs). Slowing producers protects downstream but pushes the waiting back to the source.

Queue vs log

Queues are simple for distributing work. Logs keep history, allow replay and serve many independent consumers, at the cost of partitions, offsets and retention to manage.

5. In real projects

Plan for duplicates. Most brokers guarantee "at least once": after a crash, a message may be redelivered. Every consumer must be idempotent: store processed message IDs, or make the operation naturally repeatable.

Monitor the age of the oldest message, not just queue depth. A queue of 10,000 that drains in a second is fine; 50 messages stuck for an hour are an incident.

Use the transactional outbox whenever a service must update its database and publish an event. Publishing after the commit can lose the event if the service crashes in between; publishing before can announce something that then rolls back.

Say it in the interview

Two-sentence answers

"How do you prevent message loss?" Confirmed, persistent sends from the producer; a durable, replicated broker; and consumers that acknowledge only after processing. Plus a transactional outbox when a database write and a message must go together.
"Kafka or a classic queue?" A queue for distributing tasks to competing workers; a log like Kafka for event streams that several consumers read independently, or that need replay.
"How do you keep order while scaling consumers?" Order by key, not globally: route all messages for one key to the same partition or thread, and process different keys in parallel.

Red flags interviewers notice

"The broker guarantees exactly once, so we don't need idempotency."
Auto-acknowledging messages before processing them.
No monitoring or alert on queue depth and message age.

6. Remember it

Memory hook: "Hand in, store, deliver: guard all three"A message can be lost at each step. Confirm the send, persist the store, acknowledge after the work.

How this hook works: what "hand in, store, deliver" means

What the slogan means, phrase by phrase

"Hand in": Producer to broker: wait for the broker's confirmation.
"store": Inside the broker: persist and replicate, so a crash doesn't wipe the queue.
"deliver": Broker to consumer: acknowledge only after processing succeeds.

Why this picture fits: a post room loses letters at the counter, in the sorting room or at the desk, and each needs its own procedure.

The story → the concept

Post room	Messaging
A receipt at the counter	Publisher confirm
A fireproof sorting room	Persistent, replicated broker
Signing for a letter after reading it	Acknowledge after processing
One clerk per customer's letters	Thread delegate (order per key)

The trap it saves you from

Trusting the broker to guarantee delivery end to end, when two of the three steps are in your code.

7. Common mistakes

Acknowledge, then processAuto-acknowledge removes the message before it's processed; a crash loses it for good.

Global ordering by accidentOne consumer "to keep things in order" caps throughput at one machine. Order per key instead.

Poison messagesA message that always fails, retried forever, blocks its queue. Dead-letter it after a few attempts.

Dual writesSaving to the database and publishing separately: one will eventually succeed without the other. Use an outbox.

8. Design exercise (30 minutes)

Trace one message (10 min). For one message in a system you know, write how it's protected at each of the three steps.
Find the order requirement (10 min). Which messages must be processed in order? By what key? How is that guaranteed today?
Plan the burst (10 min). If arrivals tripled for an hour, what happens to the queue, the consumers and whatever they write to?

9. Self-check quiz

A consumer crashes mid-processing and the message is gone. What prevents this?

With client acknowledgement, an unacknowledged message is redelivered.

Five teams each want to read every order event, and one may need to replay last week. Queue or log?

Logs keep messages; each consumer group reads independently and can rewind.

How do you process orders in parallel but keep each order's events in sequence?

Order per key, parallel across keys.

How does the transactional outbox work?

A relay publishes from the outbox; both happen or neither.

Which queue metric best signals real trouble?

It shows how long work is actually waiting, whatever the depth.

10. Go deeper

Resource	Time	Why
Guaranteed Delivery (Hohpe and Woolf)	10 min	Persistence at every step of a message's journey.
Transactional outbox (Chris Richardson)	10 min	The pattern for publishing reliably alongside a database write.
Kafka design (Apache Kafka docs)	40 min	How a log stores, partitions and replicates messages.