Distributed transactions and sagas
Inside one database, a transaction makes several changes happen all together or not at all. Spread those changes across services, each with its own database, and that guarantee is gone. Booking a trip (flight, hotel, card payment) now means three separate commits, any of which can fail after the others succeeded. Sagas are how distributed systems cope: a sequence of local transactions, each with a compensating step to undo it. This module covers what you give up, how sagas work, and how they fail.
1. The idea
| ACID (one database) | BASE (distributed) | |
|---|---|---|
| Atomic / Basically available | All changes or none | The system keeps answering, even with parts failing |
| Consistent / Soft state | Rules hold after every transaction | State may be in flux while updates propagate |
| Isolated / Eventually consistent | Others never see half-finished work | Copies converge once updates stop |
| Durable | Committed means saved | Each local step is durable on its own |
CAP explains why: when the network splits replicas apart (a partition), a system must either refuse some requests to stay consistent (CP) or keep answering and reconcile later (AP). Distributed architectures usually choose availability and eventual consistency for most data, and keep strict, atomic consistency inside one service's database for the data that truly needs it.
How a saga works
- The business transaction is split into steps, each a local ACID transaction in one service.
- Each step has a compensating update that semantically undoes it (cancel the booking, refund the charge). It's not a rollback: the original change happened and was visible.
- If a step fails, the completed steps are compensated in reverse order.
- Coordination is either orchestrated (a saga orchestrator) or choreographed (services react to each other's events), as in Module 21.
2. How it works
A trip-booking saga, including a compensation that fails
CAP, illustrated
Two replicas of a seat inventory, one in each data centre. The link between them fails. Watch the two possible behaviours.
3. Try it yourself
Eight kinds of saga
Sagas vary along three dimensions: how services communicate, how consistent the result must be, and how they're coordinated. Each combination behaves differently. Choose one and see its profile. The names are the memorable ones given to them in Software Architecture: The Hard Parts.
Ratings are our qualitative summary, on a five-point scale, following the book's scheme. "Atomic" here means the saga tries to behave like one transaction (nothing visible until all steps finish, coordinated undo); "eventual" lets each step commit and become visible on its own.
4. Trade-offs
Atomic vs eventual
Trying to make a distributed transaction look atomic adds locking, coordination and fragility. Eventual consistency is simpler and more scalable, if the business can accept visible intermediate states.
Compensation vs prevention
Compensating is complex and sometimes impossible (an email can't be unsent). Sometimes it's better to reorder steps so the risky one goes first, or hold resources with a reservation that expires.
Two-phase commit vs sagas
Two-phase commit gives real atomicity across databases but blocks while waiting and needs every participant to support it. Most services and cloud databases don't, so sagas are the norm.
Keep it in one service
The cheapest distributed transaction is the one you don't need. If data must change atomically, consider keeping it in one service (the integrator from Module 16).
5. In real projects
Compensations fail too. A refund API can be down, a hotel can refuse a late cancellation. Every compensation needs retries, idempotency and, in the end, a queue of cases for a person to resolve. Assuming compensation always works is one of the classic fallacies.
Design the visible states. Between steps, other users and systems can see half-finished work: a seat shown as held, a "booking confirmed" email for a trip that's about to be cancelled. Use pending states, delay irreversible side effects (emails, shipping) to the end, and word messages carefully.
Use durable workflow tooling. Orchestrated sagas run well on workflow engines (Temporal, Step Functions, Camunda) that persist the saga's state, retry steps and survive crashes.
Say it in the interview
- "How do you do a transaction across services?" A saga: local transactions in each service, each with a compensating step, run in reverse if a later step fails. Coordinated by an orchestrator for complex flows, or by events for simple ones.
- "What if a compensation fails?" Retry it idempotently with backoff; if it still fails, park it for manual resolution and alert. And avoid irreversible steps until the saga is sure to complete.
- "Explain CAP." During a network partition, a distributed store must choose between consistency (refuse some requests) and availability (answer from possibly stale data). Without a partition, you can have both.
- Assuming you can roll back across services like in one database.
- Compensations with no failure handling.
- Sending the confirmation email in the first step of the saga.
6. Remember it
How this hook works: what "book, book, fail, cancel, cancel" means
What the slogan means, phrase by phrase
- "Book, book"
- Each step commits locally and is visible straight away.
- "fail"
- A later step fails; there's no global rollback.
- "cancel, cancel"
- Compensating steps undo the completed ones, in reverse order, and may themselves fail.
Why this picture fits: everyone who has untangled a holiday booking knows that cancelling is a separate action, sometimes with fees, and that emails already sent stay sent.
The story → the concept
| Booking a holiday | A saga |
|---|---|
| Flight, hotel, car booked separately | Local transactions in separate services |
| Cancelling the hotel | A compensating update |
| The non-refundable fee | A compensation that can't fully undo |
| The confirmation email | A visible intermediate state |
The trap it saves you from
Designing a distributed workflow as if one database transaction wrapped it.
7. Common mistakes
8. Design exercise (30 minutes)
- Write the saga (10 min). For an order (reserve stock, charge card, book courier, email), list each step and its compensation.
- Reorder it (10 min). Which step is most likely to fail? Which is irreversible? Reorder to fail early and commit irreversibly late.
- Break a compensation (10 min). Pick one compensation and design what happens when it fails three times in a row.
9. Self-check quiz
The third of four saga steps fails. How are the first two undone?
How does a saga differ from an ACID transaction regarding isolation?
During a network partition, CAP says a system must:
A refund compensation keeps failing. What's the right design?
When should the confirmation email be sent?
10. Go deeper
| Resource | Time | Why |
|---|---|---|
| Software Architecture: The Hard Parts | book | The eight saga types, transactional sagas and their trade-offs in depth. |
| Saga pattern (Chris Richardson) | 15 min | Orchestrated and choreographed sagas with examples. |
| Sagas (Garcia-Molina and Salem, 1987) | 40 min | The original paper that named the idea. |