Workflows, sagas and the outbox

Placing an order means reserving stock, charging a card, booking a courier and sending an email, often in four different services, some owned by other companies. There's no single transaction that spans them all, any step can fail, and the server running the process can crash halfway. This module shows how to make multi-step processes reliable: undo instead of rollback, never lose an event, and let a workflow engine remember where you were.

1. The idea

• A distributed transaction (two-phase commit, 2PC) would lock every participant until all agree. It's slow, a stuck coordinator blocks everyone, and payment providers and other companies' APIs don't support it. Modern systems mostly avoid it across services.
• A saga is a sequence of local steps, each with a compensating action that semantically undoes it (release the stock, refund the card). If a step fails, run the compensations for the completed steps, in reverse order.
• Sagas promise "eventually all done, or all undone", not isolation: for a while an order is visibly "pending", and other requests can see the in-between state. Design the states on purpose.
• The process must survive crashes of the coordinator itself. That needs durable state: which steps completed, with what results.

Two ways to coordinate a saga

2. How it works

An order saga that fails at step 3

The dual-write problem and the outbox

Almost every step above does two things: update its own database, and tell the world (publish an event). Doing those as two separate writes is the dual-write problem: if the process crashes between them, the database says "charged" but no event was published, and the saga stalls forever, or the event goes out for a change that was rolled back.

The transactional outbox fixes it. In the same local transaction as the business change, insert the event into an outbox table. A separate relay (a poller, or change data capture such as Debezium reading the database log) publishes outbox rows to the message broker and marks them sent. The change and the event now commit together or not at all; publishing is at-least-once, so consumers stay idempotent (Module 18). Try both in section 3.

BEGIN;
  UPDATE payments SET status = 'charged' WHERE order_id = 881;
  INSERT INTO outbox (id, topic, payload) VALUES (gen_random_uuid(), 'payments', '{"order":881,"event":"PaymentCharged"}');
COMMIT;   -- both rows, or neither
-- relay: SELECT ... FROM outbox WHERE sent = false → publish → mark sent (or Debezium streams the inserts)

Durable execution: workflows as ordinary code

Workflow engines remove most of the hand-built machinery. In Temporal (and similar engines), you write the saga as a normal function. Every step that touches the outside world is an activity whose result the engine records in an event history. If the worker crashes, another worker replays the function from the start; completed activities return their recorded results instead of running again, so execution continues exactly where it stopped. Waiting ("sleep 30 days", "wait for the courier to accept") costs nothing while it waits. Two rules come with it: workflow code must be deterministic (no direct clock reads, randomness or network calls; those go in activities), and activities must be idempotent, because they can be retried.

// a Temporal-style workflow (TypeScript), simplified
async function placeOrder(order) {
  const compensations = [];
  try {
    await reserveStock(order);        compensations.push(() => releaseStock(order));
    await chargeCard(order);          compensations.push(() => refundCard(order));
    await bookCourier(order);         compensations.push(() => cancelCourier(order));
    await sendConfirmation(order);
  } catch (err) {
    for (const undo of compensations.reverse()) await undo();   // also retried until they succeed
    throw err;
  }
}

3. Try it yourself

Run the saga

Choose which step fails (if any) and whether a compensation also fails, then run it.

Dual write vs outbox, with a crash

4. Trade-offs

5. In real systems

• Temporal (open source, or Temporal Cloud) is the most widely used durable-execution engine; it grew out of Uber's Cadence. Similar ideas appear in Restate, DBOS and Azure Durable Functions.
• AWS Step Functions defines workflows as JSON state machines. Standard workflows can run for up to one year; inputs and outputs are limited to 256 KiB, so pass IDs and S3 keys, not documents. Google Cloud Workflows and Azure Logic Apps are similar.
• Apache Airflow and Dagster orchestrate scheduled data pipelines (Module 23), not user-facing transactions.
• Outbox relays: Debezium's outbox event router streams an outbox table to Kafka through change data capture; many frameworks include a polling outbox.
• Versioning running workflows: changing code while thousands of workflows are mid-flight requires care. Engines let old runs keep the old logic (Temporal's patching, Step Functions' versions) while new runs use the new.

The technologies up close

Why the naive version breaks: one server running the steps in order loses all memory of progress when it crashes (charged, but stock never reserved), and a payment webhook may land on a different server. Patching that with database checkpoints and pub/sub means hand-building a state machine, a poller for stalled orders, locks so two servers don't grab the same order, retries, and still no compensation. That's the signal to use a saga on a workflow engine.

6. Remember it

7. Common mistakes

8. Design exercise (30 minutes)

1. Design a saga for booking a flight + hotel + car through three partner APIs. List each step, its compensation, and which step goes last and why.
2. Choose coordination: choreography or orchestration for (a) "user signs up → welcome email + analytics event", (b) a mortgage application with document checks and a human approval. Justify both.
3. Add an outbox to the payment service: write the table schema and the relay's loop, and say how consumers handle duplicates.
4. Handle a 7-day wait: a contract must be signed within 7 days or the order is cancelled. Describe it with a durable timer and a signal.
5. Write it down: saga = steps + compensations; outbox = change and event in one transaction.

9. Self-check quiz

10. Go deeper

11. Under the hood (optional)

How workflow engines survive crashes and how schedulers keep millions of timers. Not needed for most interviews; open it when you want to know how the products really do it. All the deep dives are listed on the Under the hood page.