InterviewKata
System design · Technology guide T10 · Builds on Module 22

Workflow engines and orchestrators

Placing an order means reserving stock, charging a card and booking a courier, any of which can fail or take hours. Loading tonight's data means running thirty jobs in the right order. Both are workflows, and writing them as plain code with retries and cron jobs breaks the first time a server dies halfway. This guide explains the two families of engines that solve this, the products, how they differ and which to choose, with a chooser and code you can run.

A 2-minute conversation: Sam asks, Priya explains
0:00 / 2:24

1. Two families

Durable execution (application workflows)Data orchestrators (pipelines)
RunsOne workflow per business event: an order, a sign-up, a refundA graph (DAG) of jobs on a schedule or when data arrives
Steps takeMilliseconds to months (waiting for a human, a timer)Minutes to hours each
VolumeThousands to millions of workflows at onceHundreds of pipelines, a few runs each a day
ExamplesTemporal, Step Functions, Durable Functions, Cloud WorkflowsAirflow, Cloud Composer, Dagster, Prefect, Argo Workflows

Both record what has finished, so a crash resumes instead of restarting. Module 22 teaches sagas and compensation; this guide is about the engines.

2. How durable execution survives a crash

3. The products

Durable execution · open source and cloud

Temporal

Workflows are ordinary code (Go, Java, TypeScript, Python and more). The server stores an event history for each workflow; if a worker dies, another replays the history to rebuild state and carries on. Workflow code must be deterministic; side effects go in activities, which are retried with your policy. Self-hosted or Temporal Cloud.

Best for: long-running, high-volume business workflows written in code.

State machines · AWS

AWS Step Functions

Workflows defined as JSON state machines (Amazon States Language) that call AWS services directly. Standard workflows run up to a year with exactly-once execution and 90 days of history; Express workflows run up to five minutes, at-least-once when asynchronous and at-most-once when synchronous, for high-volume event processing.

Best for: orchestrating AWS services and Lambda functions.

Durable code · AWS

Lambda durable functions

A newer Lambda feature for multi-step code with checkpoints, so a function can wait and resume without paying while it waits. Quotas cover running executions, operations per execution and stored state.

Best for: workflow-style code on AWS without defining a state machine.

Serverless workflows · Google Cloud

Google Cloud Workflows

A serverless service that runs steps written in YAML or JSON: calling HTTP APIs and Google Cloud services, with retries, waits and callbacks, billed per step.

Best for: gluing Google Cloud services and APIs together.

Durable code · Azure

Azure Durable Functions and Logic Apps

Durable Functions lets you write orchestrations as code on Azure Functions, replaying history like Temporal. Logic Apps builds workflows visually from hundreds of connectors.

Best for: code-first (Durable Functions) or low-code (Logic Apps) workflows on Azure.

Data orchestrator · open source

Apache Airflow

Pipelines as Python DAGs: a scheduler starts tasks when their upstream tasks finish, with retries, backfills and a UI. Airflow 3.0 (April 2025) added DAG versioning, a new UI and running tasks outside the Airflow environment. Managed as Cloud Composer, Amazon MWAA and Astronomer.

Best for: scheduled data pipelines, the industry default.

Data orchestrators · open source

Dagster and Prefect

Python orchestrators with a more modern developer experience: Dagster centres on the data assets a pipeline produces, Prefect on flows of Python functions.

Best for: data teams wanting asset-aware or Python-native pipelines.

Kubernetes-native · open source (CNCF)

Argo Workflows

Each step is a container, defined as a Kubernetes resource, with DAGs, retries and artifacts.

Best for: container-based batch and ML pipelines on Kubernetes.

4. Differences at a glance

OptionDefined asLongest runRecovers byBest at
TemporalCodeEffectively unlimitedReplaying event historyBusiness workflows at scale
Step Functions StandardJSON state machineOne yearPersisted state per transitionOrchestrating AWS services
Step Functions ExpressJSON state machineFive minutesRe-running (at-least-once)High-volume short flows
Cloud WorkflowsYAML or JSONLong-running, with callbacksPersisted stateGluing APIs on Google Cloud
Durable FunctionsCodeLong-runningReplaying historyCode-first workflows on Azure
AirflowPython DAGsPer task, hoursRe-running failed tasksScheduled data pipelines
Argo WorkflowsKubernetes YAMLPer step, hoursRe-running failed stepsContainer pipelines

5. When to choose which, and why

  1. Is it a business process per event (an order, an onboarding, a refund) with waits and compensation? A durable execution engine: Temporal, Step Functions, Durable Functions.
  2. Is it a scheduled graph of data jobs? A data orchestrator: Airflow (or managed Composer or MWAA), Dagster, Prefect.
  3. Mostly calling cloud services? The cloud's own engine (Step Functions, Cloud Workflows, Logic Apps) needs the least code.
  4. Complex logic, many languages, or several clouds? Temporal: workflows as normal code, portable between environments.
  5. Don't orchestrate a data pipeline with a durable execution engine, or an order with Airflow. Their scaling, latency and failure models are built for different jobs.

Try it: choose for your requirements

6. Why a plain retry charges twice, in code

An order workflow reserves stock, charges the card and ships. The worker crashes right after charging. A plain script that's simply run again charges the card a second time. Durable execution records each step's result; the re-run replays the first two steps from the history and only does the one that's left. The program was run in all three languages; each prints the output below it.

class Crash(Exception):
    pass


class Payments:
    def __init__(self):
        self.charges = 0

    def charge(self):
        self.charges += 1
        return f"ch_{self.charges}"


STEPS = ["reserve_stock", "charge_card", "ship_order"]


def do(step, payments):
    if step == "charge_card":
        return payments.charge()
    return "trk_81" if step == "ship_order" else "ok"


def naive(payments, crash_after=None):
    """A plain script: on failure, the whole thing is run again from the top."""
    log = []
    for step in STEPS:
        do(step, payments)
        log.append(step)
        if step == crash_after:
            raise Crash(log)
    return log


def durable(payments, history, crash_after=None):
    """Durable execution (Temporal, Step Functions, Durable Functions): each step's result is recorded,
    and a re-run replays recorded results instead of doing the step again."""
    log = []
    for step in STEPS:
        if step in history:
            log.append(f"{step} (from history)")
        else:
            history[step] = do(step, payments)
            log.append(f"{step} (ran)")
        if step == crash_after:
            raise Crash(log)
    return log


p = Payments()
try:
    naive(p, crash_after="charge_card")
except Crash as c:
    print(f"naive run 1:   {', '.join(c.args[0])} -> worker crashes")
print(f"naive run 2:   {', '.join(naive(p))} -> card charged {p.charges} times")

p, step_history = Payments(), {}
try:
    durable(p, step_history, crash_after="charge_card")
except Crash as c:
    print(f"durable run 1: {', '.join(c.args[0])} -> worker crashes")
print(f"durable run 2: {', '.join(durable(p, step_history))} -> card charged {p.charges} time")
print("history: " + ", ".join(f"{k}={v}" for k, v in step_history.items()))
class Crash extends Error {
  constructor(readonly log: string[]) { super("crash"); }
}

class Payments {
  charges = 0;
  charge(): string { return `ch_${++this.charges}`; }
}

const STEPS = ["reserve_stock", "charge_card", "ship_order"];

function run(step: string, payments: Payments): string {
  if (step === "charge_card") return payments.charge();
  return step === "ship_order" ? "trk_81" : "ok";
}

// A plain script: on failure, the whole thing is run again from the top.
function naive(payments: Payments, crashAfter?: string): string[] {
  const log: string[] = [];
  for (const step of STEPS) {
    run(step, payments);
    log.push(step);
    if (step === crashAfter) throw new Crash(log);
  }
  return log;
}

// Durable execution (Temporal, Step Functions, Durable Functions): each step's result is recorded,
// and a re-run replays recorded results instead of doing the step again.
function durable(payments: Payments, stepHistory: Map<string, string>, crashAfter?: string): string[] {
  const log: string[] = [];
  for (const step of STEPS) {
    if (stepHistory.has(step)) log.push(`${step} (from history)`);
    else { stepHistory.set(step, run(step, payments)); log.push(`${step} (ran)`); }
    if (step === crashAfter) throw new Crash(log);
  }
  return log;
}

let p = new Payments();
try { naive(p, "charge_card"); } catch (c) { if (c instanceof Crash) console.log(`naive run 1:   ${c.log.join(", ")} -> worker crashes`); }
console.log(`naive run 2:   ${naive(p).join(", ")} -> card charged ${p.charges} times`);

p = new Payments();
const stepHistory = new Map<string, string>();
try { durable(p, stepHistory, "charge_card"); } catch (c) { if (c instanceof Crash) console.log(`durable run 1: ${c.log.join(", ")} -> worker crashes`); }
console.log(`durable run 2: ${durable(p, stepHistory).join(", ")} -> card charged ${p.charges} time`);
console.log("history: " + [...stepHistory].map(([k, v]) => `${k}=${v}`).join(", "));
import java.util.*;

public class Main {
    static class Crash extends RuntimeException {
        final List<String> log;
        Crash(List<String> log) { this.log = log; }
    }

    static class Payments {
        int charges = 0;
        String charge() { return "ch_" + (++charges); }
    }

    static final List<String> STEPS = List.of("reserve_stock", "charge_card", "ship_order");

    static String run(String step, Payments payments) {
        if (step.equals("charge_card")) return payments.charge();
        return step.equals("ship_order") ? "trk_81" : "ok";
    }

    // A plain script: on failure, the whole thing is run again from the top.
    static List<String> naive(Payments payments, String crashAfter) {
        List<String> log = new ArrayList<>();
        for (String step : STEPS) {
            run(step, payments);
            log.add(step);
            if (step.equals(crashAfter)) throw new Crash(log);
        }
        return log;
    }

    // Durable execution (Temporal, Step Functions, Durable Functions): each step's result is recorded,
    // and a re-run replays recorded results instead of doing the step again.
    static List<String> durable(Payments payments, Map<String, String> stepHistory, String crashAfter) {
        List<String> log = new ArrayList<>();
        for (String step : STEPS) {
            if (stepHistory.containsKey(step)) log.add(step + " (from history)");
            else { stepHistory.put(step, run(step, payments)); log.add(step + " (ran)"); }
            if (step.equals(crashAfter)) throw new Crash(log);
        }
        return log;
    }

    public static void main(String[] args) {
        Payments p = new Payments();
        try { naive(p, "charge_card"); } catch (Crash c) { System.out.println("naive run 1:   " + String.join(", ", c.log) + " -> worker crashes"); }
        System.out.println("naive run 2:   " + String.join(", ", naive(p, null)) + " -> card charged " + p.charges + " times");

        p = new Payments();
        Map<String, String> stepHistory = new LinkedHashMap<>();
        try { durable(p, stepHistory, "charge_card"); } catch (Crash c) { System.out.println("durable run 1: " + String.join(", ", c.log) + " -> worker crashes"); }
        System.out.println("durable run 2: " + String.join(", ", durable(p, stepHistory, null)) + " -> card charged " + p.charges + " time");
        StringJoiner j = new StringJoiner(", ");
        stepHistory.forEach((k, v) -> j.add(k + "=" + v));
        System.out.println("history: " + j);
    }
}
naive run 1:   reserve_stock, charge_card -> worker crashes
naive run 2:   reserve_stock, charge_card, ship_order -> card charged 2 times
durable run 1: reserve_stock (ran), charge_card (ran) -> worker crashes
durable run 2: reserve_stock (from history), charge_card (from history), ship_order (ran) -> card charged 1 time
history: reserve_stock=ok, charge_card=ch_1, ship_order=trk_81

7. Say it in the interview

Two-sentence answers
  • "How do you coordinate checkout across services?" A saga run by a durable workflow engine such as Temporal or Step Functions: each step's result is recorded, failures retry, and a step that can't succeed triggers compensations in reverse order. Each step is idempotent, keyed by the order ID.
  • "How are the nightly jobs run?" Airflow (Cloud Composer or MWAA) DAGs, with each task idempotent and partitioned by date, so a failed day can be backfilled by re-running it.
  • "Why not cron and a queue?" They lose track of where a multi-step process got to when something dies; the engine keeps that state and the history for debugging.

8. Common mistakes

Non-deterministic workflow codeIn Temporal or Durable Functions, reading the clock, random numbers or calling APIs directly in workflow code breaks replay. Put side effects in activities.
Steps that aren't idempotentEngines retry. A step that charges or emails must use an idempotency key.
Express workflows for paymentsAsynchronous Express runs at-least-once; use Standard (exactly-once) or make every step idempotent.
Airflow for per-request workIt schedules pipelines; it isn't built for thousands of short workflows a second.
Huge payloads in workflow statePass references (an object key, a row ID), not the data itself.

9. Check yourself

A worker crashes right after charging a card. Without durable state, what happens when the job is retried from the start?

Unless every step is idempotent or the engine records finished steps.

Why must Temporal workflow code be deterministic?

Replay must take the same path; side effects go in activities.

Which Step Functions type suits a payment flow that must not run twice?

Standard is exactly-once; asynchronous Express is at-least-once.

Thirty jobs must run nightly in dependency order. Which fits?

Scheduled DAGs of data jobs are what orchestrators are built for.

10. Sources and further reading

Product facts checked on 4 November 2026. Limits and features change; check before relying on them.

SourceUsed for
Step Functions: choosing a workflow typeStandard and Express durations and execution guarantees
Lambda quotasDurable functions
Airflow 3.0 release notesDAG versioning and task execution changes
Temporal: workflowsEvent history, replay and determinism
Module 22: Workflows and sagasSagas and compensation