Workflow engines and orchestrators
Placing an order means reserving stock, charging a card and booking a courier, any of which can fail or take hours. Loading tonight's data means running thirty jobs in the right order. Both are workflows, and writing them as plain code with retries and cron jobs breaks the first time a server dies halfway. This guide explains the two families of engines that solve this, the products, how they differ and which to choose, with a chooser and code you can run.
- Sam Why do I need a workflow engine?
- Priya Because multi-step processes fail halfway. Reserving stock, charging a card and booking a courier as plain code with retries breaks the first time a server dies in the middle. An engine remembers what's finished, so a crash resumes instead of starting again.
- Sam What are the two families?
- Priya Durable execution engines run one workflow per business event, like an order, with steps that take milliseconds to months: Temporal, Step Functions, Durable Functions and Cloud Workflows. Data orchestrators run graphs of jobs on a schedule: Airflow, Cloud Composer, Dagster, Prefect and Argo Workflows.
- Sam How does durable execution survive a crash?
- Priya Each finished step's result is appended to the workflow's event history. If the worker dies, another worker gets the history and replays the workflow code. Steps already recorded return their results instead of running again, so only the remaining step runs, and the card is charged once.
- Sam What are the options?
- Priya Temporal, with workflows as ordinary code that must be deterministic. Step Functions: Standard workflows run up to a year with exactly-once execution; Express run up to five minutes, at-least-once when asynchronous. Lambda's newer durable functions. Google Cloud Workflows. Azure Durable Functions and Logic Apps. And Airflow, whose version three added DAG versioning, managed as Cloud Composer and MWAA.
- Sam How do I choose?
- Priya A business process per event: a durable execution engine. Scheduled data jobs: a data orchestrator. Mostly calling cloud services: the cloud's own engine. Complex logic or several clouds: Temporal. And don't swap them: Airflow isn't for orders, and Temporal isn't for nightly DAGs.
- Priya Your turn. Which Step Functions type suits a payment that must not run twice?
- Priya Standard. It's exactly-once; asynchronous Express is at-least-once.
- Sam And the code?
- Priya An order workflow that crashes after charging the card: a plain retry charges twice, durable replay charges once.
- Sam Common mistakes?
- Priya Non-deterministic workflow code. Steps that aren't idempotent. Express workflows for payments. Airflow for per-request work. And huge payloads in workflow state.
1. Two families
| Durable execution (application workflows) | Data orchestrators (pipelines) | |
|---|---|---|
| Runs | One workflow per business event: an order, a sign-up, a refund | A graph (DAG) of jobs on a schedule or when data arrives |
| Steps take | Milliseconds to months (waiting for a human, a timer) | Minutes to hours each |
| Volume | Thousands to millions of workflows at once | Hundreds of pipelines, a few runs each a day |
| Examples | Temporal, Step Functions, Durable Functions, Cloud Workflows | Airflow, Cloud Composer, Dagster, Prefect, Argo Workflows |
Both record what has finished, so a crash resumes instead of restarting. Module 22 teaches sagas and compensation; this guide is about the engines.
2. How durable execution survives a crash
3. The products
Temporal
Workflows are ordinary code (Go, Java, TypeScript, Python and more). The server stores an event history for each workflow; if a worker dies, another replays the history to rebuild state and carries on. Workflow code must be deterministic; side effects go in activities, which are retried with your policy. Self-hosted or Temporal Cloud.
Best for: long-running, high-volume business workflows written in code.
AWS Step Functions
Workflows defined as JSON state machines (Amazon States Language) that call AWS services directly. Standard workflows run up to a year with exactly-once execution and 90 days of history; Express workflows run up to five minutes, at-least-once when asynchronous and at-most-once when synchronous, for high-volume event processing.
Best for: orchestrating AWS services and Lambda functions.
Lambda durable functions
A newer Lambda feature for multi-step code with checkpoints, so a function can wait and resume without paying while it waits. Quotas cover running executions, operations per execution and stored state.
Best for: workflow-style code on AWS without defining a state machine.
Google Cloud Workflows
A serverless service that runs steps written in YAML or JSON: calling HTTP APIs and Google Cloud services, with retries, waits and callbacks, billed per step.
Best for: gluing Google Cloud services and APIs together.
Azure Durable Functions and Logic Apps
Durable Functions lets you write orchestrations as code on Azure Functions, replaying history like Temporal. Logic Apps builds workflows visually from hundreds of connectors.
Best for: code-first (Durable Functions) or low-code (Logic Apps) workflows on Azure.
Apache Airflow
Pipelines as Python DAGs: a scheduler starts tasks when their upstream tasks finish, with retries, backfills and a UI. Airflow 3.0 (April 2025) added DAG versioning, a new UI and running tasks outside the Airflow environment. Managed as Cloud Composer, Amazon MWAA and Astronomer.
Best for: scheduled data pipelines, the industry default.
Dagster and Prefect
Python orchestrators with a more modern developer experience: Dagster centres on the data assets a pipeline produces, Prefect on flows of Python functions.
Best for: data teams wanting asset-aware or Python-native pipelines.
Argo Workflows
Each step is a container, defined as a Kubernetes resource, with DAGs, retries and artifacts.
Best for: container-based batch and ML pipelines on Kubernetes.
4. Differences at a glance
| Option | Defined as | Longest run | Recovers by | Best at |
|---|---|---|---|---|
| Temporal | Code | Effectively unlimited | Replaying event history | Business workflows at scale |
| Step Functions Standard | JSON state machine | One year | Persisted state per transition | Orchestrating AWS services |
| Step Functions Express | JSON state machine | Five minutes | Re-running (at-least-once) | High-volume short flows |
| Cloud Workflows | YAML or JSON | Long-running, with callbacks | Persisted state | Gluing APIs on Google Cloud |
| Durable Functions | Code | Long-running | Replaying history | Code-first workflows on Azure |
| Airflow | Python DAGs | Per task, hours | Re-running failed tasks | Scheduled data pipelines |
| Argo Workflows | Kubernetes YAML | Per step, hours | Re-running failed steps | Container pipelines |
5. When to choose which, and why
- Is it a business process per event (an order, an onboarding, a refund) with waits and compensation? A durable execution engine: Temporal, Step Functions, Durable Functions.
- Is it a scheduled graph of data jobs? A data orchestrator: Airflow (or managed Composer or MWAA), Dagster, Prefect.
- Mostly calling cloud services? The cloud's own engine (Step Functions, Cloud Workflows, Logic Apps) needs the least code.
- Complex logic, many languages, or several clouds? Temporal: workflows as normal code, portable between environments.
- Don't orchestrate a data pipeline with a durable execution engine, or an order with Airflow. Their scaling, latency and failure models are built for different jobs.
Try it: choose for your requirements
6. Why a plain retry charges twice, in code
An order workflow reserves stock, charges the card and ships. The worker crashes right after charging. A plain script that's simply run again charges the card a second time. Durable execution records each step's result; the re-run replays the first two steps from the history and only does the one that's left. The program was run in all three languages; each prints the output below it.
class Crash(Exception):
pass
class Payments:
def __init__(self):
self.charges = 0
def charge(self):
self.charges += 1
return f"ch_{self.charges}"
STEPS = ["reserve_stock", "charge_card", "ship_order"]
def do(step, payments):
if step == "charge_card":
return payments.charge()
return "trk_81" if step == "ship_order" else "ok"
def naive(payments, crash_after=None):
"""A plain script: on failure, the whole thing is run again from the top."""
log = []
for step in STEPS:
do(step, payments)
log.append(step)
if step == crash_after:
raise Crash(log)
return log
def durable(payments, history, crash_after=None):
"""Durable execution (Temporal, Step Functions, Durable Functions): each step's result is recorded,
and a re-run replays recorded results instead of doing the step again."""
log = []
for step in STEPS:
if step in history:
log.append(f"{step} (from history)")
else:
history[step] = do(step, payments)
log.append(f"{step} (ran)")
if step == crash_after:
raise Crash(log)
return log
p = Payments()
try:
naive(p, crash_after="charge_card")
except Crash as c:
print(f"naive run 1: {', '.join(c.args[0])} -> worker crashes")
print(f"naive run 2: {', '.join(naive(p))} -> card charged {p.charges} times")
p, step_history = Payments(), {}
try:
durable(p, step_history, crash_after="charge_card")
except Crash as c:
print(f"durable run 1: {', '.join(c.args[0])} -> worker crashes")
print(f"durable run 2: {', '.join(durable(p, step_history))} -> card charged {p.charges} time")
print("history: " + ", ".join(f"{k}={v}" for k, v in step_history.items()))
class Crash extends Error {
constructor(readonly log: string[]) { super("crash"); }
}
class Payments {
charges = 0;
charge(): string { return `ch_${++this.charges}`; }
}
const STEPS = ["reserve_stock", "charge_card", "ship_order"];
function run(step: string, payments: Payments): string {
if (step === "charge_card") return payments.charge();
return step === "ship_order" ? "trk_81" : "ok";
}
// A plain script: on failure, the whole thing is run again from the top.
function naive(payments: Payments, crashAfter?: string): string[] {
const log: string[] = [];
for (const step of STEPS) {
run(step, payments);
log.push(step);
if (step === crashAfter) throw new Crash(log);
}
return log;
}
// Durable execution (Temporal, Step Functions, Durable Functions): each step's result is recorded,
// and a re-run replays recorded results instead of doing the step again.
function durable(payments: Payments, stepHistory: Map<string, string>, crashAfter?: string): string[] {
const log: string[] = [];
for (const step of STEPS) {
if (stepHistory.has(step)) log.push(`${step} (from history)`);
else { stepHistory.set(step, run(step, payments)); log.push(`${step} (ran)`); }
if (step === crashAfter) throw new Crash(log);
}
return log;
}
let p = new Payments();
try { naive(p, "charge_card"); } catch (c) { if (c instanceof Crash) console.log(`naive run 1: ${c.log.join(", ")} -> worker crashes`); }
console.log(`naive run 2: ${naive(p).join(", ")} -> card charged ${p.charges} times`);
p = new Payments();
const stepHistory = new Map<string, string>();
try { durable(p, stepHistory, "charge_card"); } catch (c) { if (c instanceof Crash) console.log(`durable run 1: ${c.log.join(", ")} -> worker crashes`); }
console.log(`durable run 2: ${durable(p, stepHistory).join(", ")} -> card charged ${p.charges} time`);
console.log("history: " + [...stepHistory].map(([k, v]) => `${k}=${v}`).join(", "));import java.util.*;
public class Main {
static class Crash extends RuntimeException {
final List<String> log;
Crash(List<String> log) { this.log = log; }
}
static class Payments {
int charges = 0;
String charge() { return "ch_" + (++charges); }
}
static final List<String> STEPS = List.of("reserve_stock", "charge_card", "ship_order");
static String run(String step, Payments payments) {
if (step.equals("charge_card")) return payments.charge();
return step.equals("ship_order") ? "trk_81" : "ok";
}
// A plain script: on failure, the whole thing is run again from the top.
static List<String> naive(Payments payments, String crashAfter) {
List<String> log = new ArrayList<>();
for (String step : STEPS) {
run(step, payments);
log.add(step);
if (step.equals(crashAfter)) throw new Crash(log);
}
return log;
}
// Durable execution (Temporal, Step Functions, Durable Functions): each step's result is recorded,
// and a re-run replays recorded results instead of doing the step again.
static List<String> durable(Payments payments, Map<String, String> stepHistory, String crashAfter) {
List<String> log = new ArrayList<>();
for (String step : STEPS) {
if (stepHistory.containsKey(step)) log.add(step + " (from history)");
else { stepHistory.put(step, run(step, payments)); log.add(step + " (ran)"); }
if (step.equals(crashAfter)) throw new Crash(log);
}
return log;
}
public static void main(String[] args) {
Payments p = new Payments();
try { naive(p, "charge_card"); } catch (Crash c) { System.out.println("naive run 1: " + String.join(", ", c.log) + " -> worker crashes"); }
System.out.println("naive run 2: " + String.join(", ", naive(p, null)) + " -> card charged " + p.charges + " times");
p = new Payments();
Map<String, String> stepHistory = new LinkedHashMap<>();
try { durable(p, stepHistory, "charge_card"); } catch (Crash c) { System.out.println("durable run 1: " + String.join(", ", c.log) + " -> worker crashes"); }
System.out.println("durable run 2: " + String.join(", ", durable(p, stepHistory, null)) + " -> card charged " + p.charges + " time");
StringJoiner j = new StringJoiner(", ");
stepHistory.forEach((k, v) -> j.add(k + "=" + v));
System.out.println("history: " + j);
}
}naive run 1: reserve_stock, charge_card -> worker crashes
naive run 2: reserve_stock, charge_card, ship_order -> card charged 2 times
durable run 1: reserve_stock (ran), charge_card (ran) -> worker crashes
durable run 2: reserve_stock (from history), charge_card (from history), ship_order (ran) -> card charged 1 time
history: reserve_stock=ok, charge_card=ch_1, ship_order=trk_81
7. Say it in the interview
- "How do you coordinate checkout across services?" A saga run by a durable workflow engine such as Temporal or Step Functions: each step's result is recorded, failures retry, and a step that can't succeed triggers compensations in reverse order. Each step is idempotent, keyed by the order ID.
- "How are the nightly jobs run?" Airflow (Cloud Composer or MWAA) DAGs, with each task idempotent and partitioned by date, so a failed day can be backfilled by re-running it.
- "Why not cron and a queue?" They lose track of where a multi-step process got to when something dies; the engine keeps that state and the history for debugging.
8. Common mistakes
9. Check yourself
A worker crashes right after charging a card. Without durable state, what happens when the job is retried from the start?
Why must Temporal workflow code be deterministic?
Which Step Functions type suits a payment flow that must not run twice?
Thirty jobs must run nightly in dependency order. Which fits?
10. Sources and further reading
Product facts checked on 4 November 2026. Limits and features change; check before relying on them.
| Source | Used for |
|---|---|
| Step Functions: choosing a workflow type | Standard and Express durations and execution guarantees |
| Lambda quotas | Durable functions |
| Airflow 3.0 release notes | DAG versioning and task execution changes |
| Temporal: workflows | Event history, replay and determinism |
| Module 22: Workflows and sagas | Sagas and compensation |