Batch, stream and background jobs
Much of a system's work happens when no user is waiting: encoding a video, sending a million emails, computing yesterday's revenue, spotting fraud as payments flow past. There are three shapes of that work, each with its own tools and failure modes. This module covers how to run single jobs reliably, how to crunch huge datasets in batches, and how to compute continuously over streams of events that arrive late and out of order.
1. The idea
| Shape | Input | Output | Examples | Typical tools |
|---|---|---|---|---|
| Background job | One task, triggered by a request or event | A result for that task, plus a status | Transcode a video, generate a PDF, send an email, run a long AI task | A queue (SQS, BullMQ, Celery, Sidekiq) and workers |
| Batch | A bounded dataset: "all of yesterday's orders" | A complete result, produced periodically | Nightly reports, billing runs, rebuilding a search index, training data | Spark, SQL warehouses, dbt, Airflow or Dagster to schedule |
| Stream | An unbounded, never-ending flow of events | Continuously updated results | Fraud detection, live dashboards, ad click counting, alerting | Flink, Kafka Streams, Spark Structured Streaming, managed services |
2. How it works
Background jobs, done properly
- The API validates the request, creates a job row (
status = pending) and puts the job ID on a queue. It returns202 Acceptedwith the job ID in milliseconds. - A worker takes the ID, loads the job, sets
processing, and does the work. It sends heartbeats (or extends its visibility timeout) so a crashed worker's job is picked up by another. - It stores the result (files in object storage, metadata in the database) and sets
completedorfailed. - The client polls
GET /jobs/{id}, or is notified (push, email, webhook, WebSocket). - Failures retry with backoff; after a few attempts the job goes to a dead-letter queue and an alert fires. Duplicate submissions (a user clicking three times) are stopped with an idempotency key.
- Separate fast and slow queues (or priorities) so a 5-hour export doesn't block 5-second thumbnails. Autoscale workers on queue depth, not CPU.
Scheduled jobs ("every night at 2:00") need care at scale: only one instance should fire each run (elect a leader or take a lease, Module 21), runs must be idempotent in case one is repeated, and missed runs must be noticed.
Batch: split, process, combine
When data is too big for one machine, split it into chunks, process the chunks in parallel, then combine the partial results. Google's MapReduce (2004) made the pattern famous; Spark generalised it with in-memory processing and SQL. Press Play: counting words across a huge set of documents.
Modern batch data usually lives in a data lake: files in object storage in a columnar format (Parquet), organised by an open table format (Apache Iceberg, Delta Lake, Apache Hudi) that adds schemas, transactions and time travel, so Spark, warehouses and query engines can all read the same tables.
Streams: time, windows and late events
- Event time vs processing time. A click happens at 10:00:03 on a phone (event time) but reaches your servers at 10:00:41 after the phone left a tunnel (processing time). Correct results group by event time.
- Windows turn an endless stream into finite groups: tumbling (fixed, non-overlapping: clicks per minute), sliding (overlapping: the last 5 minutes, updated every minute), and session (activity separated by gaps of inactivity).
- Watermarks are the processor's estimate of "we've probably seen everything up to time T". When the watermark passes the end of a window, the window's result is emitted. Events that arrive after that are late: allow some lateness (update the result), send them to a side output, or drop them.
- State and checkpoints. Counts, sums and joins are state kept by the processor. Flink periodically snapshots all state together with the input positions (offsets), so after a crash it restores the snapshot and replays from those offsets: exactly-once state, even with at-least-once input.
3. Try it yourself
Windows and watermarks
Sixteen click events, placed by their event time. Some arrive late. Choose the window size and how much lateness the watermark allows, and see which events count, which update a result late, and which are dropped.
Each event's dot is placed at its event time; its arrival delay is in the tooltip. The watermark trails the latest event time seen by the "waits for" amount. Blue = counted on time, yellow = counted as a late update, red = dropped.
Job, batch or stream?
4. Trade-offs
Batch: simple and complete, but late
Batches see all the data, are easy to rerun and debug, and are cheap. Results are hours old. Most reporting and billing is still batch for good reason.
Stream: fresh, but harder
Results in seconds, at the cost of reasoning about time, late data, state and exactly-once. Use it where freshness is the point: fraud, alerting, live counters.
Lambda vs Kappa
Lambda architecture runs both (stream for fast approximate results, batch to correct them later): two codebases. Kappa uses only the stream and replays the log to recompute. Many teams now pick one engine for both.
Workers: servers, containers or serverless
Long-running servers are simple and have no time limits. Serverless functions scale to zero and absorb spikes but have time limits (AWS Lambda: 15 minutes) and cold starts. Containers on Kubernetes sit in between.
5. In real systems
- Background jobs: Celery (Python), Sidekiq (Ruby), BullMQ (Node.js), or a cloud queue plus workers. GPU-heavy jobs (video, AI) run on separate worker pools from the web tier.
- Batch: Apache Spark (version 4.2 in October 2026) is the standard engine; SQL warehouses (BigQuery, Snowflake, Redshift) increasingly do batch transforms directly, often managed with dbt. Apache Airflow (3.x; 3.3 current) and Dagster schedule and monitor pipelines as graphs of tasks.
- Stream: Apache Flink (version 2.3 in October 2026) is the leading stateful stream processor, with event time, watermarks and checkpoint-based exactly-once state. Kafka Streams is a library for stream processing inside your own services; Spark Structured Streaming and Google Cloud Dataflow (Apache Beam) are common alternatives.
- Exactly-once end to end needs the sink to cooperate: transactional writes (Kafka transactions, Flink's two-phase commit sinks) or idempotent upserts keyed by event ID.
- Back-pressure: stream processors slow their sources when a downstream stage can't keep up, rather than dropping data or running out of memory. Monitor consumer lag.
The technologies up close
return a job ID, process on a worker pool, notify when done.For metrics and counters, say whether results must be live (stream, with windows and watermarks) or can be hours old (batch), and how late events are handled.
When the math says so: 1 million images a day × 10 seconds each ≈ 12 a second ≈ 120 seconds of work every second. That can't run on the web servers; it needs a worker pool on compute-optimised (or GPU) machines. Load balancers time out requests after 30–60 seconds anyway.
| Deep-dive question | Answer |
|---|---|
| A worker crashes mid-job | Heartbeats or a visibility timeout (10–30 s): the job returns to the queue for another worker. Too long delays recovery; too short marks slow jobs (a GC pause) as dead |
| A job keeps failing (poison message) | After 3–5 attempts move it to a dead-letter queue, alert, fix, re-queue |
| The user clicks "Export" three times | An idempotency key (user + action + input hash) returns the existing job ID; make the work itself idempotent too |
| Black Friday: 10× load | Back-pressure (cap queue depth, reply "busy"); autoscale workers on queue depth, not CPU |
| 5-second and 5-hour jobs share a queue | Separate fast and slow queues with their own workers, or split big jobs into chunks (scene below) |
| Jobs depend on each other (fetch → PDF → email) | Simple chains: each worker queues the next step with full context. Complex branching: a workflow engine (Module 22) |
Job queue + worker pool
The default for anything slower than a second or twoAccept quickly, queue, process asynchronously, notify when done. The web tier only validates and returns a job ID.
How it works
Covered in section 2: a job row (pending), the job ID (not the payload) on the queue, workers that load the job, set processing, store results (files in S3, metadata in the database), set completed or failed, and notify (push, email, WebSocket) or let the client poll. Each part can fail independently: queue down → jobs wait as pending; workers down → jobs wait in the queue.
Status today
Standard everywhere: SQS, BullMQ, Celery, Sidekiq, RabbitMQ, or Kafka at scale. Interview default: Kafka or SQS plus regular worker processes.
Use it when
- Video transcoding, PDFs, reports, bulk email, CSV imports, AI tasks
- Work needing different hardware (GPU) from the web tier
Skip it when
- Short jobs (tens of milliseconds): synchronous is simpler and gives clearer back-pressure
In the interview
Transcoding takes minutes, so the API returns a job ID in milliseconds; GPU workers pull from the queue and we notify the user when it's done.
- Doing slow work inside the request.
- Putting the payload on the queue instead of an ID (size limits, stale data).
- Queueing tiny jobs for no reason.
- Debating queue brands. Focus on separating concerns, retries and status.
Serverless workers (AWS Lambda and similar)
Situational · spiky loadsFunctions that run per event, scale out automatically and to zero, billed per use.
How it works
A queue or storage event triggers a function invocation per message or batch. Great for bursts; limits apply: AWS Lambda runs at most 15 minutes, has cold starts and little local disk.
Status today
Mature: AWS Lambda, Google Cloud Run functions, Azure Functions.
Use it when
- Spiky, short jobs: thumbnails, webhooks, small transforms
Skip it when
- Jobs longer than the time limit (video transcoding of long files)
- Steady heavy load, where regular servers are cheaper
In the interview
Thumbnails run on Lambda triggered by the upload event; long transcodes run on container workers.
- Lambda for a two-hour job.
Apache Spark (batch)
Situational · big bounded datasetsThe standard engine for processing large, bounded datasets in parallel across a cluster.
How it works
Data is split into partitions processed in parallel; shuffles regroup data by key (the MapReduce idea, section 2). Jobs are easy to rerun from scratch, which makes batch simple to debug.
Status today
Spark 4.2 is current (October 2026). SQL warehouses (BigQuery, Snowflake) increasingly run batch transforms directly, managed with dbt; Airflow or Dagster schedules them.
Use it when
- Nightly reports, billing runs, rebuilding indexes, ML training data
Skip it when
- Results needed within seconds (streaming)
- Small datasets a SQL query handles
In the interview
Daily revenue is a nightly Spark (or warehouse SQL) job scheduled by Airflow; nobody needs it within seconds.
- Streaming everything when hours-old results are fine: batch is cheaper and simpler.
Apache Flink (stream)
Situational · results in secondsA stateful stream processor: windows over event time, watermarks for late data, and checkpointed exactly-once state.
How it works
Reads an unbounded stream (usually Kafka), keeps state per key (counts per ad per minute), emits results as windows close, and checkpoints state so it recovers without double counting. Exactly-once end to end needs the sink to cooperate (transactional or idempotent writes).
Status today
Flink 2.3 is current (October 2026). Kafka Streams (a library inside your service), Spark Structured Streaming and Google Dataflow are the common alternatives.
Use it when
- Fraud detection, alerting, live dashboards, ad click counting
Skip it when
- Most designs: a queue and workers, or batch, is enough
- Interviews where freshness of seconds isn't a requirement
In the interview
Only when freshness is the point: Clicks are aggregated per ad per minute in Flink with a 30-second watermark; late events update the result.
- Reaching for Flink first. Start with polling alerts or batch; add streaming for the cases that need seconds.
6. Remember it
How this hook works: what "Ticket, nightly load, conveyor belt" means
What the slogan means, phrase by phrase
- "Ticket"
- Background jobs: accept fast, hand back a job ID, process asynchronously, report status, retry and park failures.
- "nightly load"
- Batch: a bounded dataset, processed in parallel chunks and combined; easy to rerun from scratch.
- "conveyor belt"
- Streams: unbounded input, results per window, watermarks to decide when a window is complete, and checkpoints to resume after a crash.
Why this picture fits: a laundry runs all three side by side, with different machines, which is exactly how real systems combine job queues, batch pipelines and stream processors.
The story → the concept
| Laundry | Processing |
|---|---|
| A collection ticket | A job ID and status endpoint |
| The overnight hotel-sheet run | A batch job over yesterday's data |
| Sorting into piles, washing piles, folding | Map, shuffle, reduce |
| The pressing conveyor | A stream processor |
| Counting shirts per hour | A tumbling window |
| A shirt found under the belt later | A late event |
| Noting the count every 10 minutes in case the belt jams | A checkpoint |
Watch it on an example
- An ad platform counts clicks per ad per minute for advertisers' live dashboards (stream: one-minute tumbling windows, 10-second watermark lag).
- Phones that were offline send clicks hours later; they're too late for the live window and go to a side output.
- A nightly batch recomputes exact daily counts from all raw clicks in the data lake, including the late ones, for billing.
- Advertisers see fast approximate numbers by day and are billed from exact ones.
The trap it saves you from
Grouping streams by arrival time (processing time), which makes results depend on network delays, and losing state on restart because nothing was checkpointed.
It still works when…
You run AI workloads: batch inference over a dataset (batch), a long agent task per user request (job), or moderating messages as they're posted (stream).
7. Common mistakes
8. Design exercise (30 minutes)
- Design a job system for "export my data" (takes 1–60 minutes): API, job table, statuses, queue, workers, notification, retries and duplicate protection.
- Batch on paper: compute "top 10 products by revenue per country, yesterday" from 2 billion order lines. Describe the map, shuffle key and reduce.
- Stream on paper: alert when a card is used in two countries within 10 minutes. Which window type? What state is kept per card? What happens with late events?
- Use the simulator: find the smallest watermark delay that drops no events with 20-second windows. What does it cost in result latency?
- Write it down: event time vs processing time, and the three window types.
9. Self-check quiz
A request starts a 10-minute video transcode. What should the API return?
Which time should stream windows usually be based on?
What does a watermark decide?
Monthly invoices computed from all of last month's usage records. Which shape?
Your workers' CPU is low but users wait minutes for thumbnails. What should autoscaling watch?
A user clicks "Export" three times. What stops three identical jobs?
10. Go deeper
| Resource | Time | Why |
|---|---|---|
| Streaming 101 and Streaming 102 (Tyler Akidau) | 1.5 h | The clearest explanation of event time, windows, watermarks and triggers. |
| MapReduce: Simplified Data Processing on Large Clusters | 45 min | The paper that started big-data batch processing. |
| Timely stream processing (Apache Flink docs) | 30 min | Event time and watermarks in a production stream processor. |
| Managing long-running tasks (Hello Interview) | 20 min | The job pattern in interview form, with the common follow-ups. |
11. Under the hood (optional)
How batch, stream, warehouse and time-series engines really organise data and recover from failure. Not needed for most interviews; open it when you want to know how the products really do it. All the deep dives are listed on the Under the hood page.
Processing, analytics and time seriesRow vs column storage with a lab, encodings, MPP and the shuffle, lineage vs checkpoints, watermarks, Gorilla compression lab, 15 products comparedabout 40 min
Transactional databases are built to find and change a few rows fast. Analytics engines, batch and stream processors and time-series databases reorganise data for a different question: read billions of values, few columns at a time, and summarise them. That changes the storage layout, the execution engine and how failures are recovered.
Row vs column storage
A row store keeps each row's columns together: ideal for "fetch order 42". A column store keeps each column together: ideal for "sum revenue by day over a year", because it reads only the columns the query names, they compress far better, and whole blocks can be skipped using their min-max statistics.
| Encoding | Idea | Works on |
|---|---|---|
| Dictionary | Store each distinct value once; the column holds small integer codes | Strings with repeats: country, status, product |
| Run-length | "PAID × 4,812" instead of 4,812 copies | Sorted or slowly changing columns |
| Delta / delta-of-delta | Store differences from the previous value | Timestamps, increasing IDs |
| Bit-packing | Use only as many bits as the largest value needs | Small integers and dictionary codes |
| General compression | LZ4, Zstandard or Snappy on top of the above | Everything |
- Vectorized execution: engines process a batch of a few thousand values from one column per operation instead of one row at a time, which keeps CPU caches warm and lets SIMD instructions work on many values at once (DuckDB, ClickHouse, Snowflake, BigQuery).
- Data skipping: Parquet row groups, ORC stripes, Snowflake micro-partitions and Redshift zone maps all keep min and max per block; a filter on a sorted or clustered column skips most blocks without reading them. That's why choosing the sort or clustering key matters more than indexes in a warehouse.
Distributing the work: MPP and the shuffle
- Massively parallel processing: each node scans its slice, computes partial results, then rows are shuffled by key so all values for one key meet on one node for the final aggregation or join. The shuffle is usually the most expensive step: every byte crosses the network and often touches disk.
- Avoid shuffles by placing data: Redshift distributes tables by a key (co-located joins), evenly, or copies small tables to all nodes; Spark broadcasts small tables to every executor; Citus co-locates on the distribution column.
- Skew: one huge key (a celebrity, a "null" bucket) makes one task run for hours while the others finish in seconds. Salt the key, or handle hot keys separately.
- Precompute: materialized views, rollups at ingestion (Druid, Pinot) and cubes trade storage and freshness for instant queries. Star schemas (a fact table of events joined to small dimension tables) keep joins cheap.
Recovering from failure: lineage vs checkpoints
- MapReduce wrote every stage's output to disk: robust, slow.
- Spark records the lineage of each partition (the transformations that produced it); a lost partition is recomputed from its inputs rather than restored. Long chains can be cut with explicit checkpoints.
- Flink checkpoints all operator state consistently while the stream keeps flowing, using barriers injected into the stream (a variant of the Chandy–Lamport snapshot algorithm). After a failure, every operator restores from the last checkpoint and sources rewind to the matching offsets.
Event time, watermarks and windows
Events arrive late and out of order (a phone offline for an hour). Stream processors group by when events happened (event time), not when they arrived, and use a watermark, an estimate that "no events older than T are still coming", to decide when a window is complete.
| Window | Shape | Example |
|---|---|---|
| Tumbling | Fixed, non-overlapping | Clicks per minute |
| Sliding (hopping) | Fixed length, overlapping (10 min every 1 min) | Moving averages, rate limits |
| Session | Closes after a gap of inactivity | A user's visit |
Triggers decide when to emit (at the watermark, early speculative results, again for late data); allowed lateness says how long to keep a window's state for stragglers. The Apache Beam model (and Google Dataflow) made this "what, where, when, how" vocabulary standard.
Exactly-once output needs the sink to cooperate: Flink's two-phase-commit sinks (for example, Kafka transactions committed when a checkpoint completes), or idempotent upserts keyed by a deterministic ID.
Time-series storage
Metrics arrive as (series, timestamp, value) at a steady rate. Time-series databases partition by time (old partitions are cheap to drop or downsample), index series by their labels, and compress each series heavily. Prometheus uses Facebook's Gorilla encoding: timestamps as delta-of-deltas and values XORed with the previous value.
- Cardinality is the cost: every unique combination of labels is a separate series with its own index entries and memory. A label like
user_idorrequest_idcreates millions of series and takes Prometheus down. - Downsampling and retention: keep raw points for days, 5-minute rollups for months, hourly for years (Thanos, Mimir, TimescaleDB continuous aggregates).
- Approximate answers are often the right ones: HyperLogLog for distinct counts, t-digest or DDSketch for percentiles, which can be merged across servers and time ranges.
Product by product
| Product | How it is organised | The detail worth knowing |
|---|---|---|
| Hadoop MapReduce | Map near the data, shuffle (sort, spill, merge), reduce | Every stage on disk: robust and slow |
| Apache Spark | DAG of stages split at shuffles; lineage recovery; Catalyst optimizer; Structured Streaming in micro-batches | In-memory caching made it MapReduce's successor |
| Apache Flink | Record-at-a-time streaming with keyed state (memory or RocksDB) | Barrier checkpoints; two-phase-commit sinks for exactly-once |
| Kafka Streams | A library inside your app; state in RocksDB backed by changelog topics | No cluster; repartition topics act as the shuffle |
| Apache Beam / Dataflow | One model for batch and stream: windows, watermarks, triggers | Separates what is computed from when it's emitted |
| Parquet and ORC | Row groups (stripes) of column chunks and pages with statistics | Readers skip row groups by min-max |
| Snowflake | Immutable micro-partitions; storage separate from compute (virtual warehouses) | Pruning by metadata replaces indexes; old micro-partitions give time travel |
| Google BigQuery | Dremel execution trees over Capacitor columnar storage, with a separate shuffle tier | Partitioning and clustering cut the bytes scanned, which is what on-demand pricing charges |
| Amazon Redshift | MPP columnar; tables distributed by key, evenly, or to all nodes; sort keys and zone maps | A poor distribution key turns joins into network shuffles |
| Apache Druid, Apache Pinot | Segments with bitmap indexes and rollup at ingestion; real-time from Kafka | Sub-second queries on fresh events; Pinot adds star-tree indexes |
| ClickHouse | MergeTree parts of sorted columns, merged in the background | Sparse primary index and data-skipping indexes |
| DuckDB | In-process vectorized columnar engine | Analytics with no server |
| Prometheus | In-memory head block with a WAL, compacted into 2-hour blocks; Gorilla-compressed chunks; label index | High-cardinality labels exhaust memory |
| InfluxDB | TSM engine in versions 1–2; version 3 rebuilt on Arrow, Parquet and DataFusion | A database that replaced its storage engine |
| TimescaleDB | PostgreSQL extension; hypertables split into time chunks; columnar compression for old chunks | Time-series speed with SQL and joins |
Say it in the interview
- "Why is the warehouse fast at SUM over a year?" It reads only the columns used, compressed 5–10×, skips blocks outside the date range by min-max statistics, and scans in parallel across nodes with vectorized execution.
- "How do you count clicks per minute when events arrive late?" Window by event time, emit when the watermark passes the window end, keep state for an allowed lateness and update the result if stragglers arrive; write results idempotently.
- "Can I add user_id as a metric label?" No: each value creates a new series. Put per-user detail in logs or traces, and keep metric labels low-cardinality.
Why does a column store scan far fewer bytes for an analytics query?
What is a watermark in stream processing?
How does Spark recover partitions lost when an executor dies?
Sources
- Spark: RDD programming guide (lineage)
- Flink: stateful stream processing and checkpoints
- Apache Beam programming guide
- The Dataflow Model (Google, 2015)
- Parquet file format
- Snowflake: micro-partitions
- BigQuery: storage overview
- Redshift: distribution and sort keys
- Druid: segments
- Prometheus: storage
- Gorilla: a fast, scalable, in-memory time series database (VLDB 2015)
- TimescaleDB: hypertables
- Why DuckDB