Design a distributed job scheduler

Cron on one machine is a ten-line problem. Cron for millions of customers is not: thousands of jobs fall due in the same second at the top of every hour, machines die halfway through a run, two schedulers must never start the same job twice, and "9:00 every weekday in Mumbai" has to mean the right instant in every time zone. This page has enough material for a full 60–90 minute interview.

1. The prompt

"Design a service that lets customers schedule jobs: run this task at 3 pm tomorrow, or every hour, or every weekday at 9:00. We run the jobs for them on our own machines."

It sounds like a timer and a queue. What makes it hard is time (everything wants to run at :00, and time zones shift twice a year), duplicates (several schedulers and workers, any of which can die at the worst moment) and many tenants sharing one fleet. Products in this space include Kubernetes CronJobs, AWS EventBridge Scheduler, Google Cloud Scheduler, and internal platforms such as Dropbox's task scheduler and Slack's job queue.

2. Frame the problem

Spend the first five minutes asking questions. Each answer below changes something specific in the design. Try to guess the answer and its consequence before opening it.

3. What it must do

1. A customer can schedule a job: once at a time, or on a recurring cron schedule in a chosen time zone. They can pause, change or delete it.
2. The system runs each due job on a worker, retrying failures with backoff and honouring the job's overlap policy.
3. A customer can see each run: status, attempts, start and end times, and the exit result.

Not now (say these out loud, with a reason):

• Job dependencies and multi-step workflows: a different product, a workflow engine.
• Sub-second timing: nobody asked for it, and it would change the whole dispatch design.
• Streaming logs from running jobs: useful, but it's a logging system bolted on, not scheduling.
• Event-triggered jobs ("run when a file lands"): an easy extension once runs can be created by an API call.

4. How well it must do it

5. Numbers that change the design

Skip estimating API traffic for creating and listing jobs. It is tiny next to dispatch, and the interviewer gains nothing from the arithmetic.

6. The nouns

• Job: what to run and when (the schedule definition).
• Run: one occurrence of a job at one scheduled time, with its attempts.
• Tenant: the customer, with quotas.
• Worker: a machine (or slot on one) that executes runs.
• Lease: a time-limited claim of a run by a worker.

The fields that matter (others, like names and descriptions, are obvious):

jobs(job_id, tenant_id, schedule {cron | at}, time_zone, task {image, args, timeout},
     retry {max_attempts, backoff}, overlap_policy, next_run_at, enabled, version)
runs(run_id, job_id, scheduled_for, status, attempt, lease_owner, lease_expires_at, fence, started_at, finished_at, result)

Note next_run_at: storing the next due instant (in UTC) is what makes "what's due now?" a simple indexed query. The cron text and time zone are kept to compute the one after.

7. The contract

REST for customers; gRPC between our own workers and the dispatcher, because it's internal, chatty and benefits from streaming and strict schemas. The tenant always comes from the API key or token, never from the request body.

POST   /v1/jobs                     Idempotency-Key: 5f1c…
       {schedule: {cron: "0 9 * * MON-FRI", time_zone: "Asia/Kolkata"} | {at: "2026-10-18T15:00:00Z"},
        task: {image: "registry/acme/report:3", args: ["--daily"], timeout_s: 900},
        retry: {max_attempts: 3, backoff: "exponential"}, overlap: "skip" | "queue" | "allow"}
       → 201 {job_id, next_run_at}
GET    /v1/jobs/{job_id}             → job, including the next 5 run times (computed)
PATCH  /v1/jobs/{job_id}             {enabled: false} | {schedule: …}      (If-Match: version)
DELETE /v1/jobs/{job_id}
POST   /v1/jobs/{job_id}/runs        → run it now
GET    /v1/jobs/{job_id}/runs?status=failed&cursor=…   → {runs: [...], next_cursor}

// internal, worker ↔ dispatcher (gRPC)
Lease(worker_id, free_slots)        → stream of {run_id, fence, task, lease_ttl}
Heartbeat(run_id, fence)            → ok | lease_lost
Complete(run_id, fence, result)     → ok | stale_fence

Data flow: (1) a job is saved with its next_run_at → (2) a scanner finds jobs falling due soon → (3) it creates a run and advances next_run_at → (4) at the scheduled time the run is handed to a worker under a lease → (5) the worker executes and heartbeats → (6) it reports the result, or the lease expires and the run is retried.

8. First working version

Build it one request at a time, with the simplest parts that work: stateless API servers, one relational database, one scheduler process, a queue and a pool of workers. Write down what you'll fix later and keep going.

8.1 Creating a job

The API validates the cron expression and time zone, computes the first next_run_at in UTC, and inserts the job. The idempotency key makes a retried "create" return the same job instead of creating two.

8.2 Finding due jobs

Once a second, a scheduler runs one transaction: select jobs with next_run_at <= now(), insert a run for each, and move each job's next_run_at forward. Because the run insert and the advance happen together, a crash can't create a run twice or skip one. Run ids then go onto a queue.

BEGIN;
SELECT job_id, next_run_at FROM jobs WHERE enabled AND next_run_at <= now() ORDER BY next_run_at LIMIT 1000 FOR UPDATE;
INSERT INTO runs (run_id, job_id, scheduled_for, status) VALUES (…, 'queued') ON CONFLICT (job_id, scheduled_for) DO NOTHING;
UPDATE jobs SET next_run_at = next_after(cron, time_zone, next_run_at) WHERE job_id IN (…);
COMMIT;

The unique key (job_id, scheduled_for) is a second safety net: whatever happens, one job can have only one run per scheduled time.

8.3 Running a job

A worker takes a run id from the queue, marks it running, starts the container with a timeout, and writes the result. A failure schedules attempt 2 after a backoff delay. When the run finishes, the API can show it.

8.4 Viewing runs

Reads go to the runs table, filtered by job and paged with a cursor on (scheduled_for, run_id). A read replica serves the dashboard, so heavy listing never slows the scheduler.

9. Make it hold up

9.1 More schedulers, never a double dispatch never twice · surviving failures

We need several scheduler processes for availability and throughput, but each due job must be picked up by exactly one of them.

SKIP LOCKED lets each claimer skip rows another transaction has already locked, instead of waiting for them. It turns a table into a safe work queue for many consumers.

9.2 Being on time at the top of the hour on time

The numbers said 200,000 runs fall due in one second. A database poll that handles 2,000 rows a second would still be dispatching the 09:00 jobs at 09:01:40. The fix is to do the slow work early.

Try all three in the lab in section 13. Watch how the waiting backlog (red) collapses as you add the look-ahead, then ready workers, then spreading.

9.3 Workers that die, and zombies that come back never lost · never twice

• Leases, not "running" flags: a worker owns a run only until lease_expires_at (say 30 s), and renews it with heartbeats every 10 s. If heartbeats stop, a reaper finds the expired lease and puts the run back for attempt 2. Nothing stays stuck.
• Fencing tokens: every claim increments fence. A worker that was only paused (a long garbage-collection pause, a network partition) and wakes up later still holds the old fence. Its heartbeat and completion are rejected (stale_fence), so a zombie can't overwrite the real result. This is the same idea as fencing for distributed locks in Module 21.
• Kill on lease loss: when a heartbeat says lease_lost, the worker stops the container. Together with fencing, this keeps the promise "never twice at the same time".
• Give the job its identity: pass run_id and attempt to the customer's code as environment variables, so it can make side effects idempotent (for example, "write report for run 7731" instead of "append a report").

9.4 Retries, timeouts and overlap never lost · fair

• Backoff with jitter: attempt n waits about base × 2ⁿ plus a random amount, so a downstream outage doesn't cause synchronised retry storms (Module 24).
• Classify errors: a non-zero exit or a timeout is retryable; "image not found" or "permission denied" is not. Don't waste three attempts on a typo.
• Dead letter: after the last attempt, mark the run failed, notify the tenant (webhook or email), and keep it visible with its attempts.
• Timeouts: each job has a maximum duration. The worker kills the container at the limit and reports a timeout.
• Overlap policy: when a run falls due while the previous run of the same job is active, apply skip (record "skipped"), queue (start when the previous finishes) or allow. Kubernetes CronJobs offer a close cousin: Allow, Forbid (skip the new run) and Replace (cancel the running one and start the new one).

9.5 Fairness and isolation between tenants fair and isolated

Isolation of code matters as much as fairness of time. Run each job in a sandbox with CPU, memory, disk and network limits: containers for trusted internal jobs, lightweight virtual machines (Firecracker, which backs AWS Lambda, is one example) for untrusted customer code. Give each run short-lived credentials scoped to that tenant.

9.6 Time zones, daylight saving and missed runs on time · never lost

• Store the rule, not just the instant: keep the cron text and an IANA time zone ("Asia/Kolkata", "America/New_York"). Compute next_run_at in that zone, then store it in UTC.
• Daylight saving: in New York, 02:30 doesn't exist on the spring-forward day and 01:30 happens twice in autumn. Pick and document a rule: run a skipped time at the next valid instant, and run a repeated time once (on its first occurrence).
• Missed runs: if the service was down from 09:00 to 09:20, what about the 09:00, 09:05 and 09:10 runs of a five-minute job? Let the job choose a misfire policy: run once now, run every missed one, or skip to the next. Kubernetes has a similar knob, startingDeadlineSeconds: "too late to bother, wait for the next one".
• Clocks: the database's clock decides what is due, not each scheduler's clock; servers run NTP. Small skew only shifts a run by milliseconds.

9.7 The whole design after the deep dives

This is the picture to leave on the whiteboard. Follow one 09:00 run through every part: sharded schedulers holding leases, a look-ahead delay queue, fair dispatch, leased and fenced workers, a reaper, and capacity planned from the schedule itself.

10. Interviewer follow-ups

These are the "what if" questions you should expect after the main design. Answer each in under a minute: name the problem, give the mechanism, state the cost.

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• At-least-once with fencing, instead of pretending to offer exactly-once.
• A relational database for jobs and runs (transactions, unique keys), and a separate fast delay queue for timing.
• Correctness on the claim, availability on the API.
• Spreading flexible jobs, at the cost of "exactly :00" for those who don't need it.

What we'd watch:

• Dispatch lag p99 (start time minus scheduled time) per region and tenant: the headline target.
• Runs waiting for a worker.
• Lease expiries per minute (dying workers).
• Failure and retry rates per tenant.
• Dead-letter count.

At 10×:

• More shards everywhere.
• Forecast-driven pre-scaling.
• Spreading on by default.
• Regional ownership of jobs.

Common mistakes:

• Designing for the average (230/s) instead of the peak (200,000 at :00).
• Saying "exactly once" without explaining how.
• Using a "running" flag with no lease, so dead workers leave runs stuck.
• Locks without fencing.
• Ignoring time zones and daylight saving.
• Turning the scheduler into a workflow engine nobody asked for.

13. Try it, check yourself, read more

Top-of-the-hour lab. A burst of jobs falls due at 09:00:00. Choose how runs reach the workers, how many worker slots are ready, and whether flexible jobs are spread across the minute. Each job runs for 20 s. The red area is runs waiting; teal bars are runs starting each second.