Design an online code judge

A coding-practice site lets anyone paste a program, runs it against hidden tests, and says "Accepted" or why not, in seconds. Every submission is code written by a stranger, executed on your machines. And once a week a contest makes a hundred thousand people submit at the same moment.

1. The prompt

"Design an online judge: users solve programming problems in the browser, submit code in many languages, and get a verdict from hidden test cases. Weekly contests have a live leaderboard."

What makes it hard:

• Running untrusted code. Some submissions will try to read other people's data, reach the network, or crash the machine.
• Fair timing. "Time limit exceeded" must mean the code was slow, not that a neighbour was busy.
• Peaks. Contests multiply load several times over, mostly in the final minutes.

2. Frame the problem

3. What it must do

1. A user can submit code and get a verdict (accepted, wrong answer, time or memory limit, runtime or compile error) with how many tests passed.
2. A user can run code against sample or custom input and see the output quickly.
3. Contests rank participants live by problems solved and time taken.

Not now: discussion forums and editorials (ordinary content), payments, mock interviews with video, AI hints.

4. How well it must do it

5. Numbers that change the design

Skip estimating problem-page traffic: static content behind a CDN.

6. The nouns

• Problem, with limits and a versioned test set.
• Submission: code, language, status and verdict.
• Language runtime: a sandbox image per language.
• Contest and participant score.

problems     problem_id, title, time_limit_ms, memory_mb, test_set_version
test_sets    problem_id, version, object_key (blob), checker (exact | float | special program)
submissions  submission_id, user_id, problem_id, contest_id?, language, code_key (blob), test_set_version,
             status (queued | running | done), verdict, tests_passed, runtime_ms, memory_kb, created_at
scores       contest_id, user_id, solved, penalty_minutes, last_accepted_at    (source of truth; leaderboard derived)

7. The contract

POST /v1/problems/{id}/submissions  {language, code}   Idempotency-Key: …
     → 202 {submission_id, queue_position}
GET  /v1/submissions/{id}           → {status, verdict, tests_passed, runtime_ms, memory_kb}
SSE  /v1/submissions/{id}/events    → queued … running test 37/100 … done
POST /v1/problems/{id}/run  {language, code, input}   → {stdout, stderr, runtime_ms}   (fast lane)
GET  /v1/contests/{id}/leaderboard?page_token=…       → [{rank, user, solved, penalty}]

Data flow: submit → save code and a queued row → queue by lane → a judge worker takes it → sandbox: compile, run each test with limits, compare output → verdict → update the row → notify the browser → contest scorer updates the leaderboard.

8. First working version

An API, a database and blob storage, one queue, and a pool of judge workers that run code inside containers.

8.1 Submitting and judging

The API stores the code, creates the row, and enqueues only the submission ID (Module 23). A worker loads the code and the test set, compiles, runs each test with time and memory limits, compares the output with the expected answer (or a special checker for problems with many correct answers), and stops at the first failure.

8.2 Getting the verdict

The browser listens on a server-sent events stream (Module 19) for progress ("running test 37 of 100"), falling back to polling. Verdicts are written once per submission and test-set version, so a retry never produces a second, different verdict.

8.3 Contest scores

When a contest submission is accepted, the scorer updates the participant's row (problems solved, penalty minutes) in one transaction, then the leaderboard.

9. Make it hold up

9.1 Running untrusted code safely contained

Also: never show the output of hidden tests, cap stdout size, and limit error messages so failing submissions can't print the test data out.

9.2 Fair, repeatable timing fair

• Measure CPU time of the program, not wall-clock time.
• Run one sandbox per dedicated core on identical machine types, so neighbours don't steal cache and memory bandwidth.
• Per-language time multipliers (a Python limit is several times C++'s), published with the problem.
• If a run lands just over the limit, rerun it before declaring "time limit exceeded".

9.3 Surviving the contest surge verdict under 15 s

9.4 Test data and runtimes on every worker verdict time

• Test sets are immutable, versioned objects in blob storage; workers cache them on local disk by version and pre-load the contest's problems before it starts.
• Language images are pre-pulled onto every worker; a new language version is a new image, rolled out gradually and rejudged against a sample of past submissions first.
• Each submission records the test-set version it was judged against, so every verdict can be reproduced.

9.5 A live leaderboard for 100,000 people leaderboard within seconds

• Rank by problems solved (more is better), then penalty time (less is better). Encode both into one sortable score and keep it in a sorted set (Module 09; Design D20): updates and "my rank" are logarithmic.
• The scores table stays the source of truth; the sorted set can be rebuilt from it after a rejudge or a cache failure.
• Serve the top pages from a cache refreshed every few seconds; many contests freeze the public board in the final hour.

9.6 Rejudges and cheating correct

Rejudge after fixing a test: create a new test-set version, re-queue affected submissions on a low-priority lane, recompute scores from verdicts, rebuild the leaderboard. Plagiarism is checked after the contest by fingerprinting normalised code (stripping names and whitespace) and comparing submissions per problem; matches go to human review.

9.7 The whole design after the deep dives

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• Strong isolation (microVMs) at some cost in overhead, offset by warm pools.
• Dedicated cores per run for fairness, at lower density.
• Pre-scaling for known peaks rather than trusting autoscaling.
• Separate lanes for contests, practice and "Run".
• A derived leaderboard, rebuilt from the scores table when needed.

What we'd watch: verdict latency by lane · queue depth and oldest age · sandbox start time · runtime variance on reference solutions · blocked-syscall alerts · leaderboard age.

At 10×: more pre-scaled capacity and regional fleets; the API, database and leaderboard barely change.

Common mistakes:

• Running user code in the API server, or as a plain process.
• Trusting a container alone as the security boundary.
• Wall-clock time limits on shared machines.
• "We'll autoscale" for a peak whose start time you know.
• A leaderboard with no source of truth behind it.
• Printing hidden test data in error messages.

13. Try it, check yourself, read more

Contest-day simulator. A 90-minute contest: 60 submissions a second, surging to 300 a second in the last 15 minutes, on top of 25 a second of practice. Each submission needs 20 CPU-seconds, spread over parallel test runs (about 3 s of wall time when there's no queue). Pick how the fleet behaves and watch the wait for a verdict (red) and the fleet size (dashed).

Model: autoscaling adds up to 15% capacity a minute after a 3-minute delay; a cold sandbox adds 3 CPU-seconds and 2 s of start-up to each submission; with separate lanes, contest submissions go first and practice uses the spare capacity. Illustrative numbers.