Design a serverless functions platform

Design a platform like AWS Lambda or Google Cloud Run functions: developers upload a function, and it runs whenever an HTTP request or an event arrives, from zero copies to thousands in seconds, billed by the millisecond. The platform runs code from strangers side by side on shared machines, so it must start sandboxes fast, keep tenants strictly apart, and pack them tightly enough to make money.

1. The prompt

"Design a serverless compute platform: customers deploy functions that run on requests or events, scale automatically, and pay only while their code runs."

What makes it hard:

• Cold starts. Scaling to zero means the first request after a quiet spell must wait for a sandbox to be created, code downloaded and the runtime started.
• Untrusted neighbours. Thousands of customers' code runs on the same hardware; a bug or an attacker in one must never reach another.
• Bursty, tiny units of work. Millions of functions, most idle most of the time, a few suddenly needing 1,000 copies.

2. Frame the problem

3. What it must do

1. Deploy a function version (code, runtime, memory, timeout, environment).
2. Invoke it synchronously or from events, running each invocation in an isolated sandbox.
3. Scale automatically from zero to the account's limit, and back to zero.
4. Meter and report: logs, metrics, and billing per millisecond of run time.

Not now: long-running services (that's a container platform), GPUs, a function marketplace, cross-region failover of individual invocations.

4. How well it must do it

5. Numbers that change the design

Illustrative numbers. Firecracker figures are from its project page (checked October 2026).

6. The nouns

• Function version: immutable code + configuration. An alias (prod) points at a version.
• Sandbox: a microVM running one function version for one account; idle, busy, or being created.
• Host: a machine running hundreds of sandboxes, with a local agent and a code cache.
• Invocation: one run, synchronous or from an event.
• Account limits: maximum concurrency, burst rate, reserved concurrency per function.

function_version  (account, name, version) → runtime, memory_mb, timeout_s, code_digest, env (encrypted)
sandbox           id → host, function_version, state (creating | idle | busy), idle_since       (in the worker manager's memory)
host              id → zone, free_memory, cached code digests, healthy
invocation        id → function_version, sandbox, start, duration_ms, status       (stream → logs, billing)
event_queue       per async function: durable queue, attempts, dead-letter destination

7. The contract

PUT  /v1/functions/{name}              {runtime, memory_mb, timeout_s, code (zip or image digest)} → version
POST /v1/functions/{name}/aliases/prod  {version}
POST /v1/functions/{name}:invoke        body → result     (sync; 429 when over the concurrency limit)
POST /v1/functions/{name}:invoke-async  body → 202        (queued; retried twice, then dead-letter)
Triggers: HTTP routes, queue and stream subscriptions, schedules
Inside the sandbox: the runtime fetches the next event from a local runtime API, runs the handler, posts the result

8. First working version

A front end authenticates calls and enforces limits. A worker manager knows, for each function version, which sandboxes exist and which are idle. A placement service chooses a host for a new sandbox. On each host, an agent creates microVMs and caches code.

8.1 Warm and cold invocations

8.2 An event-triggered function

9. Make it hold up

9.1 Making cold starts rare and short speed · cost

The customer controls the rest: a small package, few dependencies, and work done lazily rather than at start-up. The lab in section 13 shows how the keep-alive time trades cold starts against idle memory.

9.2 Strangers on one machine security

9.3 Packing hosts cost · reliability

• Memory is the unit. CPU share and network bandwidth are given in proportion to memory, so packing by memory also bounds the rest.
• Spread each function across many hosts and zones: a host failure then costs a function a few of its sandboxes, not all of them.
• Mix workloads. Functions from different customers peak at different times; packing a host with one customer's correlated functions would make it hot when that customer spikes.
• Prefer hosts with the code cached, so placement saves the fetch.
• Overcommit carefully: idle sandboxes can be squeezed (memory reclaimed), busy ones can't.

9.4 Limits and bursts fairness

Each account has a concurrency limit (say 1,000 sandboxes) and a burst rate (how fast new sandboxes may be added). Limits protect the platform from one customer's runaway loop, and protect the customer from a surprise bill. A function can have reserved concurrency: guaranteed capacity, and also a cap, which is how a customer stops a function from overwhelming their database. Over the limit, synchronous calls get 429; asynchronous events wait in their queue and are retried.

9.5 The database behind the functions reliability

A traffic spike scales the function to 1,000 sandboxes, each opening its own database connection: the platform scaled perfectly and the customer's database fell over. Recommend a connection proxy or pooler in front of the database, reserved concurrency as a cap, and short-lived connections or an HTTP data API. Functions are stateless; anything that must survive goes to a database, object storage or a queue, never to the sandbox's disk.

9.6 The whole design after the deep dives

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• MicroVMs for isolation, accepting more complexity than containers.
• Reuse idle sandboxes for minutes, trading memory for fewer cold starts; provisioned concurrency as a paid guarantee.
• One request per sandbox: simple for customers, more sandboxes for us.
• Pack hosts by memory, spread each function widely.
• Limits per account and per function, to protect the platform and the customer's own systems.

What we'd watch: cold start rate and duration (platform part) · warm overhead · host memory utilisation · sandbox creation rate per host · throttles per account · async queue age and dead-letter counts.

At 10×: more hosts and worker manager partitions. The cost lever is utilisation: better snapshots and faster creation let idle sandboxes be reclaimed sooner.

Common mistakes:

• Plain containers on a shared kernel for untrusted tenants, without saying why that's acceptable.
• Ignoring cold starts, or "keep everything warm".
• Forgetting that concurrency = rate × duration.
• No limits, so one customer's loop takes the fleet.
• Losing async events on a host failure.

13. Try it, check yourself, read more

The keep-alive lab. A function gets requests at random. Choose how often, how long each runs and how long the platform keeps an idle sandbox, and see the cold start rate and the memory held idle.

A simple model: requests arrive at random (Poisson); a request is cold if no sandbox has been used within the keep-alive time. At high rates, cold starts come from scaling up, which this model ignores. 512 MB sandboxes.