Design an API gateway
Design the single front door for a company's APIs: every request from the web app, the mobile apps and partner integrations passes through it on the way to about 300 internal services. It must check who's calling, enforce limits, send each request to the right place, and do all of that in a few milliseconds without ever becoming the reason the site is down.
1. The prompt
"Our company runs a few hundred microservices. Design the API gateway that all external traffic goes through."
What makes it hard:
- It sits in front of everything. If the gateway is down, every product is down, so it must be more available than any service behind it.
- Every millisecond it adds is paid by every request. Whatever it does (authentication, limits, logging) must happen without slow calls on the request path.
- Hundreds of teams change it. Routes and policies change many times a day, and one bad change must not break all 300 services at once.
2. Frame the problem
How much traffic?
About 1.5 million requests a second at the daily peak, across three regions.
This sets the size of the gateway fleet, and rules out any shared component that every request must call.
Who calls it?
Our web and mobile apps (users logged in with tokens), and about 20,000 partner integrations using API keys, each on a plan with a quota.
Two kinds of identity, and per-partner limits that are part of a contract.
What must it do, and what not?
TLS, routing, authentication, rate limits and quotas, request IDs and tracing, access logs. No business logic.
The gateway handles concerns every service shares, so teams don't each rebuild them. Business rules in the gateway would make it a bottleneck for every team.
Buy or build?
Assume we build on an open-source proxy (Envoy, NGINX or similar) and design the parts around it: configuration, limits, rollout.
Nobody writes an HTTP proxy from scratch in an interview; the interesting design is the control plane and the policies.
How much latency may it add?
At most 5 ms at p99, not counting the service's own time.
A page that makes 10 sequential API calls would otherwise pay that 10 times.
How often does configuration change?
Hundreds of route or policy changes a day, made by service teams themselves.
Changes must be self-service, validated and rolled out in stages.
3. What it must do
- Send each request to the right service (and the right version of it), based on host, path, method and headers.
- Check who is calling (a user token or a partner API key), reject what isn't allowed, and enforce rate limits and quotas per caller.
- Let service teams change their routes and policies themselves, applied everywhere within seconds.
- Record every request: a request ID, a trace, metrics and an access log.
Not now: a developer portal and partner billing, GraphQL federation, bot and DDoS protection (a CDN or edge security layer sits in front of us), service-to-service traffic inside the company (a service mesh's job, see 9.5).
4. How well it must do it
| Target | Why this number |
|---|---|
| Added latency ≤ 5 ms at p99 speed | Paid by every request, often several times per page. |
| Availability 99.99% or better surviving failures | Higher than any single service, because it's in front of all of them. |
| Config changes live everywhere within 30 s; a bad change affects a small slice before it's stopped safe change | Teams deploy all day; blast radius matters more than speed. |
| Fairness: one partner or one slow service can't starve the others isolation | Shared infrastructure is only safe if neighbours can't hurt each other. |
5. Numbers that change the design
| Quantity | Working | → so we… |
|---|---|---|
| Fleet size | 1.5 M req/s ÷ ~20,000 req/s per node (TLS, token checks, proxying) ≈ 75 nodes. Each region must survive another region failing, so size each for half the global peak: 750,000 ÷ 20,000 ≈ 38 nodes, plus headroom | → so about 45 stateless nodes per region, ~135 in all, behind layer-4 load balancers |
| Calls on the request path | A 1 ms network call to a shared store at 1.5 M req/s is 1.5 M calls a second to that store, and 1 ms on every request | → so tokens are verified locally, API keys and routes are cached in memory, and rate-limit counters are local with background syncing |
| Route table | 300 services × ~30 routes ≈ 9,000 routes | Small → so the whole table lives in every node's memory, compiled into a prefix tree |
| Access logs | 1.5 M req/s × ~1 KB ≈ 1.5 GB/s ≈ 130 TB a day | → so logs are buffered and shipped asynchronously to Kafka, kept hot for 7 days, sampled for long-term storage |
Illustrative numbers. A real fleet is sized by load-testing one node with your actual mix of TLS, payload sizes and policies.
6. The nouns
- Route: which requests match (host, path pattern, method, headers) and where they go.
- Upstream: a service's pool of endpoints, with health and load-balancing settings.
- Policy: what to apply on a route: authentication, rate limit, timeout, retries, header changes.
- Consumer: a caller (a partner app with API keys, or our own apps), with a plan and quota.
- Config version: a numbered, immutable snapshot of all of the above, which nodes load as a whole.
route id, match {host, path_prefix, methods[], headers{}}, upstream_id, policy_ids[], owner_team
upstream id, service, discovery (DNS or registry name), lb (least_request), timeout_ms, retry_budget
policy id, kind (jwt | api_key | rate_limit | cors | transform), settings{}
consumer id, plan (free | pro | enterprise), api_key_hashes[], quota_per_day, rate_per_sec
config_version n, created_by, created_at, routes+upstreams+policies (whole snapshot), status (canary | live | rolled_back)
7. The contract
Data plane (the internet sees this)
Any HTTPS request to api.example.com, app.example.com
Errors in one shape (RFC 9457 problem+json):
401 no or bad credentials · 403 not allowed · 404 no route
429 over the limit, with Retry-After · 503 upstream unavailable · 504 upstream too slow
Adds: X-Request-Id, traceparent, X-Caller (set by the gateway; the same header from outside is removed)
Control plane (service teams and CI use this)
PUT /v1/routes/{id} {match, upstream_id, policy_ids} → validated, queued for rollout
POST /v1/consumers {plan} → consumer id, API key (shown once)
GET /v1/config/versions/{n} → status: canary | live | rolled_back
POST /v1/config/versions/{n}:rollbackServices trust the gateway's X-Caller header only because the gateway strips any copy that arrives from outside, and services accept traffic only from the gateway (or verify a signed internal token).
8. First working version
Two halves, kept apart on purpose. The data plane is a fleet of identical, stateless gateway nodes that handle requests. The control plane stores configuration, validates changes and sends them to the nodes. The data plane must keep working when the control plane is down.
8.1 One request through the gateway
8.2 A team changes a route
Nodes receive whole snapshots, not individual edits, so every node is always on one complete, tested version. Envoy's xDS APIs work this way: the control plane streams configuration to each proxy, and the proxy acknowledges or rejects each version.
9. Make it hold up
9.1 Five milliseconds for everything speed
| Approach | Why | |
|---|---|---|
| ✗ Bad | Call the identity service to check each token, and a database to look up each API key | Two network calls per request: milliseconds each at p99, and the identity service now needs to take 1.5 M calls a second. |
| ~ Good | Cache token checks and key lookups for a minute | Faster on a hit, but every cache miss is still a remote call, and revoked keys keep working for up to a minute. |
| ✓ Great | Nothing remote on the request path: verify token signatures locally with the identity provider's public keys (refreshed in the background); push hashed API keys to every node with the config; compile routes into a prefix tree; reuse warm connections to upstreams; write logs to a buffer | The common path is pure CPU and memory. Revocations arrive as a pushed update within seconds instead of waiting for a cache to expire. |
Try the trade-offs in the latency lab in section 13.
9.2 Rate limits across 135 nodes fairness
A partner on the Pro plan may make 1,000 requests a second. Their requests are spread over every node in a region, so no single node sees the whole picture.
| Approach | Why | |
|---|---|---|
| ✗ Bad | Each node allows 1,000 ÷ 45 ≈ 22 a second | No shared state, but wrong whenever traffic is uneven: a partner with few connections lands on few nodes and gets a fraction of what they pay for. |
| ~ Good | A shared Redis counter, incremented on every request | Exact, but adds a network hop to every request, and Redis becomes something every request depends on. |
| ✓ Great | Local counters, synced in the background: each node counts locally and every ~100 ms adds its count to a shared total in Redis and reads back the global figure | No hop on the request path. The limit can be overshot by about one sync interval's traffic, which is fine for rate limits. Daily quotas (part of a contract) are checked against the shared total. |
If Redis is unreachable, nodes fall back to the local share (fail open, with a cap) rather than rejecting everyone. The algorithms themselves (token bucket, sliding window) are in D02.
9.3 A bad change must not take everything down safe change
The gateway's configuration is code that runs on every request for every service. A route with a greedy path pattern, or a policy with a typo, can send all traffic to the wrong place in seconds. Large outages have started exactly this way.
9.4 One slow service isolation
Also per upstream: a timeout shorter than the client's, retry budgets (retries may add at most 10% extra load, so retries can't multiply an outage), circuit breakers and outlier ejection (stop sending to an endpoint that keeps failing). And when the gateway itself is overloaded, it sheds the least important traffic first (a partner's bulk export before a checkout). Module 24 has the patterns.
9.5 Gateway, service mesh or BFF? architecture
| Edge API gateway | Service mesh | Backend for frontend (BFF) | |
|---|---|---|---|
| Traffic | From outside into the company | Service to service, inside | From one client type (the iOS app, the web app) |
| Does | TLS, external auth, partner keys and quotas, routing, request IDs | mTLS between services, retries, traffic splitting, telemetry, via a proxy next to each service | Shapes and combines API calls for one client's screens |
| Owned by | A platform team | A platform team | The client's team |
| Keep out | Business logic, data joins | Business logic | Cross-cutting policy (auth, limits) |
They are layers, not rivals: a request may pass the edge gateway, then a BFF, then mesh proxies between services. Combining many service calls into one response (aggregation) belongs in a BFF, not in the shared gateway.
9.6 Regions and the control plane surviving failures
- Each region is independent. GeoDNS or anycast sends users to the nearest healthy region; each region has its own gateway fleet and its own copy of the configuration.
- The data plane doesn't depend on the control plane. Nodes keep serving with the last good configuration if the control plane is unreachable, and a restarted node loads the last snapshot from local disk or object storage. This is called static stability: a failure stops changes, not traffic.
- Configuration rolls out region by region, so a bad version never reaches all regions at once.
9.7 The whole design after the deep dives
10. Interviewer follow-ups
A partner's API key leaks on a public code site.
The partner (or our scanner) revokes it through the control plane; the revocation is pushed to every node as a small update within seconds, without waiting for a full config rollout. Keys are stored only as hashes, and the partner can hold two keys at once so they can rotate without downtime.
How would you route 5% of traffic to a new version of a service?
A weighted route: 95% to the stable upstream, 5% to the canary, sticky by user ID so a user doesn't flip between versions. Promote or roll back by changing the weights.
WebSockets and long-running streams?
The gateway upgrades and proxies them, but they pin a connection to a node for hours. Drain nodes slowly during deploys (stop accepting new connections, let old ones finish or ask clients to reconnect), and count connections, not just requests, when sizing.
Why not let each service do its own auth and limits?
300 teams would implement them 300 slightly different ways, and partners would see 300 behaviours. Services still check authorisation for their own data (is this user allowed to see this order?); the gateway does authentication and coarse access.
Large file uploads?
Don't stream gigabytes through the gateway. Return a pre-signed URL so the client uploads directly to object storage; the gateway only handles the small request that creates it.
The gateway team becomes a bottleneck for every change.
Self-service: teams own their routes in their own repositories; CI validates them against the schema and shared rules (no catch-all paths, timeouts within limits) and submits them; the platform team reviews only changes to shared policy.
11. When things break
| Failure | What users see | How we detect it | What limits the damage |
|---|---|---|---|
| Bad config version | Errors on some routes | Error rate on canary nodes vs the rest | Staged rollout, automatic rollback, per-route validation |
| Control plane down | Nothing (changes are paused) | Config age per node | Nodes keep the last good version (static stability) |
| Rate-limit store down | Limits slightly looser | Sync errors | Fall back to local shares, with a cap |
| One upstream very slow | That feature fails fast; others fine | Latency and open connections per upstream | Per-upstream limits, timeouts, circuit breakers |
| A region lost | Brief errors, then slower for far users | Health checks from outside | DNS or anycast moves traffic; each region sized for half the global peak |
| Identity provider's keys rotate | Valid tokens rejected | Spike in 401s | Fetch new public keys on an unknown key ID; publish new keys before signing with them |
12. Wrap-up and the clock
Trade-offs we made:
- A stateless data plane and a separate control plane; traffic keeps flowing when the control plane fails.
- Nothing remote on the request path: local token checks, pushed keys, synced counters.
- Rate limits that can overshoot slightly, in exchange for no extra hop.
- Configuration as versioned snapshots, rolled out in stages with automatic rollback.
- Per-upstream isolation, so one service's trouble stays its own.
What we'd watch: added latency (p50, p99) · error rates by route and by cause (gateway vs upstream) · 429s per consumer · config version and age per node · open connections per upstream · rate-limit sync lag.
At 10×: more nodes (they're stateless); shard the rate-limit store by consumer; split the fleet by traffic class (partner API vs first-party apps) so each can be tuned and fail separately.
A 60-minute interview
| 0–5 | Frame: traffic, callers, latency budget, buy vs build |
| 5–10 | Must do, how well, numbers |
| 10–15 | Nouns and contract, data plane vs control plane |
| 15–25 | First version: the request path and a config change |
| 25–50 | Deep dives: latency (9.1), rate limits (9.2), safe rollout (9.3) |
| 50–60 | Isolation or regions, follow-ups, wrap-up |
A 90-minute interview
| 0–8 | Frame, including partner plans and quotas |
| 8–15 | Must do, how well, numbers |
| 15–22 | Nouns, contract, the two planes |
| 22–35 | First working version |
| 35–75 | Deep dives 9.1–9.6 and the final picture |
| 75–85 | Follow-ups: leaked keys, canaries, uploads |
| 85–90 | Trade-offs, metrics, 10× |
| Level | What a strong answer shows |
|---|---|
| Mid-level | A proxy fleet behind a load balancer, routing by path, token checks, a rate limiter, logging. |
| Senior | Data plane vs control plane, nothing remote on the request path, distributed rate limits with their error, per-upstream timeouts and circuit breakers, where a BFF and a mesh fit. |
| Staff+ | Configuration as the main outage risk (staged rollout, validation, static stability), self-service for hundreds of teams, traffic classes and load shedding, the cost of every millisecond across the company. |
Common mistakes:
- Putting business logic or data joins in the shared gateway.
- A network call to an auth service or Redis on every request, without saying what it costs.
- One global configuration push with no canary.
- A single shared connection pool, so one slow service takes the gateway down.
- Forgetting to strip identity headers that arrive from outside.
- Making the control plane a dependency of serving traffic.
13. Try it, check yourself, read more
The latency lab. Choose how the gateway does each job and see what it adds to every request. The line marks the 5 ms p99 budget.
Illustrative numbers for one request inside a data centre. Adding up p99s overstates the true p99 a little; treat the total as a cautious estimate.
How should the gateway check a user's access token on each request?
The control plane goes down. What happens to traffic?
One service becomes very slow. How do you stop it taking the gateway down?
The mobile app needs one call that combines a user's profile, orders and recommendations. Where does that go?
| Resource | Why |
|---|---|
| Envoy: dynamic configuration (xDS) | How a control plane streams versioned configuration to proxies. |
| Envoy: global rate limiting | Local and global limits, and when to use each. |
| Static stability (Amazon Builders' Library) | Why the data plane must keep working when the control plane can't. |
| Handling overload (Google SRE book) | Per-client limits, criticality and load shedding. |
| RFC 9457: Problem Details for HTTP APIs | One error shape for every route. |