Design an ad serving system

Design the system that picks which ad to show when a page or app has a slot to fill: find the campaigns allowed to show to this person here, predict how likely each one is to be clicked, run an auction, respect every advertiser's budget and frequency limits, and answer in tens of milliseconds, half a million times a second. Counting the clicks afterwards is D17; this is the decision before.

1. The prompt

"Design the ad serving system for a large app: advertisers create campaigns with targeting and budgets; when a user opens a page with an ad slot, choose the ad."

What makes it hard:

• A hard deadline. The page won't wait: if we haven't answered in about 100 ms, the slot is filled with something else and the money is lost.
• Money moves on every decision. Budgets must not be overspent by much, and advertisers are charged for what they were promised, no more.
• The best ad depends on a prediction. Who pays most per click doesn't matter if nobody clicks; ranking needs a model running inside the deadline.

2. Frame the problem

3. What it must do

1. Choose an ad for a slot: only eligible ads (targeting, budget, frequency cap, policy), ranked by expected value, chosen by auction.
2. Charge correctly: decide the price the winner pays, and record impressions and clicks for billing.
3. Pace budgets so campaigns spend through the whole day, never far beyond their budget.
4. Apply campaign changes (new ads, paused campaigns, new bids) within a minute.

Not now: the advertiser UI and reporting, creative review, external exchanges (real-time bidding with other networks), click fraud detection (part of D17).

4. How well it must do it

5. Numbers that change the design

Illustrative numbers.

6. The nouns

• Campaign: budget, schedule, targeting, bid strategy. Ad: the creative (image, text, link) inside a campaign.
• Ad request: a slot, a user (or an anonymous context), a page, a time.
• Candidate: an eligible ad for this request, with its bid and predicted click-through rate (pCTR).
• Impression and click: billable events, each tied to the auction that produced it.

campaign       id, advertiser_id, daily_budget, schedule, targeting {age, country, interests[], placements[]}, bid {type: cpc|cpm, max}
ad             id, campaign_id, creative_url, status
targeting idx  attribute value → posting list of ad ids        (in memory on every ad server)
spend          campaign_id → spent_today, allocation per ad server (pacing service)
freq           (user_id) → {campaign_id: shows_today}            (key-value store, TTL 1 day)
auction log    auction_id, request, candidates, scores, winner, price   (stream → data lake)

7. The contract

Internal: the app's backend calls
  POST /v1/ads:select   {slot_id, user_id, context {page, country, device}, count: 1}
    → {ad_id, creative, auction_id, impression_url, click_url}   or 204 (no ad: show house content)

Tracking (from the client)
  GET  /i/{auction_id}   impression rendered     → recorded once (deduplicated by auction_id)
  GET  /c/{auction_id}   click                   → redirect to the advertiser's link

Advertiser API (separate system)  campaigns, ads, budgets, bids → change events

The auction_id ties every impression and click back to the decision, the price and the model version, which is what billing, reporting and training all need.

8. First working version

A fleet of stateless ad servers, each holding the whole targeting index in memory, calling a model-serving fleet for predictions and a key-value store for frequency counts. A pacing service hands out spend. Every decision is logged to a stream.

8.1 One ad request, step by step

8.2 The auction

For each candidate, eCPM = bid per click × pCTR × 1,000 (for CPM bids, the bid itself). The highest eCPM wins. A CPC bid of £2.00 with a 1% predicted click rate is worth £20 per thousand impressions; a £0.50 bid with 5% is worth £25 and wins.

What the winner pays depends on the auction type. In a second-price auction, they pay just enough to have beaten the runner-up, which makes bidding your true value the best strategy. In a first-price auction they pay their own bid, and advertisers' tools then shade bids down. Many display exchanges moved to first-price around 2019; a single-owner network like ours can choose either, and second-price (with a quality-adjusted rank) is the classic choice for self-serve CPC ads. Try both in the lab in section 13.

9. Make it hold up

9.1 Retrieval: 5 million ads to 1,000 speed

9.2 The prediction budget speed · quality

9.3 Budget pacing money

Pacing has two parts. A plan: how much the campaign should have spent by now, following the day's traffic curve. A throttle: if spend is ahead of plan, the campaign takes part in fewer auctions (a participation probability), or bids lower; if behind, more. The pacing service recomputes each campaign's throttle every few seconds from reported spend and pushes it to the ad servers.

Overspend comes from delay: impressions already decided but not yet reported. Each ad server spends from a small local allocation of the remaining budget and asks for more as it runs low, so the total in flight is bounded. Near the end of a budget, allocations shrink. Clicks can arrive minutes after the impression, so CPC campaigns keep a reserve for clicks still to come.

9.4 Frequency caps speed · correctness

• One key-value read per request returns the user's counts for all campaigns (a single small document, not one read per ad).
• Increment on the impression (the ad was actually shown), not on the decision; many decisions never render.
• Caps are soft: a few milliseconds of race between two requests from one user can exceed a cap by one. That's acceptable; a strict cap would need a lock per user per request.
• If the store is slow, skip capped campaigns (fail closed) rather than wait past the deadline.

9.5 Campaign changes in under a minute freshness

The advertiser system publishes change events to a stream. Every ad server applies them to its in-memory index (adding and removing ad ids from posting lists) and, every hour or so, loads a full fresh snapshot to correct any drift. A pause is the most urgent change: it also goes straight to the pacing service, which sets the campaign's throttle to zero, a much smaller and faster update than an index change.

9.6 Counting money correctly money

• Every impression and click carries its auction_id; billing counts each at most once (deduplicate by id), and only those that pass fraud filtering (D17).
• The auction log records the price at decision time, so billing doesn't depend on recomputing an auction.
• Real-time spend (for pacing) and billed spend (for invoices) are separate: pacing is fast and approximate; billing is exact, from the deduplicated log, a little later.

9.7 The whole design after the deep dives

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• A funnel: index retrieval, a light model, then a heavy model on a small set, each stage with a deadline.
• eCPM ranking and a second-price auction for self-serve CPC ads.
• Pacing with local allocations: approximate in real time, bounded overspend, exact billing later.
• Soft frequency caps; fail closed when unsure.
• Every decision logged with an auction id, for billing, reporting and training.

What we'd watch: latency by stage · fill rate (requests that got an ad) · revenue per thousand requests · overspend per campaign · time from pause to zero spend · model calibration (predicted vs actual click rate).

At 10×: stateless parts scale out; the model fleet dominates cost, so candidate counts and model size are tuned against revenue.

Common mistakes:

• Ranking by bid alone.
• Scoring every ad with the full model.
• A central spend counter updated on every impression.
• Checking budgets only once a day, or not at all on the request path.
• Counting frequency on the decision instead of the impression.
• Waiting for a slow dependency past the deadline.

13. Try it, check yourself, read more

The auction lab. Three advertisers want a slot in the feed. Change their bids and see who wins and what they pay.

Predicted click rates are fixed for this user and slot: garden shop 1.0%, bike brand 5.0%, bank 0.6%. Second price: the winner pays the lowest bid that would still have won, plus £0.01.