Design a news aggregator

Collect articles from thousands of publishers and show each reader a fast, endless, up-to-date feed for their region. It has two halves with opposite characters: a background ingestion pipeline that's write-light and can take minutes, and a feed that's read-heavy, must load in under 200 ms, and must survive the whole world refreshing at once when big news breaks. It follows the interview framework phase by phase.

1. The prompt

"Design Google News: aggregate articles from publishers worldwide into a feed per region; clicking an article opens it on the publisher's site."

Ingestion is small: about 12 articles a second. The feed is the challenge: thousands of reads a second normally, a million a second during breaking news, with fresh results. The design separates the two halves and makes the read side almost entirely cached.

2. Frame the problem

3. What it must do

1. The system ingests new articles from publishers continuously.
2. A reader sees a feed of recent articles for their region (title, summary, thumbnail, publisher), with infinite scroll.
3. Clicking an article opens the publisher's page (and we count the click).

Not now:

• Deep personalisation: a light version appears in 9.6.
• Saving and sharing.
• Comments.
• Clustering duplicate stories: mentioned in follow-ups.

4. How well it must do it

5. Numbers that change the design

6. The nouns

• Publisher: with its feed URL and polling tier.
• Article.
• Regional feed: the precomputed list per region.
• Thumbnail.

publishers  publisher_id, name, feed_url, region, tier (fast | normal | slow), etag, last_modified, next_poll_at
articles    article_id (time-ordered), publisher_id, url (unique), title, summary, thumbnail_key, region, category, published_at
feed:{region}  sorted set of article_id (latest 2,000), with category in each entry

7. The contract

GET /v1/feed?region=IN&category=tech&cursor=<article_id>&limit=20 → {articles: [...], next_cursor}
GET /a/{article_id}                → 302 to the publisher's URL (logs the click)
POST /v1/publishers/{id}/webhook   (partners push new articles; signed)

Data flow: (1) poll or receive from a publisher → (2) parse, dedupe by URL, store the article → (3) download and resize its thumbnail → (4) change data capture adds it to the regional feed → (5) readers get the feed from the cache or CDN.

8. First working version

Ingestion workers, an articles table, and a feed service that queries it.

8.1 Ingesting articles

A scheduler hands publishers to workers when they're due. A worker fetches the feed, parses new items, skips URLs it has already stored, and inserts the rest.

8.2 Reading the feed

The feed service runs SELECT … WHERE region = ? ORDER BY published_at DESC LIMIT 20 OFFSET ? and returns titles with publisher-hosted image URLs.

8.3 Clicking through

A redirect endpoint logs the click and sends a 302 to the publisher.

9. Make it hold up

9.1 Infinite scroll while news keeps arriving fast

9.2 A feed in under 200 ms at a million requests a second fast · spike-proof

9.3 Getting articles in quickly fresh · polite

• Polling RSS/Atom feeds in tiers: major publishers every 5 minutes, others every 30 minutes to a few hours, using ETag and If-Modified-Since so unchanged feeds cost almost nothing (a 304 response).
• Webhooks for partners who call you when they publish: seconds of delay, far fewer requests. Keep polling as a fallback.
• Scraping sites without feeds: a last resort, brittle and subject to the site's terms; it's essentially a small crawler (Design D09).
• Ingestion jobs run per publisher on a worker pool, with retries and backoff; a failing publisher doesn't hold up the others. Try the trade-offs in section 13.

9.4 Thumbnails fast

Don't hotlink publisher images: they're slow, vanish and change. Download each image once, store resized versions in object storage, and serve them through a CDN in several sizes (srcset). The CDN absorbs nearly all image traffic.

9.5 Breaking news spike-proof · available

When a huge story breaks, traffic multiplies within minutes. The feed service is stateless and autoscales; regions are separate stacks, so a spike in one doesn't hurt others. The regional sorted set is tiny, so throughput (not memory) is the limit: add read replicas of the cache, and let the CDN serve the first page for a few seconds at a time. Being a few seconds stale is invisible; being down is not.

9.6 Categories and light personalisation fast

• Categories: store the category in each feed entry, read ~1,000 entries and filter in memory (milliseconds), rather than maintaining hundreds of separate sorted sets.
• Light personalisation: keep a small preference vector per user (say 60% technology, 30% business, 10% sport) and build the feed at read time by mixing from category lists, plus trending stories. Far cheaper than precomputing a feed per user, at the cost of coarser personalisation.

9.7 The whole design after the deep dives

An article from a partner's webhook reaching a reader's feed, through every part.

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• Separate ingestion and feed services.
• Precomputed regional feeds via CDC (fresh, never expiring) over TTL caches.
• Id cursors.
• Polling tiers plus webhooks (freshness against politeness).
• Read-time category filtering and light personalisation.

What we'd watch:

• Feed p99.
• CDN hit rate.
• Newest-article age per region.
• Per-publisher freshness.
• Poll errors.

At 10×:

• More regional stacks and replicas.
• Story clustering.

Common mistakes:

• Offset pagination.
• Querying the database per feed load.
• TTL caches that stampede.
• Polling everyone at the same rate.
• Hotlinking images.
• One global stack.

13. Try it, check yourself, read more

How quickly do new articles appear, and how many requests does ingestion make? Mix polling tiers and webhooks.

Model: an article is published at a random moment between polls, so on average it waits half the interval; webhook publishers notify within seconds. Articles are weighted toward top publishers (they publish ~20× more). Webhook publishers aren't polled.