Cheat sheet: System design

What a system is made of

“Ask, answer, keep”

The client asks, the server answers, the database keeps. Anything else in a design exists to make one of those three faster, bigger or safer.

Avoid

• Trusting the client
• State in one server's memory
• Thinking the API is the backend

Inside one machine

“The chef never waits for the delivery”

Computation is fast and waiting is slow, so a good server keeps working on other requests while some wait.

Avoid

• Blocking the event loop
• Thread pools sized by guesswork
• Forgetting memory

Networks in plain terms

“Name, address, door, conversation”

DNS turns the name into an address; IP gets you to the building; the port is the door; the protocol is the conversation you have inside.

Avoid

• Exposing internal services
• Assuming the network is reliable and instant
• Forgetting DNS caching

Storing data

“Every row a key, every fact once, every change all-or-nothing”

Three rules that keep stored data findable, consistent and safe.

Avoid

• Floats for money
• Files in the database
• Related writes without a transaction

Data structures behind systems

“Hash to find, tree to sort, sketch to count”

Exact lookup → hash. Anything ordered (ranges, top, prefix) → tree. Counting at huge scale → a sketch.

Avoid

• Thinking O(1) is always faster
• Ignoring the disk
• Sizing a Bloom filter too small

From one server to millions

“Problem first, part second”

Never add a box without naming the problem it fixes, and the number that showed the problem.

Avoid

• Scaling the wrong tier
• Skipping statelessness
• Sharding too early

How a request travels

“Find, knock, lock, ask”

DNS finds the address, TCP knocks, TLS locks, HTTP asks. The first three are paid once per connection; only the last is paid every time.

Avoid

• Trusting identity sent by the client
• Keeping session state in one server's memory
• Ignoring round trips

Latency, throughput, numbers

“Lanes, trip time, traffic jam”

Bandwidth is lanes, latency is trip time, utilization is how full the road is. Near full, every trip gets slow, and the unluckiest trip is the one users remember.

Avoid

• Reporting only the average latency
• Planning to run at 90%+ utilization
• Using 2010-era hardware numbers

Back-of-the-envelope estimation

“A day is a hundred thousand seconds”

Per day ÷ 10⁵ = per second. Then × peak, × bytes, and finish every number with "so we need…".

Avoid

• Estimating without concluding
• Sizing for the average
• Mixing up bits and bytes

Availability and SLOs

“Chains multiply down, spares add nines”

Things you need in a row multiply your availability down. Independent copies of the same thing add nines. Recovery speed counts as much as rarity.

Avoid

• Correlated "redundancy"
• Promising more nines than your dependencies
• Measuring "server is up" instead of "users succeed"

The design method

“Requirements before rectangles”

No boxes until you know what users do and how well it must work. Then breadth before depth: a working design first, the hard parts second.

Avoid

• Not finishing a working design
• Unmeasurable targets
• Premature complexity

Networking and protocols

“TCP queues the line, QUIC gives each order its own line”

TCP delivers everything in order, so one lost packet holds up all that follows. QUIC keeps streams separate, so a loss only delays its own stream.

Avoid

• Choosing UDP for data that must arrive
• Opening a new connection per request
• Relying on DNS for instant failover

API design

“Nouns, verbs, and a receipt”

Resources are nouns, methods are verbs, and every important write carries a receipt number (an idempotency key) so asking twice can't buy twice.

Avoid

• Trusting a user ID in the request
• Unbounded lists
• Errors inside a 200

Load balancing and scaling out

“Any till, open till, shortest queue”

Stateless servers so any till can serve anyone; health checks so only open tills get customers; a balancing rule that prefers the shortest queue.

Avoid

• State in server memory
• Health checks that are too deep
• Round robin with uneven requests

Caching

“Check the counter, then the cupboard; clear the counter when you restock”

Read from the cache first and fall back to the source; on every write, delete the cached copy rather than trying to update it.

Avoid

• Caching data that must be exact
• No TTL and no eviction policy
• Ignoring the cold or failed cache

Blob storage and CDNs

“Servers sign, storage carries, edges deliver”

Your servers only sign permissions and record metadata; the bytes go straight to object storage; downloads come from the CDN edge nearest the user.

Avoid

• Trusting the client's "upload finished" call
• Storing files in the database
• Presigned URLs without limits

Relational data and modeling

“One fact, one place; connect with keys”

Store each fact once, point between tables with keys, and let joins and transactions do the connecting. Copy data only for a measured read problem.

Avoid

• Business data as primary keys
• Lists stuffed into a column
• The N+1 query

Indexes and storage engines

“B-trees edit the book, LSMs write a diary”

A B-tree finds the right page and edits it in place: great for reading. An LSM only appends new entries and tidies up later: great for writing.

Avoid

• Querying without an index on a growing table
• Wrong column order in a composite index
• Functions on indexed columns

Choosing a database

“Postgres until proven otherwise”

Start relational. Move a workload to another family only when its access pattern, scale or consistency needs prove it, and keep one source of truth.

Avoid

• Using a cache or search index as the source of truth
• Choosing NoSQL "because it scales"
• Modelling a key-value store like a relational one

Replication

“One writes, many copy, someone lags”

One leader takes writes, followers copy its log, and some copy is always behind. Decide what readers see while it catches up, and what's lost if the leader dies.

Avoid

• Treating replicas as backups
• Reading your own writes from a lagging replica
• Assuming failover loses nothing

Sharding and consistent hashing

“Split by how you ask; ring it so growth moves little”

Choose the shard key from your most common query, and place shards on a hash ring (or slots) so adding capacity moves only a small slice.

Avoid

• Sharding before it's needed
• Low-cardinality or skewed keys
• Time as the only key for hot writes

Consistency, CAP and PACELC

“When the line is cut, refuse or guess; when it isn't, wait or guess”

During a partition you refuse (C) or answer with possibly stale data (A). The rest of the time you wait for replicas (C) or answer from the nearest one (latency).

Avoid

• "We'll pick CA"
• One choice for the whole system
• Last-write-wins by accident

Transactions and contention

“Check and change in one step”

The bug is a gap between reading and writing. Close it: make the check part of the write (a condition, a constraint), or hold the row while you decide (a lock), or detect that it changed and retry (a version).

Avoid

• Read, check in code, then write
• Holding locks during slow work
• Ignoring 0 rows affected

Queues and event streams

“Clip the ticket, cook it twice safely”

Producers clip tickets to the rail and move on; cooks take them at their own pace. Delivery is at-least-once, so cooking the same ticket twice must do no harm.

Avoid

• Non-idempotent consumers
• Using a queue to hide a consumer that can't keep up
• No dead-letter queue

Real-time updates

“Last mile, then routing”

Pick the last-mile protocol from the direction and urgency of updates. Then solve routing: how an update finds the server holding the user's connection, and how a reconnecting user catches up.

Avoid

• Relying on pub/sub for delivery
• WebSockets by reflex
• Business logic in connection servers

Time, ordering and IDs

“Clocks lie, counters don't”

Timestamps from different machines can't be trusted to order events. Counters that travel with messages (Lamport, vector, sequence numbers) can. For IDs, put time first only to make them sortable, never to make them correct.

Avoid

• Ordering writes by server timestamps
• Measuring durations with wall-clock time
• Random UUIDs as clustered primary keys at high write rates

Consensus and coordination

“Any two majorities share a member”

That one fact makes consensus safe: two conflicting decisions can't both win a majority. Odd cluster sizes, at most one leader per term, and fencing tokens all follow from taking it seriously.

Avoid

• Locks without fencing
• Even-sized clusters
• Spreading a consensus group across continents

Workflows, sagas and outbox

“Write it down before you do it; know how to undo it”

Record the change and its event in one local transaction (the outbox), let a durable workflow remember each completed step, and give every step a compensation.

Avoid

• Dual writes
• No compensation for a step
• Non-idempotent steps

Batch, stream and background jobs

“Ticket, nightly load, conveyor belt”

A job gets a ticket and a status; batch washes a whole bounded load at once; a stream is a belt that never stops, so it needs windows, a sense of time, and a way to resume after a jam.

Avoid

• Long work inside the request
• One queue for every job size
• Windows by processing time

Reliability patterns

“Fuses, separate circuits, a main switch”

Breakers trip on a failing dependency, bulkheads keep circuits separate, and rate limits and load shedding cap the total draw, so one fault never burns the house down.

Avoid

• No timeout
• Retrying without backoff or jitter
• Retries at every layer

Observability

“Dashboard says something's wrong, tracker says where, records say why”

Metrics detect, traces locate, logs explain, and alerts fire on what users feel, measured against the error budget.

Avoid

• High-cardinality labels on metrics
• Alerting on causes instead of symptoms
• Averages instead of percentiles

Security

“Front desk, badge, locked floors”

Authenticate once at the front desk, authorize every door with the badge (for this person and this room), and lock floors apart so one breach stays small.

Avoid

• Missing per-resource checks
• Secrets in code, config files or logs
• Long-lived tokens with no revocation

Deployment, multi-region, DR

“Taste in one branch; keep the recipes off-site”

Release to a small slice first and roll back automatically; keep tested, off-site copies of data and decide in advance how much you can lose (RPO) and how long you can be closed (RTO).

Avoid

• Never testing a restore
• Releasing to everyone at once
• Hidden single-region dependencies

Architecture styles

“One kitchen, food court, or chain of branches”

Choose how many independent pieces the system needs, draw each boundary around a business capability that owns its data, and split only when a real need (deploys, scale, isolation) outweighs the cost.

Avoid

• A shared database between services
• Splitting by technical layer
• Long synchronous chains

Designing AI systems

“An expensive, slow, brilliant, gullible dependency”

Budget it in tokens, stream it, cache and route around it, measure it with evals, and never give it more power than you'd give text from a stranger.

Avoid

• Permissions in the prompt
• No token budgets
• Waiting for the whole answer

URL shortener

Deep dives

• Generating unique short codes
• Redirects in under 50 ms, at 12,000 a second
• 301 or 302?
• Counting clicks without slowing redirects
• Aliases, expiry and abuse
• Keeping printed links alive across regions

Trade-offs

• Leased counter blocks instead of random codes
• 302 over 301
• Asynchronous analytics
• Separate create and redirect services

Rate limiter

Deep dives

• Choosing the algorithm
• Counting correctly with 100 servers
• Keeping the overhead tiny
• When the limiter itself fails
• Hot keys and abuse
• Several regions, and pushing rules

Trade-offs

• Token bucket by default
• Atomic scripts in a Redis cluster
• Two-tier local counting for hot keys
• Fail open by default

Distributed key-value store

Deep dives

• Spreading keys so nodes can come and go
• Copies and quorums
• Concurrent writes to the same key
• Finding and fixing divergence cheaply
• Membership and failure detection without a coordinator
• Deletes, and the storage engine

Trade-offs

• Availability over consistency
• Leaderless replication with hinted handoff instead of leaders per range
• Last-write-wins by default, with siblings or CRDTs where data matters
• LSM storage

Distributed cache

Deep dives

• Routing keys to nodes
• Moving data while clients keep working
• Failover without split brain
• Multi-key operations
• Hot keys, stampedes and big keys
• Cold starts, persistence and tenants

Trade-offs

• Hash slots with an authoritative map
• Smart clients over a proxy
• Asynchronous replication
• Approximate LRU/LFU by sampling

Notification system

Deep dives

• Login codes must not wait behind a campaign
• Once, from the user's point of view
• Providers fail and throttle
• Preferences, quiet hours and frequency caps
• A campaign to 20 million users
• Tracking delivery

Trade-offs

• Separate lanes per priority
• Delivery records for dedup
• Erring towards duplicates for login codes, and towards losses for marketing
• Batched, resumable campaigns

News feed

Deep dives

• Fan-out on write, on read, or both?
• Storing timelines
• Hydrating posts quickly
• Pagination and freshness
• Ranking
• Deletes and privacy changes

Trade-offs

• Hybrid fan-out
• Ids in timelines with hydration
• Eventual freshness for followers; read-your-writes for authors
• Lazy deletion through a filter

Chat and messaging

Deep dives

• Ordering and duplicates
• Getting the message to the right server
• Offline users, receipts and sync
• Storing billions of messages
• Group chats
• Presence and typing
• End-to-end encryption

Trade-offs

• Per-conversation order, not global order
• Thin gateways and pub/sub routing
• Store first, then deliver
• High-water-mark receipts

News aggregator

Deep dives

• Infinite scroll while news keeps arriving
• A feed in under 200 ms at a million requests a second
• Getting articles in quickly
• Thumbnails
• Breaking news
• Categories and light personalisation

Trade-offs

• Separate ingestion and feed services
• Precomputed regional feeds via CDC
• Id cursors
• Polling tiers plus webhooks

Web crawler

Deep dives

• The frontier: priority and politeness together
• Don't fetch twice
• Robots.txt, DNS and slow servers
• Crawler traps
• Not losing progress
• Recrawling

Trade-offs

• A Mercator-style two-stage frontier
• Partitioning by host
• A Bloom filter with a known false-positive cost
• At-least-once fetching with content hashes

Search typeahead

Deep dives

• Finding the top suggestions for a prefix
• Building the tables, and swapping safely
• Serving it fast, and mostly not serving it at all
• Trending queries
• Personalisation and safety

Trade-offs

• Precompute everything offline
• A trending stream to close the gap
• Atomic versioned swaps
• Heavy client and CDN caching

Video platform

Deep dives

• Uploading big files over bad connections
• Transcoding: one file into many
• Adaptive streaming
• Delivering 100+ Tbit/s
• Metadata, views and recommendations
• Storage cost and moderation

Trade-offs

• Multipart uploads straight to storage
• Chunked parallel transcoding as a workflow
• Adaptive streaming over HTTP
• ISP-embedded caches for egress

File storage and sync

Deep dives

• Blocks, hashes and uploading only what changed
• The metadata service is the heart
• Telling devices something changed
• Conflicts
• Durability, versions, sharing and garbage collection
• The sync engine on the client

Trade-offs

• Content-addressed blocks
• Commit-after-upload for atomicity
• Per-namespace journals and cursors
• Notifications without data

Ride hailing

Deep dives

• Storing and querying locations
• Matching without double booking
• Not losing ride requests
• Live tracking
• Pricing and trip records
• Cities, regions and failure

Trade-offs

• An in-memory geo index
• Atomic, TTL-based claims for drivers
• One offer at a time
• Durable matching workflows

Nearby places

Deep dives

• Why a normal index isn't enough, and what is
• Edges: searching neighbours too
• Where the index lives
• Dense cities and empty countryside
• Filtering, ranking and caching
• Keeping the index fresh

Trade-offs

• Cells that turn 2-D into 1-D keys
• Adaptive cells
• A small index replicated everywhere instead of sharded
• Minutes of freshness via change streams

Ticket booking

Deep dives

• Never selling a seat twice
• Holding seats during payment
• Surviving the on-sale stampede
• Seat maps and search
• Paying exactly once
• Partitioning and contention

Trade-offs

• Hold length
• Pick-your-seat vs best-available
• A waiting room
• The database as the only judge

Payment system

Deep dives

• Idempotency, end to end
• A double-entry ledger
• A state machine and webhooks
• The outbox: saved and announced together
• Reconciliation: trust, but verify
• Security

Trade-offs

• Synchronous vs asynchronous confirmation
• One PSP vs several
• A ledger over a status column
• Correctness over latency everywhere

Ad click aggregator

Deep dives

• The redirect path
• Event time, windows and watermarks
• Deduplication and exactly-once counts
• Hot ads
• Storage and queries
• The nightly recount

Trade-offs

• Freshness vs completeness
• Stream plus batch
• Exact vs approximate
• Dimensions

Metrics monitoring

Deep dives

• Pull or push?
• The time-series database
• Long-term storage and downsampling
• Cardinality: the silent killer
• Alerting that people trust
• Monitoring the monitor

Trade-offs

• Resolution vs cost
• Local vs global view
• Fast vs quiet alerts
• Bounded labels over unlimited detail

Collaborative editor

Deep dives

• Merging concurrent edits
• One document, one server
• Op log, snapshots and history
• When a document server dies
• Presence, offline and permissions
• Very busy documents

Trade-offs

• OT vs CRDT
• One owner per document
• Granularity
• Presence never stored

Top-K and leaderboards

Deep dives

• Exact leaderboards with sorted sets
• Beyond one node
• Top-K over a stream
• Sliding windows with buckets
• When memory is the limit: Count-Min Sketch
• Hot items

Trade-offs

• Exact vs approximate
• Freshness vs cost
• Sorted set in memory, database as the truth
• Shard by item so the merge stays exact

Distributed job scheduler

Deep dives

• More schedulers, never a double dispatch
• Being on time at the top of the hour
• Workers that die, and zombies that come back
• Retries, timeouts and overlap
• Fairness and isolation between tenants
• Time zones, daylight saving and missed runs

Trade-offs

• At-least-once with fencing, instead of pretending to offer exactly-once
• A relational database for jobs and runs
• Correctness on the claim, availability on the API
• Spreading flexible jobs, at the cost of "exactly :00" for those who don't need it

Live comments

Deep dives

• Getting each comment to the right servers
• The five-million-viewer stream
• Storing comments without a hot partition
• Ordering, reconnects and duplicates
• Moderation that keeps up
• Running the connection servers

Trade-offs

• Sampling instead of "everyone sees everything" on huge streams
• Rough time order instead of a global sequence
• SSE for one-way delivery
• Availability over exactness during failures

LLM chat platform

Deep dives

• Sharing GPUs: continuous batching
• Memory is the limit: KV cache and context
• Routing, priorities and admission control
• Streams that survive bad connections
• Storing conversations, and deleting them
• Safety on a stream

Trade-offs

• Continuous batching and big batches
• Cache-aware routing against perfectly even load
• Summarising old turns
• Degrading free traffic instead of failing everyone

Flash-sale checkout

Deep dives

• The hot stock counter
• Reserve first, pay second
• Absorbing the crowd at the edge
• Bots and fairness
• Keeping the rest of the shop alive
• Orders and payments after the rush

Trade-offs

• A waiting room
• Unit rows with SKIP LOCKED in the transactional store, rather than a faster but separate counter
• Reserve before payment, at the cost of handling expiries
• A yes/no availability flag instead of exact live counts

Maps and navigation

Deep dives

• Fast routes on a continental graph
• Splitting the world
• Live traffic from location pings
• Honest arrival times
• Navigation sessions and rerouting
• Tiles for the whole planet

Trade-offs

• Heavy offline preprocessing for millisecond queries
• A fixed partition with fast re-customisation, instead of rebuilding for every traffic change
• Local navigation on the phone, with the server as an advisor
• Aggregated, anonymised traffic instead of stored traces

Web search engine

Deep dives

• Splitting the index
• Fast posting lists
• Ranking in stages
• Fresh results without rebuilding everything
• Taming the fan-out's tail
• Removing a page fast

Trade-offs

• Document partitioning
• Partial results under a deadline instead of always-complete answers
• A ranking cascade
• Two indexes

Online code judge

Deep dives

• Running untrusted code safely
• Fair, repeatable timing
• Surviving the contest surge
• Test data and runtimes on every worker
• A live leaderboard for 100,000 people
• Rejudges and cheating

Trade-offs

• Strong isolation
• Dedicated cores per run for fairness, at lower density
• Pre-scaling for known peaks rather than trusting autoscaling
• Separate lanes for contests, practice and "Run"

Deployment system

Deep dives

• Waves and blast radius
• Knowing a release is bad
• The deploy system can fail; services can't
• Getting a terabyte of images to the fleet
• Configuration is a deployment too
• A rollout that survives crashes

Trade-offs

• Slower, staged rollouts with bakes, in exchange for small blast radius
• Desired state and reconciliation instead of push scripts
• Automated canary analysis, accepting occasional false rollbacks
• Configuration treated exactly like code

Q&A site

Deep dives

• Ranking answers fairly
• Finding duplicate questions
• Votes on a hot answer
• The personalised feed
• Caching pages that change
• Spam, abuse and vote fraud

Trade-offs

• Slightly stale counts and pages for huge cache hit rates
• A confidence-based ranking instead of simple vote sums
• Feeds built on read, because questions live for years
• One database first; derived stores for search, feeds and counters

Data platform

Deep dives

• Batch, streaming, or both
• Files on object storage that behave like tables
• Change data capture without surprises
• Bad data and data contracts
• Late and duplicate events
• Deleting personal data in files that never change
• Query cost and noisy neighbours

Trade-offs

• Batch by default, streaming only where minutes matter
• Raw data kept so any table can be rebuilt, at the cost of storage
• An open table format on object storage instead of one vendor's warehouse storage, for cost and engine choice
• Daily numbers that may change for a few days, for complete late data

AI support assistant

Deep dives

• Retrieval and grounded answers
• Actions that can't go wrong quietly
• Prompt injection and data leaks
• Handing off to a person
• Measuring quality
• Latency and cost
• Keeping knowledge current

Trade-offs

• Answer less often, but correctly
• Confirmation steps for every write, at the cost of one extra tap
• Retrieval instead of fine-tuning, so facts stay current and citable
• Quality gates on every change, which slows prompt tweaks

AI code assistant

Deep dives

• The completion latency budget
• Choosing the context
• Indexing a repository
• A safe agent loop
• Privacy and tenants
• Serving and cost
• Measuring quality

Trade-offs

• A small model for completions
• Tools run in the editor, so code stays local and permissions are the user's own
• Approvals that slow the agent down, for safety
• Two serving paths instead of one, to protect completion latency

Stock exchange

Deep dives

• The order book
• One sequence, one thread
• Surviving the loss of an engine
• Market data to a thousand subscribers
• Microseconds
• Risk checks before matching

Trade-offs

• One thread per book instead of parallel matching
• Acknowledge after replication within one data centre; the second site accepts a small loss window
• Shard by symbol and forbid cross-symbol transactions
• Multicast market data; subscribers repair their own gaps

Distributed message queue

Deep dives

• Replication and acknowledgements
• Who decides who leads
• Consumer groups and rebalancing
• A hot key
• Duplicates and exactly once
• A petabyte of retention

Trade-offs

• A log, not a queue
• Order per partition only
• Acknowledgement level chosen per producer
• At least once by default; exactly once only inside the log

Object storage

Deep dives

• Redundancy: copies or erasure coding
• Disks die every day
• Billions of small objects
• Metadata for 100 billion keys
• Huge uploads
• Deletes, versions and lifecycle

Trade-offs

• Metadata and bytes stored separately, each in the structure that suits it
• Erasure coding for cost, with copies for small or hot data
• Range-partitioned metadata for ordered listings, at the cost of split machinery
• Bytes first, metadata commit last

E-commerce cart and inventory

Deep dives

• When to hold stock, and for how long
• Checkout as a saga
• Retries and double taps
• Availability for 60,000 page views a second
• Carts that always accept writes
• Many warehouses

Trade-offs

• Reserve at checkout, not at add-to-cart, with a timeout we tune
• Strong consistency only where money and stock are decided; seconds of staleness on product pages
• A saga with authorise-then-capture instead of distributed transactions
• Idempotency keys on every write that matters

Email service

Deep dives

• Filtering 20 billion messages a day
• Who really sent this?
• An exabyte of mail
• Search over years of mail
• Getting our mail delivered
• Sync and push for hundreds of millions of devices

Trade-offs

• Acknowledge SMTP only after a durable queue write; process asynchronously
• A filtering funnel from cheap to expensive
• Metadata in sharded databases, content in object storage, deduplicated
• Everything partitioned by user

API gateway

Deep dives

• Five milliseconds for everything
• Rate limits across 135 nodes
• A bad change must not take everything down
• One slow service
• Gateway, service mesh or BFF?
• Regions and the control plane

Trade-offs

• A stateless data plane and a separate control plane; traffic keeps flowing when the control plane fails
• Nothing remote on the request path
• Rate limits that can overshoot slightly, in exchange for no extra hop
• Configuration as versioned snapshots, rolled out in stages with automatic rollback

Social graph

Deep dives

• Keeping both directions in step
• Where does an edge live?
• An account with 150 million followers
• Mutual follows
• Regions and reading your own follow
• Why not a graph database?

Trade-offs

• Every edge stored twice, so each user's lists are on their own shard
• One local write plus an outbox, instead of a distributed transaction; the inverse side catches up within a second
• Counts stored, not computed; split for the hottest accounts
• A cache tier answers almost every read; read-your-writes for the actor only

Friend suggestions

Deep dives

• Taming the two-hop explosion
• Fresh after a follow
• A brand-new user
• Ranking: what to optimise
• Privacy: suggestions that leak
• Feedback loops

Trade-offs

• Precompute in batch, refresh near-line, filter and lightly re-rank at serving time
• Prune the graph walk
• Several candidate sources, one ranking model optimising for lasting connections
• Privacy rules and blocks enforced centrally and again at serving time; fail closed

Ad serving

Deep dives

• Retrieval: 5 million ads to 1,000
• The prediction budget
• Budget pacing
• Frequency caps
• Campaign changes in under a minute
• Counting money correctly

Trade-offs

• A funnel
• eCPM ranking and a second-price auction for self-serve CPC ads
• Pacing with local allocations
• Soft frequency caps; fail closed when unsure

Serverless platform

Deep dives

• Making cold starts rare and short
• Strangers on one machine
• Packing hosts
• Limits and bursts
• The database behind the functions

Trade-offs

• MicroVMs for isolation, accepting more complexity than containers
• Reuse idle sandboxes for minutes, trading memory for fewer cold starts; provisioned concurrency as a paid guarantee
• One request per sandbox
• Pack hosts by memory, spread each function widely

Calendar

Deep dives

• Repeating events: a rule plus exceptions
• Time zones and clock changes
• Finding a time for twelve people
• Rooms that can't be double-booked
• Five million reminders at 09:50
• Huge meetings and phones that sync

Trade-offs

• Rules and exceptions as the truth, plus a derived 12-month index
• Local time plus a zone name, not UTC, for anything that repeats or lies in the future
• Separate read models
• Database-enforced room bookings, keeping that transaction short

Recommendation system

Deep dives

• Retrieve cheaply, rank carefully
• Features that mean the same thing in training and serving
• Reacting within the session
• New items, new shoppers and feedback loops
• Knowing it works

Trade-offs

• A funnel
• Three speeds
• A feature store with point-in-time training data, to avoid skew and leakage
• A small, explicit exploration budget, paid for in a little short-term click rate

Likes counter

Deep dives

• Double taps, retries and the toggle
• Counting without locking
• The viral photo
• Counts that drift, and how to repair them
• Bought likes

Trade-offs

• Like rows as the truth, counts derived
• Counts seconds behind for others, instant for the tapper
• Batching per photo and split counters only for hot photos
• PUT/DELETE of a state instead of "add one", so retries are harmless

ID generator

Deep dives

• When the clock goes backwards
• Running out of sequence
• IDs without clocks: handing out blocks
• Three regions
• IDs in public

Trade-offs

• 64-bit time-based IDs
• A library, not a service
• Refuse to mint when the clock goes back, rather than risk a duplicate
• Generator numbers split by region, so partitions can't cause collisions