System design · Module 14 of 29 · Level 3: Data

Replication: the same data on several machines

A database on one machine is one disk failure away from losing everything and one crash away from an outage. Replication keeps copies on other machines, so the system survives failures and can serve more reads. The hard part is keeping those copies in step: every replication design is a choice about what happens to a write while the copies disagree.

1. The idea

A head chef writes every new recipe change in a notebook, and assistant chefs copy the notebook into their own. Diners can ask any chef, but an assistant who hasn't finished copying gives yesterday's recipe. If the head chef faints, an assistant takes over, with whatever they'd copied so far. And if two chefs both think they're in charge, the kitchen gets two versions of the menu.

Replication serves three goals, often together:

Durability: data survives losing a machine or a zone.
Availability: a copy takes over when the main one fails (failover).
Read scaling and locality: read replicas spread the read load, and replicas in other regions serve nearby users.

There are three main designs:

Design	How writes flow	Examples
Leader-follower (primary-replica)	All writes go to one leader, which streams its change log to followers. Reads can go to any copy.	PostgreSQL, MySQL, MongoDB replica sets, Redis/Valkey, Kafka partitions
Multi-leader	Several leaders (often one per region) accept writes and replicate to each other; conflicts must be resolved.	Multi-region setups, offline-first apps, collaborative editing
Leaderless	Clients write to several replicas directly and read from several; quorums decide what counts.	Cassandra, ScyllaDB, Dynamo-style stores

Synchronous or asynchronous?

Synchronous: the leader waits for a follower to confirm before telling the client "committed". No committed write is lost if the leader dies, but every write pays a round trip to the follower, and a slow follower slows every write.
Asynchronous: the leader confirms immediately and ships changes in the background. Fast, but followers lag, and writes not yet shipped are lost if the leader dies.
Semi-synchronous: wait for one follower (often in another zone), let the rest be async. The common production compromise.

2. How it works

Replication lag and "where did my comment go?"

Press Play: a user posts a comment, then reloads the page, with asynchronous followers.

Consistency guarantees users notice

Read-your-writes: after you change something, you see your change. Fix: read your own recent data from the leader, or from a replica that has caught up to your last write's position.
Monotonic reads: you never see data go backwards (a comment appears, then disappears on refresh because the next read hit a more-lagged replica). Fix: send each user's reads to the same replica.
Consistent prefix: you see events in the order they happened (an answer never before its question). Fix: keep causally related writes in the same partition.

Failover, and why it's dangerous

When the leader fails, a follower must be promoted. Each step can go wrong:

Detect the failure. Usually by missed heartbeats. Too sensitive and a slow leader is replaced unnecessarily; too slow and the outage drags on.
Choose the new leader: the most up-to-date follower, chosen by a consensus system or the managed service (Module 21).
Redirect writes to it: update DNS, a proxy, or the client's view of the cluster.
Fence the old leader. If it comes back still believing it's the leader, two leaders accept writes: split brain, which corrupts data. Systems prevent it with leases and fencing tokens (Module 21).

With asynchronous replication, writes the old leader confirmed but hadn't shipped are lost. Try the numbers in section 3.

Quorums in leaderless replication

With N copies, a write succeeds when W replicas confirm it, and a read asks R replicas and takes the newest answer. If W + R > N, every read set overlaps every write set in at least one replica, so a read always includes the latest successful write. A common setting is N = 3, W = 2, R = 2 (Cassandra's QUORUM: a majority, n/2 + 1). Replicas that missed writes catch up through read repair (fix stale copies seen during a read), hinted handoff (another node holds writes for a down replica) and background anti-entropy comparisons.

Replication vs mirroring

Both keep a second copy, at different levels. Mirroring copies storage blocks: every block written to one disk or volume is written to another, below the database, which doesn't even know. Replication copies database changes: the database ships its log of changes, and the copy is a live database that can answer queries.

Checksums: catching silent corruption

Disks, memory and networks occasionally return different bytes from the ones written, without any error: a flipped bit, a misdirected write, a firmware bug. At the scale of billions of blocks, "occasionally" means "every day". Copies don't help on their own: you need to know which copy is wrong. That's what a checksum is for: a short fingerprint (CRC32C, or a hash such as SHA-256) computed from the data and stored with it.

Where you meet this: HDFS, Ceph and object stores checksum every block; ZFS and Btrfs checksum every block on a single machine; S3 lets clients send a checksum with each upload and verifies it; TCP and Ethernet carry checksums too, but weak ones, which is why storage systems add their own end to end.

Scaling reads: the progression

Reads usually grow far faster than writes (10:1, up to 100:1 or more for content apps). The goal is to take load off the database, in this order, without skipping steps:

Optimise inside the database: indexes on what you filter, join and sort by (Module 12); denormalised columns or summary tables to avoid joins; materialised views for expensive aggregates, refreshed in the background.
Scale the database out: read replicas, roughly when indexed reads pass 50,000–100,000 a second, accepting replication lag; sharding is mainly a write technique (Module 15).
Add caches: Redis cache-aside for hot objects (Module 09), and a CDN for data many users share (public posts, catalogues, search results), never for personal data.

Prompt	Read-scaling plan
URL shortener	Short → long URL in Redis with no expiry (it never changes), plus a CDN
Ticketmaster	Cache events, venues and seat maps; never cache live seat availability; replicas for browsing, the primary for purchases
News feed	Precompute feeds for active users, cache recent posts, paginate (people read only the top)
YouTube	Cache metadata; view counts eventually consistent (updated every few minutes); CDN for thumbnails and video

Don't lead with read scaling for write-heavy systems (Uber location updates), small ones ("1,000 users" → one indexed database), strongly consistent data, or real-time collaboration.

Deep-dive question	Answer
Queries slow down as data grows	Missing indexes cause full scans: check with `EXPLAIN`, add composite indexes in the right column order
Millions of reads on one hot key	Request coalescing: one in-flight fetch per key per server, so the backend sees one request per server; or key fan-out (`feed:celebrity:1…10`, clients pick one at random), at the cost of memory and harder invalidation
Stampede when a hot entry expires	One rebuilder behind a lock; probabilistic early refresh (the chance of refreshing rises near expiry); background refresh for the most critical keys
Updates must show immediately	Versioned keys bumped in the same transaction; a small "recently deleted" cache to filter feeds while invalidation runs; CDN purge APIs or short edge TTLs

3. Try it yourself

Quorum calculator

Choose the number of replicas and the write and read quorums. Click replicas to take them down.

What does async failover cost?

4. Trade-offs

Durability vs write latency

Synchronous replication to another zone adds a millisecond or two per commit; to another region, tens of milliseconds. Async is fast but can lose recent writes on failover.

Read replicas aren't free reads

They add capacity for reads that tolerate lag. Anything that must see the latest write (checkout, "did my payment go through?") goes to the leader.

Multi-leader: local writes, global conflicts

Users in each region write fast to a local leader, but two regions can change the same row at once. Last-write-wins silently discards one; CRDTs or app-specific merges are safer (Module 16).

Replication isn't backup

A bad migration or DELETE without a WHERE replicates to every copy within milliseconds. You still need backups and point-in-time recovery (Module 27).

5. In real systems

PostgreSQL streams its write-ahead log to standbys; replication is asynchronous by default, and synchronous_standby_names with synchronous_commit make chosen standbys synchronous. Tools such as Patroni automate failover using a consensus store.
Managed databases do this for you: RDS Multi-AZ keeps a standby in another zone with automatic failover (its SLA is 99.95%, Module 04); Aurora stores data across three availability zones at the storage layer.
Cassandra lets each query choose its consistency level (ONE, QUORUM, LOCAL_QUORUM, ALL), trading latency against freshness per request.
Kafka replicates each partition to several brokers; with acks=all and a minimum number of in-sync replicas, a confirmed message survives a broker failure (Module 18).
Cross-region read replicas serve local reads with lag of tens of milliseconds to seconds; writes still travel to the primary region.

The technologies up close

The default answer in an interviewSay it in two parts: replicas for availability (a synchronous standby in another zone, automatic failover) and read replicas for read scaling, accepting some lag, with read-your-writes from the primary. Replication and sharding are separate decisions; at scale you usually need both.

Read replicas (leader-follower)

The default · say both purposes

Copies of the database that replay the leader's change log. Writes go to the leader; reads can go to any copy.

How it works

Covered above. The trade-off is lag: asynchronous replicas may be milliseconds to seconds behind, so a user may not see their own write. Route a user's own recent reads to the leader (read-your-writes), and keep each user on one replica so data never goes backwards (monotonic reads).

Status today

Built into every relational database and managed service (RDS read replicas, Aurora replicas, Cloud SQL), and into MongoDB, Redis and Kafka.

Use it when

Read-heavy workloads after indexing is done
Failover: a replica becomes the new leader
Local reads in other regions

Skip it when

Write bottlenecks: every replica still applies every write
Reads that must see the latest write (go to the leader)

In the interview

Two purposes, two sentences: A synchronous standby in another zone with automatic failover for availability; async read replicas for read scaling, with a user's own reads served by the primary.

🚩 Red flags

Replicas to fix a write bottleneck. They multiply reads, not writes.
Ignoring lag: "where did my comment go?" is the follow-up.
Reading from a replica to check a seat or balance before writing.

Synchronous and semi-synchronous replication

The default for high availability

The leader waits for at least one follower to confirm each write before acknowledging it.

How it works

Fully synchronous: no committed write is lost on failover, but every write pays a round trip and a slow follower slows everything. Semi-synchronous: wait for one follower (usually in another zone), let the rest be async: the usual compromise.

Status today

Standard: PostgreSQL synchronous standbys, MySQL semi-sync, RDS Multi-AZ, Aurora's storage-level replication across three zones.

Use it when

Data you can't lose: payments, orders, bookings

Skip it when

Cross-region synchronous writes unless the requirement demands it: they add 50–150 ms to every write

In the interview

Commits wait for a standby in another zone, so failover loses nothing; read replicas stay async.

🚩 Red flags

Synchronous replication across continents for an ordinary app, without mentioning the latency.

Multi-leader replication

Specialised · multi-region writes, offline apps

Several leaders (often one per region) accept writes and replicate to each other.

How it works

Each region writes locally and fast, but two regions can change the same record at once, so conflicts must be resolved: last write wins (loses data), merging, or CRDTs (Module 16). Many teams avoid conflicts by homing each user's data in one region.

Status today

Available (DynamoDB global tables, Cassandra across data centres, some PostgreSQL extensions), but conflict handling is the hard part.

Use it when

Users in several regions who must write with low latency
Offline-first apps and collaborative editing

Skip it when

Single-region products
Data that can't tolerate conflicts (money)

In the interview

Only with a conflict plan: Each user is homed in one region, so concurrent writes to the same record don't happen.

🚩 Red flags

Multi-region active-active writes with no conflict-resolution story.

Leaderless quorums

Situational · Dynamo-style stores

Clients (or a coordinator) write to several replicas and read from several; overlapping quorums decide what's current.

How it works

With N replicas, writing to W and reading from R where W + R > N guarantees overlap (Module 13's Cassandra scene). Missed writes are repaired by hinted handoff, read repair and anti-entropy.

Status today

Cassandra, ScyllaDB and Riak; DynamoDB keeps similar ideas internally.

Use it when

Very high availability for writes, tunable per query

Skip it when

Transactions and strict ordering

In the interview

N = 3 with QUORUM reads and writes, so every read overlaps the latest write.

🚩 Red flags

Claiming W + R > N makes it linearizable. Concurrent writes and failures can still produce surprises; it's a strong default, not a proof.

Anti-entropy: read repair and Merkle trees

Situational · leaderless replicas

How replicas that missed writes catch up: fix differences when a read notices them, and compare whole datasets cheaply in the background.

How it works

Read repair: a quorum read that sees replicas disagree writes the newest version back to the stale ones.
Merkle tree: each replica hashes its keys in ranges, then hashes those hashes up to one root. Two replicas compare roots: equal means identical; different means walk down only the branches that differ. Syncing a tiny difference in billions of keys costs a few hash comparisons, not a full copy.
Together with hinted handoff (Module 13) these keep leaderless replicas convergent.

Status today

Used by Dynamo-style stores (Cassandra's repair, Riak), and the same idea powers Git, BitTorrent and certificate transparency logs.

Use it when

Leaderless or multi-leader replication
Verifying large datasets or file sync without sending everything

Skip it when

Single-leader databases, where followers replay the leader's log

In the interview

Replicas repair on read and run Merkle-tree anti-entropy nightly, so a node that was down converges without a full copy.

🚩 Red flags

Copying the whole dataset to fix a replica that missed a few writes.

Gossip protocols

Situational · membership without a coordinator

Each node periodically tells a few random peers what it knows (who's alive, their versions); news spreads through the cluster like a rumour.

How it works

Every second, each node picks one to three random peers and exchanges state with version numbers; the newest wins. Information reaches all N nodes in about log N rounds, with no central server to fail. Combined with a failure detector (Module 21), it tells every node who's in the cluster and who's down.

Status today

Used by Cassandra, ScyllaDB, Consul (for membership, alongside Raft for its data), Redis Cluster's bus and many peer-to-peer systems.

Use it when

Cluster membership and failure information in large, leaderless clusters

Skip it when

Data that must be strongly consistent (use consensus)

In the interview

Nodes learn membership by gossip, so there's no central registry to lose.

🚩 Red flags

Expecting gossip to be instant or ordered: it's eventually consistent by design.

Physical vs logical replication

Situational · upgrades, partial copies, change streams

Read replicas usually copy the database's write-ahead log byte for byte (physical replication). Logical replication instead sends the row changes themselves: "insert this row into orders", "update row 77". That makes different things possible, and different things break.

How it works

	Physical (log shipping)	Logical (row changes)
What it copies	The whole server, every database and table	Chosen tables, even chosen rows and columns
Versions	Same major version on both sides	Can go between major versions: the usual way to upgrade with seconds of downtime
Schema changes	Copied automatically	Not copied: apply them on both sides, in the right order, or replication stops
Other gaps	None: it's an exact copy	Sequences aren't copied; tables that get updates or deletes need a primary key (or a declared replica identity)
Used for	Read replicas, failover standbys	Upgrades and moves, feeding other systems (change data capture into Kafka with Debezium), merging data from several databases

Example: moving an orders database from PostgreSQL 15 to 17. Create the empty schema on the new server, subscribe it to the old one, let it copy and catch up, check row counts, set the sequences, then point the application at the new server during a short pause. Downtime is the pause, not the copy.

Status today

Checked October 2026 against the PostgreSQL documentation (logical replication restrictions: schema and sequences are not replicated). MySQL's row-based binary log plays the same role for MySQL.

Use it when

Major-version upgrades and moves between machines or clouds
Streaming changes into a search index, cache or warehouse

Skip it when

Read replicas and failover: physical replication is simpler and exact

In the interview

For the version upgrade I'd use logical replication into the new cluster, keep it in sync for a few days, then cut over in a pause of a few seconds.

🚩 Red flags

Changing the schema on the source during logical replication without applying it on the target first.
Forgetting sequences at cut-over, so new rows collide with old ids.

A delayed replica: protection against your own mistakes

Situational · cheap insurance for important data

Replicas copy mistakes as faithfully as good writes: a bad DELETE reaches every replica within a second. A delayed replica deliberately applies changes an hour (or several) late, so after an accident there's a recent, complete copy that hasn't seen it yet.

How it works

How it fits with backups. Two targets describe recovery: how much recent data you can afford to lose (the recovery point) and how long you can be down (the recovery time, Module 27). A normal replica gives a recovery point of seconds after a crash, but none at all after a bad write. Restoring a nightly backup and replaying the log up to just before the mistake works too, but on a large database that takes hours. The delayed replica turns that into minutes for any mistake noticed within its delay.

Status today

PostgreSQL's recovery_min_apply_delay and MySQL's delayed replication (SOURCE_DELAY) both do this. Checked October 2026 against the PostgreSQL documentation.

Use it when

Data where a bad migration or script would be very costly (payments, customer records)
Large databases where restoring a backup takes hours

Skip it when

Small databases that restore from a backup in minutes
As a failover target: it's an hour behind

In the interview

Besides the synchronous standby for failover, I'd keep one replica an hour behind. If a migration deletes the wrong rows, we pause it and copy them back in minutes instead of restoring a backup.

🚩 Red flags

"We have replicas, so we're safe from data loss": replicas copy the mistake.
Promoting the delayed replica in a normal failover.

Dynamo, end to end (the 2007 paper)

Classic design · SOSP 2007

Amazon's Dynamo paper put the pieces of this module together into one store whose first rule was always accept the write: a customer adding to a shopping cart must succeed even while servers, disks or links fail. Each technique above appears in it, chosen for that rule.

How it works

Problem	Dynamo's answer	Where on this site
Spreading keys	Consistent hashing with virtual nodes; each key has a preference list of nodes	Module 15
Tunable consistency	N replicas, W acks to write, R replies to read; (3, 2, 2) was common	Leaderless quorums, above
A replica is down	Sloppy quorum: write to the next healthy node with a hint, handed back later	Hinted handoff, above
Concurrent writes	Vector clocks keep both versions; the application merges them (the cart takes the union)	Module 20
Replicas drifting apart	Anti-entropy with Merkle trees	Read repair and Merkle trees, above
Who's in the cluster	Gossip-based membership and failure detection	Gossip, above

The catch the paper is honest about: merging is the application's job. A cart can take the union of items, which can bring a deleted item back. Many later systems chose simpler rules (last write wins) or, like DynamoDB, a leader per partition.

Status today

The paper's design lives on in Cassandra and Riak. Amazon DynamoDB shares the name and goals but not the mechanism: its 2022 paper describes a replication group per partition using Multi-Paxos for leader election and consensus. Facts checked October 2026 in the Dynamo paper and the DynamoDB paper.

Use it when

Explaining why a design favours availability over consistency
"Design a key-value store" (Design D03)

Skip it when

Data that must never have two versions (balances, bookings)

In the interview

Like Dynamo, writes go to the first three healthy nodes on the ring with W = 2, a hint if a home node is down, and vector clocks so concurrent cart updates are merged rather than lost.

🚩 Red flags

"We use DynamoDB, so we have vector clocks": the product and the paper differ.
Leaderless with last-write-wins on data where a lost update matters.

6. Remember it

Memory hook: "One writes, many copy, someone lags"One leader takes writes, followers copy its log, and some copy is always behind. Decide what readers see while it catches up, and what's lost if the leader dies.

How this hook works: what "One writes, many copy, someone lags" means

What the slogan means, phrase by phrase

"One writes": Leader-follower: a single leader orders all writes, which avoids conflicts.
"many copy": Followers apply the leader's change log, in the same order.
"someone lags": Copying takes time. Lag causes stale reads, and with async replication, lost writes on failover. Every design answers: wait for copies (slow, safe) or don't (fast, risky)?

Why this picture fits: it's the head chef and the assistants copying the notebook: simple to picture, and it makes the two questions (stale reads, what's lost) obvious.

The story → the concept

In the kitchen	In the database
The head chef's notebook	The leader's write-ahead log
Assistants copying it	Followers replaying the log
An assistant still on page 40	Replication lag
Waiting until one assistant has copied the page	Synchronous replication
An assistant taking over when the head chef faints	Failover
Two chefs both giving orders	Split brain

Watch it on an example

A user changes their display name. The write goes to the leader; the profile page reads from a replica 300 ms behind.
They see their old name and change it again, thinking it failed.
Fix: for 10 seconds after a user writes, read their own profile from the leader (read-your-writes).
Everyone else reading the profile can still use replicas: a few hundred ms of staleness is fine for them.

The trap it saves you from

Adding read replicas to scale and accidentally breaking every "save, then show" flow in the product.

It still works when…

You look at caches (Module 09) and CDNs: they're replicas too, just less formal ones, with the same stale-read questions.

7. Common mistakes

Treating replicas as backupsMistakes replicate instantly. Keep backups and point-in-time recovery, and test restores.

Reading your own writes from a lagging replicaUsers see their change vanish. Route recent writers' reads to the leader or a caught-up replica.

Assuming failover loses nothingWith async replication, the last moments of confirmed writes can disappear. Use synchronous replication for data where that's unacceptable.

No fencing during failoverAn old leader that comes back and keeps accepting writes causes split brain. Fence it before promoting a new one.

W + R ≤ N while expecting fresh readsWith N = 3, W = 1, R = 1, a read can miss the latest write entirely. Use quorums that overlap when freshness matters.

8. Design exercise (25 minutes)

Choose sync or async for: a bank's ledger, a social app's likes, an e-commerce order table, a game's leaderboard. Justify each in one line.
Route the reads. A marketplace has a primary and three read replicas. Which of these go where: product page, "my orders" right after checkout, search results, seller payout report?
Set quorums. A Cassandra cluster with N = 3 serves a feature that must survive one replica down for both reads and writes, with fresh reads. Pick W and R, and check them in the calculator.
Plan a failover drill. List what you'd measure (detection time, data lost, time to recover) and what alerts you'd expect to fire.
Write it down: W + R > N, and the three user-visible guarantees.

9. Self-check quiz

What's the main risk of asynchronous replication?

If the leader dies before shipping recent changes, a promoted follower doesn't have them.

N = 5, W = 3, R = 3. Does every read see the latest successful write?

3 + 3 = 6 > 5, so any read set overlaps any write set in at least one replica.

What is split brain?

Both "leaders" accept conflicting writes. Fencing and consensus-based leader election prevent it.

Best fix for "I edited my profile but still see the old version"?

Read-your-writes only matters to the writer, so route just their reads to the leader briefly.

Your database is at 90% CPU from reads. What's the first thing to check?

Optimise inside the database first: a missing index turns lookups into full scans. Replicas and caches come after.

A Ticketmaster design caches seat maps. Which data must never come from the cache?

Purchases check the primary database with a conditional write; a stale cached "free" would double-sell.

Two replicas' Merkle roots are equal. What do you know?

The root summarises every key below it; equal roots mean equal data (barring a hash collision).

A migration wrongly deletes 40,000 rows. Why don't ordinary read replicas help?

Replication copies every committed change, good or bad. A delayed replica or a point-in-time restore is what helps.

What does logical replication NOT copy in PostgreSQL?

You apply schema changes on both sides yourself, and set sequences at cut-over.

In Dynamo, what happens to a write when one of the key's home replicas is down?

A sloppy quorum with hinted handoff keeps writes available through failures.

10. Go deeper

Resource	Time	Why
Designing Data-Intensive Applications, chapter 5 (Replication)	2 h	The definitive explanation of leaders, lag, multi-leader conflicts and quorums.
Log-shipping standby servers (PostgreSQL docs)	30 min	Streaming replication, synchronous standbys and failover in a real database.
Dynamo: Amazon's Highly Available Key-value Store (SOSP 2007)	1 h	The paper that introduced leaderless quorums, hinted handoff and read repair to the industry.
Dynamo architecture in Cassandra	20 min	How a production system implements tunable consistency.

11. Under the hood (optional)

How databases really copy data, acknowledge writes and fail over, and what each choice can lose. Not needed for most interviews; open it when you want to know how the products really do it. All the deep dives are listed on the Under the hood page.

Replication and failoverWhat gets copied, commit points, quorums, chain replication, failover and fencing, lag, conflicts and repair, 9 products comparedabout 35 min

"We replicate it" can mean a byte-for-byte log shipped to an identical server, or row changes streamed to a different database, acknowledged before or after the copy exists. Those choices decide what a failover loses, how stale a replica read can be, and how slow every write is.

What gets copied

Kind	What travels	Good for	Limits	Examples
Physical	The storage engine's own log (PostgreSQL WAL, Aurora redo records): which bytes changed on which page	Exact copies, fast, nothing to interpret	Same major version and platform; the whole cluster, not chosen tables	PostgreSQL streaming replication, Aurora storage, Oracle Data Guard (physical)
Logical (row-based)	Row changes: insert this row, update that key	Upgrades across versions, selected tables, feeding other systems (change data capture)	Schema changes need care; large transactions lag	MySQL row-based binlog, PostgreSQL logical replication, MongoDB oplog, Debezium
Statement-based	The SQL statement itself	Compact	Non-deterministic statements (NOW(), RAND(), unordered LIMIT) produce different results on replicas	Old MySQL default; now rarely used
Operation log	Commands or operations made idempotent	Simple to replay and resume	Must be designed so replays are safe	Redis command stream, MongoDB oplog, Elasticsearch operations

When is a write safe? The commit point

The real question is when the client hears "OK" relative to when copies exist. Try each mode with a primary in one zone and two replicas in others (about 1 ms away):

MySQL semi-sync waits until a replica has received and logged the change, not applied it, so a replica read can still be stale.
PostgreSQL chooses per standby or by quorum (synchronous_standby_names = 'ANY 1 (s1, s2)'), and synchronous_commit says how far to wait: received, flushed, or applied (remote_apply, which also gives read-your-writes on that standby).
Raft and Paxos groups (MySQL Group Replication, CockroachDB, etcd) commit once a majority has the entry, so one slow member never slows writes.

Quorums: N, W and R

In leaderless and quorum systems, each value lives on N replicas, a write waits for W of them, and a read asks R. If R + W > N, every read set overlaps every write set, so at least one replica answering a read has the latest write.

Overlap isn't the whole story: with sloppy quorums (writes accepted by stand-in nodes during failures, then handed back through hinted handoff), concurrent writes resolved by last-writer-wins timestamps, or a failed write that landed on fewer than W nodes, a quorum read can still return stale or lost data.
Aurora's quorum is on storage, not servers: 6 copies across 3 zones, writes need 4, reads 3. It survives losing a whole zone for writes, and a zone plus one more copy for reads.

Chain replication

Replicas form a chain: writes enter at the head and flow to the tail, and the tail answers reads, so every read sees only fully replicated writes (strong consistency) while the head's load stays low. CRAQ lets every node serve reads of items that aren't mid-update, asking the tail only for those that are. A chain is as slow as its slowest link, and a configuration service must repair it when a node fails.

Failure detection, election and fencing

Detection is guessing: heartbeats plus a timeout. Too short and a GC pause triggers a needless failover; too long and writes are down for longer. Phi-accrual detectors (Cassandra, Akka) turn heartbeat history into a suspicion level instead of a fixed timeout.
Election needs a majority so two sides of a partition can't both elect a leader: Raft terms, MongoDB elections, Patroni holding a leader key in etcd or Consul.
Fencing stops the old leader: every new leader gets a higher term or epoch, and storage or replicas reject writes carrying an older one. Without it, a paused old leader wakes up and keeps writing: split brain.
Failover can lose writes: with asynchronous replication, the new leader lacks whatever the old one hadn't shipped. When the old leader returns, those writes must be discarded or reconciled.

Lag and read-your-writes in practice

Measure lag in time and position: PostgreSQL's pg_stat_replication shows write, flush and replay lag; MySQL's Seconds_Behind_Source can mislead (it's based on event timestamps), so many teams write a heartbeat row every second and measure its age on each replica.
Read your own writes: after a write, read from the primary for a few seconds; or remember the write's log position (LSN, GTID, cluster time) and have the replica wait until it has replayed that far (MySQL WAIT_FOR_EXECUTED_GTID_SET, MongoDB causally consistent sessions); or route each user to one replica (sticky sessions) for monotonic reads.

Multi-leader conflicts and leaderless repair

Mechanism	What it does	Where
Last writer wins	Highest timestamp wins; the other write silently disappears	Cassandra, DynamoDB global tables (multi-region eventual consistency)
Merge in the application	Keep both versions (siblings) and let code or the user merge	Riak, CouchDB conflicts, offline-first apps
CRDTs	Data types whose concurrent updates always merge the same way (counters, sets, text)	Redis Enterprise active-active, Automerge and Yjs (D19)
Avoid conflicts	Each record has a home region that takes its writes; or synchronous multi-region writes	Per-user home regions; DynamoDB multi-region strong consistency, Spanner
Hinted handoff	A node that missed writes while down gets them from a stand-in (Cassandra keeps hints for 3 hours by default)	Cassandra, Dynamo-style stores
Read repair	A read that sees replicas disagree writes the newest value back	Cassandra, Riak
Anti-entropy repair	Replicas compare Merkle trees of their data and stream only the ranges that differ	Cassandra repair, Dynamo

Product by product

Product	What is replicated	Who waits, who leads	The detail worth knowing
PostgreSQL	WAL streamed to standbys (physical); publications and subscriptions (logical)	Async by default; synchronous per standby or by quorum; no built-in automatic failover	Patroni (with etcd or Consul) elects a leader and fences the old one
MySQL	Binary log, row-based by default; GTIDs identify each transaction	Async or semi-sync; Group Replication (InnoDB Cluster) uses a Paxos-based protocol	Semi-sync waits for receipt, not apply
MongoDB	The oplog: idempotent operations tailed by secondaries	Raft-like elections; the default write concern has been majority since 5.0	Writes acknowledged by fewer than a majority can be rolled back after a failover
Cassandra	Every replica takes writes; no leader	Per-request consistency: ONE, QUORUM, LOCAL_QUORUM, ALL	Hinted handoff, read repair and Merkle-tree repair keep replicas converging
Redis	Command stream; a backlog allows partial resync	Async; Sentinel or Cluster promotes a replica	Acknowledged writes can be lost on failover; WAIT narrows the window
Kafka	Followers fetch each partition's log from its leader	`acks=all` (the producer default since 3.0) with `min.insync.replicas`; the KRaft controller picks leaders from in-sync replicas	Unclean leader election is off by default: availability is traded for no lost messages
Amazon Aurora	Redo records to a shared storage layer, 6 copies in 3 zones	Writes need 4 of 6, reads 3 of 6	Replicas read the same storage, so lag is typically well under 100 ms
DynamoDB global tables	Item changes between regions	Every region writes; eventual by default, or multi-region strong consistency	In eventual mode, concurrent writes to one item resolve by last writer wins
Elasticsearch	Operations replayed from primary to replica shards	The master tracks in-sync copies; sequence numbers and a global checkpoint	Only in-sync copies can be promoted, so writes aren't lost silently

Say it in the interview

Answers that show depth

"Can a failover lose data?" With async replication, yes: whatever the old primary acknowledged but hadn't shipped. Semi-sync or a majority quorum closes that gap at the cost of one round trip per commit.
"Why isn't R + W > N enough?" Sloppy quorums, last-writer-wins on concurrent writes and partially failed writes can all return stale data; it's a probability improvement, not linearizability.
"How do you stop two primaries?" Elect through a majority (Raft, etcd) and fence: a higher term or epoch on every write, so the old leader's writes are rejected.

What does MySQL semi-synchronous replication guarantee before acknowledging a commit?

That protects against losing the write on failover, but a replica read can still be stale.

N = 3 and W = 2. What is the smallest R that guarantees a read overlaps the latest write?

R + W must exceed N: 2 + 2 > 3.

What stops a paused old leader from writing after a new one was elected?

Timeouts can't tell slow from dead; only rejecting stale terms makes split brain harmless.