Design file storage and sync

Save a file on your laptop and it appears on your phone a few seconds later; edit one slide in a 200 MB deck and only a sliver is uploaded. A sync service is a metadata problem dressed up as a storage problem: the bytes go to blob storage in content-addressed blocks, while a careful metadata service decides what every device should have. It follows the interview framework phase by phase.

1. The prompt

"Design Dropbox (or Google Drive): users keep a folder in sync across their devices, share files and folders with others, and can recover earlier versions."

Storing bytes is the easy half. The hard half is deciding, for every device, exactly what changed since it last looked; uploading only what changed; and never, ever losing or wrongly deleting someone's file when two devices disagree.

2. Frame the problem

3. What it must do

1. A user can upload, download, move, rename and delete files and folders from any device.
2. Changes on one device appear on the user's other devices automatically.
3. A user can share files and folders and restore earlier versions.

Not now:

• Co-editing inside documents: Design D19.
• Full-text search: an indexing job.
• Previews and thumbnails: a processing pipeline.

4. How well it must do it

5. Numbers that change the design

6. The nouns

• Block: hash → bytes; immutable.
• File version: a path and its ordered block list.
• Namespace: a user's root, or a shared folder.
• Journal entry: one change in a namespace, with a sequence number.
• Device cursor: the last journal position a device has applied.

blocks      sha256 → bytes in blob storage (+ reference counts for garbage collection)
files       (namespace_id, path) → current version
versions    version_id, namespace_id, path, size, block_list [sha256…], parent_version, author_device
journal     (namespace_id, seq) → change {path, version_id | deleted}
devices     device_id, user_id, cursor per namespace

7. The contract

REST for blocks and commits; a long-poll for change notifications. The user and device come from the session.

POST /v1/blocks/check {hashes: [...]}         → {missing: [...]}           (upload only these)
PUT  /v1/blocks/{hash}  (or a presigned URL)  → 201
POST /v1/files/commit {path, parent_rev, blocks: [...]} → {rev, seq} | 409 conflict
GET  /v1/changes?cursor=…                     → {entries: [...], cursor, has_more}
GET  /v1/changes/wait?cursor=…                → long-poll: returns when something changed
GET  /v1/blocks/{hash}                        → bytes (or a presigned URL)

8. First working version

A file API, a metadata database, blob storage, and clients that poll.

8.1 Saving a file

The client uploads the whole file to blob storage and records its path and storage key in the database.

8.2 Syncing other devices

Every 30 seconds each device asks for the list of files and their modification times, compares it with its own copy, and downloads whatever is newer.

8.3 Sharing and versions

A share row grants another user access to a folder; old storage keys are kept as versions.

9. Make it hold up

9.1 Blocks, hashes and uploading only what changed efficient · never lose data

Because a block's name is the hash of its contents, blocks are immutable: they can be cached forever, uploaded in parallel and retried safely, and a file version is just a list of hashes. Deduplicating across users has a privacy cost (an attacker can learn whether someone stores a known file by seeing whether its upload is skipped), so many services deduplicate only within one user's data or encrypt per user.

9.2 The metadata service is the heart never lose data · consistent

• Commit is the atomic step: a file only changes when the client commits a new block list, after all blocks are uploaded. A crash mid-upload leaves orphan blocks (garbage collected later), never a half-written file.
• A journal per namespace: every commit appends an entry with an increasing sequence number. A device asks "what changed after my cursor?" and gets an ordered list, far cheaper than comparing whole folder trees.
• Strongly consistent and transactional within a namespace (a relational database sharded by namespace works well). Shared folders are their own namespace, mounted into each member's tree.

9.3 Telling devices something changed fast propagation

Polling every device every few seconds wastes requests at this scale. Devices hold a long-poll (or a WebSocket, or a mobile push) to a notification service, which only says "namespace N has changes". The device then pulls the journal from its cursor. The notification carries no data, so a lost notification costs only a delay (the device also checks periodically), never correctness.

9.4 Conflicts never lose data

Each commit includes the version it was based on (parent_rev); a commit based on an old version fails with a conflict. File sync tools don't merge binary content: the usual answer is to keep both, saving the loser as "report (conflicted copy).docx", and let the person choose. Deletes need care too: a stale device must never delete a file that another device has since edited.

9.5 Durability, versions, sharing and garbage collection never lose data · private

• Blocks live in blob storage replicated or erasure-coded across zones (Module 10). Old versions keep their block lists, so history is cheap: unchanged blocks are shared.
• Deleting a file only removes it from the tree; blocks are garbage collected when no version references them, after the retention period. Count references carefully: a GC bug is a data-loss bug.
• Access checks happen in the metadata service (who can read a namespace); block downloads use short-lived presigned URLs. Encrypt at rest; consider client-side encryption for sensitive products, at the cost of server-side features.

9.6 The sync engine on the client never lose data · consistent

Most data-loss bugs live in the client: reconciling the local disk, the last synced state and the remote journal, through renames, moves, crashes and full disks. Keep a local database of the last synced tree, apply remote changes and local changes as explicit, testable steps, and test with randomised simulations of crashes and concurrent edits. Dropbox wrote publicly about rewriting its sync engine for exactly these reasons.

9.7 The whole design after the deep dives

One edit on a laptop reaching a phone through the final design.

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• Content-addressed blocks (dedup and resumability, at the cost of hashing and metadata per block).
• Commit-after-upload for atomicity.
• Per-namespace journals and cursors.
• Notifications without data.
• Conflicted copies instead of merging.
• Dedup within accounts for privacy.

What we'd watch:

• Change propagation time.
• Upload success and resume rates.
• Conflict rate.
• Journal lag per device.
• Storage per user after dedup.
• GC safety audits.

At 10×:

• More metadata shards.
• Split huge namespaces.
• Colder storage tiers.

Common mistakes:

• Whole-file uploads.
• Diffing whole folder listings.
• Writing files before all blocks exist.
• Last-writer-wins on conflicts.
• Cross-user dedup without thinking about privacy.
• Ignoring the client's sync logic.

13. Try it, check yourself, read more

This is a real 64 KB file, chunked in your browser two ways: fixed 4 KB blocks, and content-defined chunks (a gear rolling hash, average about 4 KB). Make an edit and see which chunks must be uploaded again (orange).