GitLab's 2017 database outage

On 31 January 2017, an engineer fighting a replication problem on GitLab.com deleted the data directory of the production database, believing they were on the secondary. That should have been a bad hour. Instead GitLab.com was down for about 18 hours and lost roughly six hours of data, because none of the backup methods in place could restore it. GitLab streamed the recovery live and published a detailed postmortem. Its root cause, in the postmortem's own words: nobody was responsible for testing the backups.

1. The situation

GitLab.com ran on a single PostgreSQL primary (db1) and one hot-standby secondary (db2), used only for failover. Git repositories and wikis were stored separately. The recovery procedures on paper:

2. What mattered, in hindsight

• Recovery point (RPO): how much data may be lost. There was no stated target; the result was about six hours.
• Recovery time (RTO): how long until service is back. The result was about 18 hours, mostly copying a snapshot over throttled disks.
• Operability: a manual, undocumented procedure to rebuild replication, run around 23:30 UTC under pressure. GitLab.com wasn't using WAL archiving, so the secondary had to be re-synchronised by hand.

3. The incident, step by step

1. Database load rose sharply: a spike in spam and, at the same time, a process removing a GitLab employee's account and data after a troll's abuse report.
2. Replication stopped: the primary had already removed write-ahead log segments the secondary still needed.
3. Rebuilding the secondary needs an empty data directory. An engineer ran the deletion on the primary instead. They stopped it within a second or two, but about 300 GB were gone.
4. Every backup method was tried and failed, as in the table. The six-hour-old LVM snapshot on staging was the best copy left.
5. Copying it from staging to production took about 18 hours: staging used cheaper network disks throttled to around 60 Mbps.

4. The decision

During the incident there was one real choice: restore from the LVM snapshot taken six hours earlier, or from the daily one, losing almost 24 hours. GitLab chose the six-hour snapshot. Afterwards, the postmortem's actions amount to a new decision about data durability:

ADR: Ownership and testing of database recovery

Context: Jan 31 outage: ~6 h of data lost, ~18 h down; four recovery methods
  existed and none worked when needed. No owner for backup testing.
Decision:
  - Assign an owner for data durability
  - Monitor backups (Prometheus), not just email on failure
  - Hourly LVM snapshots and Azure disk snapshots of production databases
  - Investigate point-in-time recovery with continuous WAL archiving
  - Automatically test restoring backups
  - Recover replicas; improve replication runbooks
  - Make shell prompts clearly show the host and environment
Consequences:
  + Recovery paths are exercised, so failures surface before they're needed
  - Ongoing cost of storage, restore tests and an owner's time

5. How it works

The state of every copy of the data, from the load spike to the restore.

6. Try your own backups

Choose the backup methods a database has and the kind of failure, and see what you can actually recover. "Tested and monitored" decides whether you can trust any of them.

Simplified model for a database of about 300 GB. A synchronous replica copies every change at once, including a mistaken deletion. Recovery times are typical orders of magnitude, not measurements. Off-site means a copy in another region.

7. What happened

The database was restored to its state at 17:20 UTC on 31 January, and GitLab.com was fully back around 18:00 UTC on 1 February. Changes between about 17:20 and midnight were lost: GitLab estimated at least 5,000 projects, 5,000 comments and about 700 new user accounts. Repositories and wikis were not affected, nor were self-managed GitLab installations. GitLab kept a public document of its progress and streamed the recovery on YouTube, at one point to around 5,000 viewers. It used "five whys" to find root causes, and committed to redacting names in future postmortems.

8. Lessons

• A backup you haven't restored is a hope. All four methods existed on paper. Recovery is a characteristic; test it like one, with automated restores (Module 26).
• Replicas are not backups. A replica protects against losing a machine, not against a mistake, which it replicates (DR plan template).
• Failure alerts need a channel that works. An alert sent by email that is rejected is the same as no alert. Monitor the backup's result, not the job's exit.
• Ownership is architecture. The deepest "why" was that nobody owned testing the backups. Name an owner in the DR plan (Module 09a).
• RTO depends on the slowest copy. Saving on staging disks turned a restore into an 18-hour copy. Know how long a restore takes, from where.
• Design the dangerous operation. Clear prompts per host, runbooks for rebuilding replicas, and fewer manual steps under pressure (Module 22).

9. Your turn

1. List every copy (10 min). For your main database: every backup, snapshot and replica, where it lives, and when someone last restored from it.
2. Fill in the DR plan (15 min). Use the template: RTO and RPO per service, the disasters, and who owns the restore test.
3. Find the silent failure (5 min). How would you find out today that last night's backup failed?

10. Check yourself

11. Sources and further reading