how it happened
- whoengineertired, late at night
- the gapproduction databasedeleted by accident
- thenbackupsnone had been tested
- the result~6 hours of data lostrestored from staging
what everyone assumed
GitLab had backups, the way every responsible team has backups. There was a nightly database dump to S3, a daily disk snapshot copied into staging, separate cloud disk snapshots, and a live replica standing by for failover. Four independent ways to get the data back is exactly what good practice looks like, and it is reasonable to believe that four safety nets mean you are covered.
what happened
The backups had been failing quietly for a long time before anyone reached for them. The nightly database dump ran with the wrong version of the tooling, so it produced an error and wrote nothing, and the failure email that should have raised the alarm was being rejected by the mail server and never arrived. The cloud disk snapshots had never been enabled for the database servers, on the reasonable assumption that the other methods were enough. The daily staging snapshot was working, but it was only ever meant to copy data into staging, not to bring production back. Each gap was small and sensible on the day it was made. Nobody owned the job of proving, end to end, that a restore actually worked, so for months the safety nets sat there looking like coverage.
step by step
- backups fail quietly for monthsThe dump writes nothing and the failure emails are rejected by the mail server. Disk snapshots were never turned on.the assumption breaks here
- a production database is deletedEngineers go looking for a copy.
- the S3 bucket is emptyNo recent dump anywhere.
- restored from a staging snapshot six hours oldTaken by hand, almost by chance. About 5,000 projects and 700 accounts' data lost.
how it surfaced
When the engineers went looking for the backups, the S3 bucket was empty and there was no recent dump anywhere. Of the methods on paper, the only usable copy was a staging snapshot taken by hand about six hours before the deletion, almost by chance. They restored from it. The data created in those final hours was lost for good, an estimated 5,000 projects, 5,000 comments, and roughly 700 user accounts. GitLab wrote the whole thing up in public, ran the recovery live, and put a single line at the center of the postmortem: of five backup and replication techniques deployed, none were working reliably or set up in the first place.
A backup that has never been restored is a hope.
The thing that protects you is not the copy existing somewhere, it is someone having proven it comes back.
picture it at a company like yours
The company here is made up. The other two paths are not: they happened, and each links to the story.
- in this storythe day it mattersSomeone deletes the main database by mistake late at night.what it costs youYou find the last good copy is from by luck, and recent work is gone for good.
- or it goes like thisit happenedCDK Global, 2024howYour main software vendor is hit and goes dark for two weeks.what leavessales, orders and invoicingwho ends up with itcompetitors, as customers go elsewhere
- or it goes like thisit happenedCode Spaces, 2014howBackups live where the attacker can delete them too.what leaveseverything, with nothing left to restorewho ends up with itthe business itself, which closes
sources
your turn
what you can do about it
this week
Pick your most important system and do a test restore into somewhere safe.
Your backups ran last night, and the dashboard is probably green, which is why nobody is thinking about them. The question the quiet months ask is the one GitLab could not answer at the moment they needed to: when did anyone last restore from them and watch the data come back. The GitLab gap was findable on any ordinary day by exactly that exercise, and it stayed invisible precisely because no one ran it.
go deeper
if you're not sure where to start
Run the offboarding checklist on one leaver. It covers every account and token to close when someone leaves.
open the checklist →