how it happened
- whocontent updateone field more than expected
- the gapFalcon sensorread past its input
- the result8.5M Windows machinesflights, hospitals, broadcasts down
what everyone assumed
When a security tool updates itself automatically, that is the feature working as intended, not a risk to manage. The faster it patches, the better it defends, and pausing every update to test it first would defeat the speed you paid for. Trusting the vendor’s update channel is the reasonable posture, because the channel exists to keep you safe.
what happened
Here the quiet stretch was not months. It was the minutes between 04:09 and the first crash, the window in which a single content update propagated faster than anyone could read it. The Falcon sensor expected a defined set of input fields. The update delivered one more than expected, and the sensor read past the edge of what it could safely read. There was no opt-out, because the design assumption was that staying current is always safer than waiting.
step by step
- every sensor takes updates automaticallyThere is no opt-out: staying current is assumed to be safer.the assumption breaks here
- 04:09 UTCa content update ships with an extra fieldThe sensor reads past the edge of what it can safely read.
- about 8.5 million Windows machines crashFlights grounded, hospitals on paper, broadcasters dark.
- 05:27 UTCthe update is pulledMachines already down need hands-on recovery, one by one.
how it surfaced
Roughly 8.5 million Windows machines crashed worldwide and would not restart on their own. Flights were grounded, hospitals fell back to paper, broadcasters went dark. CrowdStrike remediated the update at 05:27 UTC, but machines already down often needed hands-on recovery one at a time. This was a faulty update with no attacker involved, and reinsurance estimates put insured losses near a billion dollars.
A trusted vendor’s automatic update is a change entering your environment without your review, and the trust you place in the channel is also the speed at which a single mistake reaches everything at once.
picture it at a company like yours
The company here is made up. The other two paths are not: they happened, and each links to the story.
- in this storythe day it mattersOne faulty update reaches all of them before anyone can react.what it costs youEvery computer in the office stops working at once, and each needs fixing by hand.
- or it goes like thisit happenedSolarWinds / Sunburst, 2020howA genuine, signed update from a certified vendor carries a backdoor.what leavesa quiet way into your networkwho ends up with ita state intelligence service
- or it goes like thisit happenedtj-actions/changed-files, 2025howA build helper everyone uses is hijacked and its versions repointed.what leavessecret keys, printed into build logswho ends up with itanyone reading public logs
sources
your turn
what you can do about it
this week
Find the auto-updating software that runs on the most machines you own and ask two plain questions: can it roll out in stages rather than everywhere at once, and if a bad update arrived, how long would recovery take per machine.
Something in your environment updates itself tonight without asking: an agent, a security tool, a dependency that pulls the latest version. The CrowdStrike lesson was not about one vendor. It was that "auto-update" and "change everywhere simultaneously" can be the same sentence.
go deeper
if you're not sure where to start
Run the offboarding checklist on one leaver. It covers every account and token to close when someone leaves.
open the checklist →