In the early hours of Friday 19 July, the blue screen of death (BSOD) appeared on millions of machines around the world: because of a faulty update to CrowdStrike’s endpoint protection software, Falcon, Windows machines crashed, and kept crashing again after every restart. Airlines, hospitals, banks, TV channels and shops came to a standstill. Microsoft estimates that 8.5 million Windows devices were affected, which is less than one percent of all Windows machines – and yet this became one of the largest IT outages we have seen so far.
A week has passed, most machines are working again, and CrowdStrike has published its preliminary post-incident review. In this article we summarise what happened, and what every IT operations team can learn from it – even if they don’t use CrowdStrike.
What happened?
According to CrowdStrike, at 04:09 (UTC, 06:09 Hungarian time) on 19 July they released a so-called content update for the Windows Falcon agent. This was not a software update but a configuration file (“Channel File 291”, named C-00000291*.sys on disk), which gives the agent information about new attack patterns. Because of the file’s faulty content, the part of the agent running in the kernel read an invalid memory area, and this crashed the entire system.
CrowdStrike withdrew the update at 05:27 (UTC), but by then every machine running agent version 7.11 or later that was online during those 78 minutes had received it. Linux and macOS machines were not affected by the bug.
According to the preliminary review of 24 July:
- content updates like this are checked by an automatic validator (Content Validator) before release, and a bug in it let the faulty file through;
- content updates went out to all customers at once, and the version delay settings that customers can configure (which let you run an earlier version of the agent, for example) did not apply to them;
- for the future, CrowdStrike promises staggered releases (first to a small “canary” group, then gradually to everyone), more testing, better error handling, and that customers will be able to decide for themselves when and where content updates are installed.
The detailed root cause analysis has not been published yet.
Why did recovery take so long?
The faulty file was withdrawn within minutes, yet machines were being repaired for days. The reason is simple: you can’t push an update remotely to a machine that crashes while it is booting. The fix was manual work, machine by machine:
- Boot the machine in Safe Mode or in the Windows Recovery Environment (WinRE).
-
Delete the faulty file:
del C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys - Restart.
On an office PC this takes ten minutes, but if the machine’s disk is encrypted with BitLocker, you can only access it with the recovery key. In many places the keys were kept in a system that was itself down. Machines in a server room or a remote data centre needed console access, and for Windows machines in the cloud the system disk had to be detached, attached to another machine, the file deleted there, and the disk then reattached – AWS, Microsoft Azure and Google Cloud all published step-by-step guides for this. Microsoft also released a recovery tool that boots from a USB stick.
Lessons for IT operations
1. Staggered releases – for your own updates too
CrowdStrike’s mistake was that one change reached everywhere at once. The same risk exists in your own environment: a Windows update, a new firewall rule or a configuration change that goes out to every machine at the same time. Split your machines into groups (test, a few less critical production machines, then the rest), and leave time between the groups for problems to surface. And when you buy software, ask the vendor: how does it release its updates, and how much say do you have?
2. Access to the machines when the network can’t help
Where working console access was available, the fix took minutes. For physical servers this means IPMI, iDRAC or iLO; for virtual machines, the hypervisor console (for example, the Proxmox VE web console); in the cloud, the serial console. Check that you have this kind of access to every critical machine, that it works, and that you know its password.
3. Keep the BitLocker keys accessible
Store the recovery keys so that you can reach them even when most of your Windows environment is down: for example, in a password manager that doesn’t run on the same infrastructure, or, for the most critical servers, even printed out in a safe. You can query a machine’s key ID and recovery password like this:
manage-bde -protectors -get C:
4. Backups and snapshots, with tested restores
If you had snapshots of your virtual machines from before 04:09 on 19 July, rolling back to them was enough. This needs regular snapshots or backups, and a tested procedure for restoring many machines at once. It’s also worth deciding the restore order in advance: domain controllers and core network services first, business applications after that.
5. Don’t run everything on the same platform
Where the backup server, the administrator workstations and the password manager all ran on the same Windows environment with the same endpoint protection, the recovery tools went down too. The systems you need for recovery – backups, console access, documentation, passwords – should also be available on an independent platform; a Linux backup server or a backup stored at a separate site is worth a lot in exactly this situation.
6. Know quickly what is down
In the first hours of the outage, many people didn’t even know exactly which of their machines were affected. Working monitoring, for example a Zabbix system, shows within minutes which machines are not responding, and you can prioritise based on that. The status page for your customers and your internal communication shouldn’t run on the infrastructure that has just gone down either.
7. Account for supplier risk
Software running at kernel level (endpoint protection, backup agents, encryption, monitoring agents) has access to the most sensitive part of the system, and a single bug in it can take the whole machine down. March’s xz backdoor showed the same thing from another angle: what you get from your suppliers is your risk too. You don’t have to give them up, but you need to know what runs on your machines, and what happens if one of them fails.
What should you do now?
If your machines have already recovered, it’s worth sitting down in the next few days and thinking through what worked and what didn’t. Where did you wait for BitLocker keys? Where was there no console access? How long did it take to find out what was down? The answers become a short, written recovery plan that will save hours in the next outage – which won’t necessarily be caused by CrowdStrike. We wrote about the business impact of outages in an earlier article.
We help with designing recovery plans, backup and console access solutions, and with running servers; you can read about this on the Infrastructure and operations page.