CrowdStrike blames Windows outage on bug in update

The global Microsoft system outage on Friday was caused by a bug in a software update pushed by CrowdStrike, the company said in a preliminary post incident review.

CrowdStrike said it released an update for the Falcon software’s Windows sensor last Friday to track new threat techniques, but the update caused a Windows system crash.

CrowdStrike, founded in 2011, is a US cybersecurity firm that provides software to companies around the world and across industries. It claims to be the world’s most advanced cloud-based security technology provider.

CrowdStrike’s Falcon software is used by businesses around the world to protect Windows machines against malware and security breaches. The software update on Friday caused a global outage of neatly 8.5 million Windows systems, affecting banks, airlines, hospitals, and government offices. 

Blue computer screens displaying error messages, called “blue screens of death,” flooded social media as the outage caused mayhem at airports and businesses. 

Systems that were affected included Windows hosts running sensor version 7.11 and above that were online between 04:09-05:27 UTC on 19 July and received the update, the firm said.

Mac and Linux hosts were not impacted.

The defect in the content update was reverted at 05:27 UTC. Systems coming online after this time, or that did not connect during the window, were not impacted, Crowdstrike said. 

What Happened?

Explaining the events of 19 July, CrowdStrike said the company delivers security content configuration updates to its sensors in two ways: sensor content that is shipped with its sensor directly, and rapid response content that is designed to respond to the changing threat landscape at operational speed.

The issue on Friday involved a rapid response content update with an undetected error.

Rapid response content is delivered as “template instances,” a set of fields that can be configured to match the desired behavior such as observe, detect, or prevent.

Template instances are created and configured through the use of the content configuration system, which includes the content validator that performs validation checks on the content before it is published.

Friday’s outage was caused due to a bug in the content validator, as one of the two template instances passed validation despite containing problematic content data, CrowdStrike said in its review. 

When received by the sensor and loaded into the content interpreter, problematic content resulted in an out-of-bounds memory read triggering an exception. This unexpected exception could not be gracefully handled, resulting in a Windows operating system crash, or “blue screen of death,” the cybersecurity firm said. 

In addition to this preliminary post incident review, CrowdStrike will also publicly release the full root cause analysis once the investigation is complete.