CrowdStrike Update Created Widespread Outage

As a recap, a widespread Microsoft Windows outage began on July 19 and expanded throughout the day due to a CrowdStrike content update for Microsoft Windows hosts. Pondurance and its systems were not affected by the issue, and Pondurance continued its security services delivery without incident.

What happened

During the outage, Windows computers experienced a critical error, often resulting in a blue screen of death, which indicates a system crash where the operating system can no longer operate safely. As Pondurance initially reported, and was confirmed by CrowdStrike, the incident was not a Microsoft issue; rather, it was a change due to the update from CrowdStrike. Only organizations using CrowdStrike endpoint detection and response (EDR) agents on Microsoft operating systems were impacted, and iOS and Unix systems were not impacted. SentinelOne, Microsoft Defender EDR, and other vendor clients also were not impacted.

Caution Advised

Pondurance advised caution following the incident, knowing that adversaries and hackers would take advantage of the outage situation in a couple of different ways:

• Phishing attack risk. Following the incident, phishing emails occurred in the wild, where the sender falsely claimed to be from CrowdStrike support. CrowdStrike advised that it will not send unsolicited communications to companies and any email offering assistance is malicious. Please warn employees that your IT group has taken action and no action is needed unless a designated contact from the company instructs them to do so.

• Recovered system. If recovered systems were brought back online in any other process, such as full recovery from backup, your team needs to make sure the system is patched and the EDR is working.

The Pondurance security operations center sent out notifications to clients and made calls to clients that were impacted. Pondurance created custom dashboards in the CrowdStrike console to assist in monitoring the recovery of systems and validation of the CrowdStrike agents working in protect mode. These dashboards are available to all Pondurance managed and monitored clients. Updates from CrowdStrike were tested and validated as they became available. We worked with clients to confirm operations for systems that could not be recovered and needed to be rebuilt.

The Pondurance incident response team responded to numerous inbound business email compromise cases, often coming to us from the cyber insurance community, involving organizations that clicked on fraudulent CrowdStrike phishing emails. Wide-scale impacts were remediated quickly with minimal organization and industry impact. We suspect that phishing attacks related to this broadly related topic will continue.

Steps taken and steps to take

CrowdStrike received reports of crashes on Windows hosts related to the Falcon Sensor, and symptoms included hosts experiencing a bug check or blue screen error. CrowdStrike engineering identified a content deployment related to this issue and reverted those changes. CrowdStrike further created manual and automated ways to help clients recover. These are broadly available, but we recommend using them on the CrowdStrike website directly.  

Falcon Content Update Remediation and Guidance Hub

Observations

The approach CrowdStrike takes — integrating more deeply with the Microsoft operating system than other EDRs — often provides security and operational benefits, but obviously, it has risk. While we believe there were oversights and failures in the CrowdStrike quality assurance process, and CrowdStrike will report on them with hindsight clarity, we believe CrowdStrike will learn from this. In addition, other organizations including Pondurance will take note and make improvements that will benefit the industry. Pondurance found CrowdStrike’s public transparency and frequent updates and also the communications with its partners, including Pondurance, to be commendable. The largest risk to CrowdStrike, and risk to working with the company, will be financial impacts that we will continue to monitor, as many in the industry are. Pondurance continues to consider CrowdStrike a strategic partner.