Start your day with intelligence. Get The OODA Daily Pulse.

Home > Analysis > The Crowdstrike Incident – OODA Loop Update #4

In the spirit of the significance of tracking the global impact of disruptive events and encouraging the sharing of relevant stories for compilation, the following is our latest  tracking of the Crowdstrike Incident since our last update on 7/22 – The Crowdstrike/Microsoft Global IT Outage Debacle: Ongoing Impacts and Recent Updates and the July 2024 OODA Network Monthly Meeting: A Real-time Discussion of the Crowdstrike Global IT Outage.

The Crowdstrike Incident – OODA Loop Update #4

“Sadly, it was an interesting lesson for the bad guys. [They learned] It was one mechanism that started the entire process.”

CrowdStrike-Sponsored Mercedes F1 Team Hit by IT Outage – Business Insider

  • The Mercedes F1 team was briefly impacted by the CrowdStrike IT outage.
  • CrowdStrike is also their sponsor.
  • Mercedes implemented fixes, and operations at the track proceeded “seamlessly,” a spokesperson said.

Cybercriminals quick to exploit CrowdStrike chaos • The Register

Who loves a global outage? Phishers, fraudsters and all manner of creeps

Well that was fast. Criminals didn’t waste any time taking advantage of the CrowdStrike-Microsoft chaos and quickly got to work phishing organizations and spinning up malicious domains purporting to be fixes.  Just hours after a faulty CrowdStrike file shut down Windows machines around the globe, reports surfaced of scam emails using the outage as a lure and otherwise trying to use the massive outage as a means to pursue criminal activities.

“Some reports we have seen indicate that there may be phishing emails circulating claiming to come from ‘CrowdStrike Support’ or “CrowdStrike Security,” said Johannes Ullrich, dean of research for SANS Technology Institute and the founder of the Internet Storm Center. While he did not have any samples to share at the time, “attackers are likely leveraging the heavy media attention,” Ullrich added. “Please be careful with any ‘patches’ that may be delivered this way.”m  ICS also listed one domain that is “possibly” linked to these phishing attacks:

crowdfalcon-immed-update [ .] com

Other phony domains posing as fixing sites surfaced on social media, with security researchers warning users not to pay for a fix — there’s free support from the real CrowdStrike — as some of the fraudulent websites asked for bitcoin and PayPal “donations.”  Additionally, while CrowdStrike CEO George Kurtz, in a statement on X, assured customers “this is not a security incident or cyberattack,” the software flaw does make it that much easier for network intruders to sneak in while system admins work to implement the fix.

Falcon Content Update Remediation and Guidance Hub | CrowdStrike

Channel File 291 RCA Exec Summary

This document provides an executive summary of the findings of CrowdStrike’s Root Cause Analysis (RCA) report. The full report elaborates on the information previously shared in our preliminary Post Incident Review (PIR), providing further depth on the findings, mitigations, technical details and root cause analysis of the incident.

Download the Root Cause Analysis PDF

Introduction

CrowdStrike was founded with a mission to protect customers against today’s adversaries and stop breaches. On July 19, 2024, as part of regular operations, CrowdStrike released a content configuration update (via channel files) for the Windows sensor that resulted in a system crash. We apologize unreservedly.

We acknowledge the incredible round-the-clock efforts of our customers and partners who, working alongside our teams, mobilized immediately to restore systems and bring many back online within hours. As of July 29, 2024, at 8:00 p.m. EDT, ~99% of Windows sensors were online, compared to before the content update. We typically see a variance of ~1% week-over-week in sensor connections. To any customers still affected, please know we will not rest until all systems are restored.

What Happened

The CrowdStrike Falcon sensor delivers AI and machine learning to protect customer systems by identifying and remediating the latest advanced threats. In February 2024, CrowdStrike introduced a new sensor capability to enable visibility into possible novel attack techniques that may abuse certain Windows mechanisms. This capability pre-defined a set of fields for Rapid Response Content to gather data. As outlined in the RCA, this new sensor capability was developed and tested according to our standard software development processes.

On March 5, 2024, following a successful stress test, the first Rapid Response Content for Channel File 291 was released to production as part of a content configuration update, with three additional Rapid Response updates deployed between April 8, 2024 and April 24, 2024. These performed as expected in production.

On July 19, 2024, a Rapid Response Content update was delivered to certain Windows hosts, evolving the new capability first released in February 2024. The sensor expected 20 input fields, while the update provided 21 input fields. In this instance, the mismatch resulted in an out-of-bounds memory read, causing a system crash. Our analysis, together with a third-party review, confirmed this bug is not exploitable by a threat actor.

While this scenario with Channel File 291 is now incapable of recurring, it informs the process improvements and mitigation steps that CrowdStrike is deploying to ensure further enhanced resilience.

See also:

CrowdStrike Outage Losses Estimated at a Staggering $5.4B

Researchers track the healthcare sector as experiencing the biggest financial losses, with banking and transportation following close behind.

As the CrowdStrike Falcon outage story continues to unfold, the monetary losses to businesses from the global incident continue to rise: The volume is likely to reach $5.4 billion in costs for Fortune 500 companies, according to a report from Parametrix.  Parametrix researchers have found that roughly 25% of Fortune 500 companies experienced disruptions due to the incident, the most heavily affected industries financially being healthcare ($1.94 billion in estimated losses) and banking ($1.15 billion). In addition, a shocking 100% of the transportation and airlines sector was affected, and the group will rack up an estimated $0.86 billion in losses, according to the forecast. The $5.4 billion estimate excludes Microsoft.  The researchers noted that the outage impact to some industries, like software and IT-related services, is more likely to cause a “ripple effect beyond Fortune 500 companies,” though hard numbers were not quantified in the report.

SEC Reporting Implications for Publicly Traded Companies Impacted by CrowdStrike Defective Software Update // Cooley // Global Law Firm

There are a number of US Securities and Exchange Commission (SEC) reporting implications arising from the server-related outages caused by CrowdStrike’s defective software update on July 19, 2024, and their impacts on public companies, particularly in light of the SEC’s new cybersecurity disclosure rules. While the situation on the ground – as well as answers to these questions – is still very much evolving, public companies impacted by the CrowdStrike update should consider doing the following:

  • Ensure compliance with applicable policies and perform assessments to determine whether any impact from the CrowdStrike update is “material,” and whether any reporting is necessary or advisable.
  • Perform risk assessments and gap analyses to determine whether there are any shortcomings in systems and systems-related matters, including use of third parties and relevant oversight, monitoring, disaster recovery, and other practices.
  • Update risk factors and other disclosures, including regarding systems downtime and/or reliance on third parties to operate critical business systems.
  • Determine if the CrowdStrike update has had or is expected to have a material impact on the company, then consider if it should be discussed in the management’s discussion and analysis (MD&A) section of SEC filings, including as a known trend for future periods.
  • Be mindful of Regulation FD when communicating with analysts and investors regarding the impact of the CrowdStrike update on the company.
  • Evaluate whether the CrowdStrike update has implications for the company’s internal controls and disclosure controls and procedures.

Easterly: Potential Chinese cyberattack could unfold like CrowdStrike error | CyberScoop

CISA director calls CrowdStrike-linked outage a “dress rehearsal” for what China may have planned for U.S. critical infrastructure.

The faulty CrowdStrike Falcon update that caused millions of computers around the world to malfunction was “a useful exercise” for understanding what Chinese-linked cyber operations focused on sensitive U.S. networks could accomplish, a top U.S. cybersecurity official said Wednesday.   Jen Easterly, the director of the Cybersecurity and Infrastructure Security Agency, told a large crowd at the annual Black Hat cybersecurity conference that the fallout of the CrowdStrike situation — which disrupted medical care, canceled flights and shuttered retailers — showed what effects Chinese-linked activity tracked as Volt Typhoon could generate. 

“What was going through my mind was that, oh, this is exactly what China wants to do, but without rolling back the updates such that we could all reboot our systems,” Easterly said during a keynote address alongside top cybersecurity officials from the U.K. and Europe.   Volt Typhoon is the Microsoft-given name for suspected Chinese cyber activity targeting critical infrastructure organizations in the United States. Officials from the U.S. and other western countries have, for more than a year, warned that the Chinese-linked group aims to pre-position cyber capabilities in key networks to be able to disrupt operations in the event of military conflict or crisis involving China.

Low-level cybercriminals are pouncing on CrowdStrike-connected outage | CyberScoop

The malicious activity comes as CrowdStrike customers continue to recover from the July 18 outage.

Five days after a faulty update to CrowdStrike’s Falcon security software hobbled millions of Windows computers around the world, cybercriminals and hacktivist personas are taking advantage of the situation with newly registered domains, malware attached to files with CrowdStrike-themed names and at least one apparent instance of a data wiper.  CrowdStrike has documented multiple instances of likely criminal activity tied to the incident, including a Word document laced with the Daolpu information stealer and a zip file targeting Latin American-based CrowdStrike customers with the HijackLoader malware, which is typically used to deliver other malware packages, and a Python-based information stealer tracked as “Connecio.”

Additionally, a phishing email with a PDF purporting to explain how to remediate last week’s Falcon issue delivered a zip file laced with wiper malware, according to sandbox company ANY.RUN, which called it one of the most “sophisticated” outage-related attacks thus far. “Handala Hack,” a pro-Palestinian hacktivist persona known for attacking Israeli targets, claimed responsibility for the wiper attack mentioned by ANY.RUN. In a June 21 Telegram post, they asserted — without providing evidence — that they had targeted “thousands of Zionist organizations!”

Enrique Hernandez, threat research director at Splunk, said in a Tuesday post on X that he identified more than 2,000 CrowdStrike-related domains registered in the past seven days. An analysis of the top 25 suggests that “most of them are looking pretty funky,” Hernandez wrote.  James Spiteri, a director of product management with Elastic, wrote in a LinkedIn post Sunday that he had documented more than 141 certificates generated for what looks “like (mostly) bogus [CrowdStrike] domains. Hope this list helps folks keep a lookout for any phishing.” The list had grown to 193 by mid-afternoon Tuesday.

WSJ Coverage

CISA Director Easterly: Ode to an Outage | LinkedIn

With 36 hours of perspective, and readily acknowledging there is still much we need to learn about the event, I wanted to provide some personal thoughts on yesterday’s massive IT outage. While this was a technology incident, not a cyber-attack, in our role as National Coordinator for critical infrastructure security and resilience, CISA worked aggressively with Crowdstrike and partners across government and industry at all levels to understand the breadth of impacts to critical infrastructure and help drive remediation and risk mitigation. (Read More)

Black Hat Keynote: CrowdStrike outage a global wakeup call   | SC Media

The global impact of the flubbed CrowdStrike update and ensuing Microsoft outage was a wakeup call for European and U.S. cybersecurity leaders. The topic took center stage here at the Black Hat USA 2024 opening keynote.  Open questions included: How could a single vendor trigger such massive global disruptions, what does this portend for vital systems of democracy such as elections and how can the cybersecurity community ensure it doesn’t happen again?  “Sadly, it was an interesting lesson for the bad guys. [They learned] It was one mechanism that started the entire process,” said Hans de Vries, COO of the European Union Agency for Cybersecurity, commenting on the CrowdStrike bungled software update.

Joining de Vries on stage was Jen Easterly, director of the U.S. Cybersecurity and Infrastructure Security Agency, and Felicity Oswald OBE, CEO of the UK’s National Cyber Security Centre.  Easterly warned there has been a lot of “irresponsible noise on the CrowdStrike incident” — however, the gravity of the outage can’t be ignored. She outlined her three top takeaways from the CrowdStrike outage.  (Read More

Additional Coverage

“This is Not a Security Incident or Cyberattack”: Microsoft and Crowdstrike Scramble to Patch ‘Largest IT Outage in History’:  At approximately 3 AM EST on July 19th, reports started crossing the transom of a global IT outage impacting a broad range of industries, causing airlines, banks, media broadcasters, and shipping lines to shut down operations.  Boston’s Logan Airport was shut down this morning, Washington D.C.’s Metrorail has been impacted, and planes were grounded at many airports around the world. This post is a quick and dirty tick-tock of the incident and the response from Microsoft and Crowdstrike. For CISOs in mitigation mode, we have compiled some technical links here as well.

The Crowdstrike/Microsoft Global IT Outage Debacle: Ongoing Impacts and Recent Updates:  In an update of our initial post (assessing the early onset of the Global IT outage) on Friday, 7/19/24 at 10 AM,  included here is CISA’s formal response on Friday  at1 p.m. EST (with updates from CISA through 7/21), an interesting quick take from Beijing on “why China was largely unaffected by Friday’s IT outage”, amongst other ongoing impacts and updates from CNBC, Wired, and Interos.

The Botched Update Heard Around the World Calls for More Diversification:  Recently, a major IT outage impacted the global community.  In the midst of global geopolitical conflicts and states waging war on one another, the major disruption was not the result of hostile state cyber assets or enterprising nonstate hacktivist enclaves engaged in digital struggle.  This unprecedented glitch was caused by a faulty software update from CrowdStrike, a trusted U.S. cybersecurity vendor that has been a leader in the cybersecurity space for several years.  Per the company’s statement, a defect in a content update to its software for Windows hosts caused the disruption.  As a consequence, the company’s global customers across government services, emergency call centersairlines, hospitals, and banks, to name just a few, found themselves in a predicament, facing Microsoft’s dreaded “blue screen of death.”  In the United States alone, nearly 3,000 flights have been delayed or canceled, and some organizations are still trying to recover from the fallout.

https://oodaloop.com/archive/2024/07/31/the-july-2024-ooda-network-monthly-meetings-a-real-time-discussion-of-the-crowdstrike-global-it-outage/

Daniel Pereira

About the Author

Daniel Pereira

Daniel Pereira is research director at OODA. He is a foresight strategist, creative technologist, and an information communication technology (ICT) and digital media researcher with 20+ years of experience directing public/private partnerships and strategic innovation initiatives.