Start your day with intelligence. Get The OODA Daily Pulse.

Subscribe Sign In

The Crowdstrike/Microsoft Global IT Outage Debacle: Ongoing Impacts and Recent Updates

07/22/2024 | Written by: Daniel Pereira

In an update of our initial post (assessing the early onset of the Global IT outage) on Friday, 7/19/24 at 10 AM, included here is CISA’s formal response on Friday at1 p.m. EST (with updates from CISA through 7/21), an interesting quick take from Beijing on “why China was largely unaffected by Friday’s IT outage”, amongst other ongoing impacts and updates from CNBC, Wired, and Interos.

CISA Cybersecurity Alert

Widespread IT Outage Due to CrowdStrike Update

Release Date | July 19, 2024

Note: CISA will update this Alert with more information as it becomes available.

Update 9:45 a.m., EDT, July 21, 2024:

Microsoft released a recovery tool that uses a USB drive to boot and repair affected systems.
Microsoft also published a blog post that provides links to various remediation solutions and outlines their actions in response to the outage, which include working with CrowdStrike to expedite restoring services to disrupted systems.
In the blog post, Microsoft estimates the outage affected 8.5 million Windows devices. Microsoft notes that this number makes up less than one percent of all Windows machines.

Update 12:30 p.m., EDT, July 20, 2024:

CrowdStrike continues to provide updated guidance on yesterday’s widespread IT outage, including remediation steps for specific environments.
CrowdStrike released technical details that provide:
- A technical summary of the outage and the impact.
- Information on how the update to the CrowdStrike Falcon sensor configuration file, Channel File 291, caused the logic error that led to the outage.
- A discussion of the root cause analysis CrowdStrike is undertaking to determine how the logic error occurred.
Cyber threat actors continue to leverage the outage to conduct malicious activity, including phishing attempts. CISA continues to work closely with CrowdStrike and other private sector and government partners to actively monitor any emerging malicious activity.
- According to a new CrowdStrike blog, threat actors have been distributing a malicious ZIP archive file. This activity appears to be targeting Latin America-based CrowdStrike customers. The blog provides indicators of compromise and recommendations.

Update 7:30 p.m., EDT, July 19, 2024:

The CrowdStrike guidance is updated with additional guidance regarding impacts to specific environments, e.g., Azure, AWS.
For additional information:
Threat actors continue to use the widespread IT outage for phishing and other malicious activity. CISA urges organizations to ensure they have robust cybersecurity measures to protect their users, assets, and data against this activity.

CISA continues to monitor the situation and will update this Alert to provide continued support.

Initial Alert (11:30 a.m., EDT, July 19, 2024):

CISA is aware of the widespread outage affecting Microsoft Windows hosts due to an issue with a recent CrowdStrike update and is working closely with CrowdStrike and federal, state, local, tribal and territorial (SLTT) partners, as well as critical infrastructure and international partners to assess impacts and support remediation efforts. CrowdStrike has confirmed the outage:

Impacts Windows 10 and later systems.
Does not impact Mac and Linux hosts.
Is due to the CrowdStrike Falcon content update and not to malicious cyber activity.

According to CrowdStrike, the issue has been identified, isolated and a fix has been deployed. CrowdStrike customer organizations should reference CrowdStrike guidance and their customer portal to resolve the issue.

Of note, CISA has observed threat actors taking advantage of this incident for phishing and other malicious activity. CISA urges organizations and individuals to remain vigilant and only follow instructions from legitimate sources. CISA recommends organizations to remind their employees to avoid clicking on phishing emails or suspicious links.

Here’s why China was largely unaffected by Friday’s IT outage

“Thank you Microsoft, [I can] take off early” ranked second on Chinese social media platform Weibo when the outages began

CNBC’s Evelyn Chang reports from Beijing:

While businesses in the U.S. and Europe woke up Friday to a global IT outage that disrupted airports and hotels, China went into its weekend largely unaffected.
“The impact of Friday’s CrowdStrike incident on China was very small, with almost no impact on domestic public life,” Gao Feng, senior research director at Gartner, said in Chinese, translated by CNBC. “Only some foreign companies in China were affected.”
“This is partly because many of the security threats that CrowdStrike is designed to protect against originate from China,” said Rich Bishop, CEO of AppInChina, which publishes international software in China.

While businesses in the U.S. and Europe woke up Friday to a global IT outage that disrupted airports and hotels, China went into its weekend largely unaffected. Anecdotally, ride-hailing, e-commerce and other internet-connected systems in China were all running smoothly on Friday. Chinese state media also said Friday evening that international flights at Beijing’s two airports were running normally, and that Air China, China Eastern Airlines and China Southern Airlines had not been affected by large-scale technical system failures.

One of the most notable impacts of the IT outage — including in China — was on Microsoft Windows devices attempting to integrate an update of CrowdStrike’s Falcon product, resulting in a blue screen and a cycle of computer restarts. Microsoft products are widely used in China — Windows had about 87% of personal computer shipments in the mainland last year, according to Canalys. That’s higher than the 79% share for the rest of the world in the first quarter of this year, the research firm said. A hashtag “Thank you Microsoft, [I can] take off early” ranked second on Chinese social media platform Weibo when the outages began to escalate early Friday afternoon local time. Posts generally showed photos of the “blue screen of death” or discussed the global outage.

But the hashtag’s popularity soon gave way to others about domestic matters, including Chinese smartphone company Xiaomi’s product launch in Beijing that evening. Microsoft products Office 365 and Azure cloud are operated in China by a local company called 21Vianet. It was not immediately clear whether localization contributed to the limited impact on Friday. The two companies did not immediately respond to CNBC requests for comment.

Why don’t Chinese companies use CrowdStrike?

The U.S. and Chinese governments have in recent years pushed domestic companies to use homegrown technology and store data locally out of national security concerns. Canalys pointed out that China-made UOS, or Unity Operating System, has growing adoption among state-owned enterprises and government sectors, although Windows still dominates the domestic personal computer market.

“There’s been very little impact because CrowdStrike is barely used in China,” said Rich Bishop, CEO of AppInChina, which publishes international software in China..adding that Chinese companies typically use products from Tencent 360 and other businesses. “This is partly because many of the security threats that CrowdStrike is designed to protect against originate from China,” said Rich Bishop, CEO of AppInChina, which publishes international software in China. CrowdStrike said in its latest annual cyber threat report that last year, “China-nexus adversaries continued to operate at an unmatched pace across the global landscape, leveraging stealth and scale to collect targeted group surveillance data, strategic intelligence, and intellectual property.”

Interos Analyzes Supply Chain Impacts of Massive CrowdStrike Outage

Interos, the AI supply chain risk intelligence company, today released a comprehensive analysis of the CrowdStrike outage on enterprise customers, revealing the incident’s far-reaching consequences on international trade and business operational ecosystems.

The data shows the impact extends far beyond CrowdStrike’s and Microsoft’s immediate enterprise customers, potentially affecting millions of additional organizations who rely on Microsoft’s O365 software. The outage involved a CrowdStrike update which the company subsequently resolved.

Key Findings:

The outage impacted 674,620 direct (tier-1) enterprise customers of either Microsoft or CrowdStrike
41% of affected entities were in the U.S., with significant impact across Europe
When analyzing extended supply chain relationships (tier-3), this figure expands to over 49 million additional customer relationships at risk for potential operational disruptions
Major closures and delays occurred at ports and air freight hubs worldwide, including temporary shutdowns at ports from New York to Rotterdam
Air freight was severely affected, with thousands of flights grounded or delayed at major hubs in Europe, Asia, and North America
The outage exacerbates existing supply chain challenges amid rising global demand and freight costs

The analysis highlights the vulnerability of interconnected global supply chains and the potential long-term economic implications. Analysts are concerned it may be weeks before airlines and freight companies are fully back in service. “This incident is a stark reminder of the fragility of our interconnected global economy,” said Ted Krantz, CEO of Interos. “Our analysis demonstrates the critical need for anticipation and speed in supply chain risk management. Considering the scale of this incident, organizations must be extra vigilant as bad actors may have taken the opportunity to access secure systems over the last 24 hours, meaning this single incident may evolve into a new series of vulnerabilities weeks or months from now.”

The report also details the extensive industry ripple effect beyond technology and airlines – with multiple manufacturing sectors, including electronics and semiconductor production, and professional services, experiencing disruptions. Additionally, the widespread use of the affected software by U.S. state and local governments raises concerns about potential impacts on public services and cybersecurity. Interos’ data shows ongoing supply chain disruptions cost enterprises $100 million in annual losses on average. The company’s critical risk intelligence platform helps companies mitigate the financial impacts of multi-tier risks by continuously mapping and monitoring extended supply chains at speed and scale.

View the full report HERE. For more information about the outage impact analysis or to learn how Interos can help protect your supply chain, visit www.interos.ai.

Updated @ 6:25 AM, Monday, 7/22/24: Delta cancels hundreds more flights in struggle to recover from Microsoft outage

The Atlanta-based airline canceled more than 600 Sunday mainline flights, about 17% of its schedule and more than any other U.S. airline.
The disruptions have persisted at Delta while most other carriers have recovered.

Delta Air Lines CEO Ed Bastian apologized and offered frequent flyer miles to travelers for hundreds of flight cancellations as the carrier struggled to recover from Friday’s globe-spanning IT outage, disruptions that sparked criticism from Transportation Secretary Pete Buttigieg. The Atlanta-based airline canceled close to 1,400 mainline flights on Sunday, more than a third of its schedule, according to FlightAware, more than any other U.S. airline. More than 1,600 Delta flights were delayed. As of early Monday, Delta had already canceled another 550 flights, or 15% of its mainline operation.

The delays and cancellations are putting Delta in a rare spotlight for the carrier whose leaders pride themselves on reliability and punctuality. “We continue to receive reports of unacceptable disruptions and customer service conditions at Delta Air Lines, including hundreds of complaints filed with our Department,” Buttigieg said in an emailed statement late Sunday. “I have made clear to Delta that we expect the airline to provide prompt refunds” to customers who chose to call off their trips because of the disruptions as well as “timely reimbursements for food and overnight hotel stays to consumers affected by the delays and cancellations, as well as adequate customer service assistance to all of their passengers.”

The disruptions have persisted at Delta while most other carriers have recovered. American Airlines said it was almost back to normal by Saturday. “I want to apologize to every one of you who have been impacted by these events,” Bastian said in a message to customers. “Delta is in the business of connecting the world, and we understand how difficult it can be when your travels are disrupted.” The airline was offering flight attendants extra pay to pick up shifts, a staff memo on Sunday said. The carrier called some of them on their personal phones to come in, according to a person familiar with the matter. High demand during some one of the busiest periods of summer challenged the airline to find alternative flights for affected travelers, Bastian said in his note.

United Airlines also had elevated flight disruptions on Sunday with 9% of its schedule canceled, or 260 flights, according to FlightAware, but still below Delta’s. Delta Air Lines has a number of Microsoft tools that were impacted in the outage, “in particular one of our crew tracking-related tools was affected and unable to effectively process the unprecedented number of changes triggered by the system shutdown,” Bastian said in his note. That would make the event similar to an issue Southwest Airlines suffered, on a much greater scale, at the end of 2022 when it failed to recover from severe winter weather for days.

How One Bad CrowdStrike Update Crashed the World’s Computers

A defective CrowdStrike kernel driver sent computers around the globe into a reboot death spiral, taking down air travel, hospitals, banks, and more with it. Here’s how that’s possible.

A quick excerpt from this article recommended by OODA Network member Dr. Bilyana Lilly (above):

Only a handful of times in history has a single piece of code managed to instantly wreck computer systems worldwide: The Slammer worm of 2003. Russia’s Ukraine-targeted NotPetya cyberattack. North Korea’s self-spreading ransomware WannaCry. But the ongoing digital catastrophe that rocked the internet and IT infrastructure worldwide over the last 12 hours appears to have been triggered not by malicious code released by hackers, but by the software designed to stop them.

Two internet infrastructure disasters collided on Friday to produce disruptions around the world in airports, train systems, banks, healthcare organizations, hotels, television stations, and more. On Thursday night, Microsoft’s cloud platform Azure experienced a widespread outage. By Friday morning, the situation turned into a perfect storm when the security firm CrowdStrike released a flawed software update that sent Windows computers into a catastrophic reboot spiral. A Microsoft spokesperson tells WIRED that the two IT failures are unrelated.

The cause of one of those two disasters, at least, has become clear: Buggy code pushed out as an update to CrowdStrike’s Falcon monitoring product, essentially an antivirus platform that runs with deep system access on “endpoints” like laptops, servers, and routers to detect malware and suspicious activity that could indicate compromise.”Falcon requires permission to update itself automatically and regularly, since CrowdStrike is constantly adding detections to the system to defend against new and evolving threats. The downside of this arrangement, though, is the risk that this system, which is meant to enhance security and stability, could end up undermining it instead.

“This is Not a Security Incident or Cyberattack”: Microsoft and Crowdstrike Scramble to Patch ‘Largest IT Outage in History’

At approximately 3 AM EST, reports started crossing the transom of a global IT outage impacting a broad range of industries, causing airlines, banks, media broadcasters, and shipping lines to shut down operations. Boston’s Logan Airport was shut down this morning, Washington D.C.’s Metrorail has been impacted, and planes were grounded at many airports around the world. This post is a quick and dirty tick-tock of the incident and the response from Microsoft and Crowdstrike. For CISOs in mitigation mode, we have compiled some technical links here as well.

Additional OODA Loop Resources: The Uptick in Global IT Supply Chain Breaches (Frequency and Specific Targeting)

This all comes fast on the heels of the eerily prescient discussion in the June 2024 OODA Network Monthly Meeting on The Uptick in Global IT Supply Chain Breaches (Frequency and Specific Targeting):

At the June 2024 OODA Network Member Meeting – held on Friday, June 21, 2024 – the network discussed The Uptick in Global IT Supply Chain Breaches (Frequency and Specific Targeting), amongst other topics.

The central discussion at the June 2024 OODA Network Monthly Meeting revolved around the increasing frequency and specific targeting of supply chain breaches, with concerns raised about the rising risk associated with these attacks. Participants highlighted the supply chain as a major target for cyberattacks and emphasized the importance of addressing vulnerabilities in the supply chain to mitigate risks. The discussion also touched on the significance of supply chain attacks as a means to exploit systems beyond just ransomware, referencing previous notable incidents like Log4J and SolarWinds. The meeting emphasized the significance of supply chain security, with one participant noting that supply chains are among the most targeted in the world, underscoring the evolving threat landscape and the need for robust defenses to combat the growing menace of supply chain attacks.

Topics and themes discussed by the OODA Network which apply to this global IT outage include:

A CISO Playbook: For the CISOs on the call or in the larger OODA Network, an emphasis was put on the need for transparency or “covering oneself” to avoid a perception or narrative that the entire onus of a supply chain breach falls internally on a CISO (and not a provider or vendor or XaaS platform).
The significance of vendor reporting and the evolving landscape of community defense in dealing with supply chain vulnerabilities.
“What does outsourcing really mean? What does offshoring really mean? What does a vendor agreement mean? Are the survey-based approaches that we’ve used in the past for doing due diligence and counterparty risk still valid?”
The lack of effective third-party vendor and software security measures and tools for monitoring networks: A speaker on the call pointed out the strategic nature of such attacks, indicating that they serve as a pathway to achieving larger objectives beyond mere ransomware incidents.
Why the market doesn’t react to this increased IT supply chain activity?
- One member highlighted that such cyber incidents are not perceived as material, thus not impacting stock prices significantly.
- He also cautioned against taking advice from politicians on private sector matters, emphasizing the market’s autonomy – with insights into the market’s behavior and the significance of cybersecurity in the evolving digital landscape.
- He highlighted the challenge of determining the threshold for an event to influence the market.
The discussion ended with a network member reinforcing the importance of documentation and raising concerns about the lack of focus on remediation and basic controls in cybersecurity incidents – emphasizing the need for companies to remember and address these issues to differentiate between survival and failure in such incidents.

https://oodaloop.com/archive/2024/07/18/the-june-2024-ooda-network-monthly-meeting-the-uptick-in-global-it-supply-chain-breaches-frequency-and-specific-targeting/

About the Author

Daniel Pereira

Daniel Pereira is research director at OODA. He is a foresight strategist, creative technologist, and an information communication technology (ICT) and digital media researcher with 20+ years of experience directing public/private partnerships and strategic innovation initiatives.