Start your day with intelligence. Get The OODA Daily Pulse.
In an update of our initial post (assessing the early onset of the Global IT outage) on Friday, 7/19/24 at 10 AM, included here is CISA’s formal response on Friday at1 p.m. EST (with updates from CISA through 7/21), an interesting quick take from Beijing on “why China was largely unaffected by Friday’s IT outage”, amongst other ongoing impacts and updates from CNBC, Wired, and Interos.
Note: CISA will update this Alert with more information as it becomes available.
Update 9:45 a.m., EDT, July 21, 2024:
Update 12:30 p.m., EDT, July 20, 2024:
Update 7:30 p.m., EDT, July 19, 2024:
CISA continues to monitor the situation and will update this Alert to provide continued support.
Initial Alert (11:30 a.m., EDT, July 19, 2024):
CISA is aware of the widespread outage affecting Microsoft Windows hosts due to an issue with a recent CrowdStrike update and is working closely with CrowdStrike and federal, state, local, tribal and territorial (SLTT) partners, as well as critical infrastructure and international partners to assess impacts and support remediation efforts. CrowdStrike has confirmed the outage:
According to CrowdStrike, the issue has been identified, isolated and a fix has been deployed. CrowdStrike customer organizations should reference CrowdStrike guidance and their customer portal to resolve the issue.
Of note, CISA has observed threat actors taking advantage of this incident for phishing and other malicious activity. CISA urges organizations and individuals to remain vigilant and only follow instructions from legitimate sources. CISA recommends organizations to remind their employees to avoid clicking on phishing emails or suspicious links.
“Thank you Microsoft, [I can] take off early” ranked second on Chinese social media platform Weibo when the outages began
CNBC’s Evelyn Chang reports from Beijing:
While businesses in the U.S. and Europe woke up Friday to a global IT outage that disrupted airports and hotels, China went into its weekend largely unaffected. Anecdotally, ride-hailing, e-commerce and other internet-connected systems in China were all running smoothly on Friday. Chinese state media also said Friday evening that international flights at Beijing’s two airports were running normally, and that Air China, China Eastern Airlines and China Southern Airlines had not been affected by large-scale technical system failures.
One of the most notable impacts of the IT outage — including in China — was on Microsoft Windows devices attempting to integrate an update of CrowdStrike’s Falcon product, resulting in a blue screen and a cycle of computer restarts. Microsoft products are widely used in China — Windows had about 87% of personal computer shipments in the mainland last year, according to Canalys. That’s higher than the 79% share for the rest of the world in the first quarter of this year, the research firm said. A hashtag “Thank you Microsoft, [I can] take off early” ranked second on Chinese social media platform Weibo when the outages began to escalate early Friday afternoon local time. Posts generally showed photos of the “blue screen of death” or discussed the global outage.
But the hashtag’s popularity soon gave way to others about domestic matters, including Chinese smartphone company Xiaomi’s product launch in Beijing that evening. Microsoft products Office 365 and Azure cloud are operated in China by a local company called 21Vianet. It was not immediately clear whether localization contributed to the limited impact on Friday. The two companies did not immediately respond to CNBC requests for comment.
The U.S. and Chinese governments have in recent years pushed domestic companies to use homegrown technology and store data locally out of national security concerns. Canalys pointed out that China-made UOS, or Unity Operating System, has growing adoption among state-owned enterprises and government sectors, although Windows still dominates the domestic personal computer market.
“There’s been very little impact because CrowdStrike is barely used in China,” said Rich Bishop, CEO of AppInChina, which publishes international software in China..adding that Chinese companies typically use products from Tencent 360 and other businesses. “This is partly because many of the security threats that CrowdStrike is designed to protect against originate from China,” said Rich Bishop, CEO of AppInChina, which publishes international software in China. CrowdStrike said in its latest annual cyber threat report that last year, “China-nexus adversaries continued to operate at an unmatched pace across the global landscape, leveraging stealth and scale to collect targeted group surveillance data, strategic intelligence, and intellectual property.”
Interos, the AI supply chain risk intelligence company, today released a comprehensive analysis of the CrowdStrike outage on enterprise customers, revealing the incident’s far-reaching consequences on international trade and business operational ecosystems.
The data shows the impact extends far beyond CrowdStrike’s and Microsoft’s immediate enterprise customers, potentially affecting millions of additional organizations who rely on Microsoft’s O365 software. The outage involved a CrowdStrike update which the company subsequently resolved.
Key Findings:
The analysis highlights the vulnerability of interconnected global supply chains and the potential long-term economic implications. Analysts are concerned it may be weeks before airlines and freight companies are fully back in service. “This incident is a stark reminder of the fragility of our interconnected global economy,” said Ted Krantz, CEO of Interos. “Our analysis demonstrates the critical need for anticipation and speed in supply chain risk management. Considering the scale of this incident, organizations must be extra vigilant as bad actors may have taken the opportunity to access secure systems over the last 24 hours, meaning this single incident may evolve into a new series of vulnerabilities weeks or months from now.”
The report also details the extensive industry ripple effect beyond technology and airlines – with multiple manufacturing sectors, including electronics and semiconductor production, and professional services, experiencing disruptions. Additionally, the widespread use of the affected software by U.S. state and local governments raises concerns about potential impacts on public services and cybersecurity. Interos’ data shows ongoing supply chain disruptions cost enterprises $100 million in annual losses on average. The company’s critical risk intelligence platform helps companies mitigate the financial impacts of multi-tier risks by continuously mapping and monitoring extended supply chains at speed and scale.
View the full report HERE. For more information about the outage impact analysis or to learn how Interos can help protect your supply chain, visit www.interos.ai.
Delta Air Lines CEO Ed Bastian apologized and offered frequent flyer miles to travelers for hundreds of flight cancellations as the carrier struggled to recover from Friday’s globe-spanning IT outage, disruptions that sparked criticism from Transportation Secretary Pete Buttigieg. The Atlanta-based airline canceled close to 1,400 mainline flights on Sunday, more than a third of its schedule, according to FlightAware, more than any other U.S. airline. More than 1,600 Delta flights were delayed. As of early Monday, Delta had already canceled another 550 flights, or 15% of its mainline operation.
The delays and cancellations are putting Delta in a rare spotlight for the carrier whose leaders pride themselves on reliability and punctuality. “We continue to receive reports of unacceptable disruptions and customer service conditions at Delta Air Lines, including hundreds of complaints filed with our Department,” Buttigieg said in an emailed statement late Sunday. “I have made clear to Delta that we expect the airline to provide prompt refunds” to customers who chose to call off their trips because of the disruptions as well as “timely reimbursements for food and overnight hotel stays to consumers affected by the delays and cancellations, as well as adequate customer service assistance to all of their passengers.”
The disruptions have persisted at Delta while most other carriers have recovered. American Airlines said it was almost back to normal by Saturday. “I want to apologize to every one of you who have been impacted by these events,” Bastian said in a message to customers. “Delta is in the business of connecting the world, and we understand how difficult it can be when your travels are disrupted.” The airline was offering flight attendants extra pay to pick up shifts, a staff memo on Sunday said. The carrier called some of them on their personal phones to come in, according to a person familiar with the matter. High demand during some one of the busiest periods of summer challenged the airline to find alternative flights for affected travelers, Bastian said in his note.
United Airlines also had elevated flight disruptions on Sunday with 9% of its schedule canceled, or 260 flights, according to FlightAware, but still below Delta’s. Delta Air Lines has a number of Microsoft tools that were impacted in the outage, “in particular one of our crew tracking-related tools was affected and unable to effectively process the unprecedented number of changes triggered by the system shutdown,” Bastian said in his note. That would make the event similar to an issue Southwest Airlines suffered, on a much greater scale, at the end of 2022 when it failed to recover from severe winter weather for days.
A defective CrowdStrike kernel driver sent computers around the globe into a reboot death spiral, taking down air travel, hospitals, banks, and more with it. Here’s how that’s possible.
A quick excerpt from this article recommended by OODA Network member Dr. Bilyana Lilly (above):
Only a handful of times in history has a single piece of code managed to instantly wreck computer systems worldwide: The Slammer worm of 2003. Russia’s Ukraine-targeted NotPetya cyberattack. North Korea’s self-spreading ransomware WannaCry. But the ongoing digital catastrophe that rocked the internet and IT infrastructure worldwide over the last 12 hours appears to have been triggered not by malicious code released by hackers, but by the software designed to stop them.
Two internet infrastructure disasters collided on Friday to produce disruptions around the world in airports, train systems, banks, healthcare organizations, hotels, television stations, and more. On Thursday night, Microsoft’s cloud platform Azure experienced a widespread outage. By Friday morning, the situation turned into a perfect storm when the security firm CrowdStrike released a flawed software update that sent Windows computers into a catastrophic reboot spiral. A Microsoft spokesperson tells WIRED that the two IT failures are unrelated.
The cause of one of those two disasters, at least, has become clear: Buggy code pushed out as an update to CrowdStrike’s Falcon monitoring product, essentially an antivirus platform that runs with deep system access on “endpoints” like laptops, servers, and routers to detect malware and suspicious activity that could indicate compromise.”Falcon requires permission to update itself automatically and regularly, since CrowdStrike is constantly adding detections to the system to defend against new and evolving threats. The downside of this arrangement, though, is the risk that this system, which is meant to enhance security and stability, could end up undermining it instead.
At approximately 3 AM EST, reports started crossing the transom of a global IT outage impacting a broad range of industries, causing airlines, banks, media broadcasters, and shipping lines to shut down operations. Boston’s Logan Airport was shut down this morning, Washington D.C.’s Metrorail has been impacted, and planes were grounded at many airports around the world. This post is a quick and dirty tick-tock of the incident and the response from Microsoft and Crowdstrike. For CISOs in mitigation mode, we have compiled some technical links here as well.
This all comes fast on the heels of the eerily prescient discussion in the June 2024 OODA Network Monthly Meeting on The Uptick in Global IT Supply Chain Breaches (Frequency and Specific Targeting):
At the June 2024 OODA Network Member Meeting – held on Friday, June 21, 2024 – the network discussed The Uptick in Global IT Supply Chain Breaches (Frequency and Specific Targeting), amongst other topics.
The central discussion at the June 2024 OODA Network Monthly Meeting revolved around the increasing frequency and specific targeting of supply chain breaches, with concerns raised about the rising risk associated with these attacks. Participants highlighted the supply chain as a major target for cyberattacks and emphasized the importance of addressing vulnerabilities in the supply chain to mitigate risks. The discussion also touched on the significance of supply chain attacks as a means to exploit systems beyond just ransomware, referencing previous notable incidents like Log4J and SolarWinds. The meeting emphasized the significance of supply chain security, with one participant noting that supply chains are among the most targeted in the world, underscoring the evolving threat landscape and the need for robust defenses to combat the growing menace of supply chain attacks.
Topics and themes discussed by the OODA Network which apply to this global IT outage include:
https://oodaloop.com/archive/2024/07/18/the-june-2024-ooda-network-monthly-meeting-the-uptick-in-global-it-supply-chain-breaches-frequency-and-specific-targeting/