Registration now open for OODAcon 2024.

Home > Analysis > The City of Dallas, Over a Month After A Ransomware Attack, is Still not at Full Functionality

The City of Dallas, Over a Month After A Ransomware Attack, is Still not at Full Functionality

Amidst all the headlines and uncertainty, a story you do not want to miss is that of a major American municipality struggling to get back on its feet for an entire month after a ransomware attack in early May.  Following is the chronology of the cyber incident in Dallas – attributed to the Royal Ransomware Group – as reported by The Record and The City of Dallas. 

May 3, 2023

Official Website: City of Dallas (https://www.dallascitynews.net) 

City of Dallas statement on network outage

Update as of 5 p.m. May 3.

Wednesday morning, the City’s security monitoring tools notified our Security Operations Center (SOC) that a likely ransomware attack had been launched within our environment.

Subsequently, the City has confirmed that a number of servers have been compromised with ransomware, impacting several functional areas, including the Dallas Police Department Website. The City team, along with its vendors, are actively working to isolate the ransomware to prevent its spread, to remove the ransomware from infected servers, and to restore any services currently impacted. The Mayor and City Council was notified of the incident pursuant to the City’s Incident Response Plan (IRP).

The City is currently working to assess the complete impact, but at this time, the impact on the delivery of City services to its residents is limited. Should a resident experience a problem with a particular City service, they should contact 311. For emergencies, they should contact 911.

ITS will continue working around the clock to identify the source of the outage and isolate impacted devices. Departmental service to residents continues with minimal disruption as follows.

  • DPD & DFR: Service to residents is unaffected although the home page is offline.
  • 911: Calls continue to be received and dispatched
  • 311: Phone IVR is fully functional. 311 agents are accepting phone calls and dispatching Service Requests.
  • Courts: Court was closed Wednesday and will also be closed on Thursday. All cases will be reset and notices will be sent by mail.
  • DWU: Payment may be processed via IVR. Online payment processing may be delayed but no disconnections will be processed until the outage is resolved.

Currently, less than 200 of the City’s thousands of devices are impacted, but if any City device is at risk, it will be quarantined and blocked by ITS. For compromised machines, restoration will prioritize public safety, anything public-facing, then all other departments.

DallasCityNews.net will be updated at least once a day until the incident is resolved. (1)

The Record: Dallas city government confirms ransomware attack affecting police and other services

  • The city of Dallas confirmed on Wednesday that it is dealing with a ransomware attack that has affected numerous IT systems and shut down the website of the police department.
  • In a statement, the city told Recorded Future News that the city’s security monitoring tools notified their security operations center Wednesday morning that “a likely ransomware attack had been launched within our environment.”
  • “Subsequently, the City has confirmed that a number of servers have been compromised with ransomware, impacting several functional areas, including the Dallas Police Department Website,” a spokesperson said.
  • “The City team, along with its vendors, are actively working to isolate the ransomware to prevent its spread, to remove the ransomware from infected servers, and to restore any services currently impacted. The Mayor and City Council was notified of the incident pursuant to the City’s Incident Response Plan (IRP).”
  • A spokesperson for the Dallas Police Department (DPD) also confirmed that they are being affected by the ransomware attack.
  • The police department website is currently down, but the spokesperson would not say whether other police systems are still operating.
  • CBS News reported that the cyberattack “is impacting DPD’s computer-assisted dispatch system, called CAD, which directs police to emergencies and other calls.”
  • Operators can still take emergency calls but they have to write down information by hand and relay it to officers through their phones and radios, CBS News reported.
  • The city’s court systems were also affected by the attack. Government workers said after the ransomware attack began, their office printers began to print the following message suggesting that the Royal ransomware group was behind the incident. (2)

Image Source:  The Record

May 4, 2023

Official Website: City of Dallas (https://www.dallascitynews.net) 

Update as of 10 a.m. May 4.
ITS and its vendors continue to work around the clock to contain the outage and restore service, prioritizing public safety and public-facing departments. A group called Royal initiated the attack. Chief Zielinski will brief the Public Safety Committee Monday, May 8.

As of 10 a.m. these are service updates:

  • DPD and DFR service to residents is unaffected.
  • 911 calls continue to be received and dispatched.
  • 311 Calls are being answered but non-emergency service requests may be delayed.
  • Courts are closed and LiveChat is inaccessible. All cases will be reset; jurors do not need to report for service and notices will be sent by mail.
  • CSO – Saturday’s election is unaffected; Dallas County will share official information including results. Meeting notices are being posted and meetings may be viewed at dallascityhall.webex.com, dallascitynews.net/watch-live, and Spectrum channels 16 & 95 and AT&T U-verse at 99. Contracts may be delayed.
  • Dallas Public Library All branches are open and in-person checkouts continue; online materials are currently unavailable.
  • DWU Billing is unaffected; meter reading will be delayed. Only IVR can take credit card payments. Disconnections will be discontinued until the outage is resolved.

City Manager T.C. Broadnax has released the following statement:

“Since City of Dallas’ Information and Technology Services detected a cyber threat Wednesday morning, employees have been hard at work to contain the issue and ensure continued service to our residents. While the source of the outage is still under investigation, I am optimistic that the risk is contained. For those departments affected, emergency plans prepared and practiced in advance are paying off. We apologize for any inconvenience and thank residents for their understanding as we continue to work around the clock until this issue is addressed. For updates, please keep an eye on dallascitynews.net.”

4:30 p.m. update, Thursday, May 4

  • Services to residents continue with some delays and disruption due to the outage. Departmental updates as of 4:30 p.m. are as follows:
  • ITS isolated the issue and is gradually restoring service prioritizing public safety and resident-facing departments.
  • Dallas Water Utilities is unable to process payments. Disconnections will be discontinued until the outage is resolved.
  • Vital Statistics is issuing records with limited capacity. Some records may be unavailable, especially from prior to 2005. Residents may contact Vital Records at 214-670-3092.
  • Women, Infants & Children (WIC) is maintaining normal operations at all clinics and able to issue benefits.
  • Courts remain closed and cases will be reset; jurors do not need to report for service and notices will be sent by mail.
  • Code Compliance Services response to service requests may be delayed. Code is currently unable to process Single-Family and Multi-Tenant registrations. Garage sale permits can be issued in-person only at 3112 Canton St.
  • Dallas Animal Services is responding to injury and emergency requests and non-emergency response is delayed. DAS is handling adoptions, fosters, rescue and return to owners on a case-by-case basis at 1818 N. Westmoreland Rd. during regular business hours.
  • City Secretary’s Office Open Records Requests will be delayed.
  • Development Services, Permitting, Public Works, and Zoning applications and payments cannot be received, and permits cannot be issued.

May 5, 2023

5 p.m. update, Friday, May 5

The City of Dallas’ preparation for security threats and early response to network outages caused by a cyberattack on May 3 has maintained essential City services for residents throughout the week.

Since Wednesday morning, when a group called Royal initiated an illegal attack targeting the City, Dallas’ Information and Technology Services (ITS) department and ITS’ expert cybersecurity vendors have effectively worked nonstop to swiftly isolate a virus and gradually restore service, prioritizing public safety and resident-facing departments. Much progress has been made, but the recovery process is ongoing.

Thanks to heroic teamwork by our first responders, key public safety functions continue as usual despite technical difficulties. 911 and 311 calls are being answered and Dallas Police Department and Dallas Fire Rescue are being timely dispatched by radio.

  • Dallas Public Library branches are open, and operational, and can check out media to residents with a library card; however, residents with media due to be returned are asked to hang on to it a little longer. There will be no late fees charged for materials due during the service outage. Digital media is also available via Hoopla and Overdrive. Internet-connected computers may be limited, so users in need of online device access should call ahead to their respective branches.
  • Municipal Court remains closed Monday, May 8. There will be no court hearings and no trials. Cases scheduled during this outage will be reset, and updates will be mailed. Citation payments and documents due while Municipal Court is closed will be accepted after service is restored. Please note, no one from the City of Dallas will reach out to members of the public to ask for payment in person or by phone. Never give out your password or payment information by phone or through an email link. To protect against cyber threats please install the Dallas Secure app on your iOS or Android device. If you are contacted by someone seeking payment who claims to be from a City of Dallas department, please take note of the number they are calling from and the number they reached you on, then hang up and call the City of Dallas department they claim to be from to report this potential impersonation.
  • For years the City has monitored to ensure that residents’ and vendors’ information is not leaked. If it were, we would advise the public immediately. At this point, we do not believe that’s the case.  Please pardon the inconvenience, thank you again, and stay safe!

May 8, 2023

The Record:  Dallas courts, fire, and police networks still crippled from a ransomware incident

  • All municipal courts in Dallas will be closed on Monday due to a ransomware attack that was announced last week.
  • Despite statements from city officials claiming the recovery effort was slowly progressing, the fire and police departments told local news outlets they are facing massive issues as a result of the attack.
  • In a statement on Sunday, the city said staff and vendors worked “throughout this weekend to ensure progress toward service restoration” after the city confirmed on Wednesday that its systems had been hit by ransomware.
  • The Royal ransomware gang took credit for the attack, pushing out its ransom note through government printers throughout the week.
  • “As of Saturday, May 6, steady, measured progress continues with minimal service disruption. We are confident we have contained the source of the infection and not seen any new spread,” the city said.
  • Cybersecurity vendor CrowdStrike “continues to ensure that any City devices blocked and quarantined to prevent or contain the spread of the virus are clean before they are back in service,” the city said. “Microsoft continues work toward restoration of departmental web pages from backups, but the demand of so much activity on available systems is challenging capacity.”
  • The statement adds that officials have prioritized the restoration of the city’s Computer Aided Dispatch, one of the systems that underpins the 911 and 311 services.
  • They are also prioritizing public-facing services like government websites as well as systems for payment and permits.
  • Jim McDade, president of the Dallas Fire Fighters Association, told the Dallas Morning News that firefighters and police officers were running into several issues because dispatchers had to write down information by hand and relay it over radio networks, which were becoming overwhelmed with the number of calls.  McDade said firefighters typically received detailed information about a situation but were now going into emergency calls blind, only getting an address.
  • “Is it a mess? Yes. Is it potentially bad? Very,” McDade said. “If you ask just a regular firefighter out there, he just knows that dispatch is down. There’s also no way to communicate with everybody.”
  • Some calls are slipping through the cracks or being missed, while ambulances are being sent to the wrong location and younger officers are having to learn how things were run before systems were digitized starting in 2005. (3)

Official Website: City of Dallas (https://www.dallascitynews.net) 

4 p.m. update, Monday, May 8

As of Monday, May 8 DallasCityHall.com and DallasPolice.net are back online.

911 and 311 intake and dispatch continue via phone and radio dispatch while Computer Assisted Dispatch (CAD) components including 1900 mobile devices (1600 for DPD and 300 for DFR) and the server routing calls are tested to ensure no reinfection when redeployed. Completion of device cleaning to allow resumption of CAD is anticipated early this week

Please see below for frequently asked questions regarding the incident:

  1. Will the City pay the ransom?

  • The City is exploring all options to remediate this incident. As this is an ongoing criminal investigation, the City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker.
  1. Did the attackers send a ransom note to City printers?

  • An image of information allegedly transmitted by an attacker has been published. While some media outlets removed or blurred a URL shown, others did not. Do not attempt to visit this URL as it may pose a threat to the device or network of anyone that does.
  1. How did the breach initiate?

  • This is an ongoing criminal investigation. The City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker. The most common Ransomware attacks are initiated by exploiting vulnerabilities in software such as weak or default credentials, and social engineering (e.g. phishing) which tricks users into divulging confidential or personal information that may be used for fraudulent purposes.
  1. How many devices are affected?

  • This is an ongoing criminal investigation. The City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker.
  1. What departments are affected and how?

  1. What are next steps?

  • The City’s IT Department continues to work with the assistance of cybersecurity experts and vendors to review software, servers, and devices to ensure they are uninfected before they are returned to service.
  1. Are 9-1-1 dispatchers still having to take down information by hand and share it over a radio to respond?

  • As prepared for and practiced in advance, 9-1-1 operators continue to answer and dispatch calls utilizing back up procedures and the City’s public safety radio system. As City staff and contractors review devices, ensure they are secure, and bring them online, Computer Assisted Dispatch (CAD) functionality will increase for DPD, DFR and 311.
  1. What do people on deadline to pay citations need to do?

  • Citation payments and documents due while the Municipal Court system is down will be accepted after service is restored. Anyone who has a citation to pay or documents due while Municipal Court is closed will be granted an extension to pay or present the documents to the Court without penalty.
  1. How long this will take to get resolved?

  • The City’s IT Department will continue to work with the assistance of cybersecurity professionals and our system vendors to review software, servers, and devices to ensure they are uninfected and can be returned to service as quickly as possible.
  1. Is the prior network outage related to this attack?

  • There is no evidence that the network outage occurring on April 19, 2023 is related to this ransomware attack. The network outage of April 19 was caused by a hardware failure. The City and its network services provider AT&T resolved that outage.
  1. Has Royal made any demands from the city of Dallas? Has the city learned how Royal was able to gain access to its systems? If so, how did the group compromise those servers? If so, what is the group demanding, and what has been the city’s response?

  • This is an ongoing criminal investigation. The City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker.
  1. Should residents be concerned that their personal information could be leaked online as a result of this attack? Why or why not?

  • At this time the City has no indication that customer information such as billing data or personally identifiable information (PII) has been leaked from City systems or databases. Should this change, the City will notify potentially impacted individuals with information and instructions. Should any individual be contacted by someone claiming to represent the City of Dallas and ask for a payment or personal information, please take note of the number they are calling from and the number they are reaching, then hang up and call the City Department to report.
  1. How much is this attack costing the City?

  • As this is an ongoing criminal investigation, a determination of financial impact cannot be given at this time.
  1. Were social media channels compromised as a result of the outage?

  • The City of Dallas social media accounts have not been compromised. Updates will continue to be shared via DallasCityNews.net.

May 15, 2023

3:00 p.m., Monday, May 15 Update

Due to preparations made in advance and plans followed, restoration continues steadily, with additional services brought back online daily. Given the complexity of checking, cleaning, and restoring interoperability to remaining departmental devices, systems and applications, it will likely take weeks to get back to full functionality.

Public safety remains our highest priority. 911 has continually taken calls to dispatch Dallas Police Department (DPD) and Dallas Fire-Rescue (DFR) to protect and serve Dallas residents.

After public safety, public-facing services are being prioritized to minimize disruption of services to the public. There is still no indication that data from residents, vendors, or employees has been leaked.

The Record: Dallas says it ‘will likely take weeks to get back to full functionality’ after ransomware attack

  • It will likely take weeks for government systems in the city of Dallas to operate normally following a ransomware attack, officials said on Friday afternoon.
  • For the last two weeks, the city has been engulfed in a massive recovery effort after the Royal ransomware gang caused significant damage to systems that manage the city’s police, fire department, courts, critical infrastructure and more.
  • After significant backlash from police unions and others, the city made progress throughout last week in restoring some dispatch systems for police and firefighters. Sgt. Sheldon Smith told WFAA that there are still five- to 10-minute delays when they try to run the tags on a car or a license.
  • “Today, we’re working like it’s 1965 but it’s 2023,” Smith said. “Officers have to adapt and overcome. But we don’t have the full technology capabilities that we had three weeks ago, a month ago.”
  • Smith noted that the police are in the process of rebuilding their records system.
  • While city experts have made progress in restoring some systems over the last several days, officials said it will take weeks until everything returns to normal. Brian Gardner, the city’s chief information security officer, told The Dallas Morning News that they will “be working at this for weeks and months to do all the clean up.”
  • “There is still no indication data from residents, vendors or employees has been leaked. Given the complexity of checking, cleaning, and restoring interoperability to remaining departmental devices, systems and applications, it will likely take weeks to get back to full functionality,” the city said in a statement on Friday.
  • All court hearings, trials and payments are still disrupted by the ransomware attack and little indication has been given on when those services will return.
  • Dallas Water Utilities payment systems and meter reading software were back online by Thursday and the city controller was finally able to print checks to vendors so that payroll was not affected.
  • Public computers at libraries are also still down but WiFi is available to visitors. Materials can be checked out manually but books cannot be returned yet. The city government’s website and the website for the police department are back up and running.
  • Both Microsoft and CrowdStrike have assisted in the recovery effort alongside several Texas state agencies and the FBI. (4)

May 19, 2023

1:20 p.m. update, Friday, May 19

The City of Dallas is aware of a post from what appears to be the Royal ransomware group threatening to release city data. We continue to monitor the situation and maintain there is no evidence or indication that data has been compromised.

Measures to protect data are in place. Information will continue to be provided on DallasCityNews.net as the ongoing investigation of this criminal organization continues.

May 22, 2023

12:50 update, Monday, May 22

  • There remains no established evidence of a data leak.
  • Municipal Court is undergoing a previously planned upgrade and anticipates reopening Tuesday, May 30.
  • While Court has been closed since May 3 due to the network outage, City of Dallas residents have been able to mail payments, requests, or documents to the court.

All cases will be reset and notices will be mailed. Citation payments or documents due while Municipal Court is closed will be processed after May 30.

May 30, 2023

7 a.m. update, Wednesday, May 31

After undergoing a planned outage for an upgrade to Court systems, Dallas Municipal Court reopened as scheduled on Tuesday, May 30.

  • Payment processing and document requests are available online, by mail and in-person.
  • Walk-in court is available in-person.
  • Administrative hearings for parking violations have resumed.
  • Community Court hearings resume Wednesday, May 31.
  • Other court hearings will resume Thursday, June 1, excluding trials and jury duty.
  • Calls to 214.670.0109 may receive a busy signal due to high call volume; callers may come to Municipal Court in person or visit Courts’ website for online options or Live Chat.
  • Trials and jury duty will resume at a later date and notices will be mailed.

June 1, 2023 

The Record:  Dallas, still recovering from a ransomware attack, marks milestone with return of courts

  • Several city services in Dallas have been reinstated since a ransomware attack took down several agencies on May 3, but the pace of progress highlights how devastating the incident has been.
  • “We continue to work with our cybersecurity experts on additional steps to further enhance our security posture, including implementing additional cybersecurity software, deploying a system-wide reset across all user accounts, expediting the implementation of additional controls, and completely rebuilding impacted systems in a new, secure environment,” the spokesperson said.
  • For weeks, the city struggled with providing basic services like the issuance of birth and death records and more.
  • The city denied last week that there is evidence of a data leak despite the ransomware group’s wide access to the city’s network. But officials said they are monitoring the Royal ransomware gang’s posts to see if data will be released.
  • But in official statements published on its temporary website, the city did not rule out paying a ransom, writing that it is “exploring all options to remediate this incident.”
  • The city said it may take weeks to fully recover from the incident and faced some backlash from police officers, firefighters and emergency service workers who initially disputed claims that communication systems for emergencies had been restored.  (5

What Next? 


Threat Assessment: Royal Ransomware

An overview of the Royal Ransomware Group from the VMware Security Blog:

While the evolution of ransomware techniques is to be expected, the speed at which the Royal Ransomware Group has been able to adapt is impressive. Since it was first reported, those responsible for Royal ransomware have advanced quickly over a short period of time, leveraging old and new techniques as well as exploiting novel vulnerabilities as they are discovered. In just the last six months, they have rapidly escalated attacks targeting victims across numerous industries and countries.

The Royal Ransomware Group has also leveraged evasion techniques such as Virtual Instances [1], which made it challenging for defenders to prevent encryption once the threat actor had gained access to the targeted victim environment. Interestingly, some of the attributes were reminiscent of an older 2020 sample of Conti ransomware. External research by Vitali Kremez from AdvIntel has stated a direct relationship between Conti and Royal ransomware [2]. Comparing a recent Royal sample against Conti’s and other Royal ransomware variants over the past six months could provide insight into the threat actors’ future activity.

Who is Royal Ransomware?

Initially identified as Zeon in January of 2022, Royal ransomware has been rebranded as “Royal” since September of 2022 [3]. Since then, they have targeted companies across numerous industries such as Manufacturing, Healthcare, Food, and Education. Although over 60% of targeted companies have been in the United States, the Royal Ransomware Group has not shied away from targeting countries around the world including Europe and Latin America.

Future Threats for American Cities  – Past is Prologue 

There is a clear trend here.  Will the frequency, duration, intensity, and impact of future attacks on major American cities increase in the years ahead?  Here is our recent baseline, as reported by The Record, Dallas is not alone in its recent cyber woes – and attacks at the city and municipality level will continue to be an issue: 

  • The ransomware incident in Dallas — a city of 1.3 million people — is just the latest cyberattack affecting a major U.S. municipal government.   Since the attack on Dallas, several other municipalities have faced their own ransomware attacks, including the 200,000-resident city of Augusta, Georgia.  (5
  • The ransomware attack on Dallas – a city of 1.3 million people – is just the latest cyberattack affecting a major U.S. city. Just weeks ago, the City of Oakland was severely damaged by a wide-ranging ransomware attack that hampered city services for weeks, and leaked troves of sensitive data about city residents and government officials onto the internet.
  • Ransomware attacks on cities as large as Oakland have become rarer in recent years as governments step up their cybersecurity protections and groups target smaller governments with less resources. New OrleansAtlanta, and Baltimore dealt with major attacks in 2018 and 2019. Tulsa also reported an attack by the Conti ransomware group in 2021.
  • Atlanta was forced to spend more than $9.5 million recovering from the incident and Baltimore reportedly spent $19 million dealing with their attack. 
  • Incidents in Oregon and South Carolina
  • Oregon’s Curry County – which has a population of about 23,000 – announced on Friday that it was also attacked by the Royal ransomware gang last week.
  • County officials said they began to experience issues on Wednesday, April 26 before discovering that the server network was inaccessible, affecting all county departments. Cybersecurity firms were hired to help with the response and federal agencies were notified.
  • “County telephone service is uninterrupted, but email communications are not available at this time. The response team will be working around the clock to restore operations. At this time, we cannot estimate when full access and services will be restored,” county officials said.
  • “The County is prioritizing service returns to public safety and public-facing departments. No part of the vote counting process or the integrity of the upcoming election has been impacted by this incident.”
  • South Carolina’s Spartanburg County announced its own ransomware attack two weeks ago.    It is still causing issues for local officials.
  • The real estate industry has been severely damaged by the attack due to its reliance on the Register of Deeds department, which has been offline since the attack began. Title searches and submissions have ground to a halt due to the attack.
  • “We can’t turn anything else in which means we’re not making any money, which means the attorney can’t close the loans of the people waiting to purchase their dream home,” one attorney told local news outlet WYFF.

For more OODA Loop News Briefs and Original Analysis, see Ransomware | OODA Loop

https://oodaloop.com/archive/2023/04/04/cisas-pilot-ransomware-warning-system-and-pre-ransomware-notification-initiative/

 

Daniel Pereira

About the Author

Daniel Pereira

Daniel Pereira is research director at OODA. He is a foresight strategist, creative technologist, and an information communication technology (ICT) and digital media researcher with 20+ years of experience directing public/private partnerships and strategic innovation initiatives.