Start your day with intelligence. Get The OODA Daily Pulse.

Background

In March 2021,  OODA CTO Bob Gourley had an OODAcast conversation with Ellen McCarthy.  Ellen’s career began at the office of Naval Intelligence. She then moved to Norfolk and the Atlantic Intelligence Center and would later lead all intelligence activities for the US Coast Guard as their director of intelligence.  McCarthy then joined DoD’s office of the undersecretary of defense for intelligence working strategy and human capital management. Later she led the nonprofit public-private partnership INSA (the intelligence and national security alliance), helping make that organization what it is today.  She returned to government service as the Chief Operating Officer of the National Geospatial-Intelligence Agency (NGA), then later led the firm Noblis as its president. Ellen was then appointed the Assistant Secretary of State for Intelligence and Research (INR), where she led an organization famed for the highest quality of analysis in the US IC.

In the conversation, Ellen spoke highly of her time with the Coast Guard, including the agency’s formal entry into the intel community and the sophistication of the United States Coast Guard Cyber Command (CGCYBER):

Bob Gourley:  When people think of the Coast Guard, I think it depends on where you live in America, the Coast Guard is everywhere. They’re on like every major waterway and they’re doing safety at a low level and small craft all the way up to policing the coasts.  And they are global.  They are all over the place.

Ellen McCarthy:  So in terms of their port security mission,  they do that all around the world and they have access to ports for the Navy to this day would dream of getting into. And, and so, you know, people, a lot of folks said “why do you want to go to little old coast guard?” Well as it turned out, it was, it was probably one of the best moves I’ve ever made because it was just before 9/11.   So you talk about that port security notion, 9/11 occurs. And all of a sudden that little port security mission is kind of important as, as we were reviewing who was coming and going and providing assistance to customs. And the other thing is:  the Coast Guard has this incredible sort of Intel law enforcement/Title 10/Military side to it. It has got sort of three heads in addition to its maritime security role.  So learning that and understanding how you deliver intelligence to an organization that at one minute may be wearing its Title 10 hat and the next minute is wearing its law enforcement hat was fabulous.

Gourley:  You became the Director of Intelligence at the Coast Guard?

McCarthy:  It was Intel Ops.  Yes. So what happened was Dennis Hagar was then the head of the Intel program. A Navy guy and his charter was to get the Coast Guard into the intelligence community (IC). And I was doing policy and strategy at that moment. And so I worked with some brave Coasties to figure out how can the Coast Guard become an element in the IC?  Dennis Hagar leaves.  A woman named Fran Townsend comes in.   That’s an interesting story. We had spent two years working the Hill and working with the community management staff, trying to get the Coast Guard into the IC. She literally picked up the phone and made a call and within like two months we were part of the intelligence community. I then moved up into the Intel Ops position because there was a lot that came with being an element in the IC.

Part of the things that we’ve said we were going to do is to create some unique collection programs – which we did to establish an Intel branch or an Intel designation within the enlisted Corps of the Coast Guard.  And so it was just such an exciting time to be able to take this small, small office, under a hundred people, and really build it into a no-kidding Intel capability. And I had the opportunity to go visit about six months ago [September 2020] to see where they are now. And it is just incredible. They now have a SIGINT capability. They have a few folks that specialize in SIGINT who now sit at NSA very involved in cyber.  A very impressive CIA capability. It is pretty impressive. I’m very proud.

With the context provided by this OODAcast conversation, we return to the joint Cybersecurity Advisory (CSA) released in late June by the Cybersecurity and Infrastructure Security Agency (CISA) and the United States Coast Guard Cyber Command (CGCYBER) “to warn network defenders that cyber threat actors, including state-sponsored advanced persistent threat (APT) actors, have continued to exploit CVE-2021-44228 (Log4Shell) in VMware Horizon® and Unified Access Gateway (UAG) servers to obtain initial access to organizations that did not apply available patches or workarounds.”

This CISA/CGCYBER collaboration caught our eye and we analyze how it compares to previous joint CSAs and is differentiated by CGCYBER’s unique brand of intelligence ( which was built on MacCarthy’s formative efforts discussed above).

This joint CSA also provided a context for an OODA Loop update on the ongoing Log4Shell threat of attacks and further evidence of the success of the CISA JCDC (as VMware and Secureworks are direct contributors to this joint CSA).

Alert (AA22-174A) Malicious Cyber Actors Continue to Exploit Log4Shell in VMware Horizon® and Unified Access Gateway (UAG) Servers

We are very rigorous in our selection of the joint CSAs we choose to surface for more in-depth research and analysis.  But the CISA/CGCYBER collaboration caught our eye and we wanted to spend some time analyzing how this joint CSA is differentiated by the CGCYBER brand of intelligence. The joint CSA also provided a context for an OODA Loop update on the ongoing Log4Shell threat of attacks and further evidence of the success of the CISA JCDC (as VMware and Secureworks are direct contributors to this joint CSA).

This joint CSA is different from previous joint CSAs in three ways, which we attribute to CGCYBER (and how its modus operandi is differentiated from other intel community agencies) and the direct private sector collaboration with VMWare and Secureworks):

  • A level of incident specificity, including the release of downloadable copies of indicators of compromise (IOCs):
  • The provision of two detailed narratives (Victim 1 and Victim 2) of incident response activities at two related confirmed compromises; and
  • The advisory is rigorously cross-referenced and linked to the MITRE ATT&CK for Enterprise framework, version 11, in a way we have not necessarily seen in previous alerts.

Summary

The Cybersecurity and Infrastructure Security Agency (CISA) and the United States Coast Guard Cyber Command (CGCYBER) are releasing this joint Cybersecurity Advisory (CSA) to warn network defenders that cyber threat actors, including state-sponsored advanced persistent threat (APT) actors, have continued to exploit CVE-2021-44228 (Log4Shell) in VMware Horizon® and Unified Access Gateway (UAG) servers to obtain initial access to organizations that did not apply available patches or workarounds.

Since December 2021, multiple threat actor groups have exploited Log4Shell on unpatched, public-facing VMware Horizon and UAG servers. As part of this exploitation, suspected APT actors implanted loader malware on compromised systems with embedded executables enabling remote command and control (C2). In one confirmed compromise, these APT actors were able to move laterally inside the network, gain access to a disaster recovery network, and collect and exfiltrate sensitive data.

This CSA provides the suspected APT actors’ tactics, techniques, and procedures (TTPs), information on the loader malware, and indicators of compromise (IOCs). The information is derived from two related incident response engagements and malware analysis of samples discovered on the victims’ networks.

CISA and CGCYBER recommend all organizations with affected systems that did not immediately apply available patches or workarounds to assume compromise and initiate threat hunting activities using the IOCs provided in this CSA, Malware Analysis Report (MAR)-10382580-1, and MAR-10382254-1. If potential compromise is detected, administrators should apply the incident response recommendations included in this CSA and report key findings to CISA.

Victim 1

CGCYBER conducted a proactive threat-hunting engagement at an organization (Victim 1) compromised by actors exploiting Log4Shell in VMware Horizon. After obtaining access, threat actors uploaded malware, hmsvc.exe, to a compromised system. During malware installation, connections to IP address 104.223.34[.]198 were observed.

CISA and CGCYBER analyzed a sample of hmsvc.exe from the confirmed compromise. hmsvc.exe masquerades as a legitimate Microsoft® Windows® service (SysInternals LogonSessions software) [T1036.004] and appears to be a modified version of SysInternals LogonSessions software embedded with malicious packed code. When discovered, the analyzed sample of hmsvc.exe was running as NT AUTHORITYSYSTEM, the highest privilege level on a Windows system. It is unknown how the actors elevated privileges.

hmsvc.exe is a Windows loader containing an embedded executable, 658_dump_64.exe. The embedded executable is a remote access tool that provides an array of C2 capabilities, including the ability to log keystrokes [T1056.001], upload and execute additional payloads [T1105], and provide graphical user interface (GUI) access over a target Windows system’s desktop. The malware can function as a C2 tunneling proxy [T1090], allowing a remote operator to pivot to other systems and move further into a network.

When first executed, hmsvc.exe creates the Scheduled Task [T1053.005], C:WindowsSystem32TasksLocal Session Updater, which executes malware every hour. When executed, two randomly named *.tmp files are written to the disk at the location C:Users<USER>AppDataLocalTemp and the embedded executable attempts to connect to hard-coded C2 server 192.95.20[.]8 over port 4443, a non-standard port [TT571]. The executable’s inbound and outbound communications are encrypted with a 128-bit key [T1573.001].

For more information on hmsvc.exe, including IOCs and detection signatures, see MAR-10382254-1.

Victim 2

From late April through May 2022, CISA conducted an onsite incident response engagement at an organization (Victim 2) where CISA observed bi-directional traffic between the organization and suspected APT IP address 104.223.34[.]198. During incident response, CISA determined Victim 2 was compromised by multiple threat actor groups.

The threat actors using IP 104.223.34[.]198 gained initial access to Victim 2’s production environment in late January 2022, or earlier. These actors likely obtained access by exploiting Log4Shell in an unpatched VMware Horizon server. On or around January 30, likely shortly after the threat actors gained access, CISA observed the actors using PowerShell scripts [T1059.001] to callout to 109.248.150[.]13 via Hypertext Transfer Protocol (HTTP) [T1071.001] to retrieve additional PowerShell scripts. Around the same period, CISA observed the actors attempt to download [T1105] and execute a malicious file from 109.248.150[.]13. The activity started from IP address 104.155.149[.]103, which appears to be part of the actors’ C2 [TA0011] infrastructure.

After gaining initial access to the VMware Horizon server, the threat actors moved laterally [TA0008] via Remote Desktop Protocol (RDP) [T1021.001] to multiple other hosts in the production environment, including a security management server, a certificate server, a database containing sensitive law enforcement data, and a mail relay server. The threat actors also moved laterally via RDP to the organization’s disaster recovery network. The threat actors gained credentials [TA0006] for multiple accounts, including administrator accounts. It is unknown how these credentials were acquired.

After moving laterally to other production environment hosts and servers, the actors implanted loader malware on compromised servers containing executables enabling remote C2. The threat actors used compromised administrator accounts to run the loader malware. The loader malware appears to be modified versions of SysInternals LogonSessions, Du, or PsPing software. The embedded executables belong to the same malware family, are similar in design and functionality to 658_dump_64.exe, and provide C2 capabilities to a remote operator. These C2 capabilities include the ability to remotely monitor a system’s desktop, gain reverse shell access, exfiltrate data, and upload and execute additional payloads. The embedded executables can also function as a proxy.

CISA found the following loader malware:

  • SvcEdge.exe is a malicious Windows loader containing encrypted executable f7_dump_64.exe. When executed, SvcEdge.exe decrypts and loads f7_dump_64.exe into memory. During runtime, f7_dump_64.exe connects to hard-coded C2 server 134.119.177[.]107 over port 443.
  • odbccads.exe is a malicious Windows loader containing an encrypted executable. When executed, odbccads.exe decrypts and loads the executable into memory. The executable attempts communication with the remote C2 address 134.119.177[.]107.
  • praiser.exe is a Windows loader containing an encrypted executable. When executed, praiser.exe decrypts and loads the executable into memory. The executable attempts connection to hard-coded C2 address 162.245.190[.]203.
  • fontdrvhosts.exe is a Windows loader that contains an encrypted executable. When executed, fontdrvhosts.exe decrypts and loads the executable into memory. The executable attempts connection to hard-coded C2 address 155.94.211[.]207.
  • winds.exe is a Windows loader containing an encrypted malicious executable and was found on a server running as a service. During runtime, the encrypted executable is decrypted and loaded into memory. The executable attempts communication with hard-coded C2 address 185.136.163[.]104winds.exe has complex obfuscation, hindering the analysis of its code structures. The executable’s inbound and outbound communications are encrypted with an XOR key [T1573.001].

For more information on these malware samples, including IOCs and detection signatures, see MAR-10382580-1.

Additionally, CISA identified a Java® Server Pages (JSP) application (error_401.js) functioning as a malicious webshell [T505.003] and a malicious Dynamic Link Library (DLL) file:

  • error_401.jsp is a webshell designed to parse data and commands from incoming HTTP requests, providing a remote operator C2 capabilities over compromised Linux and Windows systems. error_401.jsp allows actors to retrieve files from the target system, upload files to the target system, and execute commands on the target system. rtelnet is used to execute commands on the target system. Commands and data sent are encrypted via RC4 [T1573.001]. For more information on error_401.jsp, including IOCs, see [MAR-10382580 2].
  • newdev.dll ran as a service in the profile of a known compromised user on a mail relay server. The malware had path: C:Users<user>AppDataRoamingnewdev.dll. The DLL may be the same newdev.dll attributed to the APT actors in open-source reporting; however, CISA was unable to recover the file for analysis.

Threat actors collected [TA0009] and likely exfiltrated [TA0010] data from Victim 2’s production environment. For a three week period, the security management and certificate servers communicated with the foreign IP address 92.222.241[.]76. During this same period, the security management server sent more than 130 gigabytes (GB) of data to foreign IP address 92.222.241[.]76, indicating the actors likely exfiltrated data from the production environment. CISA also found .rar files containing sensitive law enforcement investigation data [T1560.001] under a known compromised administrator account.

Note: the second threat actor group had access to the organization’s test and production environments, and on or around April 13, 2022, leveraged CVE-2022-22954 to implant the Dingo J-spy webshell. According to trusted third-party reporting, multiple large organizations have been targeted by cyber actors leveraging CVE-2022-22954 and CVE-2022-22960. For more information on exploitation of CVE-2022-22954 and CVE-2022-22960, see CISA CSA Threat Actors Chaining Unpatched VMware Vulnerabilities for Full System Control.

Incident Response

If administrators discover system compromise, CISA and CGCYBER recommend:

  1. Immediately isolating affected systems.
  2. Collecting and reviewing relevant logs, data, and artifacts.
  3. Considering soliciting support from a third-party incident response organization that can provide subject matter expertise, ensure the actor is eradicated from the network, and avoid residual issues that could enable follow-on exploitation.
  4. Reporting incidents to CISA via CISA’s 24/7 Operations Center ([email protected] or 888-282-0870). To report cyber incidents to the Coast Guard pursuant to 33 CFR Section 101.305,  contact the U.S. Coast Guard (USCG) National Response Center (NRC) ([email protected] or 800-424-8802).

Mitigations

CISA and CGCYBER recommend organizations install updated builds to ensure affected VMware Horizon and UAG systems are updated to the latest version.

  • If updates or workarounds were not promptly applied following VMware’s release of updates for Log4Shell in December 2021treat those VMware Horizon systems as compromised. Follow the pro-active incident response procedures outlined above prior to applying updates. If no compromise is detected, apply these updates as soon as possible.
    • See VMware Security Advisory VMSA-2021-0028.13 and VMware Knowledge Base (KB) 87073 to determine which VMware Horizon components are vulnerable.
    • Note: until the update is fully implemented, consider removing vulnerable components from the internet to limit the scope of traffic. While installing the updates, ensure network perimeter access controls are as restrictive as possible.
    • If upgrading is not immediately feasible, see KB87073 and KB87092 for vendor-provided temporary workarounds. Implement temporary solutions using an account with administrative privileges. Note that these temporary solutions should not be treated as permanent fixes; vulnerable components should be upgraded to the latest build as soon as possible.
    • Prior to implementing any temporary solution, ensure appropriate backups have been completed.
    • Verify successful implementation of mitigations by executing the vendor supplied script Horizon_Windows_Log4j_Mitigations.zip without parameters to ensure that no vulnerabilities remain. See KB87073 for details.

Additionally, CISA and CGCYBER recommend organizations:

  • Keep all software up to date and prioritize patching known exploited vulnerabilities (KEVs).
  • Minimize the internet-facing attack surface by hosting essential services on a segregated DMZ, ensuring strict network perimeter access controls, and not hosting internet-facing services non-essential to business operations. Where possible, implement regularly updated WAFs in front of public-facing services. WAFs can protect against web based exploitation using signatures and heuristics that are likely to block or alert on malicious traffic.
  • Use best practices for identity and access management (IAM) by implementing multifactor authentication (MFA), enforcing use of strong passwords, and limiting user access through the principle of least privilege.

A direct link to the joint CSA is found here:  Malicious Cyber Actors Continue to Exploit Log4Shell in VMware Horizon Systems | CISA

OODA Recommendations

OODA CTO Bob Gourley is prescriptive in his April 2022 post:  Four Urgent Actions For The C-Suite To Prepare For High-End Cyberattacks

OODA is here to help.  OODA members can contact us by replying to any of our emails or using this form.

For Bob’s conversation with Ellen McCarthy, see:  Lessons In Leadership From Ellen McCarthy and Her Journey From Junior Analyst To The Most Senior Echelons of the Intelligence Community

The conversation with Bob is also integrated into the OODAcast-themed post series:  Ellen McCarthy and Kathy and Randy Pherson on Intelligent Leadership and Critical Thinking

Stay Informed

It should go without saying that tracking threats are critical to inform your actions. This includes reading our OODA Daily Pulse, which will give you insights into the nature of the threat and risks to business operations.

Related Reading:

Explore OODA Research and Analysis

Use OODA Loop to improve your decision-making in any competitive endeavor. Explore OODA Loop

Decision Intelligence

The greatest determinant of your success will be the quality of your decisions. We examine frameworks for understanding and reducing risk while enabling opportunities. Topics include Black Swans, Gray Rhinos, Foresight, Strategy, Stratigames, Business Intelligence, and Intelligent Enterprises. Leadership in the modern age is also a key topic in this domain. Explore Decision Intelligence

Disruptive/Exponential Technology

We track the rapidly changing world of technology with a focus on what leaders need to know to improve decision-making. The future of tech is being created now and we provide insights that enable optimized action based on the future of tech. We provide deep insights into Artificial Intelligence, Machine Learning, Cloud Computing, Quantum Computing, Security Technology, Space Technology. Explore Disruptive/Exponential Tech

Security and Resiliency

Security and resiliency topics include geopolitical and cyber risk, cyber conflict, cyber diplomacy, cybersecurity, nation-state conflict, non-nation state conflict, global health, international crime, supply chain, and terrorism. Explore Security and Resiliency

Community

The OODA community includes a broad group of decision-makers, analysts, entrepreneurs, government leaders, and tech creators. Interact with and learn from your peers via online monthly meetings, OODA Salons, the OODAcast, in-person conferences, and an online forum. For the most sensitive discussions interact with executive leaders via a closed Wickr channel. The community also has access to a member-only video library. Explore The OODA Community

Daniel Pereira

About the Author

Daniel Pereira

Daniel Pereira is research director at OODA. He is a foresight strategist, creative technologist, and an information communication technology (ICT) and digital media researcher with 20+ years of experience directing public/private partnerships and strategic innovation initiatives.