How the SolarWinds Hack (almost) went Undetected

My lightning talk from the SEC-T 0x0D conference has now been published on YouTube. This 13 minute talk covers tactics and techniques that the SolarWinds hackers used in order to avoid being detected.

Some of these tactics included using DNS based command-and-control (C2) that mimicked Amazon AWS DNS traffic, blending in with SolarWind’s legitimate source code and handpicking only a small number of targets.

One thing I forgot to mention in my SEC-T talk though, was the speed at which the attackers were working to analyze incoming data from the trojanized installs and selecting organizations to target for stage two operations.

For example, just during June 2020 the attackers got more than 1300 new organizations that started beaconing in using the DNS-based C2. The beaconed data only included the organizations’ Active Directory domain name and a list of installed security applications. Based on this information the attackers had to decide whether or not they wanted to target the organization. We have previously estimated that less than 1% of the organizations were targeted, while the malicious backdoor was disabled for all the other 99% who had installed the trojanized SolarWinds Orion update.

The attackers typically decided whether or not to target an organization within one week from when they started beaconing. This means that the attackers probably had several hundred organizations in queue for a targeting decision on any given week between April and August 2020. That's a significant workload!

Posted by Erik Hjelmvik on Monday, 18 October 2021 10:30:00 (UTC/GMT)

Tags: #SolarWinds​ #SEC-T​ #video​ #backdoor​ #SUNBURST​ #Solorigate​ #STAGE2​ #Stage 2​ #DNS​ #C2​ #ASCII-art​

Targeting Process for the SolarWinds Backdoor

The SolarWinds Orion backdoor, known as SUNBURST or Solorigate, has been analyzed by numerous experts from Microsoft, FireEye and several anti-virus vendors. However, we have noticed that many of the published reports are either lacking or incorrect in how they describe the steps involved when a client gets targeted by the threat actors. We have therefore decided to publish this writeup, which is based on the analysis we did of the SolarWinds backdoor when creating our SunburstDomainDecoder tool.

UPDATE March 1, 2021

Fixed errors in the Stage 2 beacon structure and added a CyberChef recipe link.

avsvmcloud.com DNS queries are not DGA related

The DNS communication between the backdoored SolarWinds Orion clients and the authoritative name server for avsvmcloud.com is not caused by a Domain Generation Algorithm (DGA), it's actually a fully functional two-way communication C2 channel. The clients encode information, such as the internal AD domain and installed security applications into the DNS queries and the DNS responses from the name server are used to instruct the clients to continue beaconing, stop beaconing or to target a client by proceeding to what we call Stage 2 operation. Thus, the authoritative name server for avsvmcloud.com was actually the C2 server for Stage 1 and 2 operation of the SolarWinds backdoor.

Image: SolarWinds Backdoor State Diagram

Command: Continue Beaconing

The default response from the name server is the "Continue Beaconing" command, which indicates that the threat actors have not yet decided if the SolarWinds client is of interest for further activity. Receiving a DNS A record in any of the following net ranges instructs the SolarWinds backdoor to continue beaconing:

• 8.18.144.0/23
• 71.152.53.0/24
• 87.238.80.0/21
• 199.201.117.0/24

In "Stage 1" operation the SUNBURST client starts out in the "New" mode where it exfiltrates the internal AD domain name. The AD domain data is often split into multiple DNS queries to reduce the length of each DNS query. The client later proceeds to the "Append" mode when the full AD domain has been exfiltrated. In "Append" mode the client transmits a list of installed or running security applications to the DNS C2 server, as we have described in our Extracting Security Products from SUNBURST DNS Beacons blog post. The client remains in Append mode until it gets either terminated or targeted.

Note: It is also possible to reset a client back to the "New" mode with a so-called "Ipx" command, but that is out of scope for this blog post.

Command: Stop Beaconing

The stop beaconing command terminates the DNS beaconing, so that the client no longer retrieves any commands from the C2 server. The C2 communication is stopped after receiving a DNS DNS A or AAAA record in any of the following ranges:

• 20.140.0.0/15
• 96.31.172.0/24
• 131.228.12.0/22
• 144.86.226.0/24
• 10.0.0.0/8
• 172.16.0.0/12
• 192.168.0.0/16
• 224.0.0.0/3
• fc00:: - fe00::
• fec0:: - ffc0::
• ff00::

Command: Target Client

A SUNBURST client that has been "targeted" will change a flag called rec.dnssec in the source code from false to true. We call this flag the "Stage 2" flag, which must be set in order for the client to accept a CNAME record and proceed to Stage 3. Symantec refer to the Stage 2 flag as "a bit flag representing whether the previous DNS response successfully contained partial or full instructions to start the secondary HTTP communication channel".

A DNS A record in any of the following three IP ranges can be used to set the "Stage 2" flag:

• 18.130.0.0/16
• 99.79.0.0/16
• 184.72.0.0/15

The state of the Stage 2 flag is actually signaled in the avsvmcloud.com DNS queries, which is how we managed to identify the AD domains of 23 targeted organizations just by analyzing SUNBURST DNS queries.

Stage 2 DNS Request Structure

The structure of the SUNBURST DNS queries in Stage 1 is pretty well described by Prevasio and Symantec, so we will not cover those in this blog post. Instead we will focus specifically on the structure of the DNS queries transmitted in Stage 2 operation, where the clients request a CNAME record from the name server.

As we have explained previously the exfiltrated data gets base32 encoded, using the custom alphabet "ph2eifo3n5utg1j8d94qrvbmk0sal76c", in order to ensure that only valid domain name characters are used in the DNS beacons.

The structure of the Stage 2 request, before it gets base32 encoded and appended as an avsvmcloud.com subdomain, looks like this:

Image: Stage 2 beacon structure of the SolarWinds backdoor

The base32 encoding not only uses a custom alphabet, it also employs a reversed endianess and byte order compared to "normal" implementations. We have created a CyberChef recipe that performs this custom base32 decoding, so that the structure can be verified more easily. A list with 45 different Stage 2 avsvmcloud.com subdomains can be found in our Finding Targeted SUNBURST Victims with pDNS blog post. Feel free to replace the input to our CyberChef recipe with any of those subdomains.

Sleep Timers

The DNS responses from the name server not only controls how the SolarWinds backdoor should transition between the various stages, it also controls for how long the backdoor should wait before sending the next DNS beacon.

The delay is assigned by AND-ing the last octet of the received IP address with bitmask 0x54. The result from the AND operation is then used to select a sleep interval in the table below, within which the client picks a random number of minutes to sleep.

An exception to the table above is clients that have entered Stage 2, which will only wait one to three minutes before requesting a CNAME.

Example DNS C2 for a Non-Targeted Client

Below is an example of DNS queries and responses from a SUNBURST client that wasn't targeted by the threat actors. These particular queries and responses come from a post on SolarWinds' community forum.

• 2020-07-04 00:03 UTC
  Query: if9prvp9o36mhihw2hrs260g12eu1 ⇒ AD domain "omeros.local"
  Response: 8.18.145.139 ⇒ sleep 1h, then Continue
• 2020-07-04 01:08 UTC
  Query: hnhb3v1b37dvv09icg0edp0 ⇒ Carbon Black is running
  Response: 8.18.145.62 ⇒ sleep 1 day, then Continue
• 2020-07-05 01:15 UTC
  Query: ea99hr2sfen95nkjlc5g ⇒ Nothing new to report
  Response: 8.18.144.150 ⇒ sleep 1 day, then Continue
• 2020-07-06 02:42 UTC
  Query: 707gigk9vbc923hf27fe ⇒ Nothing new to report
  Response: 8.18.145.151 ⇒ sleep 1 day, then Continue
• 2020-07-07 03:52 UTC
  Query: 6eivqct649pcg0g16ol4 ⇒ Nothing new to report
  Response: 20.140.84.127 ⇒ Stop DNS beacon

Note: Queried domain names in this list are subdomains of appsync-api.eu-west-1.avsvmcloud.com.

Example DNS C2 for a Targeted Client

Disclaimer: We have very few DNS queries and responses for targeted victims, hence the transactions below are improvised based on data from VriesHd, Joe Słowik and FireEye. Please view these transactions as an example of what the communication might look like for a targeted victim rather than what actually happened to this particular target.

• 2020-06-11 04:00 UTC
  Query: r8stkst71ebqgj66ervisu10bdohu0gt ⇒ AD domain, part 1 "central.pima.g"
  Response: 8.18.144.1 ⇒ Sleep 1h, then Continue
• 2020-06-11 05:00 UTC
  Query: ulfmcf44qd58t9e82w ⇒ AD domain, part 2 "ov"
  Response: 8.18.144.2 ⇒ Sleep 1h, then Continue
• 2020-06-11 06:00 UTC
  Query: p50jllhvhmoti8mpbf6p2di ⇒ Nothing to report
  Response: 8.18.144.16 ⇒ Sleep 8h, then Continue
• 2020-06-11 14:00 UTC
  Query: (?) ⇒ Nothing new to report
  Response: 8.18.144.17 ⇒ Sleep 8h, then Continue
• 2020-06-11 22:35 UTC
  Query: j5uqlssr1hfqnn8hkf172mp ⇒ Nothing to report
  Response: 184.72.181.52 ⇒ Target client for Stage 2 operation (1-3 minutes sleep)
• 2020-06-11 22:37 UTC
  Query: 7sbvaemscs0mc925tb99 ⇒ Client in Stage 2 operation, requesting CNAME
  Response: deftsecurity.com ⇒ CNAME for Stage 3 HTTPS C2 server

Note: Queried domains in this list are subdomains of appsync-api.us-west-2.avsvmcloud.com.

Conclusions

We hope this blog post clears up any misunderstandings regarding the targeting process of the SolarWinds backdoor and highlights the significance of the Stage 2 flag.

We warmly welcome any feedback or questions you might have regarding this writeup, please feel free to contact us or reach out to us through Twitter.

Posted by Erik Hjelmvik on Wednesday, 17 February 2021 20:22:00 (UTC/GMT)

Tags: #SolarWinds​ #backdoor​ #SUNBURST​ #Solorigate​ #FireEye​ #Microsoft​ #CNAME​ #STAGE2​ #Stage 2​ #DNS​ #avsvmcloud.com​ #C2​ #CyberChef​ #ASCII-art​

Twenty-three SUNBURST Targets Identified

Remember when Igor Kuznetsov and Costin Raiu announced that two of the victims in FireEye's SUNBURST IOC list were ***net.***.com and central.***.gov on Kaspersky's Securelist blog in December? Reuters later reported that these victims were Cox Communications and Pima County.

We can now reveal that the internal AD domain of all SUNBURST deployments in FireEye's IOC list can be extracted from publicly available DNS logs published by twitter user VriesHd, a.k.a. "Kira 2.0", with help of our SunburstDomainDecoder tool. The data published by VriesHd is the most complete SUNBURST DNS collection we've seen, with over 35.000 avsvmcloud.com subdomains! Here is FireEye's IOC table completed with our findings:

Victims Targeted with SUNBURST Stage 2 Backdoor

It was not just the victims listed in FireEye's IOC that were specifically targeted by the SUNBURST operators. As explained in our Finding Targeted SUNBURST Victims with pDNS blog post, the "STAGE2" flag in SUNBURST's DNS beacons can be used to reveal additional organizations that were singled out as interesting targets by the threat actors.

We'd like to stress that the majority of all companies and organizations that have installed a backdoored SolarWinds Orion update were never targeted by the threat actors. This means the these SUNBURST backdoors never made it past what we call "Stage 1 operation", where the backdoor encodes the internal AD domain name and installed security products into DNS requests. SUNBURST backdoors in Stage 1 operation cannot accept any commands from the C2 server without first progressing into Stage 2 operation. We estimate that about 99.5% of the installed SUNBURST backdoors never progressed into Stage 2 operation.

Here is the full list of internal AD domain names from the SUNBURST deployments in VriesHd's DNS data that actually did enter Stage 2 operation according to our analysis:

• central.pima.gov (confirmed)
• cisco.com (confirmed)
• corp.qualys.com (confirmed)
• coxnet.cox.com (confirmed)
• ddsn.gov
• fc.gov
• fox.local
• ggsg-us.cisco.com (confirmed)
• HQ.FIDELIS (confirmed)
• jpso.gov
• lagnr.chevrontexaco.net
• logitech.local
• los.local
• mgt.srb.europa* (confirmed)
• ng.ds.army.mil
• nsanet.local (not the NSA)
• paloaltonetworks* (confirmed)
• phpds.org
• scc.state.va.us (confirmed)
• suk.sas.com
• vgn.viasatgsd.com
• wctc.msft
• WincoreWindows.local

Our SUNBURST STAGE2 Victim Table has now been updated with additional details about the STAGE2 signaling from these SUNBURST implants, including timestamps, avsvmcloud.com subdomains and GUID values.

Initial Microsoft Targeting FAIL

The last two entries in the AD domain list above are interesting, since they both hint that the targeted entity might be Microsoft.

The data that gets exfiltrated in DNS beacons during SUNBURST's initial stage is the internal domain the SolarWinds Orion PC is connected to and a list of installed security products on that PC. These domain names, security products and possibly also the victims' public IP addresses, was the data available to the attackers when they decided which ones they wanted to proceed to Stage 2 with and thereby activate the HTTPS backdoor built into SUNBURST.

The threat actors were probably surprised when they realized that "WincoreWindows.local" was in fact a company in West Virginia that manufactures high quality windows and doors.

The threat actors later found another backdoored SolarWinds Orion machine connected to a domain called "wctc.msft", which also sounds like it could be Microsoft. Below is a table outlining relevant events for these two SUNBURST deployments that can be extracted from VriesHd's SB2 spreadsheet with SunburstDomainDecoder.

Microsoft have been public about being hit by SUNBURST (or "Solorigate" as they call it), so we can assume that the threat actors eventually located a backdoored SolarWinds Orion installation in their networks.

Victim Notification

We spent the previous week reaching out to targeted companies and organizations, either directly or through CERT organizations. From what we understand many of these organizations were already aware that they had been targeted victims of SUNBURST, even though they might not have gone public about the breach.

The Ethical Dilemma

We have no intentions to shame the organizations that have installed a backdoored SolarWinds Orion update, regardless if they were targeted by the threat actor or not. In fact, the supply chain security problem is an extremely difficult one to tackle, even for companies and organizations with very high security standards. This could have happened to anyone!

However, since multiple passive DNS logs and SUNBURST victim lists have been circulating through publicly available channels for over a month, we felt that it was now acceptable to publicly write about the analysis we've been doing based on all this data. We'd also like to thank everyone who has helped collect and share passive DNS data, including John Bambenek, Joe Słowik, Rohit Bansal, Dancho Danchev , Paul Vixie and VriesHd. This open data has been crucial in order to develop and verify our SunburstDomainDecoder tool, which has been leveraged by numerous incident response teams to perform forensic analysis of DNS traffic from their SolarWinds Orion deployments.

More Credits

We'd like to thank CERT-SE and all other computer emergency response organizations that have helped us with the task of notifying organizations that were identified as targeted. We would also like to applaud companies and organizations like FireEye, Palo Alto Networks, Fidelis Cybersecurity, Microsoft, the U.S. Department of Energy and the U.S. Federal Courts for being transparent and publicly announcing that the SUNBURST backdoor had been used in an attempt to compromise their networks.

Posted by Erik Hjelmvik on Monday, 25 January 2021 08:25:00 (UTC/GMT)

Tags: #SUNBURST​ #FireEye​ #Solorigate​ #Microsoft​ #SolarWinds​ #FireEye​ #CNAME​ #STAGE2​ #DNS​ #Passive DNS​ #avsvmcloud.com​ #pDNS​ #Microsoft​

Robust Indicators of Compromise for SUNBURST

There has been a great deal of confusion regarding what network based Indicators of Compromise (IOC) SolarWinds Orion customers can use to self assess whether or not they have been targeted after having installed a software update with the SUNBURST backdoor. Many of the published IOCs only indicate that a backdoored SolarWinds Orion update has been installed, but the question that many security teams are trying to answer is whether or not the installed backdoor has been been used by the threat actor.

Dont trust everything you read!

There is a widespread misunderstanding that receiving a so-called “NetBios” DNS A record (for example an address in 8.18.144.0/23) in response to a *.avsvmcloud.com DNS query would mean that you’ve been targeted. Our analysis of the decompiled SUNBURST code and passive DNS data show that that receiving a “NetBios” response does not necessarily mean that the client has been targeted. Unfortunately this misunderstanding has lead to various sensationalist stories being published with long lists of companies and organizations that are claimed to be “singled out by the hacking group for the second stage of the attack”, “explicitly selected by the SolarWinds hackers for further activities” or “breached via SolarWinds and then specifically targeted by the hackers for additional internal compromise”.

Another common misunderstanding is that clients sending *.avsvmcloud.com DNS queries with encoded timestamps, and optionally a list of installed/running AV products, have been actively targeted. Our analysis of the decompiled SUNBURST code show that the timestamped “Pings” or AV service status reports get exfiltrated in DNS traffic after the client’s internal AD domain has been sent, but before the perpetrators decide whether or not they want to activate the backdoor.

Indicators of a Targeted Attack

So what network based IOC’s can incident responders, blue teams and SOC analysts use in order to see if they have been targeted by the SUNBURST operators?

The following network based events indicate that a client has been actively targeted and the SUNBURST backdoor has progressed beyond the initial mode of operation:

• Received a DNS A record for an *.avsvmcloud.com query, that points to an IP address in any of the following three networks: 18.130.0.0/16, 99.79.0.0/16 or 184.72.0.0/15
• Sent an *.avsvmcloud.com DNS query with the STAGE2 flag encoded in the subdomain.
• Received a CNAME record for a query to *.avsvmcloud.com

Another network based IOC is HTTPS communication to one of the known STAGE3 C2 domains. However, please note that the C2 domain list might not be complete. It is even possible that a unique C2 domain is used for each victim. Nevertheless, here’s a list of the SUNBURST STAGE3 C2 domains we are currently aware of:

• avsvmcloud[.]com
• databasegalore[.]com
• deftsecurity[.]com
• digitalcollege[.]org
• freescanonline[.]com
• globalnetworkissues[.]com
• highdatabase[.]com
• incomeupdate[.]com
• kubecloud[.]com
• lcomputers[.]com
• mobilnweb[.]com
• panhardware[.]com
• seobundlekit[.]com
• solartrackingsystem[.]net
• thedoccloud[.]com
• virtualwebdata[.]com
• webcodez[.]com
• websitetheme[.]com
• zupertech[.]com

Palo Alto was a Targeted SUNBURST Victim

We can now verify that Palo Alto was among the targeted SUNBURST victims, because their DNS request for "5qbtj04rcbp3tiq8bo6t.appsync.api.us.east.1.avsvmcloud.com" contains an encoded STAGE2 flag. The attack took place on September 29 at around 04:00 UTC, according to the timestamp that was also encoded into the avsvmcloud subdomain.

Image: Parsing passive DNS data from Dancho Danchev with SunburstDomainDecoder v1.9 and filtering on GUID “22334A7227544B1E”.

Palo Alto's CEO Nikesh Arora has confirmed that they were hit by SUNBURST (or "SolarStorm" as they call it), but they don’t provide much details. Here’s what Nikesh wrote on December 17:

Recently, we experienced an attempt to download Cobalt Strike on one of our IT SolarWinds servers. [...]

We thought this was an isolated incident, however, on Dec. 13, we became aware that the SolarWinds software supply chain was compromised and it became clear that the incident we prevented was an attempted SolarStorm attack.

Our SUNBURST STAGE2 Victim Table has now been updated to include Palo Alto along side the other targeted victims.

Posted by Erik Hjelmvik on Monday, 11 January 2021 10:30:00 (UTC/GMT)

Tags: #SUNBURST​ #SolarWinds​ #SolarStorm​ #avsvmcloud​ #STAGE2​ #DNS​ #CNAME​ #avsvmcloud.com​ #Cobalt Strike​ #DNS​ #FireEye​

Extracting Security Products from SUNBURST DNS Beacons

The latest version of our SunburstDomainDecoder (v1.7) can be used to reveal which endpoint protection applications that are installed on trojanized SolarWinds Orion deployments. The security application info is extracted from DNS queries for "avsvmcloud.com" subdomains, which is used by SUNBURST as a beacon and C2 channel.

Here's an example showing that City of Kingston, Ontario, Canada were running Windows Defender on their trojanized SolarWinds deployment back in June:

The "F9A9387F7D252842" value is the victim's unique SUNBURST GUID. See our blog post Reassembling Victim Domain Fragments from SUNBURST DNS for more info about how the GUID value is encoded into the DNS traffic.

You can also run SunburstDomainDecoder in Linux, with help of Mono, like this:

The file "uniq-hostnames.txt" is a publicly available SUNBURST passive DNS repository created by Bambenek Consulting.

Security Product Statistics

It is also possible to use the passive DNS data shared by Bambenek, Joe Słowik and others to compute statistics of which security products that are popular among SolarWinds' customers.

It is worth mentioning that SUNBURST does not report status for several other major endpoint protection vendors, such as Kaspersky, McAfee, Symantec, Sophos or Trend Micro.

Download SunburstDomainDecoder

Our tool SunburstDomainDecoder is released under a Creative Commons CC-BY license, and can be downloaded here:
https://www.netresec.com/files/SunburstDomainDecoder.zip

You can also read more about SunburstDomainDecoder in our blog post Reassembling Victim Domain Fragments from SUNBURST DNS.

Posted by Erik Hjelmvik on Tuesday, 29 December 2020 09:38:00 (UTC/GMT)

Tags: #SunburstDomainDecoder​ #SUNBURST​ #SolarWinds​ #Solorigate​ #DNS​ #Windows Defender​ #Carbon Black​ #FireEye​ #ESET​ #F-Secure​ #C2​ #beacon​

Reassembling Victim Domain Fragments from SUNBURST DNS

We are releasing a free tool called SunburstDomainDecoder today, which is created in order to help CERT organizations identify victims of the trojanized SolarWinds software update, known as SUNBURST or Solorigate.

SunburstDomainDecoder can be fed with DNS queries to avsvmcloud.com in order to reveal the full internal domain names of infected companies and organizations.

UPDATE December 18, 2020 (v1.1)

SunburstDomainDecoder has now been updated to automatically reassemble fragmented domain name segments in order to show the full domain in the output.

UPDATE December 19, 2020 (v1.2)

Domain names that have been base32 encoded, such as domain names with uppercase letters, can now be extracted with SunburstDomainDecoder. The queried SUNBURST subdomains are now also included in the output.

UPDATE December 21, 2020 (v1.6)

Improved parsing of base32 encoded domain names. SUNBURST victim domains like "LKDataCenter.com", "Sunkistgrowers.com" and "BrokenArrow.Local" can now be extracted.

UPDATE December 27, 2020 (v1.7)

Improved reassembly of long domain names, like "CIMBMY.CIMBDomain.com" and "BE.AJINOMOTO-OMNICHEM.AD", that get segmented into multiple parts. Extraction of time stamps and security applications, including "Windows Defender", "Carbon Black", "CrowdStrike", "FireEye", "ESET" and "F-Secure". See Sergei Shevchenko's blog post Sunburst Backdoor, Part III: DGA & Security Software for more details.

UPDATE January 4, 2021 (v1.8)

Security products (WinDefend, ESET etc.) are now included in the summary output at the end. SUNBURST stage2 victims, which accept C2 domains in CNAME responses, are indicated with a "STAGE2" tag. The previous release marked stage2 queries with a "DNSSEC" tag. Improved extraction of truncated base32 domains, such as "*TED.com".

UPDATE January 12, 2021 (v1.9)

DNS queries with encoded timestamps are tagged with either "AVProducts" or "Ping", depending on if they include an update of the installed/running security products and services or not. The summary data at the end has been modified to also show partial domain names, such as "paloaltonetworks*".

UPDATE February 16, 2021 (v2.0)

Slightly faster and even more accurate than previous versions.

Download SunburstDomainDecoder.zip

SUNBURST DNS Traffic

SUNBURST victims, who have installed one of the trojanized SolarWinds Orion software updates, will query for domain names formatted like this:

The "SUBDOMAIN" string has different values for each victim and the second half of this string actually contains an encoded domain name (encrypted with a simple substitution cipher).

RedDrip's decode.py

The RedDrip Team published a SUNBURST DGA decoding script yesterday, which can be used to identify SUNBURST victim organizations like CISCO and Belkin by decoding the domain names encoded in the outgoing DNS queries for subdomains of avsvmcloud.com.

This is what it looks like when RedDrip's decode.py script is fed with domain names from John Bambenek's uniq-hostnames.txt file.

The beauty of this approach is that passive DNS data can be used in order to reliably identify the victims. This is great news for national CERTs, because they typically have readily access to passive DNS data and can use the decoded domain names in order to identify and reach out to victims in their country.

After using the python script provided by ReadDrip Team I noticed two things:

1. The leaked domain names were internal domain names used on the victim organizations' corporate networks. Many of the domains were using the ".local" suffix.
2. Most of the extracted domains were truncated to around 15 bytes, which make it difficult to identify the victim organization.

Truncated Domains Fragmented Domains

I later learned that what seemed to be truncated domains were actually fragmented domains, where long domain names would be split into multiple queries. This revelation turns the output from RedDrip's python tool into an interesting domain name puzzle. At this point I decided to take a closer look at the malicious SolarWinds update I had downloaded from SolarWind's website a few days ago -- yes, that's right the malicious software update "SolarWinds-Core-v2019.4.5220-Hotfix5.msp" (MD5: 02af7cec58b9a5da1c542b5a32151ba1) was actually available for download from SolarWinds' website long after they had been notified about their software being backdoored!

As an example, lets' take a closer look at this DNS query from John Bambenek's passive DNS data:
r1qshoj05ji05ac6eoip02jovt6i2v0c.appsync-api.us-west-2.avsvmcloud.com

This query can be broken down into three parts:

1. r1qshoj05ji05ac6 : What is encoded here???
2. eoip02jovt6i2v0c : Base32 encoded string "city.kingston."
3. .appsync-api.us-west-2.avsvmcloud.com : DNS trailer without encoded data

So, which "City of Kingston", or "Kingston City", should we contact to let them know that they have installed a trojanized SolarWinds update? Is it Kingston Jamaica, City of Kingston NY USA, City of Kingston Ontario Canada, Kingston City Tennessee USA or City of Kingston Australia?

After analyzing the "SolarWinds.Orion.Core.BusinessLayer.dll" file (MD5: b91ce2fa41029f6955bff20079468448) from the "SolarWinds-Core-v2019.4.5220-Hotfix5.msp" I learned that the initial "r1qshoj05ji05ac6" string is representing a unique "GUID" value for the infected machine. This GUID is generated by calculating an MD5 hash of the MAC address of the first active non-Loopback network interface, the domain name and the "MachineGuid" registry key value in "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography".

This MD5 hash is then squeezed into a tiny 8 byte array by XOR'ing overlapping bytes. The "CreateSecureString" function in the trojanized SolarWinds update then "encrypts" this hash using XOR with a random key, which is prepended to the data. The XOR key and the XOR'ed data is then finally base32 encoded into what makes up the first part of the subdomain to query for. Don't let the SUNBURST source code below fool you, it is actually using base32 encoding with a custom alphabet even though the function is called "Base64Encode";

Image: SUNBURST source code generates a random value between 1 and 127 as XOR key

Each DNS lookup from an infected machine will query for a unique subdomain because a new XOR key will be generated for each request. Luckily for us, this XOR key is provided in each request, so we can use it in order to "decrypt" the subdomain and get the original 8 bytes derived from the MAC+domain+MachineGuid MD5 hash.

The output from my "SunburstDomainDecoder.exe" tool will print the "decrypted" 8 byte GUID in the first column, the decoded victim domain segment or timestamp in the second column and the queried SUNBURST subdomain in the last column. Each DNS query line read from standard input will generate a "GUID DecodedHostname SunburstSubdomain" line on standard output.

The tool then finishes off by outputting the domains that are complete or at least have the last part of their domain intact. Some of these domains are complete because they were short enough to fit in one single SUNBURST DNS query, while others have been pieced together by SunburstDomainDecoder from domain fragments arriving in separate SUNBURST DNS queries.

We can now see that it was "city.kingston.on.ca", (City of Kingston, Ontario, Canada) who had installed a trojanized SolarWinds update.

Download SunburstDomainDecoder

The C# source code and a compiled Windows binary for SunburstDomainDecoder is available here:
https://www.netresec.com/files/SunburstDomainDecoder.zip

The source code and Windows binary is shared under a Creative Commons CC-BY license, which means that you are free to:

• Share : copy and redistribute the material in any medium or format
• Adapt : remix, transform, and build upon the material for any purpose, even commercially.

Running SunburstDomainDecoder on Linux/MacOS

Wanna run SunburstDomainDecoder.exe but not in Windows? No problems, the tool runs perfectly fine in Mono. Another option is to build SunburstDomainDecoder.cs as a .NET core project in Linux.

.NET Reversing

Would you like to verify my findings or learn more about .NET reverse engineering? Cool, then I'd recommend that you download dnSpy in order to reverse engineer the SUNBURST .NET DLL (which can be extracted from the msp installer with 7zip). Or you can have a look at the already extracted OrionImprovementBusinessLayer.cs on GitHub.

Posted by Erik Hjelmvik on Thursday, 17 December 2020 22:30:00 (UTC/GMT)

Tags: #SunburstDomainDecoder​ #SUNBURST​ #SolarWinds​ #Solorigate​ #domain​ #DNS​ #pDNS​ #Windows Defender​ #Carbon Black​ #FireEye​ #ESET​ #F-Secure​ #Trojan​ #avsvmcloud​

Installing a Fake Internet with INetSim and PolarProxy

This is a tutorial on how to set up an environment for dynamic malware analysis, which can be used to analyze otherwise encrypted HTTPS and SMTPS traffic without allowing the malware to connect to the Internet. Dynamic malware analysis (or behavioral analysis) is performed by observing the behavior of a malware while it is running. The victim machine, which executes the malware, is usually a virtual machine that can be rolled back to a clean state when the analysis is complete. The safest way to prevent the malware from infecting other machines, or doing other bad things like sending SPAM or taking part in DDoS attacks, is to run the victim machine in an offline environment. However, network traffic analysis of malware is a central part of dynamic malware analysis, which is is why a “fake Internet” is needed in most malware labs.

INetSim and PolarProxy

INetSim is a software suite that simulates common internet services like HTTP, DNS and SMTP, which useful when analyzing the network behavior of malware samples without connecting them to the Internet. INetSim also has basic support for TLS encrypted protocols, like HTTPS, SMTPS, POP3S and FTPS, but requires a pre-defined X.509-certificate to be loaded at startup. This can cause malware to terminate because the Common Names (CN) in the presented certificates don’t match the requested server names. The victim machine will actually get the exact same certificate regardless of which web site it visits. INetSim’s TLS encryption also inhibits analysis of the network traffic captured in the malware lab, such as C2 traffic or SPAM runs, because the application layer traffic is encrypted. PolarProxy can solve both these issues because it generates certificates on the fly, where the CN value is dynamically set to the requested host name, and saves the network traffic in decrypted form to PCAP files. It is therefore a good idea to replace the TLS services in INetSim with PolarProxy, which will be used as a TLS termination proxy that forwards the decrypted traffic to INetSim’s cleartext services.

Install Linux

The first step is to install a Linux VM, which will act as a fake Internet to the victim machine(s). I'm using Ubuntu Server 18.04.3 LTS in this tutorial, but you can use any 64-bit linux distro. I'm adding two network interfaces to the Linux VM, one interface with Internet access and one that connects to an isolated offline network to which the victim VM's will be connected. The offline interface is configured to use the static IP 192.168.53.19.

Important: Do not bridge, bond or enable IP forwarding between the two interfaces!

Install INetSim

INetSim is available in Ubuntu's repo, so it is possible to install it with "apt install inetsim". However, I recommend installing INetSim as described in the official documentation to get the latest packaged version of INetSim.

INetSim listens on 127.0.0.1 by default, change this to INetSim's offline IP address by un-commenting and editing the service_bind_address variable in /etc/inetsim/inetsim.conf.

Also configure INetSim's fake DNS server to resolve all domain names to the IP of INetSim with the dns_default_ip setting:

Finally, disable the "start_service https" and "start_service smtps" lines, because these services will be replaced with PolarProxy:

Restart the INetSim service after changing the config.

Verify that you can access INetSim's HTTP server with curl:

It looks like INetSim's web server can be accessed alright.

Install PolarProxy

Next step is to install PolarProxy as a systemd service (as instructed here):

We will need to modify the PolarProxy service config file a bit before we start it. Edit the ExecStart setting in /etc/systemd/system/PolarProxy.service to configure PolarProxy to terminate the TLS encryption for HTTPS and SMTPS (implicitly encrypted email submission). The HTTPS traffic should be redirected to INetSim's web server on tcp/80 and the SMTPS to tcp/25.

Here's a break-down of the arguments sent to PolarProxy through the ExecStart setting above:

• -v : verbose output in syslog (not required)
• -p 10443,80,80 : listen for TLS connections on tcp/10443, save decrypted traffic in PCAP as tcp/80, forward traffic to tcp/80
• -p 10465,25,25 : listen for TLS connections on tcp/10465, save decrypted traffic in PCAP as tcp/25, forward traffic to tcp/25
• -x /var/log/PolarProxy/polarproxy.cer : Save certificate to be imported to clients in /var/log/PolarProxy/polarproxy.cer (not required)
• -f /var/log/PolarProxy/proxyflows.log : Log flow meta data in /var/log/PolarProxy/proxyflows.log (not required)
• -o /var/log/PolarProxy/ : Save PCAP files with decrypted traffic in /var/log/PolarProxy/
• --certhttp 10080 : Make the X.509 certificate available to clients over http on tcp/10080
• --terminate : Run PolarProxy as a TLS termination proxy, i.e. data forwarded from the proxy is decrypted
• --connect 192.168.53.19 : forward all connections to the IP of INetSim
• --nosni nosni.inetsim.org : Accept incoming TLS connections without SNI, behave as if server name was "nosni.inetsim.org".

Finally, start the PolarProxy systemd service:

Verify that you can reach INetSim through PolarProxy's TLS termination proxy using curl:

Yay, it is working! Do the same thing again, but also verify the certificate against PolarProxy's root CA this time. The root certificate is downloaded from PolarProxy via the HTTP service running on tcp/10080 and then converted from DER to PEM format using openssl, so that it can be used with curl's "--cacert" option.

Yay #2!

Now let's set up routing to forward all HTTPS traffic to PolarProxy's service on tcp/10443 and SMTPS traffic to tcp/10465. I'm also adding a firewall rule to redirect ALL other incoming traffic to INetSim, regardless of which IP it is destined to, with the final REDIRECT rule. Make sure to replace "enp0s8" with the name of your interface.

Verify that the iptables port redirection rule is working from another machine connected to the offline 192.168.53.0/24 network:

Yay #3!

Yay #4!

It is now time to save the firewall rules, so that they will survive reboots.

Install the Victim Windows PC

Configure a static IP address on the victim Windows host by manually setting the IP address. Set the INetSim machine (192.168.53.19) as the default gateway and DNS server.

Download the X.509 root CA certificate from your PolarProxy installation here:
http://192.168.53.19:10080/polarproxy.cer

1. Double-click on "polarproxy.cer"
2. Click [Install Certificate...]
3. Select 🔘 Local Machine and press [Next]
4. Select 🔘 Place all certificates in the following store and press [Browse...]
5. Choose "Trusted Root Certification Authorities" and press [OK], then [Next]
6. Press [Finish]

You might also want to install the PolarProxy certificate in your browser. This is how you install it to Firefox:

1. Options / Preferences
2. Press [Privacy & Security]
3. Scroll down to "Certificates" and press [View Certificates...]
4. In the "Authorities" tab, press [Import...]
5. Open "polarproxy.cer"
6. ☑ Trust this CA to identify websites. (check the box)
7. Press [OK]

Now, open a browser and try visiting some websites over HTTP or HTTPS. If you get the following message regardless of what domain you try to visit, then you've managed to set everything up correctly:

Accessing the Decrypted Traffic

PCAP files with decrypted HTTPS and SMTPS traffic are now available in /var/log/PolarProxy/

PolarProxy will start writing to a new capture file every 60 minutes. However, the captured packets are not written to disk instantly because PolarProxy uses buffered file writing in order to improve performance. You can restart the proxy service if you wish to flush the buffered packets to disk and have PolarProxy rotate to a new capture file.

I also recommend capturing all network traffic sent to INetSim with a sniffer like netsniff-ng. This way you’ll get PCAP files with traffic from INetSim’s cleartext services (like DNS and HTTP) as well.

PCAP or it didn’t happen!

Credits

I'd like to thank Thomas Hungenberg and Patrick Desnoyers for providing valuable feedback for this blog post!

Posted by Erik Hjelmvik on Monday, 09 December 2019 08:40:00 (UTC/GMT)

Tags: #PolarProxy​ #HTTPS​ #SMTPS​ #HTTP​ #SMTP​ #DNS​ #Malware​ #Sandbox​ #TLS​ #PCAP​ #proxy​ #tutorial​ #ASCII-art​

CapLoader 1.8 Released

We are happy to announce the release of CapLoader 1.8 today!

CapLoader is primarily used to filter, slice and dice large PCAP datasets into smaller ones. This new version contains several new features that improves this filtering functionality even further. To start with, the “Keyword Filter” can now be used to filter the rows in the Flows, Services or Hosts tabs using regular expressions. This enables the use of matching expressions like this:

• amazon|akamai|cdn
  Show only rows containing any of the strings “amazon” “akamai” or “cdn”.
• microsoft\.com\b|windowsupdate\.com\b
  Show only servers with domain names ending in “microsoft.com” or “windowsupdate.com”.
• ^SMB2?$
  Show only SMB and SMB2 flows.
• \d{1,3}\.\d{1,3}\.\d{1,3}\.255$
  Show only IPv4 address ending with “.255”.

For a reference on the full regular expression syntax available in CapLoader, please see Microsoft’s regex “Quick Reference”.

One popular workflow supported by CapLoader is to divide all flows (or hosts) into two separate datasets, for example one “normal” and one “malicious” set. The user can move rows between these two sets, where only one set is visible while the rows in the other set are hidden. To switch which dataset that is visible versus hidden the user needs to click the [Invert Hiding] button (or use the [Ctrl]+[Tab] key combination). With this new release we’ve also made the “Invert Hiding” functionality available by clicking the purple bar, which shows the number of rows present in the currently viewed set.

Readers with a keen eye might also notice that the purple bar charts are now also accompanied by a number, indicating how many rows that are visible after each filter is applied. The available filters are: Set Selection, BPF and Keyword Filter.

NetFlow + DNS = Great Success!

CapLoader’s main view presents the contents of the loaded PCAP files as a list of netflow records. Since the full PCAP is available, CapLoader also parses the DNS packets in the capture files in order to enrich the netflow view with hostnames. Recently PaC shared a great idea with us, why not show how many failed DNS lookups each client does? This would enable generic detection of DGA botnets without using blacklists. I’m happy to announce that this great idea made it directly into this new release! The rightmost column in CapLoader’s hosts tab, called “DNS_Fails”, shows how many percent of a client’s DNS requests that have resulted in an NXRESPONSE or SRVFAIL response.

Two packet capture files are loaded into CapLoader in the screenshot above; one PCAP file from a PC infected with the Shifu malware and one PCAP file with “normal traffic” (thanks @StratosphereIPS for sharing these capture files). As you can see, one of the clients (10.0.2.107) has a really high DNS failure ratio (99.81%). Unsurprisingly, this is also the host that was infected with the Shifu, which uses a domain generation algorithm (DGA) to locate its C2 servers.

Apart from parsing A and CNAME records from DNS responses CapLoader now also parses AAAA DNS records (IPv6 addresses). This enables CapLoader to map public domain names to hosts with IPv6 addresses.

Additional Updates

The new CapLoader release also comes with several other new features and updates, such as:

• Added urlscan.io service for domain and IP lookups (right-click a flow or host to bring up the lookup menu).
• Flow ID coloring based on 5-tuple, and clearer colors in timeline Gantt chart.
• Extended default flow-timeout from 10 minutes to 2 hours for TCP flows.
• Changed flow-timout for non-TCP flows to 60 seconds.
• Upgraded to .NET Framework 4.7.2.

Updating to the Latest Release

Users who have previously purchased a license for CapLoader can download a free update to version 1.8 from our customer portal. All others can download a free 30 day trial from the CapLoader product page (no registration required).

Credits

We’d like to thank Mikael Harmark, Mandy van Oosterhout and Ulf Holmström for reporting bugs that have been fixed in this release. We’d also like to thank PaC for the DNS failure rate feature request mentioned in this blog post.

Posted by Erik Hjelmvik on Tuesday, 28 May 2019 10:45:00 (UTC/GMT)

Tags: #CapLoader​ #NetFlow​ #regex​ #DNS​ #DGA​ #Stratosphere​