Honeypot Network Forensics

NCC Group recently released a 500 MB PCAP file containing three months of honeypot web traffic data related to the F5 remote code execution vulnerability CVE-2020-5902. In a blog post the NCC Group say that their objective is "to enable all threat intelligence researchers to gain further understanding and contribute back to the community".

The data in NCC's 500 MB capture file "f5-honeypot-release.pcap" ranges from July 7 up until September 28 and contains traffic from over 4000 unique client IP addresses. The packets are captured after having passed through a proxy, which is why all clients have IP "123.45.67.89" and the server is always displayed as "127.0.0.1". This makes it difficult to split or filter the traffic based on the client IP address. Many HTTP headers have also been masked in the capture file, but the IP portion of the "X-Forwarded-For" header is still intact. You can therefore track clients throughout the capture file by filtering on the "X-Forwarded-For" header, which contains the originating IP address of the client connecting to the proxy. As an example, you can use the following tshark command in order to count the exact number of unique clients in the released PCAP file:

Tracking a Single Actor

I decided to focus on one of the over 4000 clients in NCC's capture file in this blog post. The selected actor is primarily originating from the IP address 45.12.206.76, which is a VPN egress point according to Scamalytics. I can therefore not be certain that only a single actor is caught with this filter, but the consistent behavior across the various sessions indicates so.

Some characteristic traits of the actor's traffic is:

• Web browser User-Agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0
• Uses VPN from Think Huge Ltd, IP range announced by AS9009 "M247 LTd"
• Active between 21:55 and 05:08 UTC on weekdays as well as weekends

I started by opening the full 500 MB PCAP file from NCC Group in CapLoader. Then I narrowed the dataset down to a single client IP address by clicking "Edit -> Find Keyword" and entering the string "X-Forwarded-For: 45.12.206.76".

After pressing "Find All Matching Flows" I got a much smaller dataset (144 kB) containing only the sessions for this specific IP address.

July 7 22:16 UTC - First Contact

The actor with IP address "45.12.206.76" can be observed attacking the vulnerable F5 device on July 7 at 22:16 UTC. The attack started with an attempt at exploiting the authentication bypass vulnerability in CVE-2020-5902.

The attacker wrote the following content to "/tmp/.X11.1" by sending an HTTP POST to "fileSave.jsp":

The "/tmui/login.jsp" page can be accessed without authentication, so no credentials are needed for this request. But as explained in Orange Tsai's BlackHat 2018 talk "Breaking Parser Logic!" the TMUI Tomcat service will interpret "..;" as the parent directory, which yields URI "/tmui/locallb/workspace/fileSave.jsp" that would otherwise only be available to authenticated users.

I didn't find any attempt to execute the bash script in the "/tmp/.X11.1" file though.

July 9 04:49 UTC - Same Actor, New IP

The same actor came back again on July 9, but this time from the IP address "185.160.24.70". The actions carried out are pretty much a reiteration of the previous attempts, i.e. writing "set -e; cat /etc/shadow; cat /etc/hosts; [...]" (exact same commands as last time) to "/tmp/.X11.1". The commands, the name of the temp file and the HTTP headers are identical which is why I'm pretty confident that this is the same actor. Both IP addresses (45.12.206.76 and 185.160.24.70) also seem to originate from the same VPN service, since the networks are announced by AS9009 (M247 Ltd) and both networks are registered to "Think Huge Ltd" who run VPN's as part of their business.

July 18 21:55 UTC - Java Deserialization Attack

More than a week later there is activity from the 45.12.206.76 IP address again. But this time the attacker has changed approach to instead inject a serialized piece of Java code by posting it to "/hsqldb;" This is an attempt at triggering the deserialization vulnerability in CVE-2020–5902.

As you've probably noticed the argument provided to the org.hsqldb.util.ScriptTool.main function is ASCII-hex encoded. Decoding it gives a serialized Java blob containing a call to the org.apache.commons.collections.functors.InvokerTransformer constructor with methodName "exec" and a long string as args. You can use SerializationDumper to see the full structure. The full argument provided to the InvokerTransformer constructor was:

Base64 decoding the data reveals the following command sent to the TMOS Shell (tmsh):

The HTTP response contains a string saying "General error java.lang.IllegalArgumentException: argument type mismatch", which could indicate that the exploit failed. However that is probably just a side effect of the exploit, so the injected bash command might still have executed. Nevertheless the attacker POST'ed the exact same exploit three more times, getting the same IllegalArgumentException error message each time.

The attacker came back with a slightly modified payload for the deserialization attack 20 minutes later. The new payload executed this tmsh command instead:

July 19 02:54 UTC - Web Shell

Five hours after the first deserialization attack the actor came back, this time with a call to F5's "iControl REST API" at URI "/mgmt/tm/util/bash".

It looks like a bash command was supplied in the JSON data. The documentation for the iControl REST API confirms that this is a built in feature, not a bug or vulnerability:

The utilCmdArgs name is used to provide the command line arguments for the Advanced Shell (bash) utility. The -c option in bash is used to process any system commands [...]

However, this type of request can only be carried out by an authenticated user. The HTTP request used basic auth with "c25tcGQ6QWJDZDAwN3hzdzI=" as credential, which decodes into "snmpd:AbCd007xsw2". You probably recognize this credential from the deserialization attack, where the command "create auth user snmpd password AbCd007xsw2" was issued. The password in the basic auth header can also be observed in NetworkMiner's "Credentials" tab.

NetworkMiner can also be used to list all all occurrences of the "utilCmdArgs" parameter.

NetworkMiner shows us that the "cat /etc/hosts" command was issued four times. We can see that the command executed successfully, because the HTTP response comes back with a JSON formatted result containing the following output from the cat command:

#
# THIS IS AN AUTO-GENERATED FILE - DO NOT EDIT!!!
#
# Use the tmsh shell utility to make changes to the system configuration.
# For more information, see tmsh -a help sys global-settings.
127.0.0.1 localhost.localdomain localhost bigip1.localhost.localdomain
127.2.0.1 sccp aom AOM
127.2.0.2 bigip1.localhost.localdomain
127.1.1.255 tmm-bcast
127.1.1.253 tmm-shared
127.1.1.254 tmm
127.1.1.1 tmm0
127.1.1.2 tmm1
10.164.0.2 bigip1.localhost.localdomain
220.64.53.10 f5update.ddns.net

You probably recognize this command sequence, which was previously posted to "/tmp/.X11.1" on July 7 and 9 by attempting to exploit the authentication bypass vulnerability in CVE-2020-5902. However, this time the output from the command comes back in the HTTP response in form of a 16 kB JSON blob.

The PCAP file from NCC Group unfortunately only contain the first 8 kB of the full 16 kB JSON data returned from the server. It looks as if the data was cut of just before the contents of "/config/bigip.license" were about to be transferred. By looking at the TCP ACK numbers coming back from 45.12.206.76 it looks as if the complete 16 kB output was actually transferred to the attacker, but this traffic has been removed from the published capture file. Maybe this was done in order to protect the contents of NCC's "bigip.license" file?

Other Attacks with Similar Password Scheme

This blog post has outlined how an attacker created a user account called "snmpd" on NCC Group's F5 honeypot. The password for this user account was set to "AbCd007xsw2" by the attacker. In the dataset published by NSS I noticed two other attacks where user accounts were created with very similar passwords.

The first one creates a user account named "system" with password "ABcD007...A01". This user account was created by an attacker, coming from IP address 185.220.101.214 on July 7, with the following payload in another deserialization attack:

The IP address used in that attack (185.220.101.214) was an active Tor relay on July 7.

The other attack where a similar password was observed originated from IP address 198.13.54.223 on August 3'rd. This time the attacker attempted to authenticate to "/tmui/logmein.html" with username "admin" and password "aBcD008@@Ws0A".

However, as can be seen in the screenshot above, the login was not successful.

I'm not sure if this is a password scheme used by a specific actor or if it is an artifact of a common tool used by different actors.

Finally, there was also an actor coming in from IP address 185.201.9.198 who performed actions related to those described in this blog post. However, I'd like to cover that traffic in a separate blog post.

Posted by Erik Hjelmvik on Wednesday, 21 October 2020 12:35:00 (UTC/GMT)

Tags: #PCAP​ #CapLoader​ #NetworkMiner​