CapLoader 1.3 Released

A new version of our heavy-duty PCAP parser tool CapLoader is now available. There are many new features and improvements in this release, such as the ability to filter flows with BPF, domain name extraction via passive DNS parser and matching of domain names against a local white list.

Filtering with BPF

The main focus in the work behind CapLoader 1.3 has been to fully support the Rinse-Repeat Intrusion Detection methodology. We've done this by improving the filtering capabilities in CapLoader. For starters, we've added an input filter, which can be used to specify IP addresses, IP networks, protocols or port numbers to be parsed or ignored. The input filter uses the Berkeley Packet Filter (BPF) syntax, and is designed to run really fast. So if you wanna analyze only HTTP traffic you can simply write “port 80” as your input filter to have CapLoader only parse and display flows going to or from port 80. We have also added a display filter, which unlike Wireshark also uses BPF. Thus, once a set of flows is loaded one can easily apply different display filters, like “host 194.9.94.80” or “net 192.168.1.0/24”, to apply different views on the parsed data.

The main differences between the input filter and display filter are:

• Input filter is much faster than the display filter, so if you know beforehand what ports, protocols or IP addresses you are interested in then make sure to apply them as an input filter. You will notice a delay when applying a display filter to a view of 10.000 flows or more.
• In order to apply a new input filter CapLoader has to reload all the opened PCAP files (which is done by pressing F5). Modifying display filters, on the other hand, only requires you to press Enter or hit the “Apply” button.
• Previously applied display filters are accessible in a drop-down menu in the GUI, but no history is kept of previous input filters.

NetFlow + DNS == true

The “Flows” view in CapLoader gives a great overview of all TCP, UDP and SCTP flows in the loaded PCAP files. However, it is usually not obvious to an analyst what every IP address is used for. We have therefore added a DNS parser to CapLoader, so that all DNS packets can be parsed in order to map IP addresses to domain names. The extracted domain names are displayed for each flow, which is very useful when performing Rinse-Repeat analysis in order to quickly remove “known good servers” from the analysis.

Leveraging the Alexa top 1M list

As we've show in in our previous blog post “DNS whitelisting in NetworkMiner”, using a list of popular domain names as a whitelist can be an effective method for finding malware. We often use this approach in order to quickly remove lots of known good servers when doing Rinse-Repeat analysis in large datasets.

Therefore, just as we did for NetworkMiner 1.5, CapLoader now includes Alexa's list of the 1 million most popular domain names on the Internet. All domain names, parsed from DNS traffic, are checked against the Alexa list. Domains listed in the whitelist are shown in CapLoader's “Server_Alexa_Domian” column. This makes it very easy to sort on this column in order to remove (hide) all flows going to “normal” servers on the Internet. After removing all those flows, what you're left with is pretty much just:

• Local traffic (not sent over the Internet)
• Outgoing traffic to either new or obscure domains

Manually going through the remaining flows can be very rewarding, as it can reveal C2 traffic from malware that has not yet been detected by traditional security products like anti-virus or IDS.

Many new features in CapLoader 1.3

The new features highlighted above are far from the only additions made to CapLoader 1.3. Here is a more complete list of improvements in this release:

• Support for “Select Flows in PCAP” to extract and select 5-tuples from a PCAP-file. This can be a Snort PCAP with packets that have triggered IDS signatures. This way you can easily extract the whole TCP or UDP flow for each signature match, instead of just trying to make sense of one single packet per alert.
• Improved packet carver functionality to better carve IP, TCP and UPD packets from any file. This includes memory dumps as well as proprietary and obscure packet capture formats.
• Support for SCTP flows.
• DNS parser.
• Alexa top 1M matching.
• Input filter and display filter with BPF syntax.
• Flow Producer-Consumer-Ratio PCR.
• Flow Transcript can be opened simply by double-clicking a flow.
• Find form updated with option to hide non-matching flows instead of just selecting the flows that matched the keyword search criteria.
• New flow transcript encoding with IP TTL, TCP flags and sequence numbers to support analysis of Man-on-the-Side attacks.
• Faster loading of previously opened files, MD5 hashes don't need to be recalculated.
• A selected set of flows in the GUI can be inverted simply by right-clicking the flow list and selecting “Invert Selection” or by hitting Ctrl+I.

Downloading CapLoader 1.3

All these new features, except for the Alexa lookup of domain names, are available in our free trial version of CapLoader. So to try out these new features in CapLoader, simply grab a trial download here:
https://www.netresec.com/?page=CapLoader#trial (no registration needed)

All paying customers with an older version of CapLoader can grab a free update for version 1.3 at our customer portal.

Posted by Erik Hjelmvik on Monday, 28 September 2015 07:30:00 (UTC/GMT)

Tags: #CapLoader​ #BPF​ #Berkeley Packet Filter​ #Rinse-Repeat​ #DNS​ #Alexa​ #PCAP​ #Passive DNS​ #NetFlow​ #Malware​ #C2​

Rinse-Repeat Intrusion Detection

I am a long time skeptic when it comes to blacklists and other forms of signature based detection mechanisms. The information security industry has also declared the signature based anti-virus approach dead several times during the past 10 years. Yet, we still rely on anti-virus signatures, IDS rules, IP blacklists, malware domain lists, YARA rules etc. to detect malware infections and other forms of intrusions in our networks. This outdated approach puts a high administrative burden on IT and security operations today, since we need to keep all our signature databases up to date, both when it comes to end point AV signatures as well as IDS rules and other signature based detection methods and threat feeds. Many organizations probably spend more time and money on updating all these blacklists and signature databases than actually investigating the security alerts these detection systems generate. What can I say; the world is truly upside down...

I would therefore like to use this blog post to briefly describe an effective blacklist-free approach for detecting malware and intrusions just by analyzing network traffic. My approach relies on a combination of whitelisting and common sense anomaly detection (i.e. not the academic statistical anomaly detection algorithms that never seem to work in reality). I also encourage CERT/CSIRT/SOC/SecOps units to practice Sun Tzu's old ”know yourself”, or rather ”know your systems and networks” approach.

Know your enemy and know yourself and you can fight a hundred battles without disaster.

My method doesn't rely on any dark magic, it is actually just a simple Rinse-Repeat approach built on the following steps:

After looping through these steps a few times you'll be left with some odd network traffic, which will have a high ratio of maliciousness. The key here is, of course, to know what traffic to classify as ”normal”. This is where ”know your systems and networks” comes in.

What Traffic is Normal?

I recently realized that Mike Poor seems to be thinking along the same lines, when I read his foreword to Chris Sanders' and Jason Smith's book Applied NSM:

The next time you are at your console, review some logs. You might think... "I don't know what to look for". Start with what you know, understand, and don't care about. Discard those. Everything else is of interest.

Following Mike's advice we might, for example, define“normal” traffic as:

• HTTP(S) traffic to popular web servers on the Internet on standard ports (TCP 80 and 443).
• SMB traffic between client networks and file servers.
• DNS queries from clients to your name server on UDP 53, where the servers successfully answers with an A, AAAA, CNAME, MX, NS or SOA record.
• ...any other traffic which is normal in your organization.

Whitelisting IP ranges belonging to Google, Facebook, Microsoft and Akamai as ”popular web servers” will reduce the dataset a great deal, but that's far from enough. One approach we use is to perform DNS whitelisting by classifying all servers with a domain name listed in Alexa's Top 1 Million list as ”popular”.

You might argue that such a method just replaces the old blacklist-updating-problem with a new whitelist-updating-problem. Well yes, you are right to some extent, but the good part is that the whitelist changes very little over time compared to a blacklist. So you don't need to update very often. Another great benefit is that the whitelist/rinse-repeat approach also enables detection of 0-day exploits and C2 traffic of unknown malware, since we aren't looking for known badness – just odd traffic.

Threat Hunting with Rinse-Repeat

Mike Poor isn't the only well merited incident handler who seems to have adopted a strategy similar to the Rinse-Repeat method; Richard Bejtlich (former US Air Force CERT and GE CIRT member) reveal some valuable insight in his book “The Practice of Network Security Monitoring”:

I often use Argus with Racluster to quickly search a large collection of session data via the command line, especially for unexpected entries. Rather than searching for specific data, I tell Argus what to omit, and then I review what’s left.

In his book Richard also mentions that he uses a similar methodology when going on “hunting trips” (i.e. actively looking for intrusions without having received an IDS alert):

Sometimes I hunt for traffic by telling Wireshark what to ignore so that I can examine what’s left behind. I start with a simple filter, review the results, add another filter, review the results, and so on until I’m left with a small amount of traffic to analyze.

I personally find Rinse-Repeat Intrusion Detection ideal for threat hunting, especially in situations where you are provided with a big PCAP dataset to answer the classic question “Have we been hacked?”. However, unfortunately the “blacklist mentality” is so conditioned among incident responders that they often choose to crunch these datasets through blacklists and signature databases in order to then review thousands of alerts, which are full of false positives. In most situations such approaches are just a huge waste of time and computing power, and I'm hoping to see a change in the incident responders' mindsets in the future.

I teach this “rinse-repeat” threat hunting method in our Network Forensics Training. In this class students get hands-on experience with a dataset of 3.5 GB / 40.000 flows, which is then reduced to just a fraction through a few iterations in the rinse-repeat loop. The remaining part of the PCAP dataset has a very high ratio of hacking attacks as well as command-and-control traffic from RAT's, backdoors and botnets.

UPDATE 2015-10-07

We have now published a blog post detailing how to use dynamic protocol detection to identify services running on non-standard ports. This is a good example on how to put the Rinse-Repeat methodology into practice.

Posted by Erik Hjelmvik on Monday, 17 August 2015 08:45:00 (UTC/GMT)

Tags: #Rinse-Repeat​ #PCAP​ #NSM​ #PCAP​ #Intrusion Detection​ #IDS​ #network forensics​