NetworkMiner 2.3.2 was released this morning, and there was much rejoicing!
Image: U.S. Navy photo by Stuart Phillips (source)
This new release primarily fixes bugs related to extraction of
emails and
VoIP calls.
We have also corrected a bug affecting the json/CASE
export function in NetworkMiner Professional.
The OSINT domain name lookup in NetworkMiner Professional has also been extended with the
crt.sh Certificate Search and DNSTrails has been replaced with
SecurityTrails.
We’d like to thank Carlos Kasprzykowski for notifying us about the VoIP bug,
which caused lots of files to be written to the %TEMP% directory
when there were more than 50 simultaneous SIP+RTP calls.
We also wanna thank Josh Wilczek for reporting a bug in the “User Defined Port-to-Protocol Mappings” in NetworkMiner Professional’s Settings window, which also has been fixed in the 2.3.2 release.
Upgrading to Version 2.3.2
Users who have purchased a license for NetworkMiner Professional 2.x can download a free update to version 2.3.2 from our
customer portal.
Those who instead prefer to use the free and open source version can grab the latest version
of NetworkMiner from the official
NetworkMiner page.
⛏ FOR GREAT JUSTICE! ⛏
Posted by Erik Hjelmvik on Monday, 27 August 2018 09:23:00 (UTC/GMT)
The free and open source network forensics tool NetworkMiner now comes with improved extraction of files
and metadata from several protocols as well as a few GUI updates.
But the biggest improvements for version 2.3 are in the commercial tool NetworkMiner Professional,
which now supports VoIP call audio extraction and playback as well as OSINT lookups of file hashes,
IP addresses, domain names and URLs.
I’m happy to announce that NetworkMiner 2.3 now does an even better job than before at extracting files
and metadata from several protocols. Improvements have been made in the parsers for the following protocols:
HTTP, IEC-104, IPv4, Modbus, SIP, SMB, SMB2, SMTP and SSL/TLS.
We have also added support for the SNMP protocol in NetworkMiner 2.3,
so that SNMP community strings can be extracted and displayed on the Parameters and Credentials tabs.
Another change is that timestamps are now displayed using the UTC time zone instead of using the local time zone.
We have also fixed a few GUI quirks in order to further improve the usability of the tool.
NetworkMiner Professional
The commercial version of NetworkMiner, i.e. NetworkMiner Professional,
comes with several additional improvements which are presented below.
VoIP Call Playback
NetworkMiner Professional has received a new tab called “VoIP”,
which enables audio playback of VoIP calls that are using SIP and RTP with G.711 μ-law or A-law encoding
(u-Law is primarily used in North America and Japan while A-law is used in Europe and most other parts of the world).
Video: Audio playback and extraction to WAV from the “SIP_CALL_RTP_G711” PCAP file in the
Wireshark Sample Captures.
The audio streams from the VoIP calls are also extracted to disk as .WAV files when codecs
G.729 or
G.711
(u-Law and A-Law) is used. NetworkMiner Professional also attempts to reassemble RTP streams encoded with
G.722 to .au files.
OSINT Lookups of IP Addresses, Domains, URLs and File Hashes
Right-clicking a network host in NetworkMiner Professional’s Hosts tab brings up a context menu with options for performing lookups of IP
and domain names using external sources. We refer to this method as open-source intelligence (OSINT) because the accessed data resides at
publicly available sources.
Clicking on an OSINT provider brings up a webpage with more detailed information about the selected IP address, such as
IBM X-Force,
mnemonic Passive DNS,
Shodan,
UrlQuery or
VT.
However, if you’re lazy like me, then you’ll probably click the “All above!” option instead,
which will bring up all of the sources in separate tabs in your browser.
The full list of OSINT providers available for IP lookups includes
APNIC Whois, BFK Passive DNS,
Censys,
Cymon,
DNSTrails,
ExoneraTor, Google Public DNS, GreenSnow.co, Hurricane Electric, IBM X-Force, Internet Storm Center, mnemonic Passive DNS,
PacketTotal,
Shodan,
ThreatCrowd,
ThreatMiner,
UrlQuery and VirusTotal.
The domain name lookup menu contains a similar set of providers:
BFK Passive DNS, Cymon,
DNSTrails, Google Public DNS, Google Safe Browsing,
Hybrid Analysis, IBM X-Force Exchange, mnemonic Passive DNS,
MXToolBox, MyWOT, Norton Safe Web,
PacketTotal,
ThreatCrowd,
ThreatMiner,
URL Void, UrlQuery, VirusTotal, Website Informer, Webutation and Whoisology.
Right-clicking a URL in the Browsers tab brings up a similar context menu, which additionally includes the following services for URL lookups:
Google Safe Browsing, IBM X-Force,
ThreatMiner,
URLhaus and
UrlQuery.
Finally, right-clicking on one of the files that NetworkMiner has extracted from a PCAP file brings up a menu for doing OSINT lookups based on
the MD5 or SHA256 hash of the file. The sources used for lookups of hashes include IBM X-Force,
PacketTotal,
ThreatCrowd,
TotalHash,
UrlQuery, VirScan.org, Comodo Valkyrie,
AlienVault OTX,
Hybrid Analysis,
ThreatMiner and VirusTotal.
Hybrid Analysis API Integration
Did you know that the malware analysis service Hybrid Analysis provides free API keys to people in the IT security community?
This is a great move by the Hybrid Analysis team, and we’re happy to announce that we have leveraged their API in NetworkMiner Professional
in order to submit files for analysis directly from within the NetworkMiner GUI.
The API integration also enables you to query for an IP on Hybrid Analysis to see which previously submitted samples has communicated
with that particular IP address.
Here are the steps required to enable the Hybrid Analysis API integration:
Start NetworkMiner Pro, open the Tools > Settings menu and input your API key
Credits
I would like to thank
Chris Sistrunk, Mats Karlsson and Michael Nilsson for suggesting several of the protocol and GUI improvements
that have been incorporated into this new release.
I’d also like to thank Doug Green and Ahmad Nawawi for discovering and reporting bugs in the
IP and SSL parser respectively.
Upgrading to Version 2.3
Users who have purchased a license for NetworkMiner Professional 2.x can download a free update to version 2.3 from our
customer portal.
Those who instead prefer to use the free and open source version can grab the latest version of NetworkMiner from the official
NetworkMiner page.
⛏ FOR GREAT JUSTICE! ⛏
Posted by Erik Hjelmvik on Tuesday, 03 April 2018 06:27:00 (UTC/GMT)
We are today happy to announce the release of CapLoader 1.5.
This new version of CapLoader parses pcap and pcap-ng files even faster than before and comes with new features,
such as a built-in TCP stream reassembly engine, as well as support for Linux and macOS.
Support for ICMP Flows
CapLoader is designed to group packets together that belong to the same bi-directional flow,
i.e. all UDP, TCP and SCTP packets with the same
5-tuple (regardless of direction)
are considered being part of the same flow.
5-tuple
/fʌɪv ˈtjuːp(ə)l/
noun
A combination of source IP, destination IP, source port, destination port and transport protocol (TCP/UDP/SCTP)
used to uniquely identify a flow or layer 4 session in computer networking.
The flow concept in CapLoader 1.5 has been extended to also include ICMP.
Since there are no port numbers in the ICMP protocol CapLoader sets the source and destination port of ICMP flows to 0.
The addition of ICMP in CapLoader also allows input filters and display filters like “icmp” to be leveraged.
Image: CapLoader 1.5 showing only ICMP flows due to display filter 'icmp'.
TCP Stream Reassembly
One of the foundations for making CapLoader a super fast tool for reading and filtering PCAP files is that it doesn’t attempt to reassemble TCP streams.
This means that CapLoader’s Transcript view will show out-of-order segments in the order they were received and retransmitted segments will be displayed twice.
The basic concept has been to let other tools do the TCP reassembly, for example by exporting a PCAP for a flow from CapLoader to
Wireshark or NetworkMiner.
The steps required to reassemble a TCP stream to disk with Wireshark are:
Right-click a TCP packet in the TCP session of interest.
Select “Follow > TCP Stream”.
Choose direction in the first drop-down-list (client-to-server or server-to-client).
Change format from “ASCII” to “Raw” in the next drop-down-menu.
Press the “Save as...” button to save the reassembled TCP stream to disk.
Unfortunately Wireshark fails to properly reassemble some TCP streams.
As an example the current stable release of Wireshark (version 2.2.5) shows
duplicate data in “Follow TCP Stream” when there are retransmissions with partially overlapping segments.
We have also noticed some additionalbugs related to TCP stream reassembly in other recent releases of Wireshark.
However, we’d like to stress that Wireshark does perform a correct reassembly of most TCP streams;
it is only in some specific situations that Wireshark produces a broken reassembly.
Unfortunately a minor bug like this can cause serious consequences,
for example when the TCP stream is analyzed as part of a digital forensics investigation or when the extracted data is being used as input for further processing.
We have therefore decided to include a TCP stream reassembly engine in CapLoader 1.5.
The steps required to reassemble a TCP stream in CapLoader are:
Double click a TCP flow of interest in the “Flows” tab to open a flow transcript.
Click the “Save Client Byte Stream” or “Save Server Byte Stream” button to save the data stream for the desired direction to disk.
Extracting TCP streams from PCAP files this way not only ensures that the data stream is correctly reassembled,
it is also both faster and simpler than having to pivot through Wireshark’s Follow TCP Stream feature.
PCAP Icon Context Menu
The PCAP icon in CapLoader is designed to allow easy drag-and-drop operations in order to open a set of selected flows in an external packet analysis tool, such as
Wireshark or NetworkMiner.
Right-clicking this PCAP icon will bring up a context menu,
which can be used to open a PCAP with the selected flows in an external tool or copy the PCAP to the clipboard.
This context menu has been extended in CapLoader 1.5 to also include a “Save As” option.
Previous versions of CapLoader required the user to drag-and-drop from the PCAP icon to a folder in order to save filtered PCAP data to disk.
Faster Parsing with Protocol Identification
CapLoader can identify over 100 different application layer protocols,
including HTTP, SSL, SSH, RTP, RTCP and SOCKS, without relying on port numbers.
The protocol identification has previously slowed down the analysis quite a bit, which has caused many users to disable this powerful feature.
This new release of of CapLoader comes with an improved implementation of the port-independent protocol identification feature,
which enables PCAP files to be loaded twice as fast as before with the “Identify protocols” feature enabled.
Works in Linux and macOS
One major improvement in CapLoader 1.5 is that this release is compatible with the
Mono framework, which makes CapLoader platform independent.
This means that you can now run CapLoader on your Mac or Linux machine if you have Mono installed.
Please refer to our previous blog posts about how to run NetworkMiner in
various flavors of Linux and macOS to find out how to install Mono on your computer.
You will, however, notice a performance hit when running CapLoader under Mono instead of using Windows
since the Mono framework isn't yet as fast as Microsoft's .NET Framework.
Image: CapLoader 1.5 running in Linux (Xubuntu).
Credits
We’d like to thank Sooraj for reporting a bug in the “Open With” context menu of CapLoader’s PCAP icon.
This bug has been fixed in CapLoader 1.5 and Sooraj has been awarded an official
“PCAP or it didn’t happen” t-shirt for reporting the bug.
Image: PCAP or it didn't happen t-shirt
Have a look at our Bug Bounty Program if you also wanna get a PCAP t-shirt!
Downloading CapLoader 1.5
Everything mentioned in this blog post, except for the protocol identification feature, is available in our free trial version of CapLoader.
To try it out, simply grab a copy here: https://www.netresec.com/?page=CapLoader#trial (no registration needed)
All paying customers with an older version of CapLoader can download a free update to version 1.5 from our
customer portal.
Posted by Erik Hjelmvik on Tuesday, 07 March 2017 09:11:00 (UTC/GMT)
Our heavy-duty PCAP analyzer CapLoader comes with a feature called ”Port Independent Protocol Identification”, a.k.a. PIPI
(see Richard Bejtlich's PIPI blog post from 2006).
Academic research in the Traffic Measurement field often use the term ”Traffic Classification”, which is similar but not the same thing.
Traffic Classification normally group network traffic in broad classes, such as Email, Web, Chat or VoIP.
CapLoader, on the other hand, identifies the actual application layer protocol used in each flow.
So instead of classifying a flow as ”VoIP” CapLoader will tell you if the flow carries SIP, Skype, RTP or MGCP traffic.
This approach is also known as “Dynamic Protocol Detection”.
Being able to identify application layer protocols without relying on the TCP or UDP port number is crucial
when analyzing malicious traffic, such as malware Command-and-Control (C2) communication,
covert backdoors and rouge servers, since such communication often use services on non-standard ports.
Some common examples are:
Many botnet C2 protocols communicate over port TCP 443, but using a proprietary protocol rather than HTTP over SSL.
Backdoors on hacked computers and network devices typically wither run a standard service like SSH on a port other than 22 in order to hide.
More advanced backdoors use port knocking to run a proprietary C2 protocol on a standard port (SYNful knock runs on TCP 80).
This means that by analyzing network traffic for port-protocol anomalies,
like an outgoing TCP connection to TCP 443 that isn't SSL, you can effectively detect intrusions without having IDS signatures
for all C2 protocols. This analysis technique is often used when performing
Rinse-Repeat Intrusion Detection, which is a blacklist-free approach for
identifying intrusions and other form of malicious network traffic.
With CapLoader one can simply apply a BPF filter like “port 443” and scroll through the displayed flows to make sure they
are all say “SSL” in the Protocol column.
Image: Miuref/Boaxxe Trojan C2 traffic to "1.web-counter[.]info" on TCP 443 doesn't use SSL (or HTTPS)
Statistical Analysis
CapLoader relies on statistical analysis of each TCP, UDP and SCTP session's behavior in order to compare it to previously
computed statistical models for known protocols.
These statistical models are generated using a multitude of metrics, such as inter-packet delays, packet sizes and payload data.
The port number is, on the other hand, a parameter that is intentionally not used by CapLoader to determine
the application layer protocol.
The PIPI/Dynamic Protocol Detection feature in CapLoader has been designed to detect even encrypted and obfuscated binary protocols,
such as Tor and
Encrypted BitTorrent (MSE).
These protocols are designed in order to deceive protocol detection mechanisms,
and traditional signature based protocol detection algorithms can't reliably detect them.
The statistical approach employed by CapLoader can, on the other hand, actually detect even these highly obfuscated protocols.
It is, however, important to note that being a statistical method it will never be 100% accurate.
Analysts should therefore not take for granted that a flow is using the protocol stated by CapLoader.
There are some situations when it is very difficult to accurately classify an encrypted protocol,
such as when the first part of a TCP session is missing in the analyzed data.
This can occur when there is an ongoing session that was established before the packet capture was started.
Identified Protocols
The following protocols are currently available for detection in CapLoader's protocol database:
AOL Instant Messenger
BACnet
BitTorrent
BitTorrent Encrypted - MSE
CCCam
CUPS
DAYTIME
DHCP
DHCPv6
Diameter
DirectConnect
DNS
Dockster
DropBox LSP
eDonkey
eDonkey Obfuscated
EtherNet-IP
FTP
Gh0st RAT
Gnutella
Groove LAN DPP
HSRP
HTTP
IMAP
IRC
ISAKMP
iSCSI
JavaRMI
Kelihos
Kerberos
L2TP
LDAP
LLC
Meterpreter
MgCam
MGCP
MikroTik NDP
Modbus TCP
MSN Messenger
MS RPC
MS-SQL
MySQL
NAT-PMP
NetBIOS Datagram Service
NetBIOS Name Service
NetBIOS Session Service
NetFlow
NTP
OsCam
Pcap-over-IP
Poison Ivy RAT
POP3
QUIC
Ramnit
Reverse Shell
RTCP
RTP
RTSP
Shell
SIP
Skype
SLP
SMTP
SNMP
Socks
SopCast P2P
Spotify P2P
Spotify Server
SSH
SSL
Syslog
TeamViewer
TeamViewer UDP
Telnet
Teredo
TFTP
TFTP Data
TPKT
VNC
WS-Discovery
XMPP Jabber
ZeroAccess
Zeus TCP
Zeus UDP
The list of implemented protocols is constantly being increased with new protocols.
PIPI in NetworkMiner
NetworkMiner Professional, which is the commercial version of NetworkMiner,
also comes with an implementation of our protocol detection mechanism.
Even though NetworkMiner Professional doesn't detect as many protocols as CapLoader,
the PIPI feature built into NetworkMiner Pro still helps a lot when analyzing HTTP traffic
on ports other that 80 or 8080 as well as in order to reassemble files downloaded from FTP or TFTP servers
running on non-standard ports.
Posted by Erik Hjelmvik on Tuesday, 06 October 2015 09:05:00 (UTC/GMT)
Image: NetworkMiner Professional 2.3.2 with “vm_win7.pcap” from University of Twente’s Data Exfiltration Malware dataset loaded.
Image: Miuref/Boaxxe Trojan C2 traffic to "1.web-counter[.]info" on TCP 443 doesn't use SSL (or HTTPS)
