What is PCAP over IP?

PCAP-over-IP is a method for reading a PCAP stream, which contains captured network traffic, through a TCP socket instead of reading the packets from a PCAP file.

A simple way to create a PCAP-over-IP server is to simply read a PCAP file into a netcat listener, like this:

The packets in “sniffed.pcap” can then be read remotely using PCAP-over-IP, for example with tshark like this (replace 192.168.1.2 with the IP of the netcat listener):

But there’s an even simpler way to read PCAP-over-IP with Wireshark and tshark, which doesn’t require netcat.

The Wireshark name for this input method is “TCP socket” pipe interface, which is available in Linux, Windows and macOS builds of Wireshark as well as tshark.

It is also possible to add a PCAP-over-IP interface from Wireshark's GUI. Open Capture/Options, Manage Interfaces, Pipes Tab and then enter a Local Pipe Path such as TCP@127.0.0.1:57012 and click OK. This setting will disappear when you close Wireshark though, since pipe settings don't get saved.

Live Remote Sniffing

Sniffed traffic can be read remotely over PCAP-over-IP in real-time simply by forwarding a PCAP stream with captured packets to netcat like this:

Tcpdump is not available for Windows, but dumpcap is since it is included with Wireshark.

Note how TCP port 57012 is purposely filtered out using BPF when capturing in order to avoid a snowball effect, where the PCAP-over-IP traffic otherwise gets sniffed and re-transmitted through the PCAP-over-IP stream, which again gets sniffed etc.

A more sophisticated setup would be to let the service listening on TCP port 57012 spawn the sniffer process, like this:

Or even better, let the listening service reuse port 57012 to allow multiple incoming PCAP-over-IP connections.

Reading PCAP-over-IP with NetworkMiner

We added PCAP-over-IP support to NetworkMiner in 2011 as part of NetworkMiner 1.1, which was actually one year before the TCP socket sniffing feature was included in Wireshark.

Image: Live remote sniffing with NetworkMiner 2.7.3 using PCAP-over-IP

This PCAP-over-IP feature is actually the recommended method for doing real-time analysis of live network traffic when running NetworkMiner in Linux or macOS, because NetworkMiner’s regular sniffing methods are not available on those platforms.

Reading Decrypted TLS Traffic from PolarProxy

One of the most powerful use-cases for PCAP-over-IP is to read decrypted TLS traffic from PolarProxy. When PolarProxy is launched with the argument “--pcapoverip 57012” it starts a listener on TCP port 57012, which listens for incoming connections and pushes a real-time PCAP stream of decrypted TLS traffic to each client that connects. PolarProxy can also make active outgoing PCAP-over-IP connections to a specific IP address and port if the “--pcapoveripconnect <host>:<port>” argument is provided.

In the video PolarProxy in Windows Sandbox I demonstrate how decrypted TLS traffic can be viewed in NetworkMiner in real-time with help of PCAP-over-IP. PolarProxy’s PCAP-over-IP feature can also be used to read decrypted TLS traffic from PolarProxy with Wireshark as well as to send decrypted TLS traffic from PolarProxy to Arkime (aka Moloch).

Replaying PCAP-over-IP to an Interface

There are lots of great network monitoring products and intrusion detection systems that don’t come with a built-in PCAP-over-IP implementation, such as Suricata, Zeek, Security Onion and Packetbeat, just to mention a few. These products would greatly benefit from having access to the decrypted TLS traffic that PolarProxy can provide. Luckily we can use netcat and tcpreplay to replay packets from a PCAP-over-IP stream to a network interface like this:

But for permanent installations we recommend creating a dedicated dummy interface, to which the traffic gets replayed and sniffed, and then deploy a systemd service that performs the replay operation. See our blog post Sniffing Decrypted TLS Traffic with Security Onion for an example on how to deploy such a systemd service. In that blog post we show how decrypted TLS traffic from PolarProxy can be replayed to a local interface on a Security Onion machine, which is being monitored by Suricata and Zeek.

Nils Hanke has also compiled a detailed documentation on how decrypted TLS packets from PolarProxy can be replayed to Packetbeat and Suricata with help of tcpreplay.

In these setups netcat and tcpreplay act as a generic glue between a PCAP-over-IP service and tools that can sniff packets on a network interface, but there are a few drawbacks with this approach. One drawback is that tcpreplay requires root privileges in order to replay packets to an interface. Another drawback is that extra complexity is added to the solution and two additional single point of failures are introduced (i.e. netcat and tcpreplay). Finally, replaying packets to a network interface increases the risk of packet drops. We therefore hope to see built-in PCAP-over-IP implementations in more network monitoring solutions in the future!

FAQ for PCAP-over-IP

Q: Why is it called “PCAP-over-IP” and not “PCAP-over-TCP”?

Good question, we actually don’t know since we didn’t come up with the name. But in theory it would probably be feasible to read a PCAP stream over UDP or SCTP as well.

Q: What is the standard port for PCAP-over-IP?

There is no official port registered with IANA for PCAP-over-IP, but we’ve been using TCP 57012 as the default port for PCAP-over-IP since 2011. The Wireshark implementation, on the other hand, uses TCP port 19000 as the default value.

Q: Which software comes with built-in PCAP-over-IP servers or clients?

The ones we know of are: Arkime, NetworkMiner, PolarProxy, tshark and Wireshark. There is also a PCAP-over-IP plugin for Zeek (see update below).

Q: Is there some way to encrypt the PCAP-over-IP transmissions?

Yes, we recommend encrypting PCAP-over-IP sessions with TLS when they are transmitted across a non-trusted network. NetworkMiner’s PCAP-over-IP implementation comes with a “Use SSL” checkbox, which can be used to receive “PCAP-over-TLS”. You can replace netcat with socat or ncat in order to establish a TLS encrypted connection to NetworkMiner.

Q: Is there a tool that can aggregate multiple PCAP-over-IP streams into one?

No, none that we’re aware of. However, multiple PCAP-over-IP streams can be merged into one by specifying multiple PCAP-over-IP interfaces in dumpcap and then forwarding that output to a netcat listener, like this:

Update 2023-04-13

Erich Nahum has published zeek-pcapovertcp-plugin, which brings native PCAP-over-IP support to Zeek.

Erich's plugin can be installed as a zeek package through zkg.

After installing the plugin, a command like this reads a PCAP stream from a remote source:

Posted by Erik Hjelmvik on Monday, 15 August 2022 08:05:00 (UTC/GMT)

Tags: #PCAP-over-IP​ #PCAP​ #tcpdump​ #Wireshark​ #tshark​ #NetworkMiner​ #PolarProxy​ #Suricata​ #Zeek​ #Arkime​ #tcpreplay​ #netcat​ #ASCII-art​

Real-time PCAP-over-IP in Wireshark

Did you know that it is possible to stream captured packets from a remote device or application to Wireshark in real-time using PCAP-over-IP? This blog post explains how you can configure Wireshark to read decrypted TLS packets directly from PolarProxy over a TCP socket.

PolarProxy

PolarProxy is a TLS proxy that decrypts and re-encrypts TLS traffic, while also saving the decrypted traffic in a PCAP file. Users who wish to inspect the decrypted TLS traffic in Wireshark typically open this file from disk, but that doesn’t allow for a real-time view of the traffic.

PolarProxy comes with a feature called PCAP-over-IP, which provides a real-time PCAP stream with decrypted packets to connecting clients. If you start PolarProxy with “--pcapoverip 57012” then a PCAP-over-IP listener will be set up on TCP port 57012. I have previously demonstrated how this decrypted stream can be read by NetworkMiner, but it was not until recently that I learned that the same thing can be done with Wireshark as well.

PCAP-over-IP in Wireshark

There’s a little known feature in Wireshark that allows a PCAP stream to be read from a TCP socket, which is exactly what PCAP-over-IP is! To connect to a PolarProxy PCAP-over-IP service on the local PC, do as follows:

1. Capture > Options (or Ctrl+K)
2. “Manage Interfaces...”
3. Select the “Pipes” tab
4. Click the “+” button
5. Name the pipe “TCP@127.0.0.1:57012” and press ENTER to save it.
   
6. Click “OK” in the Manage Interface window.
7. Click “Start” to inspect decrypted traffic from PolarProxy in real-time.

This setup works on Windows, Linux and macOS. Just remember to replace 127.0.0.1 with the IP of PolarProxy in case it is running on a remote machine.

Image: Real-time view of HTTP2 packets from decrypted TLS traffic

It’s also possible to read PCAP-over-IP with the command line tool tshark like this:

The PCAP-over-IP params can also be supplied to Wireshark on the command line in a similar manner:

Happy sniffing!

Posted by Erik Hjelmvik on Tuesday, 24 May 2022 14:00:00 (UTC/GMT)

Tags: #pcapoverip​ #Wireshark​ #PolarProxy​ #PCAP​

PolarProxy in Windows Sandbox

In this video I demonstrate how PolarProxy can be run in a Windows Sandbox to intercept and decrypt outgoing TLS communication. This setup can be used to inspect otherwise encrypted traffic from malware or suspicious Windows applications, which communicate over HTTPS or some other TLS encrypted protocol.

The Windows Sandbox WSB file used in the demo can be downloaded from here: https://www.netresec.com/?download=PolarProxySandbox

Note: Windows Pro or Enterprise is required to run WSB files

Parsing Decrypted TLS Traffic with NetworkMiner

This sandbox also includes NetworkMiner, primarily because it can be used to read a real-time PCAP-over-IP stream with decrypted traffic from PolarProxy. As shown in the video, this feature can be used in order to extract files, images or parameters from the decrypted TLS traffic in near real-time.

For more info about how to run NetworkMiner in Windows Sandbox, please see our blog post Running NetworkMiner in Windows Sandbox.

Configuring a Proxy Server in Windows Sandbox

Windows’ built-in proxy settings are unfortunately not available in Windows Sandbox, which is why I installed a third-party proxy client that redirects all outgoing network traffic to PolarProxy’s SOCKS server. I used Proxifier in the video, which has the additional benefit of being able to redirect all traffic to the proxy, even from applications that aren’t proxy aware. This feature is crucial when attempting to intercept and decrypt TLS traffic from malware that doesn’t respect the proxy settings configured in the operating system.

Command Log

Start PolarProxy with a PCAP-over-IP listener on TCP 57012, SOCKS server on TCP 1080, HTTP proxy on 8080 and a transparent TLS proxy on port 443:

Test PolarProxy’s SOCKS server by sending an unencrypted HTTP request through the proxy:

Test PolarProxy’s SOCKS server by sending an HTTPS request through the proxy:

Test PolarProxy’s HTTP CONNECT proxy server by sending an HTTPS request through the proxy:

Start Menu Search

As shown in the video, text typed into Windows’ start menu gets sent to Microsoft. For more information about this behavior, and how it can be disabled, check out our Start Menu Search video and blog post.

Posted by Erik Hjelmvik on Monday, 31 January 2022 09:50:00 (UTC/GMT)

Tags: #PolarProxy​ #NetworkMiner​ #SOCKS​ #proxy​ #Windows Sandbox​ #Sandbox​ #PCAP-over-IP​ #pcapoverip​ #Windows​ #TLS​ #HTTPS​

PolarProxy 0.9 Released

PolarProxy was previously designed to only run as a transparent TLS proxy. But due to popular demand we’ve now extended PolarProxy to also include a SOCKS proxy and a HTTP CONNECT proxy. PolarProxy automatically decrypts all proxied SSL and TLS traffic, regardless if the remote server is running on TCP 443 or some other port, as long as the traffic passes through PolarProxy. As from now we also release a Windows build of PolarProxy, alongside the Linux x64, ARM and ARM64 builds.

SOCKS Proxy

Use the command line argument “--socks [port]” to start PolarProxy’s SOCKS proxy server. This SOCKS proxy supports multiple versions of the SOCKS protocol, including SOCKS 4, SOCKS 4a, SOCKS 5 and SOCKS 5h.

As an example, the command below starts a SOCKS server on TCP port 1080 and passes a copy of the decrypted TLS traffic as a PCAP stream to tshark.

You can then use curl to run some HTTPS traffic through the SOCKS proxy:

After doing this you should see the decrypted HTTP/2 traffic in tshark’s output.

HTTP CONNECT Proxy

We’ve also added a HTTP proxy to PolarProxy 0.9, but it only supports the CONNECT request method. This means that normal unencrypted HTTP requests, like GET or POST requests, will be rejected by PolarProxy. Most web traffic is TLS encrypted nowadays anyway, so we don't consider this limitation to be a big issue.

The HTTP CONNECT proxy service is activated with the “--httpconnect” argument. Decrypted TLS traffic from PolarProxy’s HTTP CONNECT proxy can be forwarded to tshark just like in the SOCKS example, but the traffic from these proxies can also be accessed through PCAP-over-IP like this:

You can then connect to PolarProxy’s PCAP-over-IP service with NetworkMiner by clicking File, Receive PCAP over IP, select “Connect to IP/port”, enter “localhost” and click the “Start Receiving” button. You’ll now be able to see a real-time feed of all the traffic that PolarProxy decrypts. As an example, let’s download the PolarProxy logo over HTTPS to see if NetworkMiner can extract it from PolarProxy’s decrypted PCAP-over-IP stream:

The PolarProxy logo immediately shows up in NetworkMiner’s images tab:

Port-Independent TLS Protocol Detection

When PolarProxy is running as a transparent TLS proxy all incoming traffic can be expected to be TLS. But that’s not the case when, for example, PolarProxy is running as a SOCKS proxy. We have therefore added port-independent TLS protocol detection for proxied traffic, so that TLS traffic can be detected and decrypted even when it runs on other ports than the standard 443, 465, 853, 990, 993, 995 and 5061 ones.

There is one crucial limitation to the automatic SSL/TLS protocol detection though, it doesn’t support explicit TLS traffic that relies on opportunistic encryption features like STARTTLS, which bootstraps TLS into an already established application layer session.

Allow Non-TLS Traffic

SOCKS and HTTP CONNECT proxies can both be used to transport other protocols than TLS. PolarProxy blocks all non-TLS traffic by default, but this setting can be overridden with the “--nontls allow” argument to allow any traffic to be proxied. The allow non-TLS override has no effect on PolarProxy’s transparent proxy though, because it will need to see a valid SNI field in order to know whereto the traffic should be forwarded.

Windows Build

There wasn’t much need for a Windows build of PolarProxy prior to the release of version 0.9, because the Windows firewall can’t be configured to redirect outgoing port 443 traffic to a local service. However, now that PolarProxy also includes SOCKS and HTTP CONNECT services, the situation is completely different. There are many ways to configure a Windows PC, as well as web browsers and other applications, to use a local proxy server.

You can use the Proxy settings window in Windows 10 and 11 to enable a local HTTP proxy like this:

Another option is to run “inetcpl.cpl” (Internet Options), open the “Connections” tab and click the “LAN settings” button to configure an HTTP proxy.

You can, of course, also configure your browser to use a local SOCKS or HTTP proxy in Windows, just as you’d do on any other operating system.

But don’t forget to configure your OS and/or browser to trust your PolarProxy instance’s root CA certificate first, as explained in the “Trusting the PolarProxy root CA” section of our PolarProxy documentation.

The Windows version of PolarProxy is a .NET framework-dependent application, which requires the .NET 6 runtime to be installed. The PolarProxy releases for other platforms (Linux x64, ARM and ARM64) are all self-contained applications, which are published with the .NET runtime built-in.

Visit our PolarProxy page to download and install PolarProxy.

Posted by Erik Hjelmvik on Thursday, 13 January 2022 10:15:00 (UTC/GMT)

Tags: #PolarProxy​ #proxy​ #SOCKS​ #SOCKS5​ #TLS​ #SSL​ #decrypt​ #Windows​ #PCAP-over-IP​ #pcapoverip​

Start Menu Search Video

In this video I demonstrate that text typed into the Windows 10 start menu gets sent to Microsoft and how that traffic can be intercepted, decrypted and parsed.

What Was Sent?

The XML files shown in the video were sent by Cortana's "SmartSearch" app to https://www.bing.com/threshold/xls.aspx in HTTP/2 POST requests. As shown in the video, the POST'ed keystrokes can be found inside requestInfo XML tags that have a "RawQuery" key.

The following tcpdump and grep commands can be used to list the RawQuery data sent to Bing in these HTTP/2 requests:

Running that command on the PolarProxy PCAP file from the video gives the following output:

The same data also gets sent in the query string variable "qry" of GET requests for https://www.bing.com/AS/API/WindowsCortanaPane/V2/Suggestions, as shown in this NetworkMiner screenshot.

How to Intercept, Decrypt and Decode HTTPS Traffic

The following section presents the technical details regarding my setup, so that others can reproduce and verify these findings.

My first step was to install PolarProxy on a Linux machine on the local network. PolarProxy is a TLS proxy, which can intercept and decrypt TLS traffic. This TLS proxy is primarily designed to decrypt traffic from malware and hackers, but can also be used to decrypt legitimate traffic when needed.

PolarProxy was configured to listen for incoming TLS connections on TCP port 443 and output PCAP data with the decrypted traffic as if it had been transmitted over TCP 80. The decrypted traffic was accessible as a real-time stream through a PCAP-over-IP service running on port 57012. Here's the full command that was used to start PolarProxy:

In the video I showed the Windows 10 client's modified hosts file, which included an entry for www.bing.com pointing to the PolarProxy machine. What was not shown in the video though, is that PolarProxy's own CA certificate had been added to the Win10 machine's list of trusted root CA's, as explained in the "Trusting the PolarProxy root CA" section of the PolarProxy installation instructions. With these two changes in place all HTTPS requests for www.bing.com from the Win10 PC got diverted through the PolarProxy TLS inspection service, which then decrypted and re-encrypted the traffic before forwarding it to Bing.

The decrypted Bing requests could be accessed either locally on the Linux machine, or remotely using the PCAP-over-IP service on TCP port 57012. I used NetworkMiner to read the live PCAP stream with decrypted traffic from port 57012 and extract all files being sent and received in real-time.

Is it Possible to Disable the Cortana Search?

When Ars Technica reporters asked Microsoft back in 2015 if there was any way to disable this communication, Microsoft replied with the following statement:

As part of delivering Windows 10 as a service, updates may be delivered to provide ongoing new features to Bing search, such as new visual layouts, styles and search code. No query or search usage data is sent to Microsoft, in accordance with the customer's chosen privacy settings.

There are plenty of how-to guides online with instructions on how the Cortana search feature can be disabled. Most of these guides suggest disabling the AllowCortana setting in group policies or in the registry. We've tried several of the settings suggested in these how-to guides, but none of them seem to prevent Windows from sending keystrokes to Bing.

If you know how to successfully disable Cortana's Bing searches, then please feel free to reach out to us so that we can update this blog post.

UPDATE 210928 - How to Actually Disable Cortana Search

Twitter user @GeorgeProfonde3 reached out to suggest a fix that might prevent the start menu from sending data to Bing. We have now verified this fix and we're happy to announce that it works (at least for us).

1. Start regedit.exe
2. Open the following registry key:
   HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Search
3. Ensure that the value for CortanaConsent is set to 0
4. Create a new DWORD registry entry called "BingSearchEnabled" with value 0

You should no longer see any connections to www.bing.com when interacting with the start menu after implementing this fix.

UPDATE 211015 - Another way to Disable Cortana Search

You may need to use a different method to disable the start meny search, depending on your Windows version and build. Kimberly (@StopMalvertisin) suggested the following method, which seems to work on Windows 11:

1. Start regedit.exe
2. Create a registry key for:
   HKEY_CURRENT_USER\SOFTWARE\Policies\Microsoft\Windows\Explorer
3. Create a new DWORD registry entry called "DisableSearchBoxSuggestions" with value 1

Disabling Start Menu Search from Group Policy

There are also a few different methods for disabling start menu searches using GPO. However, please note that your success will vary depending on your Windows version and build.

GPO Method #1

1. Start gpedit.msc
2. Open the following branch:
   User configuration\Administrative templates\Windows components\File Explorer
3. Enable the following group policy:
   "Turn off display of recent search entries in the File Explorer search box"

GPO Method #2

• Start gpedit.msc
• Open the following branch:
  User Configuration\Administrative Templates\Start Menu and Taskbar
• Enable the following group policy: "Do not search communications"

Posted by Erik Hjelmvik on Tuesday, 28 September 2021 08:24:00 (UTC/GMT)

Tags: #PCAP​ #NetworkMiner​ #PolarProxy​ #Microsoft​ #video​ #videotutorial​ #pcapoverip​ #PCAP-over-IP​ #HTTP/2​ #http2​

Capturing Decrypted TLS Traffic with Arkime

The latest version of Arkime (The Sniffer Formerly Known As Moloch) can now be fed with a real-time stream of decrypted HTTPS traffic from PolarProxy. All that is needed to enable this feature is to include "pcapReadMethod=pcap-over-ip-server" in Arkime's config.ini file and start PolarProxy with the "--pcapoveripconnect 127.0.0.1:57012" option. PolarProxy will then connect to Arkime's PCAP-over-IP listener on TCP port 57012 and send it a copy of all TLS packets it decrypts.

Note: The required PCAP-over-IP feature is available in Arkime 2.7.0 and PolarProxy 0.8.16.

Arkime is an open source packet capture solution that indexes the PCAP data it collects. Arkime also comes with a web frontend for browsing and searching through the captured, and indexed, network traffic. The Arkime project recently changed name from Moloch, probably in an attempt to convince users that the tool doesn't eat children.

This guide demonstrates how TLS traffic, or more specifically HTTPS traffic, can be decrypted and ingested in real-time into Arkime.

The TLS decryption is performed with PolarProxy, which is a transparent TLS interception proxy that is freely available under a Creative Commons BY-ND 4.0 license.

PolarProxy and Arkime can be installed on a server to intercept, decrypt, index and store decrypted TLS network traffic from multiple clients on a network. It is even possible to install PolarProxy and Arkime on separate servers, so that PolarProxy forwards a stream of decrypted traffic to the Arkime server. However, to avoid unnecessary complexity, Arkime and PolarProxy are installed locally on a Linux client in this howto guide. The Linux client is a Ubuntu 20.04.1 machine, but the instructions can also be used on other Linux flavors that use systemd, such as Arch, CentOS, Debian, Fedora, SUSE and Red Hat Linux.

Download and Install Arkime

Arkime can be downloaded as a pre-built installation packages for CentOS and Ubuntu here: https://arkime.com/#download

Note: You can alternatively visit the Arkime GitHub page if there is no pre-built installation package for your Linux distro or you prefer to build Arkime from source.

After installing the Arkime package, configure Arkime by running:

• Enter "none" as the interface to monitor (the interface setting will be ignored when Arkime gets configured as a PCAP-over-IP server)
• Install the ElasticSearch server by typing "yes" when prompted

Edit /data/moloch/etc/config.ini and add "pcapReadMethod=pcap-over-ip-server" to configure Arkime to listen for PCAP-over-IP connections.

Next, enable and start the ElasticSearch systemd service.

Initiate the Arkime search cluster.

Create a new admin user.

You can now enable and start the Moloch capture and viewer services.

Verify that Arkime now listens for incoming connections on TCP port 57012.

Install PolarProxy to Decrypt TLS Traffic

Create a user for PolarProxy's systemd service and download PolarProxy like this:

Copy the default PolarProxy service config to the systemd location.

Modify /etc/systemd/system/PolarProxy.service by adding "--pcapoveripconnect 127.0.0.1:57012" at the end of the ExecStart command.

It's now time to enable and start the PolarProxy service.

Verify that PolarProxy has connected to Arkime's PCAP-over-IP listener on TCP port 57012.

Take it For a Test Run

PolarProxy is listening for incoming TLS connections on TCP port 10443. We can therefore run traffic through the TLS decryption proxy with this curl command:

The decrypted traffic will show up in Arkime if everything is working. Open http://localhost:8005/sessions in a browser and look for a connection to www.netresec.com.

Note: The Arkime username and password is admin/THEPASSWORD if you've followed the instructions in this tutorial.

Also: You might have to wait a minute or two for the traffic to appear in Arkime's user interface.

Trust PolarProxy's Root CA Certificate

The root CA certificate used by your PolarProxy service must be trusted by both the operating system and browser in order to run TLS traffic through the decryption proxy without errors. Follow these instructions to add trust the root CA:

• Select the "extra/PolarProxy-root-CA.crt" Certificate Authority
• Press <Ok>

Start Firefox

• Download the root CA certificate from: http://localhost:10080/polarproxy.cer
• Open: about:preferences#privacy
• Scroll down to "Certificates" and click "View Certificates"
• Import > Select "polarproxy.cer"
• Select: ☑ Trust this CA to identify websites

Configure Firewall Redirect of Outgoing HTTPS Traffic

The final step in this tutorial is to redirect the local user's outgoing HTTPS traffic to the PolarProxy service listening on TCP port 10443. Add the following lines at the top of /etc/ufw/before.rules (before the "*filter" section) to redirect outgoing HTTPS traffic to the local PolarProxy service listening on port 10443.

Note: The UFW config in "before.rules" is equivalent to running "iptables -t nat -A OUTPUT -m owner --uid 1000 -p tcp --dport 443 -j REDIRECT --to 10443"

Make sure to modify the uid value (1000) in the firewall rule to match that of the local user that PolarProxy should decrypt the HTTPS traffic for. You can see your uid value by running the command "id -u". You can even redirect traffic from several users to PolarProxy, but it's important that you DON'T forward the outgoing HTTPS traffic from the "proxyuser" account. You will otherwise generate an infinite firewall redirect loop, where outgoing HTTPS traffic from PolarProxy is redirected back to PolarProxy again. You can check the proxyuser's uid with the command "id -u proxyuser".

After saving before.rules, reload UFW to activate the port redirection.

Surf 'n' Snoop

Your Linux machine is now configured to send decrypted HTTPS traffic to Arkime for inspection. Open Firefox and visit some websites, then go back to Arkime and have a look at the traffic. Again, remember that there might be a few minutes' delay before the traffic appears in Arkime's user interface

You'll probably notice that the majority of all HTTPS traffic is actually using the HTTP/2 protocol. Unfortunately Arkime's http2 support is still quite limited, but I'm hoping it will improve in future releases.

Luckily, both Wireshark and NetworkMiner (which runs fine in Linux by the way) can be used to parse and extract contents from HTTP/2 traffic. Just hit Arkime's "Download PCAP" button and open the capture file in a tool of your choice.

Image: NetworkMiner in Linux with files extracted from decrypted HTTP/2 traffic

Posted by Erik Hjelmvik on Tuesday, 01 December 2020 07:50:00 (UTC/GMT)

Tags: #Arkime​ #PolarProxy​ #TLS​ #HTTPS​ #decrypt​ #PCAP​ #systemd​ #systemctl​ #UFW​ #http2​ #HTTP/2​ #PCAP-over-IP​ #pcapoverip​ #ASCII-art​

PolarProxy 0.8.16 Released

Client Certificates

PolarProxy now supports client-authenticated TLS handshakes for outgoing connections to support sites that require mutual TLS (mTLS) authentication. The following example uses the PKCS#12 client certificate "client.p12" with password "pwd" to authenticate PolarProxy when connecting to "https://api.example.com":

Thanks to Peter Lambrechtsen for the idea!

Bypassing Decryption for Specific Domains

There are situations when it isn't appropriate to decrypt the traffic passing through PolarProxy. The traffic might, for example, contain personal or confidential information. It might also not be possible to decrypt the traffic for technical reasons, such as when clients use certificate pinning or certificate transparency to validate the server certificate. We therefore recommend that such sites are put on a "bypass" list, i.e. a list of domains for which PolarProxy should let the encrypted traffic pass untouched to preserve the end-to-end encryption between the client and server.

PolarProxy's "--bypass <file>" option, which can be used to provide a regular expression list of domains not to decrypt, has now been acompanied by "--bypassexact <file>". The new --bypassexact option simply matches domains against the lines in <file> using string matching of the full domain name, no fancy-pants regex involved.

PCAP-over-IP Client

The new "--pcapoveripconnect" option can be used to let PolarProxy connect to a PCAP-over-IP listener and send it a live PCAP stream of decrypted traffic over TCP. This option complements PolarProxy's "--pcapoverip" option, which sets up a PCAP-over-IP listener that serves clients with the same PCAP stream. Thanks to Andy Wick for suggesting adding a PCAP-over-IP connector to PolarProxy!

The following command instructs PolarProxy to send a live PCAP stream with decrypted traffic to a local PCAP-over-IP listener:

PolarProxy will automatically attempt to re-establish the PCAP-over-IP connection every 10 seconds if it goes down or cannot be established for some reason.

Only Store Packets When Instructed

PolarProxy no longer writes hourly rotated pcap files with decrypted packets to disk unless explicitly instructed to do so with "-o <directory>" or "-w <file>".

Flushing Buffered Packets to Disk

PolarProxy now periodically flushes buffered packets to disk every 60 seconds. The flush interval can be controlled with the "--autoflush <seconds>" option. The auto flush can also be disabled with "--autoflush 0".

No More Out-of-Quota Issues

We have also improved the quota handling for our privileged users, who have a license key that allows them to decrypt more than 10 GB or 10 000 TLS sessions per day. You should now be able to use your full daily quota without issues!

UPDATE 2022-12-08

Peter Lambrechtsen's talk IoT your Pet from Kawaiicon 2022 is on YouTube! In this talk Peter explains how he used PolarProxy to MITM traffic between an IoT device and a cloud service running on Amazon AWS. Check out Peter's Pet Hub Local project for more details.

Posted by Erik Hjelmvik on Monday, 30 November 2020 07:45:00 (UTC/GMT)

Tags: #Netresec​ #PolarProxy​ #PCAP​ #TLS​ #bypass​ #PCAP-over-IP​ #pcapoverip​ #certificate​

PolarProxy in Podman

Podman is a daemonless Linux container engine, which can be used as a more secure alternative to Docker. This blog post demonstrates how to run PolarProxy in a rootless container using Podman. If you still prefer to run PolarProxy in Docker, then please read our blog post "PolarProxy in Docker" instead.

Install Podman and fuse-overlayfs

Install Podman according to the official Podman installation instructions. Then install fuse-overlayfs, which is an overlay file system for rootless containers. Fuse-overlayfs can be installed in Debian/Ubuntu with "sudo apt install fuse-overlayfs" and in CentOS with "sudo yum install fuse-overlayfs".

Create a Podman Image for PolarProxy

Create a Dockerfile with the following contents:

Save the Docker file as "Dockerfile" (no extension) in an empty directory and start a shell in that directory. Build a PolarProxy Podman image with:

Test the PolarProxy Podman Image

Take the polarproxy Podman image for a test run. Start it with:

Establish an HTTPS connection through PolarProxy by running this curl command from another shell on the same machine:

If everything works alright, then curl should output HTML and the interactive Podman session running the polarproxy image should print something like:

<6>[10443] 127.0.0.1 -> N/A Connection from: 127.0.0.1:44122
<6>[10443] 127.0.0.1 -> www.netresec.com Connection request for: www.netresec.com from 127.0.0.1:44122
<6>[10443] 127.0.0.1 -> www.netresec.com Action: DECRYPT

Create a Podman Container for PolarProxy

Create directories "pcap" and "polarproxy", where PolarProxy should store the decrypted network traffic and its root CA certificate.

Create a container called "polarproxy", which has the "pcap" and "polarproxy" directories mounted as volumes. The service on TCP 10080 will serve the proxy's public root cert over HTTP. The localhost:57012 service is a Pcap-over-IP server, from which the decrypted network traffic can be streamed in real-time.

Create and enable a systemd user service that will run the container.

Start the systemd user service to activate the PolarProxy container.

Verify that the service is running and that you can view the logs from PolarProxy.

Expose PolarProxy to the Network

Try making an HTTPS connection via PolarProxy from another PC on the network.

Don't forget to save the firewall redirect rule if it is working as desired!

Redirect HTTPS and Trust the Root CA

You can now redirect outgoing TCP 443 traffic from your network to your Podman/PolarProxy host. Review the "Routing HTTPS Traffic to the Proxy" section on the PolarProxy page for recommendations on how to redirect outgoing traffic to PolarProxy.

Finally, configure the operating system, browsers and other applications that will get their TLS traffic proxied by PolarProxy to trust the root CA of the PolarProxy service running in your Podman container. Follow the steps in the "Trusting the PolarProxy root CA" section of the PolarProxy documentation in order to install the root cert.

Accessing Decrypted TLS Traffic

You should be able to access PCAP files with the decrypted HTTPS traffic in the "pcap" directory.

It is also possible view the decrypted traffic in real-time by using netcat and tcpdump as a Pcap-over-IP client like this:

It probably makes more sense to forward the decrypted traffic to an IDS or other type of network security monitoring tool though. See our blog posts "Sniffing Decrypted TLS Traffic with Security Onion" and "Capturing Decrypted TLS Traffic with Arkime" for instructions on how to forward the decrypted network traffic to a network monitoring solution like Security Onion or Arkime.

PolarProxy in Podman on ARM Linux

PolarProxy can also run on ARM Linux installations, such as a Raspberry Pi. However, the Dockerfile must be modified slightly in order to do so.

Don't know if you're running a 32-bit or 64-bit OS? Run "uname -m" and check if the output says "armv7*" (arm32), "armv8*" (arm64) or "aarch64" (arm64).

See our blog post "Raspberry PI WiFi Access Point with TLS Inspection" for more details about deploying PolarProxy on a Raspberry Pi.

ʕ•ᴥ•ʔ + 🦭 = 💜

Posted by Erik Hjelmvik on Tuesday, 27 October 2020 18:33:00 (UTC/GMT)

Tags: #PolarProxy​ #Docker​ #TLS​ #HTTPS​ #Proxy​ #curl​ #PCAP​ #Dockerfile​ #DNAT​ #container​ #arm32​ #arm64​ #AArch64​ #PCAP-over-IP​ #pcapoverip​ #systemctl​ #systemd​