Building a monitoring solution - Parsing Palo Alto 200 - TRAFFIC (Firewall) Logs

In this post, we will leverage Splunk – which I installed previously – to build a dashboard that allows us to get a quick overview of our Palo Alto “Traffic” Logs.

One of first things to do when building a Splunk dashboard or parsing events from any log source, is to understand the data within the events. 

Below is a sample of our Palo Alto “TRAFFIC” events as seen by Splunk

Jul 8 20:52:09 192.168.1.5 Jul 8 08:49:07 1,2016/07/08 08:49:07,001606042988,TRAFFIC,end,1,2016/07/08 08:49:05,192.168.1.30,8.8.8.8,99.229.40.63,8.8.8.8,Web Traffic,,,dns,vsys1,TRUSTED_LAN_L3,INTERNET,vlan,ethernet1/1,Security Monitoring,2016/07/08 08:49:06,45165,1,31249,53,3123,53,0x400000,udp,allow,203,64,139,2,2016/07/08 08:48:36,0,any,0,4811734,0x0,192.168.0.0-192.168.255.255,United States,0,1,1

By looking at the Palo Alto documents in the reference section, you would see the position of each value. This information is helpful to contribute to successful parsing. However, it is not absolutely needed. However, I do recommend you look at it.

Now that we have our logs, let’s build out all of the fields that are of importance to us using the "rex" command.

* | rex field=_raw ".*,TRAFFIC,.*?,\d{1,5},.*?,(?<fw_srcIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,(?<fw_rule>.*?),.*?,.*?,(?<fw_application>.*?),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_srcPort>\w+),(?<fw_dstPort>\w+),.*?,.*?,.*?,(?<fw_protocol>.*?),(?<fw_action>\w+),(?<fw_total_bytes>\d+),(?<fw_bytes_sent>\d+),(?<fw_bytes_received>\d+),(?<fw_total_packets>\d+),.*?,(?<fw_duration>\d+),.*?,.*?,.*?,.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_srcIP, fw_dstIP, fw_rule, fw_application, fw_srcPort, fw_dstPort, fw_protocol, fw_action, fw_total_bytes, fw_bytes_sent, fw_bytes_received, fw_total_packets, fw_duration, fw_geolocation | sort count | reverse

To extract the fields above, the following “rex” command was used.
* | rex field=_raw ".*,TRAFFIC,.*?,\d{1,5},.*?,(?<fw_srcIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,(?<fw_rule>.*?),.*?,.*?,(?<fw_application>.*?),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_srcPort>\w+),(?<fw_dstPort>\w+),.*?,.*?,.*?,(?<fw_protocol>.*?),(?<fw_action>\w+),(?<fw_total_bytes>\d+),(?<fw_bytes_sent>\d+),(?<fw_bytes_received>\d+),(?<fw_total_packets>\d+),.*?,(?<fw_duration>\d+),.*?,.*?,.*?,.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_srcIP, fw_dstIP, fw_rule, fw_application, fw_srcPort, fw_dstPort, fw_protocol, fw_action, fw_total_bytes, fw_bytes_sent, fw_bytes_received, fw_total_packets, fw_duration, fw_geolocation | sort count | reverse

Below shows an examples of all the important fields build out using the search query above.

By building out the fields above we can now leverage this information to pick specific fields to gain greater insights.

Looking at all traffic sourced from the local network let's look at the geolocation which were seen.

* | rex field=_raw ".*,TRAFFIC,.*?,\d{1,5},.*?,(?<fw_srcIP>192.168\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,(?<fw_rule>.*?),.*?,.*?,(?<fw_application>.*?),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_srcPort>\w+),(?<fw_dstPort>\w+),.*?,.*?,.*?,(?<fw_protocol>.*?),(?<fw_action>\w+),(?<fw_total_bytes>\d+),(?<fw_bytes_sent>\d+),(?<fw_bytes_received>\d+),(?<fw_total_packets>\d+),.*?,(?<fw_duration>\d+),.*?,.*?,.*?,.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_geolocation | sort count | reverse

Below we see the results of this command above.

Ultimately once you have built all of this out, you should add them to a dashboard so that there is a central view of the traffic in your network.

See you in the next post where we look at parsing the "THREAT" logs.

All Posts In This Series.

Building a monitoring solution – Hardening the OS - CentOS 7 (Linux)

Building a monitoring solution – Forwarding Palo Alto Logs

Building a monitoring solution - Parsing Palo Alto 200 - TRAFFIC (Firewall) Logs

Building a Splunk Dashboard by parsing Palo Alto events – “THREAT” Logs

Reference: http://docs.splunk.com/Documentation/Splunk/6.0.2/Data/Configureindex-timefieldextraction
https://www.paloaltonetworks.com/documentation/61/pan-os/pan-os/reports-and-logging/syslog-field-descriptions.html
https://www.paloaltonetworks.com/documentation/70/pan-os/pan-os/monitoring/view-the-log-files.html