Sunday, July 10, 2016

Building a Splunk Dashboard by parsing Palo Alto events – “THREAT” Logs


In this post, we will leverage Splunk – which I installed previously – to build a dashboard that allows us to get a quick overview of our Palo Alto “Threats” Logs.

In the previous post we looked at parsing the “TRAFFIC” Logs In this post we look at parsing the “THREAT” logs. Once again, before we can parse a log we need to understand the structure of the log file. From below we can see that this log event is “,” delimited. So looking at this log, let’s do our thing. If you looked at the previous post, the "THREAT" logs is structured basically the same as the "TRAFFIC" logs except for a few fields where the value differs such as the "sub type"

Jul  8 07:00:45 192.168.1.5 Jul  7 18:57:46 1,2016/07/07 18:57:46,001606042988,THREAT,vulnerability,1,2016/07/07 18:57:41,192.168.1.29,107.xx.xx.21,99.xx.xx.63,107.21.226.21,Web Traffic,,,web-browsing,vsys1,TRUSTED_LAN_L3,INTERNET,vlan,ethernet1/1,Security Monitoring,2016/07/07 18:57:46,7691,1,60316,80,30074,80,0x400000,tcp,alert,"",HTTP OPTIONS Method(30520),any,informational,client-to-server,825,0x0,192.168.0.0-192.168.255.255,United States,0,

using the search query below we can parse the Threat logs

host="192.168.1.5" THREAT | rex field=_raw ".*,THREAT,(?<fw_threat_subtype>\w+),.*?,(?<fw_srcIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_dstPort>\w+),.*?,.*?,.*?,.*?,(?<fw_protocol>\w+),(?<fw_action>\w+),.*?,(?<fw_msg>.*?),.*?,.*?,(?<fw_direction>\w+),.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_threat_subtype, fw_srcIP, fw_dstIP, fw_dstPort, fw_protocol, fw_action, fw_msg, fw_direction, fw_geolocation | sort count | reverse



Taking another example, let’s look at vulnerabilities which have been seen over the past … however long we got some logs for …

host="192.168.1.5" THREAT | rex field=_raw ".*,THREAT,(?<fw_threat_subtype>\w+),.*?,(?<fw_srcIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_dstPort>\w+),.*?,.*?,.*?,.*?,(?<fw_protocol>\w+),(?<fw_action>\w+),.*?,(?<fw_msg>.*?),.*?,.*?,(?<fw_direction>\w+),.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_msg | sort count | reverse



Ok enough of this. In this series we built a monitoring solution and then used Splunk to prase some Palo Alto logs.

Hope you enjoyed and do leave a comment if you find this series useful.

All Posts In This Series.


Building a monitoring solution – Hardening the OS - CentOS 7 (Linux)
Building a monitoring solution – Forwarding Palo Alto Logs
Building a monitoring solution - Parsing Palo Alto 200 - TRAFFIC (Firewall) Logs
Building a Splunk Dashboard by parsing Palo Alto events – “THREAT” Logs

Building a monitoring solution - Parsing Palo Alto 200 - TRAFFIC (Firewall) Logs



In this post, we will leverage Splunk – which I installed previously – to build a dashboard that allows us to get a quick overview of our Palo Alto “Traffic” Logs.

One of first things to do when building a Splunk dashboard or parsing events from any log source, is to understand the data within the events. 

Below is a sample of our Palo Alto “TRAFFIC” events as seen by Splunk

Jul 8 20:52:09 192.168.1.5 Jul 8 08:49:07 1,2016/07/08 08:49:07,001606042988,TRAFFIC,end,1,2016/07/08 08:49:05,192.168.1.30,8.8.8.8,99.229.40.63,8.8.8.8,Web Traffic,,,dns,vsys1,TRUSTED_LAN_L3,INTERNET,vlan,ethernet1/1,Security Monitoring,2016/07/08 08:49:06,45165,1,31249,53,3123,53,0x400000,udp,allow,203,64,139,2,2016/07/08 08:48:36,0,any,0,4811734,0x0,192.168.0.0-192.168.255.255,United States,0,1,1


By looking at the Palo Alto documents in the reference section, you would see the position of each value. This information is helpful to contribute to successful parsing. However, it is not absolutely needed. However, I do recommend you look at it.

Now that we have our logs, let’s build out all of the fields that are of importance to us using the "rex" command.
* | rex field=_raw ".*,TRAFFIC,.*?,\d{1,5},.*?,(?<fw_srcIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,(?<fw_rule>.*?),.*?,.*?,(?<fw_application>.*?),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_srcPort>\w+),(?<fw_dstPort>\w+),.*?,.*?,.*?,(?<fw_protocol>.*?),(?<fw_action>\w+),(?<fw_total_bytes>\d+),(?<fw_bytes_sent>\d+),(?<fw_bytes_received>\d+),(?<fw_total_packets>\d+),.*?,(?<fw_duration>\d+),.*?,.*?,.*?,.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_srcIP, fw_dstIP, fw_rule, fw_application, fw_srcPort, fw_dstPort, fw_protocol, fw_action, fw_total_bytes, fw_bytes_sent, fw_bytes_received, fw_total_packets, fw_duration, fw_geolocation | sort count | reverse

To extract the fields above, the following “rex” command was used.
* | rex field=_raw ".*,TRAFFIC,.*?,\d{1,5},.*?,(?<fw_srcIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,(?<fw_rule>.*?),.*?,.*?,(?<fw_application>.*?),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_srcPort>\w+),(?<fw_dstPort>\w+),.*?,.*?,.*?,(?<fw_protocol>.*?),(?<fw_action>\w+),(?<fw_total_bytes>\d+),(?<fw_bytes_sent>\d+),(?<fw_bytes_received>\d+),(?<fw_total_packets>\d+),.*?,(?<fw_duration>\d+),.*?,.*?,.*?,.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_srcIP, fw_dstIP, fw_rule, fw_application, fw_srcPort, fw_dstPort, fw_protocol, fw_action, fw_total_bytes, fw_bytes_sent, fw_bytes_received, fw_total_packets, fw_duration, fw_geolocation | sort count | reverse



Below shows an examples of all the important fields build out using the search query above.

By building out the fields above we can now leverage this information to pick specific fields to gain greater insights.

Looking at all traffic sourced from the local network let's look at the geolocation which were seen.

* | rex field=_raw ".*,TRAFFIC,.*?,\d{1,5},.*?,(?<fw_srcIP>192.168\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,(?<fw_rule>.*?),.*?,.*?,(?<fw_application>.*?),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_srcPort>\w+),(?<fw_dstPort>\w+),.*?,.*?,.*?,(?<fw_protocol>.*?),(?<fw_action>\w+),(?<fw_total_bytes>\d+),(?<fw_bytes_sent>\d+),(?<fw_bytes_received>\d+),(?<fw_total_packets>\d+),.*?,(?<fw_duration>\d+),.*?,.*?,.*?,.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_geolocation | sort count | reverse



Below we see the results of this command above.