Sunday, July 10, 2016

Building a monitoring solution - Parsing Palo Alto 200 - TRAFFIC (Firewall) Logs



In this post, we will leverage Splunk – which I installed previously – to build a dashboard that allows us to get a quick overview of our Palo Alto “Traffic” Logs.

One of first things to do when building a Splunk dashboard or parsing events from any log source, is to understand the data within the events. 

Below is a sample of our Palo Alto “TRAFFIC” events as seen by Splunk

Jul 8 20:52:09 192.168.1.5 Jul 8 08:49:07 1,2016/07/08 08:49:07,001606042988,TRAFFIC,end,1,2016/07/08 08:49:05,192.168.1.30,8.8.8.8,99.229.40.63,8.8.8.8,Web Traffic,,,dns,vsys1,TRUSTED_LAN_L3,INTERNET,vlan,ethernet1/1,Security Monitoring,2016/07/08 08:49:06,45165,1,31249,53,3123,53,0x400000,udp,allow,203,64,139,2,2016/07/08 08:48:36,0,any,0,4811734,0x0,192.168.0.0-192.168.255.255,United States,0,1,1


By looking at the Palo Alto documents in the reference section, you would see the position of each value. This information is helpful to contribute to successful parsing. However, it is not absolutely needed. However, I do recommend you look at it.

Now that we have our logs, let’s build out all of the fields that are of importance to us using the "rex" command.
* | rex field=_raw ".*,TRAFFIC,.*?,\d{1,5},.*?,(?<fw_srcIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,(?<fw_rule>.*?),.*?,.*?,(?<fw_application>.*?),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_srcPort>\w+),(?<fw_dstPort>\w+),.*?,.*?,.*?,(?<fw_protocol>.*?),(?<fw_action>\w+),(?<fw_total_bytes>\d+),(?<fw_bytes_sent>\d+),(?<fw_bytes_received>\d+),(?<fw_total_packets>\d+),.*?,(?<fw_duration>\d+),.*?,.*?,.*?,.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_srcIP, fw_dstIP, fw_rule, fw_application, fw_srcPort, fw_dstPort, fw_protocol, fw_action, fw_total_bytes, fw_bytes_sent, fw_bytes_received, fw_total_packets, fw_duration, fw_geolocation | sort count | reverse

To extract the fields above, the following “rex” command was used.
* | rex field=_raw ".*,TRAFFIC,.*?,\d{1,5},.*?,(?<fw_srcIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,(?<fw_rule>.*?),.*?,.*?,(?<fw_application>.*?),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_srcPort>\w+),(?<fw_dstPort>\w+),.*?,.*?,.*?,(?<fw_protocol>.*?),(?<fw_action>\w+),(?<fw_total_bytes>\d+),(?<fw_bytes_sent>\d+),(?<fw_bytes_received>\d+),(?<fw_total_packets>\d+),.*?,(?<fw_duration>\d+),.*?,.*?,.*?,.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_srcIP, fw_dstIP, fw_rule, fw_application, fw_srcPort, fw_dstPort, fw_protocol, fw_action, fw_total_bytes, fw_bytes_sent, fw_bytes_received, fw_total_packets, fw_duration, fw_geolocation | sort count | reverse



Below shows an examples of all the important fields build out using the search query above.

By building out the fields above we can now leverage this information to pick specific fields to gain greater insights.

Looking at all traffic sourced from the local network let's look at the geolocation which were seen.

* | rex field=_raw ".*,TRAFFIC,.*?,\d{1,5},.*?,(?<fw_srcIP>192.168\.\d{1,3}\.\d{1,3}),(?<fw_dstIP>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),.*?,.*?,(?<fw_rule>.*?),.*?,.*?,(?<fw_application>.*?),.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,.*?,(?<fw_srcPort>\w+),(?<fw_dstPort>\w+),.*?,.*?,.*?,(?<fw_protocol>.*?),(?<fw_action>\w+),(?<fw_total_bytes>\d+),(?<fw_bytes_sent>\d+),(?<fw_bytes_received>\d+),(?<fw_total_packets>\d+),.*?,(?<fw_duration>\d+),.*?,.*?,.*?,.*?,.*?,(?<fw_geolocation>.*?)," | stats count by fw_geolocation | sort count | reverse



Below we see the results of this command above.




3 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Thank you for sharing this useful information. I wrote this below search which produces consistent results for PA logs field extraction. Keep the good work up.


    Suggested Search:

    sourcetype="pan:traffic"
    | eval field=split(_raw, ",")
    | eval type=mvindex(field,3), subtype=mvindex(field,4), src_ip=mvindex(field,7), dst_ip=mvindex(field,8), nat_src=mvindex(field,9), nat_dst =mvindex(field,10), rule_name=mvindex(field,11), app=mvindex(field,14),src_zone=mvindex(field,16),dst_zone=mvindex(field,17), ingress_if =mvindex(field,18), egress_if=mvindex(field,19), log_action=mvindex(field,20), src_port=mvindex(field,24), dst_port=mvindex(field,25), proto =mvindex(field,29), action=mvindex(field,30), bytes_sent=mvindex(field,31), bytes_received=mvindex(field,32), packets=mvindex(field,33), session_end_reason=mvindex(field,46), device=mvindex(field,52)

    | table _time type subtype src_ip dst_ip nat_src nat_dst rule_name app src_zone dst_zone ingress_if egress_if log_action src_port dst_port proto action bytes_sent bytes_received packets session_end_reason device


    Reference: https://docs.paloaltonetworks.com/pan-os/8-0/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/traffic-log-fields

    ReplyDelete
    Replies
    1. Hi Harwinder , Thank you for the great query , I am wondering if you know how to correlate the traffic with threat intel IP feeds and only show logs for traffic that might be bad ? can be based on source and destination. Thanks again

      Delete