Sunday, May 3, 2015
PFSense + Splunk - Security on the cheap - Parsing Firewall logs
Splunk allows you to build dashboards which can be the view you see as you enter Splunk.
The searches below should be plugged into your dashboards as a panel, giving you a quick environment overview.
Sample Firewall Log
May 1 19:43:35 pfsense_firewall May 1 23:43:25 filterlog: 78,16777216,,1000002761,eth0,match,pass,out,4,0x0,,63,55879,0,DF,6,tcp,60,10.0.0.2,10.0.0.3,52008,443,0,S,4088643738,,14600,,mss;sackOK;TS;nop;wscaleFrom the above we can see that the firewall logs are flagged via "filterlog:". Using this information we will build out the information to monitor for the firewalls.
First let's build a table that allows us to get a view of all information flow seen by the firewall
Our search filter:
host="pfsense-firewall" "filterlog:" | rex field=_raw "filterlog:\s[0-9]*,[0-9]*,,[0-9]*,(?<interface>[0-9A-Za-z]*),(?<status>[A-Za-z]*),(?<pass_block>[A-Za-z]*),(?<in_out>[A-Za-z]*),[0-9]*,[0-9A-Za-z\s]*,[0-9]*,[0-9]*,[0-9]*,[0-9]*,[0-9A-Za-z]*,(?<protocol_number>[0-9]*),(?<protocol>[A-Za-z0-9]*),[0-9]*,(?<s_ip>[A-Za-z0-9\.\:]*),(?<d_ip>[A-Za-z0-9\.\:]*),(?<s_port>[0-9]*),(?<d_port>[0-9]*)"
The result of the above is shown below:
Now that we have the overview of the traffic, let's expand on our security monitoring by being a bit more focused. Let's look at the more active IPs
Our search filter:
host="pfsense_firewall" "filterlog:" ,match,pass, NOT *v6 | rex field=_raw "filterlog:\s[0-9]*,[0-9]*,,[0-9]*,(?<interface>[0-9A-Za-z]*),(?<status>[A-Za-z]*),(?<pass_block>[A-Za-z]*),(?<in_out>[A-Za-z]*),[0-9]*,[0-9A-Za-z\s]*,[0-9]*,[0-9]*,[0-9]*,[0-9]*,[0-9A-Za-z]*,(?<protocol_number>[0-9]*),(?<protocol>[A-Za-z0-9]*),[0-9]*,(?<s_ip>[A-Za-z0-9\.\:]*),(?<d_ip>[A-Za-z0-9\.\:]*),(?<s_port>[0-9]*),(?<d_port>[0-9]*)" | stats count by s_ip | sort count | reverse
Results of our search
While monitoring successful IPs is important, monitoring dropped IPs is just as important. These dropped IPs can serve as an early warning system
Our search filter:
host="pfsense_firewall" "filterlog:" ,match,block, NOT *v6 | rex field=_raw "filterlog:\s[0-9]*,[0-9]*,,[0-9]*,(?<interface>[0-9A-Za-z]*),(?<status>[A-Za-z]*),(?<pass_block>[A-Za-z]*),(?<in_out>[A-Za-z]*),[0-9]*,[0-9A-Za-z\s]*,[0-9]*,[0-9]*,[0-9]*,[0-9]*,[0-9A-Za-z]*,(?<protocol_number>[0-9]*),(?<protocol>[A-Za-z0-9]*),[0-9]*,(?<s_ip>[A-Za-z0-9\.\:]*),(?<d_ip>[A-Za-z0-9\.\:]*),(?<s_port>[0-9]*),(?<d_port>[0-9]*)" | stats count by s_ip | sort count | reverse
Search Results:
Monitoring allowed ports
Monitoring the allowed ports is just as important as monitoring the blocked ports. The objective here is that you get an idea of what ports are allowed through your network. This information may help you to recognise a misconfigured firewall or even a host that is infected or behaving in a suspicious manner.
Our search filter:
host="pfsense_firewall" "filterlog:" ,match,pass, NOT *v6 | rex field=_raw "filterlog:\s[0-9]*,[0-9]*,,[0-9]*,(?<interface>[0-9A-Za-z]*),(?<status>[A-Za-z]*),(?<pass_block>[A-Za-z]*),(?<in_out>[A-Za-z]*),[0-9]*,[0-9A-Za-z\s]*,[0-9]*,[0-9]*,[0-9]*,[0-9]*,[0-9A-Za-z]*,(?<protocol_number>[0-9]*),(?<protocol>[A-Za-z0-9]*),[0-9]*,(?<s_ip>[A-Za-z0-9\.\:]*),(?<d_ip>[A-Za-z0-9\.\:]*),(?<s_port>[0-9]*),(?<d_port>[0-9]*)" | stats count by d_port | sort count | reverse
Search results:
monitoring blocked ports
Similarly to monitoring allowed ports, we also need to monitor blocked ports. This helps us to understand which hosts may be trying to communicate on ports which are not allowed on the network helping to add context to the environment.
Our search filter:
host="pfsense_firewall" "filterlog:" ,match,block, NOT *v6 | rex field=_raw "filterlog:\s[0-9]*,[0-9]*,,[0-9]*,(?<interface>[0-9A-Za-z]*),(?<status>[A-Za-z]*),(?<pass_block>[A-Za-z]*),(?<in_out>[A-Za-z]*),[0-9]*,[0-9A-Za-z\s]*,[0-9]*,[0-9]*,[0-9]*,[0-9]*,[0-9A-Za-z]*,(?<protocol_number>[0-9]*),(?<protocol>[A-Za-z0-9]*),[0-9]*,(?<s_ip>[A-Za-z0-9\.\:]*),(?<d_ip>[A-Za-z0-9\.\:]*),(?<s_port>[0-9]*),(?<d_port>[0-9]*)" | stats count by d_port | sort count | reverse
Search results:
The above should give us a very good idea about the hosts that are communicating through our firewall and how the traffic is distributed. This is extremely important as it helps to add context to your environment and where possible create a baseline.
Hope you enjoyed this post. See you in the post on parsing snort alert logs.
In this series:
1. PFSense + Splunk - Security on the cheap
2. PFSense + Splunk - Security on the cheap - Parsing Firewall logs
3. PFSense + Splunk - Security on the cheap - Parsing ARPWatch Logs
4. PFSense + Splunk - Security on the cheap - Parsing Snort Logs
5. PFSense + Splunk - Security on the cheap - Parsing DHCP Server Logs
Subscribe to:
Post Comments (Atom)
The rex does not function properly when there is ICMP.
ReplyDeleteSend me a couple of sample logs with ICMP and I will write the the query out for you.
DeleteJan 4 09:31:04 filterlog: 91,16777216,,1000005911,sk0,match,pass,out,4,0x0,,64,20798,0,none,1,icmp,86,217.30.178.237,64.4.54.22,request,40417,486466
ReplyDeleteJan 4 09:17:18 filterlog: 82,16777216,,1451688253,ovpnc1,match,pass,in,4,0x0,,127,33,0,none,1,icmp,60,192.168.1.179,10.99.0.11,request,2,2010740
These are now "echo requests" aka pings so I assume the word "request" would be replaced in case of other icmp.
ReplyDeleteLet me know if this works for you
Deletefilterlog icmp | rex field=_raw "filterlog:\s\d{1,},\d{1,},,\d{1,},(?.*?),.*?,(?.*?),(?.*?),.*,icmp,\d{1,},(?\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),(?.*?)," | stats count by interface, action, direction, src_ip, dst_ip, icmp_message
hi, this command is not creating a table for me. I am still getting the raw format
ReplyDeletehost="pfsense-firewall" "filterlog:" | rex field=_raw "filterlog:\s[0-9]*,[0-9]*,,[0-9]*,(?[0-9A-Za-z]*),(?[A-Za-z]*),(?[A-Za-z]*),(?[A-Za-z]*),[0-9]*,[0-9A-Za-z\s]*,[0-9]*,[0-9]*,[0-9]*,[0-9]*,[0-9A-Za-z]*,(?[0-9]*),(?[A-Za-z0-9]*),[0-9]*,(?[A-Za-z0-9\.\:]*),(?[A-Za-z0-9\.\:]*),(?[0-9]*),(?[0-9]*)"
post a sample log entry and a sample of your splunk filter.
Delete