Monday, August 21, 2017

Still Splunking Parsing TinyProxy logs – Building a monitoring system on the cheap

Having a proxy in your infrastructure, is essential for many different reasons. The first two to come to mind is bandwidth management and from a security perspective it gives excellent visibility into the domains and URLs being accessed by resources on your network. 

In this post, we will continue our building a monitoring system on the cheap by leveraging Splunk (free version) to identify domains and URLs which are detected, refused and allowed on our infrastructure via the TinyProxy proxy server.

Let’s get going!

First up with Splunk, let’s identify the log source so that we can focus on this traffic. In my case Tinyproxy has “source = /var/log/tinyproxy/tinyproxy.log” and “sourcetype = Tinyproxy”. Using either or both of these we can focus our search and filters.

Let’s filter our TinyProxy event types using “* sourcetype=Tinyproxy | rex field=_raw "(?<event_type>.*?\s+)" | stats count by event_type | sort count | reverse

This produces the following:

Now that we have the different event types, let’ save this and then focus on each of these to build our dashboard out.

Let’s first look at the hosts connecting to our proxy. This can be achieved through the use of the following search “* sourcetype=Tinyproxy CONNECT | rex field=_raw ".*\[(?<requesting_host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\]$" | stats count by requesting_host | sort count | reverse

The above search produced:






















Next up let’s look at the HTTP methods which are being seen. The search “* sourcetype=Tinyproxy CONNECT | rex field=_raw ".*\):\s+(?<http_method>[A-Z]*?\s+)" | stats count by http_method | sort count | reverse” helps us to gather this information as seen below.














Next up let’s identify the URLs which are being requested via GET or POST methods using the search “
* sourcetype=Tinyproxy CONNECT | rex field=_raw ".*\):\s+(?<http_method>(GET|POST)*?\s+)(?<url>.*?)HTTP" | stats count by url | sort count | reverse” we get the following:

Next up let’s identify the domains which are being allowed by leveraging the search “* sourcetype=Tinyproxy CONNECT established | rex field=_raw "Established\s+connection\s+to\s+host\s+\"(?<allowed_domains>.*?)\"\s+" | stats count by allowed_domains | sort count | reverse”. This produces the following

Now that we have the URLs as well as the domains being requested, let’s now figure out the domains and URLs which are being rejected.
First let’s look at the domains using “* sourcetype=Tinyproxy NOTICE | rex field=_raw ".*filtered\s+url\s+\"(?<filtered_url>(http|https).*?)\"" | stats count by filtered_url | sort count | reverse

This produces:
 
Focusing in specifically on the domains using the search “* sourcetype=Tinyproxy NOTICE | rex field=_raw ".*filtered\s+url\s+\"(?<filtered_domains>.*:(80|443)?)\"" | stats count by filtered_domains | sort count | reverse” we get

 
So that’s it for this post. Hope this helped you to make better use of your Splunk Dashboard skillz.

Reference:

1 comment:

  1. really cool post, highly informative and professionally written and I am glad to be a visitor of this perfect blog, thank you for this rare info!
    splunk training in hyderabad

    ReplyDelete