Visualizing your Zeek (Bro) data with Splunk - http.log (http logs)

The HTTP logs be it from your web server or any other source should be an area where great focus is placed. We do a large number of communications online and with the continued push to the cloud, monitoring this traffic will become even more critical. There is obviously a challenge that comes with this also. There is a greater push to ensure more privacy on the internet and thus there is probably more HTTPS (encrypted HTTP) traffic now online rather than HTTP (unencrypted). There are even browsers that have started to mark HTTP sites as unsecure. Thus there are interesting challenges ahead with monitoring of network traffic. However, while we still have visbility into these logs, let's make the most of them.

Similarly to the connections logs (conn.log), we need to understand the structure of the http.log file. Once again, let's leverage "bro-cut"

root@securitynik-host:/opt/bro/logs/current# bro-cut -C < http.log | head --lines=10 --verbose

#fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       trans_depth     method  host    uri     referrer        version user_agent    request_body_len        response_body_len       status_code     status_msg      info_code       info_msg        tags    username        password     proxied  orig_fuids      orig_filenames  orig_mime_types resp_fuids      resp_filenames  resp_mime_types
#types  time    string  addr    port    addr    port    count   string  string  string  string  string  string  count   count   count   string  count   stringset[enum]       string  string  set[string]     vector[string]  vector[string]  vector[string]  vector[string]  vector[string]  vector[string]
1541354421.339883       CvkNpf1KBPCrkB72k1      192.168.0.22    52512   134.19.176.32   80      1       GET     s2.startv.biz   /stalker_portal/server/load.php?type=watchdog&action=get_events&cur_play_type=0&event_active_id=0&init=0&JsHttpRequest=1-xml& -       1.1     Mozilla/5.0 (QtEmbedded; U; Linux; C) AppleWebKit/533.3 (KHTML, like Gecko) MAG200 stbapp ver: 2 rev: 250 Safari/533.3        0       164     200     OK      -       -       (empty) -       -       -    --       -       FAO2AESJgAeFeJRHl       -       text/json
1541354421.319882       CbqzWP1bslQsNBC8qe      192.168.0.22    55654   134.19.176.32   80      2       GET     s2.startv.biz   /stalker_portal/server/load.php?type=watchdog&action=get_events&cur_play_type=0&event_active_id=0&init=0&JsHttpRequest=1-xml& -       -       Mozilla/5.0 (QtEmbedded; U; Linux; C) AppleWebKit/533.3 (KHTML, like Gecko) MAG200 stbapp ver: 2 rev: 250 Safari/533.3        0       0       -       -       -       -       (empty) -       -       -    --       -       -       -       -

Now that we have the structure, the following search can be used to extract those field in Splunk.

index=_* OR index=* sourcetype=Bro-Security-Monitoring source="/opt/bro/logs/current/http.log" NOT "#fields"
|  rex field=_raw "(?<ts>.*?\t)(?<uid>.*?\t)(?<orig_h>.*?\t)(?<orig_p>.*?\t)(?<resp_h>.*?\t)(?<resp_p>.*?\t)(?<trans_depesponse_body_len>.*?\t)(?<status_code>.*?\t)(?<status_msg>.*?\t)(?<info_code>.*?\t)(?<info_msg>.*?\t)(?<tags>.*?\t)(?<u_fuids>.*?\t)(?<resp_filenames>.*?\t)(?<resp_mime_types>.*?\t)" 
|  stats count by ts,uid,orig_h,orig_p,resp_h,resp_p,trans_depesponse_body_len,status_code,status_msg,info_code,info_msg,tags,u_fuids,resp_filenames,resp_mime_types

The output below represents a snapshot of all the fields extracted.

Once again, once all the fields have been extracted we are then in a position to gain statistis on each field. I always am a big believer in tracking user agents, they provide insights into tools and applications being seen in your environment. This search filter allows you to track those user agents and the IPs they are associated with the applications on which they are being used.

index=_* OR index=* sourcetype=Bro-Security-Monitoring source="/opt/bro/logs/current/http.log" NOT "#fields"
|  rex field=_raw "(?<ts>.*?\t)(?<uid>.*?\t)(?<orig_h>.*?\t)(?<orig_p>.*?\t)(?<resp_h>.*?\t)(?<resp_p>.*?\t)(?<trans_depesponse_body_len>.*?\t)(?<status_code>.*?\t)(?<status_msg>.*?\t)(?<info_code>.*?\t)(?<info_msg>.*?\t)(?<tags>.*?\t)(?<u_fuids>.*?\t)(?<resp_filenames>.*?\t)(?<resp_mime_types>.*?\t)" 
|  stats count by orig_h,u_fuids 
|  dedup orig_h,u_fuids 
| sort -count

The search filter above produces:

Let's wrap this up by looking at the "rare" or better yet unique user agents. These unique user agents can help you detect sooner the signs of a possible compromise. If not a compromise, it can help you recognize when tools are used against your environment. The following filter will help you with identifying the unique user agents.

index=_* OR index=* sourcetype=Bro-Security-Monitoring source="/opt/bro/logs/current/http.log" NOT("microsoft.com" OR "dell.com" OR "adobe.com" OR "splunk.com" OR "firefox.com" OR "portableapps.com" OR "stariptv" OR "blogspot" OR "comodo.com" OR "WINDOWS.COM")
|  rex field=_raw "(?<ts>.*?\t)(?<uid>.*?\t)(?<orig_h>.*?\t)(?<orig_p>.*?\t)(?<resp_h>.*?\t)(?<resp_p>.*?\t)(?<trans_depesponse_body_len>.*?\t)(?<status_code>.*?\t)(?<status_msg>.*?\t)(?<info_code>.*?\t)(?<info_msg>.*?\t)(?<tags>.*?\t)(?<u_fuids>.*?\t)(?<resp_filenames>.*?\t)(?<resp_mime_types>.*?\t)" 
|  stats count by status_msg 
| sort -count 
|  rare limit=50 status_msg

Note that this filter whitelists some values such as anything to do with "microsoft.com", "dell.com", etc. The search then produced.

Well that's it for this post. Once again, we first extracted all the fields. As a result of that extraction, we are now able to utilize any of the fields as we see fit.


See you in the next post where we look at DNS logs.


References:
https://security.googleblog.com/2018/02/a-secure-web-is-here-to-stay.html
https://blog.chromium.org/2017/04/next-steps-toward-more-connection.html

Posts in this series:
Visualizing your Zeek (Bro) data with Splunk - The Setup
Visualizing your Zeek (Bro) data with Splunk - conn.log (connection logs)
Visualizing your Zeek (Bro) data with Splunk - http.log (http logs)
Visualizing your Zeek (Bro) data with Splunk - dns.log (connection logs)
Visualizing your Zeek (Bro) data with Splunk - x509.log (connection logs)