Friday, February 5, 2021

Snort3 on Ubuntu 20 - Housekeeping - AppID, RNA, Performance Monitoring, Profiling, JSON Logging, Other config, etc.

Now that your Snort3 has been installed and you have confirmed all your tests are working as expected, and you then fed the pig, your next step is to configure Snort3 for your specific environment. This philosophy should also be the same for any security tool you are using. Let's customize Snort3 for our environment.

Note: After every (small) modification, you should test your configuration. No need to wait until the end to test. If you wait until the end, you will more than likely run into (probably a ton of) problems. Here are two ways to test your configuration.

Option 1. Being verbose.

securitynik@snort3:~/snort-files$ sudo snort -c /usr/local/etc/snort/snort.lua
--------------------------------------------------
o")~   Snort++ 3.1.0.0
--------------------------------------------------
Loading /usr/local/etc/snort/snort.lua:
...
Snort successfully validated the configuration (with 0 warnings).
o")~   Snort exiting

Option 2. Being quiet.

securitynik@snort3:~/snort-files$ sudo snort -c /usr/local/etc/snort/snort.lua -q

If no information is returned, it is more likely your configuration is fine.

Let's edit the Snort3 configuration file. Change the HOME_NET from ...

HOME_NET = 'any'

... to reflect the networks you monitor and or own. For this purpose, let's say those are.

HOME_NET = [[10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16]]

Interestingly, in Snort2.x our EXTERNAL_NET is normally configured as EXTERNAL_NET = !$HOME_NET. However, it seems this might have changed in Snort3 and thus we leave EXTERNAL_NET = 'any'

Update: I recently saw in this document that we can use the following for External_NET.

EXTERNAL_NET = "!$HOME_NET"

Let's take advantage of reputation block list by removing "--[[" and "--]]" just before and after "reputation" respectively. This is what mine looks like after the change.

reputation =
{
    -- configure one or both of these, then uncomment reputation
    blacklist = BLACK_LIST_PATH .. "/default.blocklist"
    --whitelist = 'whitelist file name with ip lists'
}

Next up, let's make the configuration changes which allows us to take advantage of Hyperscan. This must be configured after reputation but before Section 3 configure bindings.

search_engine = { search_method = "hyperscan" } 

detection = { 
		hyperscan_literals = true, 
		pcre_to_regex = true 
	    } 

Let's now configure Snort3 to enable decoder and inspector alerts.

ips =
{
    mode = tap,
    -- use this to enable decoder and inspector alerts
    enable_builtin_rules = true,
    include = RULE_PATH .. "/local.rules",
    variables = default_variables
}

Because we may wish to send this data to other tools such as Elastic Stack, like we did with Zeek in this blog, let's enable logging in JSON format. 

In section "7. configure outputs", we add the following lines:

alert_json = 
{
    file = true, -- write the output to a file
    limit = 100, -- Create a new file after every 100M
    fields = 'seconds action class b64_data dir dst_addr dst_ap dst_port eth_dst eth_len \
    eth_src eth_type gid icmp_code icmp_id icmp_seq icmp_type iface ip_id ip_len msg mpls \
    pkt_gen pkt_len pkt_num priority proto rev rule service sid src_addr src_ap src_port \
    target tcp_ack tcp_flags tcp_len tcp_seq tcp_win tos ttl udp_len vlan timestamp',
}

Note I used the continuation character "\" above for clarity, if you encounter any issues, remove those and put everything on one line:

Here is what a sample of your JSON output may look like once an alert is triggered.

securitynik@snort3:~$ sudo tail --lines 1 /var/log/snort/alert_json.txt
{ "seconds" : 1612230402, "action" : "allow", "class" : "none", "dir" : "UNK", \
 "dst_ap" : ":0", "eth_dst" : "52:54:00:12:35:02", "eth_len" : 42, \
 "eth_src" : "08:00:27:2A:BA:15", "eth_type" : "0x806", "gid" : 112, \
 "iface" : "securitynik-sample.pcap", "msg" : "(arp_spoof) unicast ARP request", \
 "pkt_gen" : "raw", "pkt_len" : 0, "pkt_num" : 21043, "priority" : 3, \
"proto" : "ARP", "rev" : 1, "rule" : "112:1:1", "service" : "unknown", \
 "sid" : 1, "src_ap" : ":0", "vlan" : 0, "timestamp" : "02/01-20:46:42.210696" }
....

{ "seconds" : 1612458702, "action" : "allow", "class" : "none", "dir" : "S2C",\
 "dst_addr" : "10.0.2.15", "dst_ap" : "10.0.2.15:38500", "dst_port" : 38500, \
"eth_dst" : "08:00:27:2A:BA:15", "eth_len" : 60, "eth_src" : "52:54:00:12:35:02",\
 "eth_type" : "0x800", "gid" : 119, "iface" : "enp0s3:enp0s8", "ip_id" : 888, \
"ip_len" : 20, "msg" : "(http_inspect) HTTP header line terminated by LF without a CR", \
 "mpls" : 0, "pkt_gen" : "raw", "pkt_len" : 40, "pkt_num" : 549, "priority" : 3,\
 "proto" : "TCP", "rev" : 1, "rule" : "119:13:1", "service" : "http", "sid" : 13,\
 "src_addr" : "172.217.1.179", "src_ap" : "172.217.1.179:443", "src_port" : 443,\
 "tcp_ack" : 2000257859, "tcp_flags" : "***A****", "tcp_len" : 20, "tcp_seq" : 179136002,\
 "tcp_win" : 65535, "tos" : 0, "ttl" : 64, "vlan" : 0, \
"timestamp" : "02/04-12:11:42.995132" }

If you are looking to profile your Snort3 deployment, you can enable profiling for rules, memory, modules, etc., by changing 

-- profiler = { }

to

profiler = { }

This will give you an output similar to ...

--------------------------------------------------
module profile (all, depth 255, sorted by total_time)
#                      module layer    checks   time(us)  avg/check  %/caller  %/total
=                      ====== =====    ======   ========  =========  ========  =======
 1                     eventq     1     22755    1036104         45     56.41    56.41
 2                      other     1     21044     277173         13     15.09    15.09
 3                        daq     1     21374     267906         12     14.59    14.59
 4                 stream_tcp     1     10155      63083          6      3.43     3.43
 5                     decode     1     21044      59652          2      3.25     3.25
...
 19              back_orifice     1       859        578          0      0.03     0.03
 20                 arp_spoof     1       256        470          1      0.03     0.03
 21                       ssl     1       131        225          1      0.01     0.01
--                      total    --     21044    1836589         87        --   100.00
--------------------------------------------------
memory profile (all, depth 255, sorted by total_used)
#                  module layer   allocs   used (kb) avg/allocation  %/caller  %/total
=                  ====== =====   ======   ========= ==============  ========  =======
 1             stream_tcp     1     2316     1673.50          739.9     50.54    50.54
 2                 stream     1      447     1121.50         2569.2     33.87    33.87
 3           http_inspect     1      544      292.25          550.1      8.83     8.83
 4                  appid     1      444      222.00          512.0      6.70     6.70
 5                    ssl     1       14        1.75          128.0      0.05     0.05
--                  total    --     3765     3311.00          900.5        --   100.00
--------------------------------------------------
rule profile (all, sorted by total_time)
#       gid   sid rev    checks matches alerts time (us) avg/check avg/match avg/non-match timeouts suspends
=       ===   === ===    ====== ======= ====== ========= ========= ========= ============= ======== ========
1         1     4   5         1       1      1       262       262       262             0        0        0


Looking at the performance monitor statistics via perf_monitor, we change 

-- perf_monitor = { }

to enable everything via

perf_monitor = { }

Alternatively, we can be a bit more selective.

perf_monitor = {  
    modules = { },
    base = true,
    cpu = true,
    flow = true,
    flow_ip = true,
    packets = 10000,
    seconds = 60,
    flow_ports = 1023,
    output = console, 
    format = text,    
    summary = true,
    
    enable_flow_ip_profiling,
    
    packets = 10000,
    flow_tracker_creates,
    flow_tracker_total_deletes,
    flow_tracker_reload_deletes,
    flow_tracker_prunes
   }

Here is an example of some of those files.

securitynik@snort3:~/snort-files$ ls /var/log/snort/*.csv
/var/log/snort/perf_monitor_base.csv  /var/log/snort/perf_monitor_flow.csv
/var/log/snort/perf_monitor_cpu.csv   /var/log/snort/perf_monitor_flow_ip.csv

Here is the contents of one of the perf_monitor_flow_ip.csv

securitynik@snort3:~/snort-files$ sudo cat /var/log/snort/perf_monitor_flow_ip.csv
#timestamp,flow_ip.ip_a,flow_ip.ip_b,flow_ip.tcp_packets_a_b,flow_ip.tcp_bytes_a_b,flow_ip.tcp_packets_b_a,flow_ip.tcp_bytes_b_a,flow_ip.udp_packets_a_b,flow_ip.udp_bytes_a_b,flow_ip.udp_packets_b_a,flow_ip.udp_bytes_b_a,flow_ip.other_packets_a_b,flow_ip.other_bytes_a_b,flow_ip.other_packets_b_a,flow_ip.other_bytes_b_a,flow_ip.tcp_established,flow_ip.tcp_closed,flow_ip.udp_created
1612227193,10.0.2.15,91.189.89.199,0,0,0,0,12,1080,12,1080,0,0,0,0,0,0,12
1612227193,10.0.2.15,52.216.249.236,428,24196,225,307879,0,0,0,0,0,0,0,0,1,1,0
1612227193,10.0.2.15,140.82.112.4,566,32293,173,123010,0,0,0,0,0,0,0,0,2,2,0
1612227193,10.0.2.15,35.224.170.84,156,11527,145,12818,0,0,0,0,0,0,0,0,29,29,0
1612227193,10.0.2.15,99.86.62.122,5159,280212,2047,1390660,0,0,0,0,0,0,0,0,1,1,0
....

Obtaining and Installing Open AppID. I downloaded it to my workstation and then transferred it to my Snort3 box via SCP.

C:\users\securitynik\downloads>scp snort-openappid.tar.gz securitynik@10.0.0.116:~/snort-files`
securitynik@10.0.0.116's password:
snort-openappid.tar.gz                                 100%  409KB   4.6MB/s   00:00

Verify the hash which is posted online.

securitynik@snort3:~$ md5sum snort-openappid.tar.gz
3d407b77a0b58d71a14235e0960f86c4  snort-openappid.tar.gz

With the hash confirmed, extract the files.

securitynik@snort3:~/snort-files$ tar --extract --verbose --gzip --file snort-openappid.tar.gz

Copy the "odp" directory to "/usr/local/etc/odp"

securitynik@snort3:~/snort-files$ sudo mkdir --parents /usr/local/etc/odp
securitynik@snort3:~/snort-files$ sudo cp odp/* /usr/local/etc/odp/ --recursive --verbose

Modify the "appid" section to point to the "odp" folder. Here is what my config looks like after the change

appid =
{
    -- appid requires this to use appids in rules
    app_detector_dir = '/usr/local/etc/'
}

appid_listener =
{
    json_logging = true,
    file = "/var/log/snort/appid_json.txt",
}

Next up, let's take advantage or Realtime Network Awareness (RNA) host discovery. This is a feature I really liked as part of supporting Sourcefire IPS. I'm happy to see this is now part of Snort3 also.

First let's create a config file named in "/usr/local/etc/rna.conf"

securitynik@snort3:~$ cat /usr/local/etc/rna.conf
-- Discover host information for any IPv4 and IPv6 hosts
-- As of this writing, only host discovery has been implemented
-- Hopefully soon they will implement application and user
-- Then again, we can get some of that application information
-- from the APPID we previously configured

config Analyze 0.0.0.0/0 -1
config Analyze ::/0 -1


Continuing to modifying our Snort config (snort.lua) to implement the RNA inspector. 

...
dce_http_proxy = { }
dce_http_server = { }

-- information for RNA
host_cache = { dump_file = '/var/log/snort/rna.dump'} -- Store discovered host infomration
rna = {
    rna_conf_path = '/usr/local/etc/rna.conf',  -- Configuration file path
    dump_file = '/var/log/snort/rna_mac_cache.dump',  -- Dump RNA MAC Cache on Shutdown
    icmp_bidirectional,                         -- Count bidirectional ICMP flows received
    icmp_new,                                   -- Count new ICMP flows
    ip_new,                                     -- Count of new IP flows received
    udp_bidirectional,                          -- Count bidirectional UDP flows
    tcp_syn,                                    -- Count the number of TCP SYNs received
    tcp_syn_ack,                                -- Count the number of TCP SYN-ACKs received
    other_packets,                              -- Count packets received withiut session tracking
    change_host_update                          -- Count number of change hosts events

}

Running snort to see the RNA config in effect.

securitynik@snort3:~$ sudo /usr/local/bin/snort -c /usr/local/etc/snort/snort.lua \
--snaplen 65535 -k none -l /var/log/snort -i enp0s3:enp0s8 -A cmg
--------------------------------------------------
o")~   Snort++ 3.1.0.0
--------------------------------------------------
Loading /usr/local/etc/snort/snort.lua:
Loading snort_defaults.lua:
Finished snort_defaults.lua:
Loading file_magic.lua:
Finished file_magic.lua:
        ...
        ftp_server
        port_scan
        rna
        dce_http_server
        ...

With RNA loaded, generate some traffic that Snort3 can see, then shutdown snort after a few minutes (or hours if you wish) and look at the stats on the console.

--------------------------------------------------
rna
             appid_change: 77
       icmp_bidirectional: 2
                 icmp_new: 2
        udp_bidirectional: 23
                  udp_new: 23
                  tcp_syn: 9
              tcp_syn_ack: 8
            other_packets: 12
--------------------------------------------------

Looks good! Let's look to see if the files were created and their contents.

securitynik@snort3:~$ ls /var/log/snort/*rna* -al
-rw------- 1 root root 1271 Feb  4 12:05 /var/log/snort/rna.dump
-rw------- 1 root root  178 Feb  4 12:23 /var/log/snort/rna_mac_cache.dump

Taking a peek at the rna.dump file, we see ...

securitynik@snort3:~$ sudo head --lines 11 /var/log/snort/rna.dump
Current host cache size: 2915 bytes, 5 trackers

IP: 172.217.1.19
    type: Host, ttl: 64, hops: 255, time: 2021-02-04 17:03:42
macs size: 1
    mac: 52:54:00:12:35:02, ttl: 64, primary: 0, time: 2021-02-04 17:01:21
services size: 1
    port: 80, proto: 6, appid: 676, vendor: ghs, vendor: GSE
network proto: 2048
transport proto: 1, 6
....

And peeking into the rna_mac_cache.dump file, we see two mac addresses.

securitynik@snort3:~$ sudo cat /var/log/snort/rna_mac_cache.dump
Current mac cache size: 228 bytes, 2 trackers

MAC: 52:54:00:12:35:02
 Key: 90520731923714
 Network proto: 2054

MAC: 08:00:27:2a:ba:15
 Key: 8796750133781
 Network proto: 2054

Let's move on from the snort.lua configuration file and setup PulledPork to run automatically.

securitynik@snort3:~$ crontab -l
no crontab for securitynik

securitynik@snort3:~$ crontab -e
no crontab for securitynik - using an empty one

Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.tiny
  3. /bin/ed

Choose 1-3 [1]: 1



10 1 * * * /usr/local/bin/pulledpork.pl -c /usr/local/etc/pulledpork/pulledpork.conf -l -P -E -H SIGHUP

crontab: installing new crontabcrontab: installing new crontab


Before configuring Snort3 to start as a service, let's look to see if the network card has any type of offloading configured.

securitynik@snort3:~$ ethtool --show-offload enp0s3 | grep --perl-regexp "^generic|^tcp-segmentation|^large"
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]


securitynik@snort3:~$ ethtool --show-offload enp0s8 | grep --perl-regexp "^generic|^tcp-segmentation|^large"
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]

Create a SystemD Service to ensure turning off the ones flagged as "on" above.

securitynik@snort3:~$ sudo touch /lib/systemd/system/ethtool.service

securitynik@snort3:~$ cat /lib/systemd/system/ethtool.service
[Unit]
Description=Ethtool Configration for Network Interface

[Service]
Requires=network.target
Type=oneshot
ExecStart=/sbin/ethtool -K enp0s3 gro off
ExecStart=/sbin/ethtool -K enp0s3 lro off
ExecStart=/sbin/ethtool -K enp0s3 tso off
ExecStart=/sbin/ethtool -K enp0s3 gso off

ExecStart=/sbin/ethtool -K enp0s8 gro off
ExecStart=/sbin/ethtool -K enp0s8 lro off
ExecStart=/sbin/ethtool -K enp0s8 tso off
ExecStart=/sbin/ethtool -K enp0s8 gso off

[Install]
WantedBy=multi-user.target

We see below that various offloading features have been disabled ("off") for the interfaces I'm monitoring.

securitynik@snort3:~$ ethtool --show-offload enp0s3 | grep --perl-regexp "^generic|^tcp-segmentation|^large"
tcp-segmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]

securitynik@snort3:~$ ethtool --show-offload enp0s8 | grep --perl-regexp "^generic|^tcp-segmentation|^large"
tcp-segmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]

Enable and start the ethtool service.

securitynik@snort3:~$ sudo systemctl enable --now ethtool.service
Created symlink /etc/systemd/system/multi-user.target.wants/ethtool.service → /lib/systemd/system/ethtool.
service.

Before configuring the Snort3 service.

Create a group named "snort" to assign the "snort" user.

securitynik@snort3:~$ sudo groupadd snort
securitynik@snort3:~$ sudo useradd --system --shell /sbin/nologin --comment "Snort3_IDS/IPS User" --gid snort snort

Change the ownership and permission on the /var/log/snort folder and verify its ownership.

securitynik@snort3:~$ sudo chmod --recursive 5775 /var/log/snort/
securitynik@snort3:~$ sudo chown --recursive snort:snort /var/log/snort/

securitynik@snort3:~$ ls /var/log/snort/ -al
total 22292
drwsrwxr-t  2 snort snort      4096 Feb  3 14:52 .
drwxrwxr-x 14 root  syslog     4096 Feb  3 09:49 ..
-rwxrwxr-t  1 snort snort  22782434 Feb  3 14:52 alert_json.txt
...

Next up, create the Snort3 SystemD service

securitynik@snort3:~$ sudo vi /lib/systemd/system/snort3.service

securitynik@snort3:~$ cat /lib/systemd/system/snort3.service
[Unit]
Description=Snort3 NIDS Daemon
After=syslog.target network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/snort -c /usr/local/etc/snort/snort.lua --snaplen 65535 \
-k none -l /var/log/snort -D -u snort -g snort -i enp0s3:enp0s8 -m 0x1b --create-pidfile

[Install]
WantedBy=multi-user.target

Enable and start the Snort3 service.

securitynik@snort3:~$ sudo systemctl enable --now snort3
Created symlink /etc/systemd/system/multi-user.target.wants/snort3.service → /lib/systemd/system/snort3.service.

securitynik@snort3:~$ systemctl status snort3.service
● snort3.service - Snort3 NIDS Daemon
     Loaded: loaded (/lib/systemd/system/snort3.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-02-03 16:30:24 EST; 41s ago
   Main PID: 77578 (snort)
      Tasks: 2 (limit: 4654)
     Memory: 114.6M
     CGroup: /system.slice/snort3.service
             └─77578 /usr/local/bin/snort -c /usr/local/etc/snort/snort.lua --snaplen 65535 -k none -l /v>

Feb 03 16:30:24 snort3 systemd[1]: Started Snort3 NIDS Daemon.


No comments:

Post a Comment