Silk is one of the tools used to analyze network flow data and something we teach in the SANS SEC503, Network Monitoring and Threat Detection. In this post, I am walking through some of the tools within the SiLK suite, to show their basic and somewhat common usage. There is no specific order to their usage and at times, you may even see the same tool being used multiple times but in different ways.
Get SiLK version, compile information, etc. via silk_config.
1 2 3 4 5 6 7 8 9 10 | sans@sec503:~/nik$ silk_config silk-version: 3.19.2 compiler: gcc cflags: -I/usr/local/include -DNDEBUG -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -I/usr/local/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fno-strict-aliasing -O3 include: -I/usr/local/include -DNDEBUG -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -I/usr/local/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include libsilk-libs: -L/usr/local/lib -lsilk -lz -lm libsilk-thrd-libs: -L/usr/local/lib -lsilk-thrd -lsilk -lz -lm libflowsource-libs: -L/usr/local/lib -lflowsource -lsilk-thrd -lsilk -L/usr/local/lib -lfixbuf -lpthread -lgthread-2.0 -pthread -lglib-2.0 -lz -lm data-rootdir: /data python-site-dir: /usr/lib/python3/dist-packages |
Get information about the sensors in the site via rwsiteinfo.
1 2 3 4 5 6 | sans@sec503:~$ rwsiteinfo --fields sensor,describe-sensor Sensor| Sensor-Description| Internal| Backbone ERS| Perimeter| Perimeter collector| ERS|Avaya ERS Switch Stack| internal| STIFortunes| |
A different view of the sensors information
1 2 3 | sans@sec503:~$ rwsiteinfo --fields=sensor:list Sensor:list| Internal,Perimeter,ERS,internal| |
Get information from a particular sensor via rwsiteinfo.
1 2 3 4 5 6 7 | sans@sec503:~$ rwsiteinfo --sensor=Internal --fields type,repo-file-count,repo-start-date,repo-end-date Type|File-Count| Start-Date| End-Date| in| 5828|2018/10/01T17:00:00|2022/07/03T19:00:00| out| 5093|2018/10/01T01:00:00|2022/07/03T19:00:00| inweb| 5059|2018/10/04T22:00:00|2022/07/03T19:00:00| outweb| 1781|2018/10/03T08:00:00|2019/05/03T14:00:00| ... |
Get information on classes, types and their default values. The "+" mark rows for the default class and "*" mark rows for a default type
1 2 3 4 5 6 | sans@sec503:~/nik$ rwsiteinfo --sensor=Perimeter --fields class,type,mark-default Class| Type|Defaults| all| in| +*| all| out| +*| all| inweb| +*| all| outweb| +*| |
Get the start and end date of the repo.
1 2 3 | sans@sec503:~/nik$ rwsiteinfo --fields=repo-start,repo-end Start-Date| End-Date| 2018/10/01T01:00:00|2022/07/03T19:00:00| |
Leverage rwcount, to count the number of flow records, their bytes and packets.
1 2 3 4 | sans@sec503:~/nik$ rwcount /tmp/attack-trace.rw Date| Records| Bytes| Packets| 2019/04/20T03:28:00| 2.60| 2218.09| 16.36| 2019/04/20T03:28:30| 9.40| 176342.91| 331.64| |
Leverage rwfilter, to retrieve information based on start and end date for all IP protocols relating to all traffic types and specifically for the host with address 8.8.8.8. Match on the first successful 100 records and save those to a file named 8.rw.
1 | sans@sec503:~/nik$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --protocol=0- \--type=all --any-address=8.8.8.8 --max-pass=100 --pass=8.rw |
Get information on the 8.rw file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | sans@sec503:~/nik$ rwfileinfo 8.rw 8.rw: format(id) FT_RWIPV6ROUTING(0x0c) version 16 byte-order littleEndian compression(id) none(0) header-length 176 record-length 88 record-version 1 silk-version 3.19.2 count-records 100 file-size 8976 command-lines 1 rwfilter --start=2022/01/05 --end=2022/07/01T23 --protocol=0- --type=all --any-address=8.8.8.8 --max-pass=100 --pass=8.rw |
Accessing the file just saved, by using the rwcut tool, while view a few fields.
1 2 3 4 5 6 | sans@sec503:~/nik$ rwcut 8.rw --fields sip,sPort,dIP,dPort sIP|sPort| dIP|dPort| 8.8.8.8| 53| 172.28.10.137|56213| 8.8.8.8| 53| 172.28.10.137|55171| 8.8.8.8| 53| 172.28.10.137|54512| .... |
Confirming the number of records in the file 8.rw.
1 2 | sans@sec503:~/nik$ rwcut 8.rw --no-title | wc --lines 100 |
Using rwcut, to get more details from a flow file named attack-trace.rw.
1 2 3 4 | sans@sec503:~/nik$ rwcut attack-trace.rw --fields=sIP,sPort,dIP,dPort,bytes,stime --num-recs=2 sIP|sPort| dIP|dPort| bytes| sTime| 98.114.205.102| 1821| 192.150.11.111| 445| 168|2019/04/20T03:28:28.374| 192.150.11.111| 445| 98.114.205.102| 1821| 128|2019/04/20T03:28:28.375| |
Removing the space to the left with ipv6=policy-ignore. We could have also set the environment variable SILK_IPV6_POLICY=ignore.
1 2 3 4 | sans@sec503:~/nik$ rwcut attack-trace.rw --fields=sIP,sPort,dIP,dPort,bytes,stime --num-recs=2 --ipv6-policy=ignore sIP|sPort| dIP|dPort| bytes| sTime| 98.114.205.102| 1821| 192.150.11.111| 445| 168|2019/04/20T03:28:28.374| 192.150.11.111| 445| 98.114.205.102| 1821| 128|2019/04/20T03:28:28.375| |
rwcut can be used without specifying fields. In the example below, it shows 12 fields by default.
1 2 3 4 | sans@sec503:~/nik$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --protocol=0- --type=all --any-address=8.8.8.8 --max-pass=100 --pass=stdout | rwcut --num-recs=2 sIP| dIP|sPort|dPort|pro| packets| bytes| flags| sTime| duration| eTime| sensor| 8.8.8.8| 172.28.10.137| 53|56213| 17| 1| 218| |2022/02/08T14:26:40.723| 0.001|2022/02/08T14:26:40.724| Internal| 8.8.8.8| 172.28.10.137| 53|55171| 17| 1| 102| |2022/02/08T14:27:10.329| 0.013|2022/02/08T14:27:10.342| Internal| |
Using rwcut, to get a CSV file from the retrieved data. Maybe you want to get this data in your machine learning algorithms, something we teach in the SANS SEC595: Applied Data Science and AI/Machine Learning for Cybersecurity Professionals or maybe you would like to import them into Pandas or Excel.
1 2 3 4 | sans@sec503:~/nik$ rwcut attack-trace.rw --fields=sIP,sPort,dIP,dPort,bytes,stime \--num-recs=2 --ipv6-policy=ignore --no-columns --delimited=, --no-final-delimiter sIP,sPort,dIP,dPort,bytes,sTime 98.114.205.102,1821,192.150.11.111,445,168,2019/04/20T03:28:28.374 192.150.11.111,445,98.114.205.102,1821,128,2019/04/20T03:28:28.375 |
Get information on a particular bytes-range.
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwfilter --start=2022/01/05T0 --end=2022/07/01 --protocol=0- --pass=stdout --type=all --bytes=0-30 \--max-pass=5 | rwuniq --fields=sIP,dIP,bytes,packets sIP| dIP| bytes| packets| Records| 10.200.223.7| 172.28.10.1| 28| 1| 1| 10.200.223.7| 172.28.20.1| 28| 1| 1| 10.200.223.7| 172.28.1.1| 28| 1| 1| 10.200.223.7| 172.28.30.64| 28| 1| 1| 10.200.223.7| 172.28.30.65| 28| 1| 1| |
Group data in 24 hours bin/buckets
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwfilter --start=2022/01/05T0 --end=2022/07/01 --protocol=0- \--pass=stdout --type=all --bytes=0-30 | rwuniq --bin-time=86400 --fields stime,type \--values=records --sort-output sTime| type| Records| 2022/02/12T00:00:00| in| 4136| 2022/02/12T00:00:00| out| 52| 2022/02/13T00:00:00| in| 2469| 2022/02/14T00:00:00| in| 4307| 2022/02/14T00:00:00| out| 7| ... |
Grouping data in 1 hour bins/buckets.
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwfilter --start=2022/01/05T0 --end=2022/07/01 --protocol=0- \--pass=stdout --type=all --bytes=0-30 | rwuniq --bin-time=3600 --fields stime,type \--values=records --sort-output sTime| type| Records| 2022/02/12T19:00:00| in| 2| 2022/02/12T20:00:00| in| 3674| 2022/02/12T20:00:00| out| 52| 2022/02/12T21:00:00| in| 14| 2022/02/12T22:00:00| in| 446| .... |
Get the number of bytes within the hours.
1 2 3 4 5 6 7 8 9 10 11 | sans@sec503:~/nik$ rwfilter --start=2022/01/05T0 --end=2022/07/01 --protocol=0- \--pass=stdout --type=all --bytes=0-30 | rwuniq --bin-time=86400 --fields stime,type \--values=bytes --sort-output | head --lines=10 sTime| type| Bytes| 2022/02/12T00:00:00| in| 115808| 2022/02/12T00:00:00| out| 1456| 2022/02/13T00:00:00| in| 69132| 2022/02/14T00:00:00| in| 120596| 2022/02/14T00:00:00| out| 196| 2022/02/16T00:00:00| out| 120| 2022/02/17T00:00:00| in| 5527373| 2022/02/17T00:00:00| out| 882| 2022/02/18T00:00:00| in| 29| |
Extending further, grabbing the count of the distinct source and destination IPs.
1 2 3 4 5 6 7 8 9 10 11 | sans@sec503:~/nik$ rwfilter --start=2022/01/05T0 --end=2022/07/01 --protocol=0- --pass=stdout \--type=all --bytes=0-30 | rwuniq --bin-time=86400 --fields stime,type --values=bytes,sip,dip \--sort-output | head --lines=10 sTime| type| Bytes| sIP-Distinct| dIP-Distinct| 2022/02/12T00:00:00| in| 115808| 2| 3268| 2022/02/12T00:00:00| out| 1456| 1| 1| 2022/02/13T00:00:00| in| 69132| 1| 2387| 2022/02/14T00:00:00| in| 120596| 1| 4199| 2022/02/14T00:00:00| out| 196| 7| 1| 2022/02/16T00:00:00| out| 120| 1| 4| 2022/02/17T00:00:00| in| 5527373| 3| 13| 2022/02/17T00:00:00| out| 882| 7| 21| 2022/02/18T00:00:00| in| 29| 1| 1| |
By default rwuniq has a value of records, ie --value=records. This represents which values are counted in the bin.
1 2 3 4 | sans@sec503:~/nik$ rwuniq attack-trace.rw --fields sIP sIP| Records| 192.150.11.111| 6| 98.114.205.102| 6| |
Above is the same as --value=records means the records are counted in the bin.
1 2 3 4 | sans@sec503:~/nik$ rwuniq attack-trace.rw --fields sIP --values=records sIP| Records| 192.150.11.111| 6| 98.114.205.102| 6| |
Expand rwuniq to extract the stime and source IP fields. Group by the bytes and sort the output.
1 2 3 4 | sans@sec503:~/nik$ rwuniq attack-trace.rw --fields stime,sip --values=bytes --sort-output --bin-time=600 sTime| sIP| Bytes| 2019/04/20T03:20:00| 98.114.205.102| 171264| 2019/04/20T03:20:00| 192.150.11.111| 7297| |
Group by packets with a bin size of 10 minutes
1 2 3 4 | sans@sec503:~/nik$ rwuniq attack-trace.rw --fields stime,sip --values=packets \--sort-output --bin-time=600 sTime| sIP| Packets| 2019/04/20T03:20:00| 98.114.205.102| 195| 2019/04/20T03:20:00| 192.150.11.111| 153| |
Group by both packets and bytes
1 2 3 4 | sans@sec503:~/nik$ rwuniq attack-trace.rw --fields stime,sip --values=bytes,packets --sort-output --bin-time=600 sTime| sIP| Bytes| Packets| 2019/04/20T03:20:00| 98.114.205.102| 171264| 195| 2019/04/20T03:20:00| 192.150.11.111| 7297| 153| |
Assuming the input has been sorted, we can pass --presorted-input to the rwuiq command.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwuniq attack-trace.rw --fields sip,stime --values=bytes,packets --presorted-input --bin-time=600 sIP| sTime| Bytes| Packets| 98.114.205.102|2019/04/20T03:20:00| 168| 4| 192.150.11.111|2019/04/20T03:20:00| 128| 3| 98.114.205.102|2019/04/20T03:20:00| 4777| 14| 192.150.11.111|2019/04/20T03:20:00| 1590| 17| ... |
Once again, use --ipv6-policy=true to remove the space on the left.
1 2 3 4 5 6 | sans@sec503:~/nik$ rwuniq attack-trace.rw --fields sip,stime --values=bytes,packets \--presorted-input --bin-time=600 --ipv6-policy=ignore sIP| sTime| Bytes| Packets| 98.114.205.102|2019/04/20T03:20:00| 168| 4| 192.150.11.111|2019/04/20T03:20:00| 128| 3| 98.114.205.102|2019/04/20T03:20:00| 4777| 14| |
Finding the most commonly used protocols with rwstats.
rwstats group records into time bin either by field or fields.
rwstats can count the top N and lower N number of bins. rwuniq cannot do this.
rwstats can also compute summary percentage.
Find the top 10 protocols in a 10 minute span.
1 2 3 4 5 | sans@sec503:~/nik$ rwstats attack-trace.rw --fields=protocol,stime \--count=10 --bin-time=600 --values=bytes INPUT: 12 Records for 1 Bin and 178561 Total Bytes OUTPUT: Top 10 Bins by Bytes pro| sTime| Bytes| %Bytes| cumul_%| 6|2019/04/20T03:20:00| 178561|100.000000|100.000000| |
Grab the top 5 bins within a 5 minutes span. Group by bytes.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwfilter --protocol=0- --start-date=2022/01/01 \--end-date=2022/05/01 --pass=stdout --max-pass=100 | rwstats --field=stime,sIP \--count=5 --values=bytes --bin-time=300 INPUT: 100 Records for 5 Bins and 17964 Total Bytes OUTPUT: Top 5 Bins by Bytes sTime| sIP| Bytes| %Bytes| cumul_%| 2022/02/08T14:40:00| 8.8.8.8| 13785| 76.736807| 76.736807| 2022/02/08T14:35:00| 8.8.8.8| 2166| 12.057448| 88.794255| 2022/02/08T14:25:00| 8.8.8.8| 1101| 6.128925| 94.923180| 2022/02/08T14:30:00| 8.8.8.8| 836| 4.653752| 99.576932| 2022/02/08T14:40:00| 17.253.26.125| 76| 0.423068|100.000000| |
Top 5 records by bytes.
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwfilter --protocol=0- --start-date=2022/01/01 --end-date=2022/05/01 \--pass=stdout --max-pass=1000 | rwstats --field=protocol --count=5 --values=bytes \--bin-time=300 INPUT: 1000 Records for 4 Bins and 5287991 Total Bytes OUTPUT: Top 5 Bins by Bytes pro| Bytes| %Bytes| cumul_%| 6| 5142342| 97.245665| 97.245665| 17| 134037| 2.534743| 99.780408| 1| 11500| 0.217474| 99.997882| 58| 112| 0.002118|100.000000| |
Top 5 records by packets.
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwfilter --protocol=0- --start-date=2022/01/01 --end-date=2022/05/01 \--pass=stdout --max-pass=1000 | rwstats --field=protocol --count=5 --values=packets \--bin-time=300 INPUT: 1000 Records for 4 Bins and 10277 Total Packets OUTPUT: Top 5 Bins by Packets pro| Packets| %Packets| cumul_%| 6| 9235| 89.860854| 89.860854| 17| 915| 8.903376| 98.764231| 1| 125| 1.216308| 99.980539| 58| 2| 0.019461|100.000000| |
Top 5 records, by records which are the default when no values are specified.
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwfilter --protocol=0- --start-date=2022/01/01 --end-date=2022/05/01 \--pass=stdout --max-pass=1000 | rwstats --field=protocol --count=5 --bin-time=300 INPUT: 1000 Records for 4 Bins and 1000 Total Records OUTPUT: Top 5 Bins by Records pro| Records| %Records| cumul_%| 17| 886| 88.600000| 88.600000| 6| 109| 10.900000| 99.500000| 1| 3| 0.300000| 99.800000| 58| 2| 0.200000|100.000000| |
Get the overall stats via summary parameters
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | sans@sec503:~/nik$ rwstats attack-trace.rw --overall-stats | more FLOW STATISTICS--ALL PROTOCOLS: 12 records *BYTES min 40; max 165088 quartiles LQ 150.00000 Med 504.00000 UQ 4000.00000 UQ-LQ 3850.00000 interval_max|count<=max|%_of_input| cumul_%| 40| 1| 8.333333| 8.333333| 60| 1| 8.333333| 16.666667| 100| 0| 0.000000| 16.666667| 150| 1| 8.333333| 25.000000| 256| 2| 16.666667| 41.666667| 1000| 3| 25.000000| 66.666667| 10000| 3| 25.000000| 91.666667| 100000| 0| 0.000000| 91.666667| 1000000| 1| 8.333333|100.000000| 4294967295| 0| 0.000000|100.000000| *PACKETS min 1; max 159 quartiles LQ 3.00000 Med 10.00000 UQ 17.50000 UQ-LQ 14.50000 interval_max|count<=max|%_of_input| cumul_%| 3| 3| 25.000000| 25.000000| 4| 1| 8.333333| 33.333333| 10| 2| 16.666667| 50.000000| 20| 4| 33.333333| 83.333333| 50| 0| 0.000000| 83.333333| 100| 0| 0.000000| 83.333333| 500| 2| 16.666667|100.000000| 1000| 0| 0.000000|100.000000| 10000| 0| 0.000000|100.000000| 4294967295| 0| 0.000000|100.000000| ... |
Look at the top 5 by bytes, this time includes the "distinct"/unique source and destination IPs.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwstats attack-trace.rw --count=5 --fields=bytes --values=bytes,distinct:sip,dip INPUT: 12 Records for 12 Bins and 178561 Total Bytes OUTPUT: Top 5 Bins by Bytes bytes| Bytes| sIP-Distinct| dIP-Distinct| %Bytes| cumul_%| 165088| 165088| 1| 1| 92.454679| 92.454679| 4777| 4777| 1| 1| 2.675276| 95.129956| 4488| 4488| 1| 1| 2.513427| 97.643382| 1590| 1590| 1| 1| 0.890452| 98.533834| 801| 801| 1| 1| 0.448586| 98.982421| |
Set a threshold for the number of records that must be found before a flow can be reported.
1 2 3 4 5 6 | sans@sec503:~/nik$ rwstats attack-trace.rw --threshold=6 --fields=sIP INPUT: 12 Records for 2 Bins and 12 Total Records OUTPUT: Top 2 bins by Records (threshold 6) sIP| Records| %Records| cumul_%| 98.114.205.102| 6| 50.000000| 50.000000| 192.150.11.111| 6| 50.000000|100.000000| |
Set a threshold for the number of bytes that must be match in a flow to 1500.
1 2 3 4 5 6 | sans@sec503:~/nik$ rwstats attack-trace.rw --threshold=1500 --fields=sIP --values=bytes INPUT: 12 Records for 2 Bins and 178561 Total Bytes OUTPUT: Top 2 bins by Bytes (threshold 1500) sIP| Bytes| %Bytes| cumul_%| 98.114.205.102| 171264| 95.913441| 95.913441| 192.150.11.111| 7297| 4.086559|100.000000| |
Both records above match that criterion. Let's change this to a threshold of 7298 to get just one record.
1 2 3 4 5 | sans@sec503:~/nik$ rwstats attack-trace.rw --threshold=7298 --fields=sIP --values=bytes INPUT: 12 Records for 2 Bins and 178561 Total Bytes OUTPUT: Top 1 bins by Bytes (threshold 7298) sIP| Bytes| %Bytes| cumul_%| 98.114.205.102| 171264| 95.913441| 95.913441| |
Above shows, with our threshold, only one record was returned. Removing the two right most columns. The percentage fields.
1 2 3 4 5 | sans@sec503:~/nik$ rwstats attack-trace.rw --threshold=7298 --fields=sIP \--values=bytes --no-percents INPUT: 12 Records for 2 Bins and 178561 Total Bytes OUTPUT: Top 1 bins by Bytes (threshold 7298) sIP| Bytes| 98.114.205.102| 171264| |
Characterizing traffic by time. view records in 20 seconds buckets.
1 2 3 4 | sans@sec503:~/nik$ rwcount attack-trace.rw --bin-size=20 Date| Records| Bytes| Packets| 2019/04/20T03:28:20| 8.26| 100551.36| 212.18| 2019/04/20T03:28:40| 3.74| 78009.64| 135.82| |
rwcount default bin size is 30 seconds.
1 2 3 4 | sans@sec503:~/nik$ rwcount attack-trace.rw Date| Records| Bytes| Packets| 2019/04/20T03:28:00| 2.60| 2218.09| 16.36| 2019/04/20T03:28:30| 9.40| 176342.91| 331.64| |
You can skip flows with zero bytes, flows or packets by using --skip-zeroes. I don't have any 0s below. At the same time, I've changed the --bin-size to 20 seconds rather than the default 30.
1 2 3 4 | sans@sec503:~/nik$ rwcount attack-trace.rw --bin-size=20 --skip-zeroes Date| Records| Bytes| Packets| 2019/04/20T03:28:20| 8.26| 100551.36| 212.18| 2019/04/20T03:28:40| 3.74| 78009.64| 135.82| |
Reverse sort all records by destination IP, protocol and bytes. rwsort binary output cannot be written to the screen. Hence the pipe to rwcut.
1 2 3 4 5 6 7 8 9 10 11 12 13 | sans@sec503:~/nik$ rwsort attack-trace.rw --fields=dip,protocol,bytes \--reverse | rwcut --fields=dip,protocol,bytes,stime --num-recs=10 --ipv6-policy=ignore dIP|pro| bytes| sTime| 192.150.11.111| 6| 165088|2019/04/20T03:28:34.516| 192.150.11.111| 6| 4777|2019/04/20T03:28:28.509| 192.150.11.111| 6| 798|2019/04/20T03:28:33.576| 192.150.11.111| 6| 381|2019/04/20T03:28:30.466| 192.150.11.111| 6| 168|2019/04/20T03:28:28.374| 192.150.11.111| 6| 52|2019/04/20T03:28:44.593| 98.114.205.102| 6| 4488|2019/04/20T03:28:34.517| 98.114.205.102| 6| 1590|2019/04/20T03:28:28.509| 98.114.205.102| 6| 801|2019/04/20T03:28:33.457| 98.114.205.102| 6| 250|2019/04/20T03:28:30.466| |
Perform the reverse sort based on the bytes.
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwsort attack-trace.rw --fields=bytes,dip,protocol --reverse | \rwcut --fields=dip,protocol,bytes,stime --num-recs=10 --ipv6-policy=ignore dIP|pro| bytes| sTime| 192.150.11.111| 6| 165088|2019/04/20T03:28:34.516| 192.150.11.111| 6| 4777|2019/04/20T03:28:28.509| 98.114.205.102| 6| 4488|2019/04/20T03:28:34.517| 98.114.205.102| 6| 1590|2019/04/20T03:28:28.509| 98.114.205.102| 6| 801|2019/04/20T03:28:33.457| 192.150.11.111| 6| 798|2019/04/20T03:28:33.576| |
Create a set of IP addresses from flow data using a combination of rwfilter and rwset. This can be used for export from flow and import into other security tools such as SIEM, Firewall, etc.
1 | sans@sec503:~/nik$ rwfilter --type=all --pass=stdout --proto=0- \--start-date=2022/04/1T00 --end-date=2022/04/04 --bytes-per-packet=70 \--max-pass=100 | rwset --any-file=ip_from_flow.set
|
Validate the exported records, by leveraging rwsetcat.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwsetcat ip_from_flow.set 8.8.8.8 18.118.192.126 34.193.254.175 35.168.220.189 172.28.10.137 172.28.30.2 172.28.50.2 192.225.158.1 |
Reverse this process, using rwsetbuild. Create a set of IPs from a txt file. This can be used for ignoring future flows via an allow/permit list.
1 | sans@sec503:~/nik$ rwsetbuild --ip-ranges ip.txt ip.set |
Read the created set via rwsetcat.
1 2 3 4 5 6 | sans@sec503:~/nik$ rwsetcat ip.set 1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5 |
Get statistics on the IP addresses.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwsetcat --print-statistics ip.set Network Summary minimumIP = 1.1.1.1 maximumIP = 5.5.5.5 5 hosts (/32s), 0.000000% of 2^32 5 occupied /8s, 1.953125% of 2^8 5 occupied /16s, 0.007629% of 2^16 5 occupied /24s, 0.000030% of 2^24 5 occupied /27s, 0.000004% of 2^27 |
Get a snapshot view of the network structure with rwsetcat.
1 2 | sans@sec503:~/nik$ rwsetcat ip.set --network-structure TOTAL| 5 hosts in 5 /8s, 5 /16s, 5 /24s, and 5 /27s |
Get a different view of the network structure with rwsetcat.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwsetcat ip.set --network-structure=24 1.1.1.0/24| 1 2.2.2.0/24| 1 3.3.3.0/24| 1 4.4.4.0/24| 1 5.5.5.0/24| 1 |
Do a resolve IP addresses to host names using rwresolve, taking the data from rwsetcat output.
1 2 3 4 5 6 | sans@sec503:~/nik$ rwsetcat ip.set | rwresolve one.one.one.one 2.2.2.2 3.3.3.3 4.4.4.4 dynamic-005-005-005-005.5.5.pool.telefonica.de |
About to do another resolve. Review the data first via rwcut.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwcut 8.rw --fields=sip,dip --num-recs=5 --ipv6-policy=ignore sIP| dIP| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| |
Doing the resolve by specifying the getnameinfo resolver.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwcut 8.rw --fields=sip,dip --num-recs=5 --ipv6-policy=ignore | \rwresolve --ip-fields=1,2 --resolver=getnameinfo sIP| dIP| dns.google| 172.28.10.137| dns.google| 172.28.10.137| dns.google| 172.28.10.137| dns.google| 172.28.10.137| dns.google| 172.28.10.137| |
Find the top 5 DNS Servers seen within the flows using rwfilter and rwstats.
Interesting that a public DNS server is seen as the device with highest number of packets. I was expecting to see an internal DNS server. Then again, it could be the location of this sensor.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=17 --pass=stdout --type=in --sport=53 | rwstats --values=packets \--fields sIP --count=5 --ipv6-policy=ignore INPUT: 1215737 Records for 276 Bins and 1348651 Total Packets OUTPUT: Top 5 Bins by Packets sIP| Packets| %Packets| cumul_%| 8.8.8.8| 1330414| 98.647760| 98.647760| 199.212.0.63| 9296| 0.689281| 99.337041| 199.180.180.63| 2011| 0.149112| 99.486153| 204.61.216.50| 1272| 0.094316| 99.580470| 205.251.199.83| 586| 0.043451| 99.623920| |
Looking at the DNS communication from the bytes perspective using rwfilter and rwstats.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=17 --pass=stdout --type=in --sport=53 | rwstats --values=bytes \--fields sIP --count=5 --ipv6-policy=ignore INPUT: 1215737 Records for 276 Bins and 212797321 Total Bytes OUTPUT: Top 5 Bins by Bytes sIP| Bytes| %Bytes| cumul_%| 8.8.8.8| 209509379| 98.454895| 98.454895| 199.212.0.63| 758309| 0.356353| 98.811248| 199.180.180.63| 710009| 0.333655| 99.144903| 204.61.216.50| 449465| 0.211217| 99.356120| 193.0.9.10| 205880| 0.096749| 99.452870| |
Looking at it from the number of records packets using rwfilter and rwstats.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=17 --pass=stdout --type=in --sport=53 | rwstats --values=records \--fields sIP --count=5 --ipv6-policy=ignore INPUT: 1215737 Records for 276 Bins and 1215737 Total Records OUTPUT: Top 5 Bins by Records sIP| Records| %Records| cumul_%| 8.8.8.8| 1200021| 98.707286| 98.707286| 199.212.0.63| 7931| 0.652361| 99.359648| 199.180.180.63| 2008| 0.165167| 99.524815| 204.61.216.50| 1271| 0.104546| 99.629361| 193.0.9.10| 581| 0.047790| 99.677151| |
Above relates to traffic coming in the enterprise. What about traffic going out to DNS Servers?
Looking at it from a different perspective using rwfilter and rwstats.
1 2 3 4 5 6 7 8 9 10 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=17 --pass=stdout --type=out,outweb --dport=53 | rwstats --values=bytes \--fields sIP --count=5 --ipv6-policy=ignore INPUT: 1225733 Records for 10 Bins and 110746579 Total Bytes OUTPUT: Top 5 Bins by Bytes sIP| Bytes| %Bytes| cumul_%| 172.28.10.137| 110716457| 99.972801| 99.972801| 172.28.30.2| 13824| 0.012483| 99.985284| 172.28.20.3| 12528| 0.011312| 99.996596| 172.28.20.5| 960| 0.000867| 99.997463| 172.28.30.5| 680| 0.000614| 99.998077| |
The number of flow records using rwfilter and rwstats.
1 2 3 4 5 6 7 8 9 10 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 --protocol=17 --pass=stdout --type=out,outweb --dport=53 | rwstats --values=records --fields sIP --count=5 --ipv6-policy=ignore INPUT: 1225733 Records for 10 Bins and 1225733 Total Records OUTPUT: Top 5 Bins by Records sIP| Records| %Records| cumul_%| 172.28.10.137| 1225669| 99.994779| 99.994779| 172.28.30.2| 29| 0.002366| 99.997145| 172.28.20.3| 24| 0.001958| 99.999103| 172.28.10.89| 3| 0.000245| 99.999347| 172.28.30.5| 2| 0.000163| 99.999510| |
Digging deeper to see what the host at 172.28.10.137 is doing, using rwfilter and rwstats
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=17 --pass=stdout --type=out,outweb --dport=53 | rwstats --values=bytes \--fields sIP,dip,dport --count=5 --ipv6-policy=ignore INPUT: 1225733 Records for 359 Bins and 110746579 Total Bytes OUTPUT: Top 5 Bins by Bytes sIP| dIP|dPort| Bytes| %Bytes| cumul_%| 172.28.10.137| 8.8.8.8| 53| 108039369| 97.555491| 97.555491| 172.28.10.137| 199.212.0.63| 53| 1382580| 1.248418| 98.803909| 172.28.10.137| 192.175.48.6| 53| 300216| 0.271084| 99.074993| 172.28.10.137| 192.175.48.42| 53| 299097| 0.270073| 99.345066| 172.28.10.137| 199.180.180.63| 53| 153126| 0.138267| 99.483333| |
What's your conclusion of above?
What sensor is this traffic coming from? Using rwfilter and rwstats
1 2 3 4 5 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 --protocol=17 --pass=stdout --type=out,outweb --dport=53 | rwstats --values=records \--fields sensor --count=5 --ipv6-policy=ignore --no-percent INPUT: 1225733 Records for 1 Bin and 1225733 Total Records OUTPUT: Top 5 Bins by Records sensor| Records| Internal| 1225733| |
Looking at it from a different perspective via rwfilter and rwuniq.
1 2 3 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=17 --pass=stdout --type=out,outweb --dport=53 | rwuniq --values=flows \--fields=sensor sensor| Records| Internal| 1225733| |
1 2 3 4 5 6 7 8 9 10 11 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=0- --pass=stdout --type=all --any-address=172.28.10.137 | \rwuniq --values=flows,distinct:sip,distinct:dip,distinct:sport,distinct:dport \--fields type,protocol --sort type|pro| Records| sIP-Distinct| dIP-Distinct|sPort|dPort| in| 1| 160| 8| 1| 1| 1| in| 6| 408473| 8| 1|21465|65477| in| 8| 1| 1| 1| 1| 1| in| 17| 1245841| 280| 1| 279|14462| in| 63| 1| 1| 1| 1| 1| out| 1| 139| 1| 8| 1| 1| out| 6| 12106| 1| 8| 21| 9732| out| 17| 1230067| 1| 355| 8708| 99| inweb| 6| 10791| 294| 1| 185| 7857| |
Finding the top 3 unique destination ports for traffic going outbound using rwfilter and rwuniq.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=0- --pass=stdout --type=out,outweb | rwstats --value=flows \--fields=dport --count=3 INPUT: 1894243 Records for 31811 Bins and 1894243 Total Records OUTPUT: Top 3 Bins by Records dPort| Records| %Records| cumul_%| 53| 1225733| 64.708329| 64.708329| 443| 120219| 6.346546| 71.054875| 9573| 9755| 0.514981| 71.569857| |
Since I've done some work with port 53 above, let's look at port 443.
Find the top 5 source IP communicating via port 443 with traffic greater than 250 bytes in their flows. Note the rwfilter --bytes-per-packet=250-. Once again, using rwfilter and rwstats.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=0- --pass=stdout --type=out,outweb --dport=443 --bytes-per-packet=250- | \rwstats --value=bytes --fields=sip --count=5 --ipv6-policy=ignore INPUT: 120213 Records for 10 Bins and 691383585 Total Bytes OUTPUT: Top 5 Bins by Bytes sIP| Bytes| %Bytes| cumul_%| 172.28.20.6| 130274858| 18.842631| 18.842631| 172.28.30.5| 114087108| 16.501275| 35.343906| 172.28.30.2| 104686092| 15.141536| 50.485442| 172.28.30.3| 90222518| 13.049560| 63.535002| 172.28.20.3| 72788328| 10.527922| 74.062925| |
Looking at flow records with 0-250 bytes per packet. Note the rwfilter --bytes-per-packet=0-250. Adding the duration to this activity. Interesting this activity all have 0 time. Scanning?
1 2 3 4 5 6 7 8 9 10 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=0- --pass=stdout --type=out,outweb --dport=443 --bytes-per-packet=0-250 | \rwstats --value=bytes --fields=sip,dip,sport,dport,packets,duration --count=5 \--ipv6-policy=ignore --no-percent INPUT: 6 Records for 6 Bins and 744 Total Bytes OUTPUT: Top 5 Bins by Bytes sIP| dIP|sPort|dPort| packets|durat| Bytes| 172.28.30.4| 23.58.146.215|57496| 443| 1| 0| 124| 172.28.30.3| 23.58.146.216|65523| 443| 1| 0| 124| 172.28.30.3| 23.58.146.216|56311| 443| 1| 0| 124| 172.28.20.6| 23.58.146.215|49308| 443| 1| 0| 124| 172.28.30.4| 184.51.157.69|58644| 443| 1| 0| 124| |
Find flows where the duration is 0 and the bytes-per-packet is less than 250 using rwfilter and rwstats.
1 2 3 4 5 6 7 8 9 | sans@sec503:~$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 --protocol=0- --pass=stdout \--type=out,outweb --dport=443 --bytes-per-packet=0-250 --duration=0 | rwstats --value=bytes \--fields=stime,sip,dip,sport,dport,packets,duration --count=5 --ipv6-policy=ignore --no-percent --bin=3600 INPUT: 6 Records for 6 Bins and 744 Total Bytes OUTPUT: Top 5 Bins by Bytes sTime| sIP| dIP|sPort|dPort| packets|durat| Bytes| 2022/02/18T18:00:00| 172.28.20.6| 23.58.146.215|49308| 443| 1| 0| 124| 2022/02/17T17:00:00| 172.28.20.6| 23.58.146.216|56289| 443| 1| 0| 124| 2022/03/24T15:00:00| 172.28.30.4| 184.51.157.69|58644| 443| 1| 0| 124| 2022/03/04T17:00:00| 172.28.30.3| 23.58.146.216|56311| 443| 1| 0| 124| 2022/03/06T21:00:00| 172.28.30.3| 23.58.146.216|65523| 443| 1| 0| 124| |
Add the type column to validate the direction of the traffic. Still using rwfilter and rwstats.
1 2 3 4 5 6 7 8 9 10 | sans@sec503:~$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 --protocol=0- --pass=stdout --type=out,outweb \--dport=443 --bytes-per-packet=0-250 --duration=0 | rwstats --value=bytes --fields=stime,sip,dip,sport,dport,packets,duration,type \--count=5 --ipv6-policy=ignore --no-percent --bin=3600 INPUT: 6 Records for 6 Bins and 744 Total Bytes OUTPUT: Top 5 Bins by Bytes sTime| sIP| dIP|sPort|dPort| packets|durat| type| Bytes| 2022/03/04T17:00:00| 172.28.30.3| 23.58.146.216|56311| 443| 1| 0| out| 124| 2022/03/24T15:00:00| 172.28.30.4| 184.51.157.69|58644| 443| 1| 0| out| 124| 2022/02/17T17:00:00| 172.28.20.6| 23.58.146.216|56289| 443| 1| 0| out| 124| 2022/02/18T18:00:00| 172.28.20.6| 23.58.146.215|49308| 443| 1| 0| out| 124| 2022/03/06T21:00:00| 172.28.30.3| 23.58.146.216|65523| 443| 1| 0| out| 124| |
Do we have similar traffic on the inside? Removing the type from rwfilter.
1 2 3 4 5 6 7 8 9 | sans@sec503:~$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 --protocol=0- --pass=stdout --dport=443 \--bytes-per-packet=0-250 --duration=0 | rwstats --value=bytes --fields=stime,sip,dip,sport,dport,packets,duration,type \--count=5 --ipv6-policy=ignore --no-percent --bin=3600 INPUT: 147455 Records for 147297 Bins and 92351786 Total Bytes OUTPUT: Top 5 Bins by Bytes sTime| sIP| dIP|sPort|dPort| packets|durat| type| Bytes| 2022/03/17T21:00:00| 10.200.223.2| 172.28.3.173|34796| 443| 32| 0| inweb| 1920| 2022/03/17T21:00:00| 10.200.223.2| 172.28.2.183|54576| 443| 32| 0| inweb| 1920| 2022/03/17T21:00:00| 10.200.223.2| 172.28.14.48|33192| 443| 32| 0| inweb| 1920| 2022/03/17T21:00:00| 10.200.223.2| 172.28.12.198|58278| 443| 32| 0| inweb| 1920| 2022/03/17T21:00:00| 10.200.223.2| 172.28.14.69|48240| 443| 32| 0| inweb| 1920| |
Taking a different view. Looking for smaller outbound transfers. Note the --type=out,outweb. Maybe beaconing? We also talk about detecting beaconing in SANS SEC595: Applied Data Science and AI/Machine Learning for Cybersecurity Professionals using Fast Fourier Transform. The bytes below are all consistent for the 3 unique hosts.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=0- --pass=stdout --type=out,outweb --dport=443 --bytes-per-packet=0-250 | \rwstats --value=bytes --fields=sip --count=5 --ipv6-policy=ignore INPUT: 6 Records for 3 Bins and 744 Total Bytes OUTPUT: Top 5 Bins by Bytes sIP| Bytes| %Bytes| cumul_%| 172.28.20.6| 248| 33.333333| 33.333333| 172.28.30.4| 248| 33.333333| 66.666667| 172.28.30.3| 248| 33.333333|100.000000| |
Get some additional protocol statistics via rwfilter and rwstats. Note the --print-statistics for rwfilter.
1 2 3 4 5 6 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=0- --pass=stdout --type=out,outweb --dport=443 --bytes-per-packet=0-250 \--print-statistics | rwstats --value=bytes --fields=sip --count=5 --ipv6-policy=ignore \--detail-proto-stat s=6 | grep "min" Files 1235. Read 1894243. Pass 6. Fail 1894237. *BYTES min 124; max 124 *PACKETS min 1; max 1 *BYTES/PACKET min 124; max 124 |
Revisiting the source IPs with low byte count. What destination are they communicating with? Adding the destination field to rwstats.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=0- --pass=stdout --type=out,outweb --dport=443 --bytes-per-packet=0-250 | \rwstats --value=bytes --fields=sip,dip,sport,dport,packets --count=5 --ipv6-policy=ignore \--no-percent INPUT: 6 Records for 6 Bins and 744 Total Bytes OUTPUT: Top 5 Bins by Bytes sIP| dIP|sPort|dPort| packets| Bytes| 172.28.20.6| 23.58.146.216|56289| 443| 1| 124| 172.28.30.3| 23.58.146.216|56311| 443| 1| 124| 172.28.30.4| 184.51.157.69|58644| 443| 1| 124| 172.28.20.6| 23.58.146.215|49308| 443| 1| 124| 172.28.30.4| 23.58.146.215|57496| 443| 1| 124| |
Obviously something is wrong above. There is just too much commonality there. Let's see, 20 (IP header length) + (assume) 20 (TCP header) = 84 bytes. Each of these packets have ~84 bytes of IP TCP data. Looks at the IPs also to find the commonality.
Resolve those IP addresses of the hosts above, using rwresolve.
Focus on one particular address using the --any-address flag with rwfilter. Pipe the output to rwstats.
Resolve those IP addresses of the hosts above, using rwresolve.
1 2 3 4 5 6 7 8 9 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 --protocol=0- --pass=stdout --type=out,outweb \--dport=443 --bytes-per-packet=0-250 | rwstats --value=bytes --fields=sip,dip,sport,dport,packets --count=5 --ipv6-policy=ignore \--no-percent | rwresolve INPUT: 6 Records for 6 Bins and 744 Total Bytes OUTPUT: Top 5 Bins by Bytes sIP| dIP|sPort|dPort| packets| Bytes| 172.28.20.6|a23-58-146-216.deploy.static.akamaitechnologies.com|56289| 443| 1| 124| 172.28.30.3|a23-58-146-216.deploy.static.akamaitechnologies.com|56311| 443| 1| 124| 172.28.30.4|a184-51-157-69.deploy.static.akamaitechnologies.com|58644| 443| 1| 124| 172.28.20.6|a23-58-146-215.deploy.static.akamaitechnologies.com|49308| 443| 1| 124| 172.28.30.4|a23-58-146-215.deploy.static.akamaitechnologies.com|57496| 443| 1| 124| |
Focus on one particular address using the --any-address flag with rwfilter. Pipe the output to rwstats.
1 2 3 4 5 6 7 8 | sans@sec503:~$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 --protocol=0- --pass=stdout --dport=443 \--bytes-per-packet=0-250 --duration=0- --any-address=23.58.146.216 | rwstats --value=bytes \--fields=stime,sip,dip,sport,dport,packets,duration,type,proto --count=5 --ipv6-policy=ignore --no-percent \--bin=3600 INPUT: 3 Records for 3 Bins and 372 Total Bytes OUTPUT: Top 5 Bins by Bytes sTime| sIP| dIP|sPort|dPort| packets|durat| type|pro| Bytes| 2022/03/06T21:00:00| 172.28.30.3| 23.58.146.216|65523| 443| 1| 0| out| 17| 124| 2022/03/04T17:00:00| 172.28.30.3| 23.58.146.216|56311| 443| 1| 0| out| 17| 124| 2022/02/17T17:00:00| 172.28.20.6| 23.58.146.216|56289| 443| 1| 0| out| 17| 124| |
Above is interesting, as the traffic is all on UDP 443 rather than TCP.. QUIC?
Keeping it simple by finding the first 10 records that match a particular query using rwfilter and rwuniq.
1 2 3 4 5 6 7 8 9 10 11 12 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 --protocol=6 --pass-destination=stdout \--max-pass=10 | rwuniq --fields sip,sport,dip,dport sIP|sPort| dIP|dPort| Records| 52.109.88.36| 443| 172.28.10.89|56674| 1| 10.200.223.4|50494| 172.28.10.5| 22| 1| 142.250.72.10| 443| 172.28.20.5|53715| 1| 172.28.10.5| 22| 10.200.223.4|50494| 1| 52.109.88.36| 443| 172.28.10.89|56673| 1| 10.200.223.4|50673| 172.28.1.1| 22| 1| 142.250.72.35| 443| 172.28.30.5|53821| 1| 20.50.73.10| 443| 172.28.10.89|56669| 1| 20.189.173.13| 443| 172.28.10.80|64499| 1| 142.250.72.35| 443| 172.28.20.5|53714| 1| |
Find the first 10 records that fails (think grep --invert-match or grep -v) the query, using rwfilter and rwuniq. Notice rather than --pass-destination it is now --fail-destination..
1 2 3 4 5 6 7 8 9 10 11 12 | sans@sec503:~$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 \--protocol=6 --fail-destination=stdout --max-fail=10 | rwuniq \--fields sip,sport,dip,dport --ipv6-policy=ignore sIP|sPort| dIP|dPort| Records| 8.8.8.8| 53| 172.28.10.137|56104| 1| 8.8.8.8| 53| 172.28.10.137|54512| 1| 8.8.8.8| 53| 172.28.10.137|55382| 1| 8.8.8.8| 53| 172.28.10.137|55171| 1| 8.8.8.8| 53| 172.28.10.137|56350| 1| 8.8.8.8| 53| 172.28.10.137|55339| 1| 8.8.8.8| 53| 172.28.10.137|54864| 1| 8.8.8.8| 53| 172.28.10.137|56290| 1| 8.8.8.8| 53| 172.28.10.137|55359| 1| 8.8.8.8| 53| 172.28.10.137|56213| 1| |
Combining the rwfilter --pass-destination and --fail-destination as well as writing --pass-destination and --fail-destination to file.
1 2 3 4 5 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/04/01 --protocol=6 --pass-destination=stdout \--max-pass=10 --fail-destination=6-fail.rw --max-fail=10 | rwfilter stdin --aport=443 --fail-destination=stdout \--pass-destination=pass-443 | rwuniq --fields sip,sport,dip,dport sIP|sPort| dIP|dPort| Records| 10.200.223.4|50494| 172.28.10.5| 22| 1| 172.28.10.5| 22| 10.200.223.4|50494| 1| 10.200.223.4|50673| 172.28.1.1| 22| 1| |
Find 5 unique sessions that were initiated by the client. That is the device sending the SYN packet. Note the --flags-initial with rwfilter. S/SA means we are looking to see if the SYN flag is set while testing the SYN and ACK flags.
1 2 3 4 5 6 7 | sans@sec503:~$ rwfilter --start-date=2019/04/01T0 --end-date=2022/05/01 --protocol=6 --pass-destination=stdout --aport=443 --flags-initial=S/SA \--max-pass=5 | rwuniq --fields stime,sIP,dIP,flags,initialflags,duration --values=records sTime| sIP| dIP| flags|initialF|durat| Records| 2019/05/02T16:30:15| 172.16.10.13| 13.107.5.88| SRPA | S | 0| 1| 2019/05/02T16:29:56| 172.16.10.13| 65.55.44.108|FSRPA | S | 132| 1| 2019/05/02T16:31:04| 172.16.10.13| 65.55.44.109| SRPA | S | 4| 1| 2019/05/02T16:30:45| 172.16.10.13| 157.55.135.128|FS PA | S | 19| 1| 2019/05/02T16:30:15| 172.16.10.13| 13.107.3.128| SRPA | S | 0| 1| |
Similarly find the devices acting as a server. Meaning, the device responded to a SYN with a SYN/ACK. Notice the rwfilter --flags-initial=SA/SA now shows test SYN/ACK to see if both SYN and ACK are set.
1 2 3 4 5 6 7 | sans@sec503:~$ rwfilter --start-date=2019/04/01T0 --end-date=2022/05/01 --protocol=6 --pass-destination=stdout --aport=443 --flags-initial=SA/SA \--max-pass=5 | rwuniq --fields stime,sIP,dIP,flags,initialflags,duration --values=records sTime| sIP| dIP| flags|initialF|durat| Records| 2019/05/02T16:30:15| 13.107.3.128| 172.16.10.13| S A | S A | 0| 1| 2019/05/02T16:30:45| 157.55.135.128| 172.16.10.13|FSRPA | S A | 19| 1| 2019/05/02T16:31:04| 65.55.44.109| 172.16.10.13| S PA | S A | 4| 1| 2019/05/02T16:29:56| 65.55.44.108| 172.16.10.13| S PA | S A | 132| 1| 2019/05/02T16:30:15| 13.107.5.88| 172.16.10.13| S A | S A | 0| 1| |
Find 5 unique sessions that seems to have been fully completed. Notice the rwfilter --flags-
all=SAFP/FSRPA tests the FIN, SYN, RST, PUSH and ACK flags to see if SYN, ACK, FIN and PUSH are set.
1 2 3 4 5 6 7 | sans@sec503:~$ rwfilter --start-date=2019/04/01T0 --end-date=2022/05/01 --protocol=6 --pass-destination=stdout --dport=22,80,443,4444 \--flags-all=SAFP/FSRPA --max-pass=5 | rwuniq --fields stime,sIP,dIP,dport,flags,type --values=records sTime| sIP| dIP|dPort| flags| type| Records| 2019/05/02T16:38:30| 172.16.10.13| 192.96.162.110| 80|FS PA | outweb| 1| 2019/05/02T16:38:32| 172.16.10.13| 192.96.162.33| 80|FS PA | outweb| 2| 2019/05/02T16:38:32| 172.16.10.13| 23.33.106.133| 80|FS PA | outweb| 1| 2019/05/02T16:30:45| 172.16.10.13| 157.55.135.128| 443|FS PA | outweb| 1| |
Look at the last 5 sessions again, this time add duration field to rwuniq. Added flows and bytes to the --values.
1 2 3 4 5 6 | sans@sec503:~$ rwfilter --start-date=2019/04/01T0 --end-date=2022/05/01 --protocol=6 --pass-destination=stdout --dport=22,80,443,4444 --flags-all=SAFP/FSRPA --max-pass=5 | \rwuniq --fields stime,sIP,dIP,dport,flags,type,duration --values=flows,bytes,packets sTime| sIP| dIP|dPort| flags| type|durat| Records| Bytes| Packets| 2019/05/02T16:38:32| 172.16.10.13| 192.96.162.33| 80|FS PA | outweb| 78| 2| 977| 14| 2019/05/02T16:38:32| 172.16.10.13| 23.33.106.133| 80|FS PA | outweb| 78| 1| 505| 7| 2019/05/02T16:30:45| 172.16.10.13| 157.55.135.128| 443|FS PA | outweb| 19| 1| 6297| 16| 2019/05/02T16:38:30| 172.16.10.13| 192.96.162.110| 80|FS PA | outweb| 108| 1| 575| 7| |
Leveraging rwbag. Preparing the data via rwfilter, then redirect it to rwbag.
1 2 | sans@sec503:~/nik$ rwfilter --start-date=2019/04/01T0 --end-date=2022/05/01 --protocol=6 \--pass-destination=stdout --dport=22,80,443,4444 --max-pass=5 | \ rwbag --bag-file=sipv4,sum-bytes,/tmp/test.bag
|
Viewing the contents in the bag created via rwbag.
1 2 3 | sans@sec503:~/nik$ rwbagcat test.bag 172.16.10.13| 8106| 172.16.40.12| 80| |
Leveraging rwscan to identify potential scanning IPs.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/05/01 --protocol=6 --pass-destination=stdout | \rwsort --fields sip,protocol,dip | rwscan --scan-model=2 sip| proto| stime| etime| flows| packets| bytes| 10.200.223.2| 6| 2022-03-07 11:59:46| 2022-04-30 23:49:43| 1061401| 57208212|3107887558| 10.200.223.3| 6| 2022-02-09 12:51:11| 2022-04-30 16:36:16| 413308| 10588224| 583916721| 10.200.223.4| 6| 2022-02-08 14:26:13| 2022-04-30 00:14:41| 3749647| 65736155|4056928153| 10.200.223.5| 6| 2022-02-11 20:55:22| 2022-04-30 15:11:11| 2776970| 7508499| 406143689| 10.200.223.7| 6| 2022-02-08 15:50:32| 2022-03-24 22:18:35| 149108| 4259383| 241338192| 10.200.223.8| 6| 2022-02-11 21:06:29| 2022-04-30 03:01:23| 232009| 3673446| 177412489| 172.28.20.3| 6| 2022-02-18 16:23:53| 2022-04-30 23:15:50| 299| 1430| 181320| 172.28.20.4| 6| 2022-02-24 18:32:19| 2022-04-29 23:05:43| 224| 1207| 170436| 172.28.20.6| 6| 2022-02-16 16:05:19| 2022-04-21 01:39:55| 8202| 24551| 1342732| 172.28.30.2| 6| 2022-02-16 16:26:12| 2022-04-26 16:05:56| 544| 2724| 351448| 172.28.30.3| 6| 2022-02-10 17:42:20| 2022-03-19 18:53:36| 168| 497| 25844| 172.28.30.4| 6| 2022-02-08 20:47:57| 2022-04-28 16:56:30| 525| 2626| 350572| 172.28.30.5| 6| 2022-02-10 18:05:52| 2022-04-30 15:16:06| 446| 2428| 334660| 172.28.50.2| 6| 2022-02-10 18:41:21| 2022-04-29 15:55:32| 580| 2883| 367628| |
Narrowing above down to only the IPs and storing them in the bag. First get the data via rwfilter, rwsort and rwscan. The pipe this data into cut.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/05/01 \--protocol=6 --pass-destination=stdout | rwsort --fields sip,protocol,dip | \rwscan --scan-model=2 --no-title --output-path=stdout | cut --fields=1,5 --delimiter='|' 10.200.223.2| 1061401 10.200.223.3| 413308 10.200.223.4| 3749647 10.200.223.5| 2776970 10.200.223.7| 149108 10.200.223.8| 232009 172.28.20.3| 299 172.28.20.4| 224 172.28.20.6| 8202 172.28.30.2| 544 172.28.30.3| 168 172.28.30.4| 525 172.28.30.5| 446 172.28.50.2| 580 |
Create the bag consisting of the IPs shows above. Reading directly from rwfilter. Pipe it into rwsort, then rwscan then rwbagbuild. After building the bag, use rwbagcat to view the records.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/05/01 \--protocol=6 --pass-destination=stdout | rwsort --fields sip,protocol,dip | \rwscan --scan-model=2 --no-title --output-path=stdout | \cut --fields=1,5 --delimiter='|' | rwbagbuild --bag-input=stdin --key-type=sipv4 \--counter-type=records | rwbagcat 10.200.223.2| 1061401| 10.200.223.3| 413308| 10.200.223.4| 3749647| 10.200.223.5| 2776970| 10.200.223.7| 149108| 10.200.223.8| 232009| 172.28.20.3| 299| 172.28.20.4| 224| 172.28.20.6| 8202| 172.28.30.2| 544| 172.28.30.3| 168| 172.28.30.4| 525| 172.28.30.5| 446| 172.28.50.2| 580| |
Alternatively, group the bags by IPs. Notice --bin-ips to rwbagcat.
1 2 | sans@sec503:~/nik$ rwfilter --start-date=2022/01/01T0 --end-date=2022/05/01 \--protocol=6 --pass-destination=stdout | rwsort --fields sip,protocol,dip | \rwscan --scan-model=2 --no-title --output-path=stdout | cut --fields=1 \--delimiter='|' | sort --unique | rwbagbuild --bag-input=stdin --key-type=sipv4 \--counter-type=records | rwbagcat --bin-ips 1| 14| |
Introducing rwnetmask. Maybe you have a network where communication looks like this.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwfilter --start-date=2022/04/01T0 --end-date=2022/05/01 --protocol=6 \--pass-destination=stdout --max-pass=5 | rwuniq --fields sip,dip sIP| dIP| Records| 52.167.17.97| 172.28.20.4| 1| 20.72.205.209| 172.28.30.3| 1| 52.109.88.35| 172.28.30.4| 1| 52.167.17.97| 172.28.30.4| 1| 20.72.205.209| 172.28.10.10| 1| |
Rather than getting the full IP, you decide you would like to have a 24 bit mask of the IP address. Using the rwnetmask, we see we were able to change the IP address to /24 networks.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwfilter --start-date=2022/04/01T0 --end-date=2022/05/01 \--protocol=6 --pass-destination=stdout --max-pass=5 | rwnetmask \--4sip-prefix-length=24 --4dip-prefix-length=24 | rwcut --fields sip,dip sIP| dIP| 52.167.17.0| 172.28.20.0| 52.167.17.0| 172.28.30.0| 20.72.205.0| 172.28.10.0| 52.109.88.0| 172.28.30.0| 20.72.205.0| 172.28.30.0| |
Find the well-known TCP ports on the network which seems to be the busiest, via rwfilter and rwuniq.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwfilter --protocol=6 --dport=0-1023 \--start-date=2022/01/01 --end-date=2022/05/01 --pass=stdout --max-pass=1000 | \rwuniq --fields dport --values flow,bytes,packets --sort dPort| Records| Bytes| Packets| 21| 1| 156| 3| 22| 39| 248109713| 4847875| 25| 17| 1047287| 809| 80| 4| 6315| 104| 443| 939| 912196| 20670| |
More detail to understand the type of data and the sensor involved.
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwfilter --protocol=6 --dport=0-1023 --start-date=2022/01/01 \--end-date=2022/05/01 --pass=stdout --max-pass=1000 | rwuniq \--fields dport,type,sensor --values flow,bytes,packets --sort dPort| type| sensor| Records| Bytes| Packets| 21| in| Internal| 1| 156| 3| 22| in| Internal| 33| 247919188| 4845368| 22| out| Internal| 6| 190525| 2507| 25| in| Internal| 17| 1047287| 809| 80| inweb| Internal| 4| 6315| 104| 443| inweb| Internal| 939| 912196| 20670| |
I find it quite interesting, that the majority of this traffic is on port 22, typically associated with SSH.
So far I've been specific about fields such as --fields=sip. How about grabbing all the fields with rwcut.
1 2 3 4 | sans@sec503:~/nik$ rwcut --all-fields attack-trace.rw --num-recs=2 sIP| dIP|sPort|dPort|pro| packets| bytes| flags| sTime| duration| eTime| sensor| in| out| nhIP|initialF|sessionF|attribut|appli|cla| type| sTime+msec| eTime+msec| dur+msec|iTy|iCo| 98.114.205.102| 192.150.11.111| 1821| 445| 6| 4| 168|FS A |2019/04/20T03:28:28.374| 0.354|2019/04/20T03:28:28.728| Internal| 0| 0| 0.0.0.0| S |F A | | 0|all| in|2019/04/20T03:28:28.374|2019/04/20T03:28:28.728| 0.354| | | 192.150.11.111| 98.114.205.102| 445| 1821| 6| 3| 128|FS A |2019/04/20T03:28:28.375| 0.353|2019/04/20T03:28:28.728| Internal| 0| 0| 0.0.0.0| S A |F A | | 0|all| in|2019/04/20T03:28:28.375|2019/04/20T03:28:28.728| 0.353| | | |
Revisit the timestamps via rwcut.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwcut --fields stime attack-trace.rw --num-recs=5 sTime| 2019/04/20T03:28:28.374| 2019/04/20T03:28:28.375| 2019/04/20T03:28:28.509| 2019/04/20T03:28:28.509| 2019/04/20T03:28:30.466| |
Use the legacy timestamp instead with rwcut, rather than the default.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwcut --fields stime attack-trace.rw --legacy-timestamp --num-recs=5 sTime| 04/20/2019 03:28:28| 04/20/2019 03:28:28| 04/20/2019 03:28:28| 04/20/2019 03:28:28| 04/20/2019 03:28:30| |
Or maybe get rwcut to produce the time in epoch time.
1 2 3 4 5 6 | sans@sec503:~$ rwfilter --start=2022/01/05T0 --end=2022/07/01 --protocol=0- \--pass=stdout --type=all --bytes=0-30 | rwuniq --bin-time=86400 --fields stime,type \--values=records --sort-output --timestamp-format=epoch | head --lines=5 sTime| type| Records| 1644624000| in| 4136| 1644624000| out| 52| 1644710400| in| 2469| 1644796800| in| 4307 |
Revisit rwcut formatting.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwcut --fields stime,duration,sip,dport attack-trace.rw --num-recs=5 sTime| duration| sIP|dPort| 2019/04/20T03:28:28.374| 0.354| 98.114.205.102| 445| 2019/04/20T03:28:28.375| 0.353| 192.150.11.111| 1821| 2019/04/20T03:28:28.509| 4.938| 98.114.205.102| 445| 2019/04/20T03:28:28.509| 4.938| 192.150.11.111| 1828| 2019/04/20T03:28:30.466| 3.100| 98.114.205.102| 1957| |
Remove the columns, make it pipe delimited.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwcut --fields stime,duration,sip,dport attack-trace.rw --num-recs=5 --no-columns sTime|duration|sIP|dPort| 2019/04/20T03:28:28.374|0.354|98.114.205.102|445| 2019/04/20T03:28:28.375|0.353|192.150.11.111|1821| 2019/04/20T03:28:28.509|4.938|98.114.205.102|445| 2019/04/20T03:28:28.509|4.938|192.150.11.111|1828| 2019/04/20T03:28:30.466|3.100|98.114.205.102|1957| |
Revisit creating a file from rwfilter. This time, set the --compression-method to none.
1 | sans@sec503:~/nik$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --protocol=0- \--type=all --max-pass=100 --compression-method=none --pass=uncompressed.rw
|
Leveraging rwfilter compression when creating files. Set the --compression-method to best.
1 2 | sans@sec503:~/nik$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --protocol=0- \--type=all --max-pass=100 --compression-method=best --pass=compressed.rw
|
Review the files to created by rwfilter, confirm the compression
1 2 3 | sans@sec503:~/nik$ ls *compressed* -l -rw-rw-r-- 1 sans sans 1177 Jun 13 01:51 compressed.rw -rw-rw-r-- 1 sans sans 8976 Jun 13 01:51 uncompressed.rw |
Find echo replies by leveraging rwfilter --icmp-type and --icmp-code parameters. Specifically look at ICMP type 0 and code 0.
1 2 3 4 5 6 | sans@sec503:~$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --icmp-type=0 --icmp-code=0 --type=all --max-pass=100000 \--pass-destination=stdout | rwcut --num-recs=4 --fields sip,dip,proto,packets,bytes --icmp-type-and-code sIP| dIP|pro| packets| bytes|sPort|dPort| fe80::250:56ff:fead:e8b6| ff02::2| 58| 1| 56| 0| 0| fe80::250:56ff:fead:445| ff02::2| 58| 1| 56| 0| 0| 66.35.60.78| 172.28.30.5| 1| 10| 920| 0| 0| 66.35.60.78| 172.28.30.2| 1| 15| 1380| 0| 0| |
Note above in the --fields section, I do not have sport or dport. However, we see these values for the ICMP type and codes. Do note, ICMP does not use the concept of ports. Trick question I ask at interviews, "What protocol and port does Ping use TCP or UDP?" :-)
Are there any echo requests to match those replies?
1 2 | sans@sec503:~$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --icmp-type=8 --icmp-code=0 --type=all --max-pass=100000 \--pass-destination=stdout | rwcut --num-recs=4 --fields sip,dip,proto,packets,bytes --icmp-type-and-code sIP| dIP|pro| packets| bytes|sPort|dPort| |
That's interesting! No records returned for echo requests. How can that be?! Very interesting! Did I miss something? Leave me a note in the comment section.
Looking at the rwfilter --print-volume-statistics to see if there are any clues as to why no packets were returned
1 2 3 4 5 | sans@sec503:~$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --icmp-type=8 --icmp-code=0 --type=all --max-pass=100000 --print-volume-statistics | Recs| Packets| Bytes| Files| Total| 25713809| 688813257| 402383908447| 13064| Pass| 0| 0| 0| | Fail| 25713809| 688813257| 402383908447| | |
Going back further in time, just to see what the ICMP echo request output looks like.
1 2 3 4 5 6 7 | sans@sec503:~$ rwfilter --start=2012/01/05 --end=2022/07/01T23 --icmp-type=8 --icmp-code=0 --type=all --max-pass=100000 --pass-destination=stdout | rwcut --num-recs=4 --fields sip,dip,proto,packets,bytes --icmp-type-and-code sIP| dIP|pro| packets| bytes|sPort|dPort| 192.168.2.166| 192.168.2.1| 1| 31| 2604| 8| 0| 192.168.2.166| 192.168.2.1| 1| 9| 756| 8| 0| 192.168.2.166| 192.168.2.1| 1| 10| 840| 8| 0| 192.168.2.166| 192.168.2.1| 1| 30| 2520| 8| 0| |
We now see four records of ICMP Type 8 and Code 0.
Get the filenames via rwfilter --print-file-names.
1 2 3 4 5 6 | sans@sec503:~$ rwfilter --start=2022/01/05 --end=2022/07/01T23 \--icmp-type=8 --icmp-code=0 --type=all --max-pass=100000 --print-volume-statistics \--print-filenames | more /data/in/2022/02/08/in-internal_20220208.14 /data/out/2022/02/08/out-internal_20220208.14 /data/inweb/2022/02/08/iw-internal_20220208.14 /data/ext2ext/2022/02/08/ext2ext-internal_20220208.14 ... |
Find the missing files via rwfilter --print-missing-files.
1 2 3 4 5 6 7 8 9 | sans@sec503:~$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --icmp-type=8 \--icmp-code=0 --type=all --max-pass=100000 --print-volume-statistics \--print-missing-files | more Missing /data/out/2022/01/20/out-Internal_20220120.13 Missing /data/out/2022/01/20/out-Perimeter_20220120.13 Missing /data/out/2022/01/20/out-ERS_20220120.13 Missing /data/out/2022/01/20/out-internal_20220120.13 Missing /data/inweb/2022/01/20/iw-Internal_20220120.13 Missing /data/inweb/2022/01/20/iw-Perimeter_20220120.13 ... |
Leveraging rwfglob.
1 2 | sans@sec503:~/nik$ rwfglob --start-date=2012/01/01 --end-date=2022/07/01 \--no-file-names globbed 24574 files; 0 on tape |
Revisiting rwcount bin sizes from the time perspective. Below shows the time is at 30 minutes interval. This seems to be the default --bin-size.
1 2 3 4 5 6 7 8 9 10 11 | sans@sec503:~/nik$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --dport=443 \--type=all --max-pass=5 --pass-destination=stdout | rwcount Date| Records| Bytes| Packets| 2022/02/08T15:28:30| 1.00| 6390.00| 5.00| 2022/02/08T15:29:00| 0.00| 0.00| 0.00| 2022/02/08T15:29:30| 0.00| 0.00| 0.00| 2022/02/08T15:30:00| 0.00| 0.00| 0.00| 2022/02/08T15:30:30| 0.00| 0.00| 0.00| 2022/02/08T15:31:00| 0.00| 0.00| 0.00| 2022/02/08T15:31:30| 1.00| 6390.00| 5.00| 2022/02/08T15:32:00| 0.00| 0.00| 0.00| 2022/02/08T15:32:30| 3.00| 19170.00| 15.00| |
Adjusting the rwcount --bin-size by using terminal to do arithmetic. Changing the --bin-size to two minutes interval.
1 2 3 4 5 | sans@sec503:~/nik$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --dport=443 \--type=all --max-pass=5 --pass-destination=stdout | rwcount --bin-size=$((2*60)) Date| Records| Bytes| Packets| 2022/02/08T15:28:00| 1.00| 6390.00| 5.00| 2022/02/08T15:30:00| 1.00| 6390.00| 5.00| 2022/02/08T15:32:00| 3.00| 19170.00| 15.00| |
Changing the --bin-size to 5 minutes interval.
1 2 3 4 | sans@sec503:~/nik$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --dport=443 \--type=all --max-pass=5 --pass-destination=stdout | rwcount --bin-size=$((5*60)) Date| Records| Bytes| Packets| 2022/02/08T15:25:00| 1.00| 6390.00| 5.00| 2022/02/08T15:30:00| 4.00| 25560.00| 20.00| |
Using the time as a range via rwfilter --stime.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | sans@sec503:~$ rwfilter --start-date=2022/02/09T16 --stime=2022/02/09T16:00:00-2022/02/09T16:02:00 \--type=all --pass-destination=stdout --protocol=0- | rwcut --fields=stime,sip,dip sTime| sIP| dIP| 2022/02/09T16:00:19.850| 8.8.8.8| 172.28.10.137| 2022/02/09T16:00:35.377| 17.253.26.125| 172.28.10.137| 2022/02/09T16:01:29.106| 8.8.8.8| 172.28.10.137| 2022/02/09T16:01:29.817| 8.8.8.8| 172.28.10.137| 2022/02/09T16:00:19.850| 172.28.10.137| 8.8.8.8| 2022/02/09T16:00:35.377| 172.28.10.137| 17.253.26.125| 2022/02/09T16:01:29.106| 172.28.10.137| 8.8.8.8| 2022/02/09T16:01:29.817| 172.28.10.137| 8.8.8.8| 2022/02/09T16:01:29.214| 52.109.20.75| 172.28.30.4| 2022/02/09T16:01:29.892| 52.109.8.20| 172.28.30.4| 2022/02/09T16:01:29.892| 52.109.8.20| 172.28.30.4| 2022/02/09T16:00:19.852| 72.21.81.240| 172.28.10.25| |
Find completed flows, by looking at the SYN, ACK, FIN and RST flags. Note the --flags-all=SAF/SAF,SAR/SAR parameters for rwfilter.
1 2 3 4 5 6 7 | sans@sec503:~$ rwfilter --start-date=2022/02/09T16 --type=all --pass-destination=stdout --protocol=6 --flags-all=SAF/SAF,SAR/SAR | \rwcut --fields=stime,sip,dip,flags --num-recs=5 sTime| sIP| dIP| flags| 2022/02/09T16:00:19.852| 72.21.81.240| 172.28.10.25|FS PA | 2022/02/09T16:05:25.018| 52.167.17.97| 172.28.30.5|FS PA | 2022/02/09T16:02:31.816| 52.167.249.196| 172.28.10.89|FS PA E | 2022/02/09T16:02:39.571| 142.250.72.10| 172.28.30.5|FS PA | 2022/02/09T16:03:15.843| 142.250.72.35| 172.28.50.2|FS PA | |
Print rwcut TCP flags as integers via --integer-tcp-flags.
1 2 3 4 5 6 7 | sans@sec503:~$ rwfilter --start-date=2022/02/09T16 --type=all --pass-destination=stdout --protocol=6 --flags-all=SAF/SAF,SAR/SAR | \rwcut --fields=stime,sip,dip,flags --num-recs=5 --integer-tcp-flags sTime| sIP| dIP|fla| 2022/02/09T16:00:19.852| 72.21.81.240| 172.28.10.25| 27| 2022/02/09T16:05:25.018| 52.167.17.97| 172.28.30.5| 27| 2022/02/09T16:02:31.816| 52.167.249.196| 172.28.10.89| 91| 2022/02/09T16:02:39.571| 142.250.72.10| 172.28.30.5| 27| 2022/02/09T16:03:15.843| 142.250.72.35| 172.28.50.2| 27| |
Change rwcut format of the IP address to decimal via --ip-format=decimal.
1 2 3 4 5 6 7 | sans@sec503:~$ rwfilter --start-date=2022/02/09T16 --type=all --pass-destination=stdout \--protocol=6 | rwcut --fields=sip,dip --num-recs=5 --ip-format=decimal sIP| dIP| 879563851| 2887523844| 879560724| 2887523844| 879560724| 2887523844| 879563851| 2887523844| 879870852| 2887523844| |
Convert the decimal values by to dotted notation via num2dot --ip-field
1 2 3 4 5 6 7 | sans@sec503:~$ rwfilter --start-date=2022/02/09T16 \--type=all --pass-destination=stdout --protocol=6 | \rwcut --fields=sip,dip --num-recs=5 --ip-format=decimal | num2dot --ip-field=1,2 sIP| dIP| 52.109.20.75| 172.28.30.4| 52.109.8.20| 172.28.30.4| 52.109.8.20| 172.28.30.4| 52.109.20.75| 172.28.30.4| 52.113.195.132| 172.28.30.4| |
Show the IP addresses as hexadecimal via rwcut --ip-format=hexadecimal
1 2 3 4 5 6 7 | sans@sec503:~$ rwfilter --start-date=2022/02/09T16 --type=all \--pass-destination=stdout --protocol=6 | rwcut --fields=sip,dip --num-recs=5 \--ip-format=hexadecimal sIP| dIP| 346d144b| ac1c1e04| 346d0814| ac1c1e04| 346d0814| ac1c1e04| 346d144b| ac1c1e04| 3471c384| ac1c1e04| |
Leveraging rwaddrcount to get information about the records in the file.
1 2 3 4 | sans@sec503:~/nik$ rwaddrcount attack-trace.rw --print-recs sIP| Bytes| Packets| Records| Start_Time| End_Time| 192.150.11.111| 7297| 153| 6| 2019/04/20T03:28:28| 2019/04/20T03:28:44| 98.114.205.102| 171264| 195| 6| 2019/04/20T03:28:28| 2019/04/20T03:28:44| |
Get some additional file statistics via rwaddrcount.
1 2 3 | sans@sec503:~/nik$ rwaddrcount attack-trace.rw --print-stat | sIP_Uniq| Bytes| Packets| Records| Total| 2| 178561| 348| 12| |
What are the 2 actual unique source IP values in that file? Continuing with rwaddrcount.
1 2 3 4 | sans@sec503:~/nik$ rwaddrcount attack-trace.rw --print-ips sIP 192.150.11.111 98.114.205.102 |
Leveraging rwappend, to create a new flow file, consisting of 2 existing flow files.
Use rwfilter to create a file consisting of TCP flows.
1 | sans@sec503:~/nik$ rwfilter --start-date=2022/02/09T16 --max-pass=2 --type=all \--pass-destination=tcp_file.rw --protocol=6
|
Use rwfilter to create a file consisting of UDP flows.
1 | sans@sec503:~/nik$ rwfilter --start-date=2022/02/09T16 --max-pass=2 --type=all \ --pass-destination=udp_file.rw --protocol=17
|
Combine the TCP and UDP flow files created by rwfilter using rwappend.
1 | sans@sec503:~/nik$ rwappend --create tcp_udp.rw tcp_file.rw udp_file.rw |
Use rwcut to see the contents of the rwappend merged files.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwcut tcp_udp.rw --num-recs=5 sIP| dIP|sPort|dPort|pro| packets| bytes| flags| sTime| duration| eTime| sensor| 52.109.20.75| 172.28.30.4| 443|50137| 6| 9| 8048| S PA |2022/02/09T16:01:29.214| 1.191|2022/02/09T16:01:30.405| Internal| 52.109.8.20| 172.28.30.4| 443|50138| 6| 7| 6533| S PA |2022/02/09T16:01:29.892| 0.513|2022/02/09T16:01:30.405| Internal| 8.8.8.8| 172.28.10.137| 53|55874| 17| 1| 289| |2022/02/09T16:00:19.850| 0.002|2022/02/09T16:00:19.852| Internal| 17.253.26.125| 172.28.10.137| 123| 123| 17| 1| 76| |2022/02/09T16:00:35.377| 0.034|2022/02/09T16:00:35.411| Internal| |
Deduplicating two files into one via rwdedupe.
1 | sans@sec503:~/nik$ rwdedupe --buffer-size=88000 8.rw attack-trace.rw \--output=deduped-data.rw
|
Use rwfilter --ip-version to track IPv6 addresses.
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwfilter --start=2022/01/05 --end=2022/07/01T23 --ip-version=6 \--type=all --max-pass=5 --pass-destination=stdout | rwcut --fields=sip,dip,dport sIP| dIP|dPort| fe80::250:56ff:fead:e8b6| ff02::2| 0| fe80::250:56ff:fead:445| ff02::2| 0| fe80::250:56ff:fead:e8b6| ff02::2| 0| fe80::250:56ff:fead:445| ff02::2| 0| fe80::250:56ff:fead:e8b6| ff02::2| 0| |
Leveraging rwpcut to convert .pcap files to ASCII.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | sans@sec503:~/nik$ rwpcut attack-trace.pcap 2>/dev/null | more {'version': False, 'columns': False, 'delimiter': '|', 'epoch_time': False, 'fields': ['time ', 'sip', 'dip', 'sport', 'dport', 'proto', 'payhex'], 'integer_ips': False, 'zero_pad_ips': False, 'files': ['attack-trace.pcap']} reading from file attack-trace.pcap, link-type EN10MB (Ethernet), snapshot length 65535 time|sip|dip|sport|dport|proto|payhex| 2019-04-20 03:28:28.374595|98.114.205.102|192.150.11.111|1821|445|6|450000303b9f40007106d24a 6272cd66c0960b6f071d01bd08cb8066000000007002faf0fa440000020405b401010402| 2019-04-20 03:28:28.375059|192.150.11.111|98.114.205.102|445|1821|6|450000300000400040063eea c0960b6f6272cd6601bd071d5c3ba87408cb8067701216d0d9a40000020405b401010402| 2019-04-20 03:28:28.493653|98.114.205.102|192.150.11.111|1821|445|6|450000283bad40007106d244 6272cd66c0960b6f071d01bd08cb80675c3ba8755010faf022480000000000000000| 2019-04-20 03:28:28.508770|98.114.205.102|192.150.11.111|1821|445|6|450000283bae40007106d243 6272cd66c0960b6f071d01bd08cb80675c3ba8755011faf022470000000000000000| ... |
Ooops, that looks nasty. Making it cleaner by leveraging --fields, --columnar and --delimiter.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | sans@sec503:~/nik$ rwpcut attack-trace.pcap --fields=sip,sport,dip,dport --columnar --delimiter=" | " \--zero-pad-ips 2>/dev/null| more {'version': False, 'columns': True, 'delimiter': ' | ', 'epoch_time': False, 'fields': [' sip', 'sport', 'dip', 'dport'], 'integer_ips': False, 'zero_pad_ips': True, 'files': ['attac k-trace.pcap']} reading from file attack-trace.pcap, link-type EN10MB (Ethernet), snapshot length 65535 sip | sport | dip | dport | 098.114.205.102 | 1821 | 192.150.011.111 | 445 | 192.150.011.111 | 445 | 098.114.205.102 | 1821 | 098.114.205.102 | 1821 | 192.150.011.111 | 445 | 098.114.205.102 | 1821 | 192.150.011.111 | 445 | 098.114.205.102 | 1828 | 192.150.011.111 | 445 | 192.150.011.111 | 445 | 098.114.205.102 | 1828 | 192.150.011.111 | 445 | 098.114.205.102 | 1821 | ... |
Converting a pcap to SiLK flow via rwptoflow. Then redirect the output to rwcut.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwptoflow attack-trace.pcap | rwcut --num-recs=5 --fields=sip,dip,flags sIP| dIP| flags| 98.114.205.102| 192.150.11.111| S | 192.150.11.111| 98.114.205.102| S A | 98.114.205.102| 192.150.11.111| A | 98.114.205.102| 192.150.11.111|F A | 98.114.205.102| 192.150.11.111| S | |
Write the rwptoflow converted flow data to a file. At the same time, for the records that were used to create the flow, create another pcap file. Get the statistics when everything is done. Add a comment also.
1 2 3 4 5 6 7 8 9 10 | sans@sec503:~/nik$ rwptoflow attack-trace.pcap --flow-output rwp_flow_file.rw \--note-add "Converted from attacktrace.pcap" --compression-method=zlib \--packet-pass-output=rwp.pcap --print-statistics --set-sensorid=1 Packet count statistics for attack-trace.pcap 348 read 0 rejected: too short to get information 0 rejected: not IPv4 348 total written 0 total fragmented packets 0 zero-packet of a fragment 0 incomplete (no ports and/or flags) |
Validate the rwp.pcap file.
1 2 | sans@sec503:~/nik$ file rwp.pcap rwp.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Ethernet, capture length 65535) |
Leveraging rwrandomizeip to randomize IPs.
First take 5 IPs from the 8.rw file.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwcut 8.rw --fields=sip,dip --num-recs=5 sIP| dIP| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| |
Now randomize the first 5 records via rwrandomizeip.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ rwrandomizeip 8.rw | rwcut --fields=sip,dip --num-recs=5 sIP| dIP| 10.255.111.99| 10.39.63.221| 10.215.197.155| 10.56.240.34| 10.189.217.143| 10.192.119.61| 10.12.82.4| 10.251.82.128| 10.78.26.161| 10.173.1.103| |
Convert SiLK flow data to IPFIX using rwsilk2ipfix.
1 2 | sans@sec503:~/nik$ rwsilk2ipfix 8.rw --ipfix-output rw-2-2ipfix.dat --print-statistics rwsilk2ipfix: Wrote 100 IPFIX records to 'rw-2-2ipfix.dat' |
View a sample of the rwsilk2ipfix converted data using yafscii.
1 2 3 4 5 6 7 | sans@sec503:~/nik$ yafscii --in=rw-2-2ipfix.dat --out=- | more 2022-02-08 14:26:40.723 - 14:26:40.724 (0.001 sec) udp 8.8.8.8:53 => 172.28.10.137:56213 (1/218 ->) 2022-02-08 14:27:10.329 - 14:27:10.342 (0.013 sec) udp 8.8.8.8:53 => 172.28.10.137:55171 (1/102 ->) 2022-02-08 14:27:43.431 - 14:27:43.433 (0.002 sec) udp 8.8.8.8:53 => 172.28.10.137:54512 (1/213 ->) 2022-02-08 14:28:29.633 - 14:28:29.646 (0.013 sec) udp 8.8.8.8:53 => 172.28.10.137:55359 (1/100 ->) 2022-02-08 14:28:30.328 - 14:28:30.396 (0.068 sec) udp 8.8.8.8:53 => 172.28.10.137:54864 (1/108 ->) ... |
Taking a different view of the IPFIX record information via ipfixDump.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | sans@sec503:~/nik$ ipfixDump --yaf --in=rw-2-2ipfix.dat --out=- | more --- Message Header --- export time: 2023-06-15 16:04:04 observation domain id: 0 message length: 952 sequence number: 0 (0) --- template record --- header: tid: 40404 (0x9dd4) field count: 21 scope: 0 fields: ent: 0 id: 152 type: millisec len: 8 flowStartMilliseconds ent: 0 id: 153 type: millisec len: 8 flowEndMilliseconds ent: 0 id: 2 type: uint64 len: 4 packetDeltaCount ent: 0 id: 1 type: uint64 len: 4 octetDeltaCount ent: 0 id: 10 type: uint32 len: 2 ingressInterface ent: 0 id: 14 type: uint32 len: 2 egressInterface ent: 6871 id: 33 type: uint16 len: 2 silkAppLabel ent: 6871 id: 31 type: uint16 len: 2 silkFlowSensor ent: 6871 id: 30 type: uint8 len: 1 silkFlowType ent: 6871 id: 32 type: uint8 len: 1 silkTCPState ent: 0 id: 4 type: uint8 len: 1 protocolIdentifier ent: 0 id: 210 type: octet len: 1 paddingOctets ent: 0 id: 7 type: uint16 len: 2 sourceTransportPort ent: 0 id: 11 type: uint16 len: 2 destinationTransportPort ent: 0 id: 210 type: octet len: 1 paddingOctets ent: 0 id: 6 type: uint16 len: 1 tcpControlBits ent: 6871 id: 14 type: uint16 len: 1 initialTCPFlags ent: 6871 id: 15 type: uint16 len: 1 unionTCPFlags ent: 0 id: 8 type: ipv4 len: 4 sourceIPv4Address ent: 0 id: 12 type: ipv4 len: 4 destinationIPv4Address ent: 0 id: 15 type: ipv4 len: 4 ipNextHopIPv4Address --- template record --- header: tid: 40657 (0x9ed1) field count: 17 scope: 0 fields: ent: 0 id: 152 type: millisec len: 8 flowStartMilliseconds ent: 0 id: 153 type: millisec len: 8 flowEndMilliseconds ent: 0 id: 2 type: uint64 len: 4 packetDeltaCount ent: 0 id: 1 type: uint64 len: 4 octetDeltaCount ent: 0 id: 10 type: uint32 len: 2 ingressInterface ent: 0 id: 14 type: uint32 len: 2 egressInterface ent: 6871 id: 33 type: uint16 len: 2 silkAppLabel ent: 6871 id: 31 type: uint16 len: 2 silkFlowSensor ent: 6871 id: 30 type: uint8 len: 1 silkFlowType ent: 6871 id: 32 type: uint8 len: 1 silkTCPState ent: 0 id: 4 type: uint8 len: 1 protocolIdentifier ent: 0 id: 210 type: octet len: 1 paddingOctets ent: 0 id: 210 type: octet len: 2 paddingOctets ent: 0 id: 139 type: uint16 len: 2 icmpTypeCodeIPv6 ent: 0 id: 27 type: ipv6 len: 16 sourceIPv6Address ent: 0 id: 28 type: ipv6 len: 16 destinationIPv6Address ent: 0 id: 62 type: ipv6 len: 16 ipNextHopIPv6Address ... |
Convert the IPFIX file back to SiLK format using rwipfix2silk. Rather than writing the output to a file, write instead to stdout and use rwcut to see the values.
1 2 3 4 5 6 7 8 | sans@sec503:~/nik$ rwipfix2silk --silk-output=- rw-2-2ipfix.dat | rwcut --fields sip,dip --num-recs=5 sIP| dIP| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| 8.8.8.8| 172.28.10.137| ... |
Split a flow file into multiple files with rwsplit.
1 | sans@sec503:~/nik$ rwsplit --basename=nik_split_ --compression=best --flow-limit=4 \--max-outputs=2 --note-add="Files created with rwsplit" attack-trace.rw
|
Validate the rwsplit files were created.
1 2 | sans@sec503:~/nik$ ls nik_split_.0000000* nik_split_.00000000.rwf nik_split_.00000001.rwf |
Use rwfileinfo to get information on one of the rwsplit created files.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | sans@sec503:~/nik$ rwfileinfo nik_split_.00000001.rwf nik_split_.00000001.rwf: format(id) FT_RWIPV6ROUTING(0x0c) version 16 byte-order littleEndian compression(id) zlib(1) header-length 264 record-length 88 record-version 1 silk-version 3.19.2 count-records 4 file-size 404 command-lines 1 rwsplit --basename=nik_split_ --compression=best --flow-limit=4 --max-outputs=2 --note-add=Files created with rwsplit attack-trace.rw annotations 1 Files created with rwsplit |
Changing the byte order of the file with rwswapbytes.
Get the current byte order of the file 8.rw
1 2 3 | sans@sec503:~/nik$ rwfileinfo 8.rw --fields=byte-order 8.rw: byte-order littleEndian |
Change the byte order using rwswapbytes
1 | sans@sec503:~/nik$ rwswapbytes --big-endian \--note-add="Byte order swapped from little endian" 8.rw 8-swappped.rwf
|
Validate the byte order has been changed.
1 2 3 | sans@sec503:~/nik$ rwfileinfo 8-swappped.rwf --fields=byte-order 8-swappped.rwf: byte-order BigEndian |
Get some totals with rwtotal. Looking at the first 8 bytes of the destination IPs.
1 2 3 4 5 | sans@sec503:~/nik$ rwtotal attack-trace.rw --summation --skip-zero --dip-first-8 dIP_First8| Records| Bytes| Packets| 98| 6| 7297| 153| 192| 6| 171264| 195| TOTALS| 12| 178561| 348| |
Instead look at the first 24 bytes of the source IP.
1 2 3 4 5 | sans@sec503:~/nik$ rwtotal attack-trace.rw --summation --skip-zero --sip-first-24 sIP_First24| Records| Bytes| Packets| 98.114.205| 6| 171264| 195| 192.150. 11| 6| 7297| 153| TOTALS| 12| 178561| 348| |
Use rwtotal to learn what are the protocols seen on the network?
1 2 3 4 | sans@sec503:~/nik$ rwtotal attack-trace.rw --proto --summation --skip-zero protocol| Records| Bytes| Packets| 6| 12| 178561| 348| TOTALS| 12| 178561| 348| |
Looking at the destination ports
1 2 3 4 5 6 7 8 9 10 11 12 13 | sans@sec503:~/nik$ rwtotal attack-trace.rw --dport --summation --skip-zero --print-filenames attack-trace.rw dPort| Records| Bytes| Packets| 445| 2| 4945| 18| 1080| 1| 165088| 159| 1821| 1| 128| 3| 1828| 1| 1590| 17| 1924| 1| 250| 6| 1957| 1| 381| 6| 2152| 1| 4488| 112| 8884| 2| 841| 15| 36296| 2| 850| 12| TOTALS| 12| 178561| 348| |
Reference:
It is really nice post. Thank you for that.
ReplyDeleteDo you have any other tips, source for learning SEC503? How can I practice and learn more than the course itself?