Saturday, June 13, 2020

Installing Zeek 3.1.4 on Ubuntu 20.04

In the SANS SEC503 Intrusion Detection in Depth class, we teach you quite a lot to get you started with Zeek Network Security Monitoring. One of the things we cannot do because of time, is walk you through the installation, upgrading, etc., of Zeek. In this post, we help you to install Zeek 3.1.4, the current version as of this writing on Ubuntu 20.04. The hope is with the knowledge we provide to you in the SEC 503 class as well as what you get here, you are in a better position to maximize your Zeek deployment.

First, let us confirm my version of Ubuntu. As seen below, I am running Ubuntu 20.04 LTS.


securitynik@securitynik-zeek:~$ lsb_release --all
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal

Next, let us install Zeek's dependencies. As we are using Ubuntu, we need the following dependencies:


cmake 
make 
gcc 
g++ 
flex 
bison 
libpcap-dev 
libssl-dev 
python-dev 
swig 
zlib1g-dev

Let's install them by using:


securitynik@securitynik-zeek:~$sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev

Let's now install an optional dependency. Specifically, we will install libmaxminddb (for geolocating IP addresses). To make the most of this, you will need to register with MaxMind before you can download the GeoIP dataset. Once registered and logged in, download the GeoLite2 City database.

Once downloaded, you may need to copy the file to the remote machine running Zeek. In my example, I downloaded the file on a Windows system, then used SCP to transfer it to the device running Zeek.

If you don't wish to use geo-data, feel free to skip this step.


securitynik@securitynik-zeek:~$ sudo apt-get install libmaxminddb-dev

Copy the GeoIP data file from Windows 10 system to the system running Zeek, using the native SCP client in Windows 10.


C:\Users\securitynik\Downloads>scp GeoLite2-City_20200609.tar.gz nik@192.168.0.4:~/

nik@192.168.0.4's password:     
GeoLite2-City_20200609.tar gz         100%   29MB  10.4MB/s   00:02

Once the file is on the system running Zeek, we then extract the files:


securitynik@securitynik-zeek:~$ tar --extract --verbose --gunzip --file GeoLite2-City_20200609.tar.gz
GeoLite2-City_20200609/
GeoLite2-City_20200609/README.txt
GeoLite2-City_20200609/COPYRIGHT.txt
GeoLite2-City_20200609/GeoLite2-City.mmdb
GeoLite2-City_20200609/LICENSE.txt

Next copy the GeoLite2-City_20200609/GeoLite2-City.mmdb into the /usr/share/GeoIP folder. In my case I had to create this folder.


securitynik@securitynik-zeek:~$ sudo mkdir --parents /usr/share/GeoIP
securitynik@securitynik-zeek:~$ sudo cp GeoLite2-City_20200609/GeoLite2-City.mmdb /usr/share/GeoIP/

Once the dependencies have been installed successfully, we then transition to the Zeek installation. For our install, we use the latest version at the time of this writing from Zeek's GitHub page.


securitynik@securitynik-zeek:~$ git clone --recursive https://github.com/zeek/zeek

Upon completion of the cloning, we then change directory (cd) into the zeek directory. Once we changed into the zeek directory, we then specify "--with-geoip" argument for the configure script so as to customize our installation. In this example, we are pointing it to the previously created folder which holds or geo-data file.


securitynik@securitynik-zeek:~/zeek$ cd zeek/
securitynik@securitynik-zeek:~/zeek$ ./configure --with-geoip=/usr/share/GeoIP

If everything went ok, you may see something similar to:


....
====================|  Zeek Build Summary  |====================

Build type:        RelWithDebInfo
Build dir:         /home/nik/zeek/build
Install prefix:    /usr/local/zeek
Zeek Script Path:  /usr/local/zeek/share/zeek
Debug mode:        false
Unit tests:

CC:                /usr/bin/cc
CFLAGS:             -Wall -Wno-unused -O2 -g -DNDEBUG
CXX:               /usr/bin/c++
CXXFLAGS:           -Wall -Wno-unused -Wno-register -Werror=vla -std=c++17 -O2 -g -DNDEBUG
CPP:               /usr/bin/c++

ZeekControl:       true
Aux. Tools:        true

libmaxminddb:      true
Kerberos:          false
gperftools found:  false
        tcmalloc:  false
       debugging:  false
jemalloc:          false

Fuzz Targets:
Fuzz Engine:

================================================================

-- Configuring done
-- Generating done
-- Build files have been written to: /home/nik/zeek/build

Next up, make and make install Zeek. Grab a quick coffee, beer or Guinness. Basically whatever you drink as depending on your system resources, this may take a while to complete.


securitynik@securitynik-zeek:~/zeek$ sudo make 
securitynik@securitynik-zeek:~/zeek$ sudo make install

Now that the install has completed, test to see if zeek can be executed.


securitynik@securitynik-zeek:~$which zeek
securitynik@securitynik-zeek:~$

Looks like zeek is not in the path. Let's add a symbolic link and verify Zeek is now in the path


securitynik@securitynik-zeek:~$ sudo ln --symbolic /usr/local/zeek/bin/zeek /usr/bin/zeek
securitynik@securitynik-zeek:~$ which zeek
/usr/bin/zeek

Looks good! Similarly, let's setup a symbolic link for zeekctl


securitynik@securitynik-zeek:~$ sudo ln --symbolic /usr/local/zeek/bin/zeekctl /usr/bin/zeekctl
securitynik@securitynik-zeek:~$ which zeekctl
/usr/bin/zeekctl

While the above is cool, we need to make this a bit more permanent. More importantly, there are other files in the /usr/local/zeek/bin/ directory such as zeek-cut that we more than likely need to use. Thus adding all of these symbolic links may not be the best use of our time. Let's address this by first modifying the path.


securitynik@securitynik-zeek:~$ export PATH=$PATH:/usr/local/zeek/bin
securitynik@securitynik-zeek:~$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/loal/zeek/bin:/usr/local/zeek/bin

This now allows us to access the other binaries in the /usr/local/zeek/bin/ directory such as zeek-cut

Let's now make it permanent. 


securitynik@securitynik-zeek:~$ echo "export PATH=$PATH:/usr/local/zeek/bin" >> .bashrc
securitynik@securitynik-zeek:~$ tail --lines 1 .bashrc
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/loal/zeek/bin:/usr/local/zeek/bin:/usr/local/zeek/bin

At this point you should be able to execute Zeek with its default configuration in standalone mode. However, for us, let move away from the defaults.

First edit the networks.cfg using your favourite editor (vi, nano, gedit, mousepad, etc.), to ensure it reflects your protected network(s). Here is what mine looks like.


securitynik@securitynik-zeek:/usr/local/zeek/etc$ cat networks.cfg
# List of local networks in CIDR notation, optionally followed by a
# descriptive tag.
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.

#10.0.0.0/8          Private IP space
#172.16.0.0/12       Private IP space
192.168.0.0/24     SecurityNik Protected Network

For demonstration purposes, I commented out 10.0.0.0/8 and 172.16.0.0/12 networks and left the 192.168.0.0/16. I also changed the description to suit my needs. You should also add any other networks you have to this file.

Once that is completed, you should then configure the node.cfg file. Here is what mine looks like. I commented out the standalone section to simulate a distributed deployment. Everything is on one host but the host has two interfaces. One of those is the loopback and the other enp0s25. Note, you don't really need to do it the way I did. I am only doing it this way to demonstrate how you can use different workers on different systems and or interfaces.


#[zeek]
#type=standalone
#host=securitynik
#interface=enp0s25

## Below is an example clustered configuration. If you use this,
## remove the [zeek] node above.

#[logger-1]
#type=logger
#host=localhost
#
[securitynik-zeek-manager]
type=manager
host=192.168.0.4
#
[securitynik-zeek-proxy]
type=proxy
host=192.168.0.4
#
[securitynik-zeek-worker-enp0s25]
type=worker
host=192.168.0.4
interface=enp0s25
#
[securitynik-zeek-worker-lo]
type=worker
host=localhost
interface=lo

Once you have modified your node.cfg to reflect your configuration needs, your next step is to make any necessary changes to your zeekctl.cfg. At a minimum in this file you may want to change the recipient email address. Here is a snapshot of my file:


securitynik@securitynik-zeek:/usr/local/zeek/etc$ cat zeekctl.cfg | more
## Global ZeekControl configuration file.

###############################################
# Mail Options

# Recipient address for all emails sent out by Zeek and ZeekControl.
MailTo = admin@securitynik.local
......

At this point, we should be good to go. Once you modify your files, you should deploy your changes. Let's do that.


securitynik@securitynik-zeek:/usr/local/zeek/etc$ sudo zeekctl deploy
checking configurations ...
installing ...
removing old policies in /usr/local/zeek/spool/installed-scripts-do-not-touch/site ...
removing old policies in /usr/local/zeek/spool/installed-scripts-do-not-touch/auto ...
creating policy directories ...
installing site policies ...
generating cluster-layout.zeek ...
generating local-networks.zeek ...
generating zeekctl-config.zeek ...
generating zeekctl-config.sh ...
stopping ...
stopping workers ...
stopping proxy ...
stopping manager ...
starting ...
starting manager ...
starting proxy ...
starting workers ...

Looks like everything started properly. Let's confirm this:


securitynik@securitynik-zeek:/usr/local/zeek/etc$ sudo zeekctl status
Name                        Type    Host             Status    Pid    Started
securitynik-zeek-manager         manager 192.168.0.4      running   207505 13 Jun 14:13:33
securitynik-zeek-proxy           proxy   192.168.0.4      running   207554 13 Jun 14:13:35
securitynik-zeek-worker-enp0s25  worker  192.168.0.4      running   207624 13 Jun 14:13:37
securitynik-zeek-worker-lo       worker  localhost        running   207621 13 Jun 14:13:37

Looks like we are good to go. To further confirm, let's see if any logs are being created.


securitynik@securitynik-zeek:/usr/local/zeek/etc$ ls ../logs/current
broker.log   conn.log  files.log  loaded_scripts.log  reporter.log  stats.log   stdout.log  weird.log
cluster.log  dns.log   http.log   packet_filter.log   ssl.log       stderr.log  syslog.log  x509.log

Nice. Looks like we are making progress. 

Finally, you may wish to have Zeek start as a service via Systemd. Let's make that happen.

Create a file name zeek.zervice and verify its existence as follow.


securitynik@securitynik-zeek:~$ sudo touch /etc/systemd/system/zeek.service
securitynik@securitynik-zeek:~$ ls /etc/systemd/system/zeek.service*
/etc/systemd/system/zeek.service

Next edit the file and add the following contents:


securitynik@securitynik-zeek:~$ sudo vi /etc/systemd/system/zeek.service
securitynik@securitynik-zeek:~$ cat /etc/systemd/system/zeek.service
Description=Zeek Network Security Monitoring

Wants=network.target
After=syslog.target network-online.target

[Service]
Type=forking
ExecStart=zeekctl deploy
Restart=on-failure
RestartSec=10
KillMode=process

[Install]
WantedBy=multi-user.target

Next reload systemd


securitynik@securitynik-zeek:~$ sudo systemctl daemon-reload

Enable the zeek service (zeek.zervice)


securitynik@securitynik-zeek:~$ sudo systemctl enable --now zeek.service
Created symlink /etc/systemd/system/multi-user.target.wants/zeek.service → /etc/systemd/system/zeek.service.

Start the service 


securitynik@securitynik-zeek:~$ sudo systemctl start zeek.service
securitynik@securitynik-zeek:~$

Looks like the Zeek service started without errors. Let's confirm the service is now running.


securitynik@securitynik-zeek:~$ systemctl status zeek.service
● zeek.service
     Loaded: loaded (/etc/systemd/system/zeek.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2020-06-13 14:42:21 EDT; 1min 26s ago
    Process: 215455 ExecStart=/usr/bin/zeekctl start (code=exited, status=0/SUCCESS)
   Main PID: 215455 (code=exited, status=0/SUCCESS)
      Tasks: 50 (limit: 2225)
     Memory: 293.0M
     CGroup: /system.slice/zeek.service
             ├─215481 /usr/bin/bash /usr/local/zeek/share/zeekctl/scripts/run-zeek -1 -U .status -p zeekctl -p zeekctl->
             ├─215487 /usr/local/zeek/bin/zeek -U .status -p zeekctl -p zeekctl-live -p local -p securitynik-zeek-manager lo>
             ├─215533 /usr/bin/bash /usr/local/zeek/share/zeekctl/scripts/run-zeek -1 -U .status -p zeekctl -p zeekctl->
             ├─215539 /usr/local/zeek/bin/zeek -U .status -p zeekctl -p zeekctl-live -p local -p securitynik-zeek-proxy loca>
             ├─215592 /usr/bin/bash /usr/local/zeek/share/zeekctl/scripts/run-zeek -1 -i enp0s25 -U .status -p zeekctl >
             ├─215600 /usr/bin/bash /usr/local/zeek/share/zeekctl/scripts/run-zeek -1 -i lo -U .status -p zeekctl -p ze>
             ├─215605 /usr/local/zeek/bin/zeek -i enp0s25 -U .status -p zeekctl -p zeekctl-live -p local -p securitynik-zeek>
             └─215609 /usr/local/zeek/bin/zeek -i lo -U .status -p zeekctl -p zeekctl-live -p local -p securitynik-zeek-work>

Jun 13 14:42:14 securitynik systemd[1]: Started zeek.service.
Jun 13 14:42:21 securitynik zeekctl[215455]: starting manager ...
Jun 13 14:42:21 securitynik zeekctl[215455]: starting proxy ...
Jun 13 14:42:21 securitynik zeekctl[215455]: starting workers ...
Jun 13 14:42:21 securitynik systemd[1]: zeek.service: Succeeded.

Above we can see zeek.serviceSuceeded. However, if you look at the Active above, we see Inactive dead. However, I was able to confirm Zeek is running as expected even after stopping Zeek with zeekctl stop. I then restarted the system and run sudo zeekctl status along with systemctl status zeek.service and everything is working as expected.


At this point, I will close off this post. Have fun, stay safe and I look forward to seeing you in one of  the upcoming SANS SEC503 Intrusion Detection in Depth class, where we teach you how to maximize your security monitoring with Zeek.

Reference:

19 comments:

  1. Thank you! Just what I needed.... Lab VMs didn't come with my class for some reason, so I'm recreating the environment ...

    ReplyDelete
    Replies
    1. Jim,
      Glad you found it helpful. Let me know if you have any questions.

      Delete
  2. This is great!!!!!
    Thank you very much

    ReplyDelete
  3. Amazing tutorial. thank you very much for making our lives easier. Can't thank you enough. your hardwork is helping novice like me.

    ReplyDelete
    Replies
    1. Hey Anonymous, that is the whole idea. If I was able to help someone else learn this content then I've done at least one good for today :-)

      Let me know if there is anything else I can help with.

      Delete
  4. Hello I am a novice zeek/linux user.
    When i run the following command, I continue to get the following error. Can you please help me ?

    securitynik@securitynik-zeek:~$sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev


    ERROR RECEIVED

    -- Found SWIG: /usr/bin/swig4.0 (found version "4.0.1")
    -- Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Development) (found version "3.8")
    -- Could NOT find PythonDev (missing: PYTHON_INCLUDE_DIR)
    CMake Error at auxil/zeekctl/auxil/pysubnettree/CMakeLists.txt:29 (message):
    Could not find prerequisite package 'PythonDev'


    CMake Error at auxil/zeekctl/auxil/pysubnettree/CMakeLists.txt:31 (message):
    Configuration aborted due to missing prerequisites


    -- Configuring incomplete, errors occurred!
    See also "/home/wajid/bro/build/CMakeFiles/CMakeOutput.log".

    ------------------I ran following commands but I was not able to get the above issue resolved


    george@ubuntu:~/bro$ sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Note, selecting 'python-dev-is-python2' instead of 'python-dev'
    bison is already the newest version (2:3.5.1+dfsg-1).
    cmake is already the newest version (3.16.3-1ubuntu1).
    flex is already the newest version (2.6.4-6.2).
    g++ is already the newest version (4:9.3.0-1ubuntu2).
    gcc is already the newest version (4:9.3.0-1ubuntu2).
    libpcap-dev is already the newest version (1.9.1-3).
    make is already the newest version (4.2.1-1.2).
    python-dev-is-python2 is already the newest version (2.7.17-4).
    swig is already the newest version (4.0.1-5build1).
    libssl-dev is already the newest version (1.1.1f-1ubuntu2.1).
    zlib1g-dev is already the newest version (1:1.2.11.dfsg-2ubuntu1.2).
    The following package was automatically installed and is no longer required:
    libfprint-2-tod1
    Use 'sudo apt autoremove' to remove it.
    0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    george@ubuntu:~/bro$ sudo apt-get install python-dev
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Note, selecting 'python-dev-is-python2' instead of 'python-dev'
    python-dev-is-python2 is already the newest version (2.7.17-4).
    The following package was automatically installed and is no longer required:
    libfprint-2-tod1
    Use 'sudo apt autoremove' to remove it.
    0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

    See also "/home/wajid/bro/build/CMakeFiles/CMakeError.log".
    Also, before re-running configure, consider cleaning the cache (removing the build directory) via `make distclean`

    ReplyDelete
    Replies
    1. Hello George,
      For starters, it seems you may have to install some Python3 and some dependencies based on this:
      Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Development) (found version "3.8")
      -- Could NOT find PythonDev (missing: PYTHON_INCLUDE_DIR)
      CMake Error at auxil/zeekctl/auxil/pysubnettree/CMakeLists.txt:29 (message):
      Could not find prerequisite package 'PythonDev'

      I recommend you resolve those and start from there.

      Delete
    2. Nik I do not think you answered his question, I used your instructions months ago and everything worked great. Now jan 2021 I follow the same directions, and get the Python-dev not found error also. so it appears the github/zeek/zeek changed or ubuntu 20.04 changed. Either way your awesome instructions no longer work because something else changed. I think his question was how do we deal with the dependancy issue. whne I try to install python 3 it sayes it is installed already, but same issue remains, any help greatly appreciated.

      Delete
    3. Hmmm! I can try to see if I can recreate this issue but I'm wondering if something changed with Zeek. I don't want to believe it is a change in Ubuntu 20.04. I can't promise I will look at this soon as I have some things I have to take care of that is going to take some time. However, if anyone finds the solution before I update this post, I would definitely appreciate you posting your findings.

      Delete
    4. I had the same problem. After installing the dependencies in the zeek/INSTALL file of the current version, everything worked fine! This file calls for python3 and python3-dev instead of pyhon-dev.

      Delete
    5. Running the following command solves the problem:
      sudo apt install python-dev-is-python3
      sudo apt autoremove

      Delete
  5. Hi, Do we need to install Zeek as a standalone instance or we can install this on an existing ELK instance? And would it be compatible with Ubuntu 18.04.3? Thank You!

    ReplyDelete
    Replies
    1. Hello Anonymous,
      Zeek and Elastic are installed on the same device in this post. However, you can install Zeek on it's own device and install Filebeat on that device to forward the logs to your remote Elastic instance. As for Ubuntu 18.04.3, I have not tried it but I don't expect the install to be much different.

      If you do install it on Ubuntu 18.x let me know how it goes.

      Thanks

      Delete
  6. Hi, thanks for the tutorial. I've managed to make the service "running" by adding Type=forking in the Service stanza as explained here:
    https://unix.stackexchange.com/questions/137028/why-is-my-systemd-unit-loaded-but-inactive-dead

    ReplyDelete
    Replies
    1. Hmmm! Thanks! Very interesting!! Not sure how that slipped through. I updated the post and the reference.

      Very much appreciated!

      Delete
  7. I just completed SEC503. This filled a gap. Much appreciated. It was one of the best classes I've taken. Explained a lot of things, I only assumed in the past.

    ReplyDelete