Monday, October 5, 2020

Security On The Cheap - Beginning Elastic Stack - Providing Basic Security to Elastic and Kibana 7.9 communication on Ubuntu 20.04

In the first post, we installed Elasticsearch. In the second post, we installed Kibana. In this post, we now provide some basic security to the communication between the Elasticsearch and Kibana. Note, there is a lot more you can do to secure this environment, taking advantage of keystores, etc. 

In generating the Certification Authority (CA) certificate, I choose to use PKCS#12 format. In this format, this file contains both the CA certificate and its private key. This may be a cause for concern in some environments.

Additionally, I created one certificate file to be used by each component. This means the various components need to be able to read the file. This obviously makes the data in the file accessible by those users.  In this post, I am keeping it simple. If you are looking at thoroughly securing your environment, this post is something you can essentially build on if you wish.

If you are wondering why we need to secure the communication between Elastic and Kibana, here is a simple reason why.

root@securitynik-monitoring:~# tcpdump -nnti any host 10.0.0.1 and port 9200 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
IP 10.0.0.1.60100 > 10.0.0.1.9200: Flags [P.], seq 3727928959:3727929473, ack 22660352, win 9203, options [nop,nop,TS val 4105791837 ecr 4105786838], length 514

E..6.G@.@.!"..........#..3...Y....#........
..i]..U.POST /.reporting-*/_search HTTP/1.1
content-type: application/json
Host: 10.0.0.1:9200
Content-Length: 374
Connection: keep-alive

{"seq_no_primary_term":true,"_source":{"excludes":["output.content"]},"query":{"bool":{"filter":{"bool":{"minimum_should_match":1,"should":[{"term":{"status":"pending"}},{"bool":{"must":[{"term":{"status":"processing"}},{"range":{"process_expiration":{"lte":"2020-08-14T01:02:35.764Z"}}}]}}]}}}},"sort":[{"priority":{"order":"asc"}},{"created_at":{"order":"asc"}}],"size":1}

IP 10.0.0.1.9200 > 10.0.0.1.60100: Flags [P.], seq 1:247, ack 514, win 512, options [nop,nop,TS val 4105791839 ecr 4105791837], length 246
E..*..@.@...........#....Y...3.......u.....
..i_..i]HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 159
....

As shown above, using tcpdump, we were able to sniff the traffic on the wire, thus gaining visibility into actual communication occurring on the network. At this point, if we can see the data, anyone else can.

With that out of the way, let's get to providing some basic security to this environment via Transport Layer Security (TLS).

First up, we modify our "elasticsearch.yml" file to include "xpack.security.enabled: true". Here is what my configuration looks like.

root@securitynik-monitoring:~# cat /etc/elasticsearch/elasticsearch.yml | grep "xpack.security.enabled: true"
xpack.security.enabled: true

For TLS to work properly, we need a certificate. There are many ways to get certificates. However, for us, we will use the built in "elasticsearch-certutil" utility, to generate our own certification authority.

root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-certutil ca ca-dn securitynik.local
This tool assists you in the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'. This will create a new X.509 certificate and private key that can be used to sign certificate when running in 'cert' mode.


Use the 'ca-dn' option if you wish to configure the 'distinguished name' of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:

    * The CA certificate
    * The CA's private key


If you elect to generate PEM format certificates (the -pem option), then the output will be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]: SecurityNik-CA.p12

Enter password for SecurityNik-CA.p12 :

Above, my CA cert is named "SecurityNik-CA.p12". Also you have the option to specify a password for the CA cert. I did not specify a password. In my example, the file was stored in:

root@securitynik-monitoring:~# ls /usr/share/elasticsearch/SecurityNik-CA.p12 -al
-rw------- 1 root root 2527 Aug 13 21:49 /usr/share/elasticsearch/SecurityNik-CA.p12

With the CA certificate generated, let's now generate a certificate for our node at "10.0.0.1".

root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /usr/share/elasticsearch/SecurityNik-CA.p12  --days 1825 --dns monitoring,monitoring.securitynik.local --ip 10.0.0.1 --keysize 2048 --name 10.0.0.1 --out /usr/share/elasticsearch/10.0.0.1.p12 --pass "" --silent

Enter password for CA (/usr/share/elasticsearch/SecurityNik-CA.p12) :

Below we see the file which was created.

root@securitynik-monitoring:~# ls -al /usr/share/elasticsearch/10.0.0.1.p12 -al
-rw------- 1 root root 3529 Aug 13 22:23 /usr/share/elasticsearch/10.0.0.1.p12

Copy this certificate to the "/etc/elasticsearch/" folder.

root@securitynik-monitoring:~# cp /usr/share/elasticsearch/10.0.0.1.p12 /etc/elasticsearch/ -v 
'/usr/share/elasticsearch/10.0.0.1.p12' -> '/etc/elasticsearch/10.0.0.1.p12'

Next I changed the permission of the certificate so it is world readable. Not necessarily the best thing to do but this is for simplicity.

root@securitynik-monitoring:~# chmod 644 /etc/elasticsearch/10.0.0.1.p12
root@securitynik-monitoring:~# ls -al /etc/elasticsearch/10.0.0.1.p12
-rw-r--r-- 1 root elasticsearch 3529 Aug 13 22:32 /etc/elasticsearch/10.0.0.1.p12

As our certificate is in PKCS#12 format, we will add the following lines to our "elasticsearch.yml".

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: 10.0.0.1.p12
xpack.security.transport.ssl.truststore.path: 10.0.0.1.p12

Here is what my Elasticsearch configuration now looks like after those changes.

root@securitynik-monitoring:~# tail --lines 6 /etc/elasticsearch/elasticsearch.yml
# Configuration added by Nik for security
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: "10.0.0.1.p12"
xpack.security.transport.ssl.truststore.path: "10.0.0.1.p12"

Time to restart Elasticsearch and pray that everything works as expected.

root@securitynik-monitoring:~# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-08-13 22:40:44 EDT; 3min 50s ago
       Docs: https://www.elastic.co
   Main PID: 24533 (java)
      Tasks: 61 (limit: 4563)
     Memory: 1.2G
     CGroup: /system.slice/elasticsearch.service
             ├─24533 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddres>
             └─24725 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 13 22:40:17 securitynik-monitoring systemd[1]: Starting Elasticsearch...
Aug 13 22:40:44 securitynik-monitoring systemd[1]: Started Elasticsearch.

Above, all looks well. Looking at the network ports via "ss". Note you can also use "netstat". 

root@securitynik-monitoring:~# ss --numeric --listen --tcp | grep 9200
LISTEN  0       4096     [::ffff:10.0.0.1]:9200              *:*

Awesome! 

Next step, let's now generate a certificate for HTTP communication. For example, between Kibana and Elastic and ultimately between our browser and Kibana. Once again, we use "elasticsearch-certutil", this time with the "http" argument.

root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-certutil http --silent
## Elasticsearch HTTP Certificate Utility
## Do you wish to generate a Certificate Signing Request (CSR)?
Generate a CSR? [y/N]N
## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?
Use an existing CA? [y/N]Y
## What is the path to your CA?
CA Path: /usr/share/elasticsearch/SecurityNik-CA.p12
Password for SecurityNik-CA.p12:
## How long should your certificates be valid?
For how long should your certificate be valid? [5y]
## Do you wish to generate one certificate per node?
Generate a certificate per node? [y/N]N
## Which hostnames will be used to connect to your nodes?

securitynik-monitoring

securitynik-monitoring.securitynik.local

You entered the following hostnames.

 - securitynik-monitoring
 - securitynik-monitoring.securitynik.local

Is this correct [Y/n]y

## Which IP addresses will be used to connect to your nodes?
10.0.0.1

You entered the following IP addresses.

 - 10.0.0.1
Is this correct [Y/n]y
## Other certificate options
Key Name: securitynik-monitoring
Subject DN: CN=securitynik-monitoring

Key Size: 2048

Do you wish to change any of these options? [y/N]n
## What password do you want for your private key(s)?
Provide a password for the "http.p12" file:  [<ENTER> for none]
## Where should we save the generated files?

What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]

Confirming the file was successfully created.

root@securitynik-monitoring:~# ls /usr/share/elasticsearch/elasticsearch-ssl-http.zip -l
-rw------- 1 root root 7334 Aug 13 23:22 /usr/share/elasticsearch/elasticsearch-ssl-http.zip

First we install "unzip". With unzip installed, we are now able to look into the zip file

root@securitynik-monitoring:/usr/share/elasticsearch# apt install unzip
root@securitynik-monitoring:/usr/share/elasticsearch# unzip -l elasticsearch-ssl-http.zip
Archive:  elasticsearch-ssl-http.zip
  Length      Date    Time    Name
---------  ---------- -----   ----

        0  2020-08-13 23:22   elasticsearch/
     1091  2020-08-13 23:22   elasticsearch/README.txt
     3499  2020-08-13 23:22   elasticsearch/http.p12
      657  2020-08-13 23:22   elasticsearch/sample-elasticsearch.yml
        0  2020-08-13 23:22   kibana/
     1306  2020-08-13 23:22   kibana/README.txt
     1200  2020-08-13 23:22   kibana/elasticsearch-ca.pem
     1056  2020-08-13 23:22   kibana/sample-kibana.yml
---------                     -------
     8809                     8 files

Extracting the contents from the "elasticsearch-ssl-http.zip" into a folder named "certs" and verifying the extraction.

root@securitynik-monitoring:/usr/share/elasticsearch# unzip -d certs/ elasticsearch-ssl-http.zip
Archive:  elasticsearch-ssl-http.zip
   creating: certs/elasticsearch/
  inflating: certs/elasticsearch/README.txt
  inflating: certs/elasticsearch/http.p12
  inflating: certs/elasticsearch/sample-elasticsearch.yml
   creating: certs/kibana/
  inflating: certs/kibana/README.txt
  inflating: certs/kibana/elasticsearch-ca.pem
  inflating: certs/kibana/sample-kibana.yml
root@securitynik-monitoring:/usr/share/elasticsearch# ls certs/
elasticsearch  kibana

Time to copy "certs/elasticsearch/http.p12" file to "/etc/elasticsearch/" folder. 

root@securitynik-monitoring:~# cp /usr/share/elasticsearch/certs/elasticsearch/http.p12 /etc/elasticsearch/ -v 
'/usr/share/elasticsearch/certs/elasticsearch/http.p12' -> '/etc/elasticsearch/http.p12'

I then added the following lines to the bottom of my "elasticsearch.yml".

# This turns on SSL for the HTTP (Rest) interface
xpack.security.http.ssl.enabled: true
# This configures the keystore to use for SSL on HTTP
xpack.security.http.ssl.keystore.path: "http.p12"

For consistency I also rename the "elasticsearch-ca.pem" file to "SecurityNik-CA.pem"

root@securitynik-monitoring:/usr/share/elasticsearch/certs# cd /etc/kibana/
root@securitynik-monitoring:/etc/kibana# mv elasticsearch-ca.pem SecurityNik-CA.pem
root@securitynik-monitoring:/etc/kibana#

Next the file "kibana/SecurityNik-ca.pem" was copied to the "/etc/kibana" folder

root@securitynik-monitoring:/usr/share/elasticsearch/certs# cp /usr/share/elasticsearch/certs/kibana/SecurityNik-ca.pem /etc/kibana/ -v
'/usr/share/elasticsearch/certs/kibana/SecurityNik-ca.pem' -> '/etc/kibana/SecurityNik-ca.pem'

Next I modified my "kibana.yml" file changing "elasticsearch.hosts: ["http://10.0.0.1:9200"]" to "elasticsearch.hosts: ["https://10.0.0.1:9200"]". Note the https. I also added "elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/SecurityNik-CA.pem" ]"

With these configurations now out of the way, let's restart both Elasticsearch and Kibana.

root@securitynik-monitoring:/etc/kibana# systemctl stop elasticsearch.service
root@securitynik-monitoring:/etc/kibana# systemctl start elasticsearch.service
root@securitynik-monitoring:/etc/kibana# systemctl stop kibana.service
root@securitynik-monitoring:/etc/kibana# systemctl start kibana.service

Let's now configure Kibana so that we connect to it via HTTPS. 

First I copy the "10.0.0.1.p12" node certificate which was previously created to the Kibana folder. 

root@securitynik-monitoring:~# cp /etc/elasticsearch/10.0.0.1.p12 /etc/kibana/ -v 
'/etc/elasticsearch/10.0.0.1.p12' -> '/etc/kibana/10.0.0.1.p12'

Once again, modifying "kibana.yml", we add the following lines.

server.ssl.keystore.path: "/etc/kibana/10.0.0.1.p12"
server.ssl.enabled: true
server.ssl.keystore.password: ""

Once completed, restarted Kibana 

root@securitynik-monitoring:/etc/kibana# systemctl stop kibana.service
root@securitynik-monitoring:/etc/kibana# systemctl start kibana.service

Before we login, let's setup the default users using "elasticsearch-setup-passwords".

root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive --silent

Please confirm that you would like to continue [y/N]y
Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:

Now that the users are setup, the Kibana configuration file must now reflect the values for username and password for "kibana_system" account.

root@securitynik-monitoring:~# cat /etc/kibana/kibana.yml | grep "kibana_system" --after-context 1
elasticsearch.username: "kibana_system"
elasticsearch.password: "WelcomeToSecurityNikElastic"

Below represents some of those changes I've made to the "kibana.yml"

root@securitynik-monitoring:~# tail --lines 8 /etc/kibana/kibana.yml

# Below added by Nik
server.ssl.enabled: true
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/SecurityNik-CA.pem"]
server.ssl.keystore.path: "/etc/kibana/10.0.0.1.p12"
server.ssl.keystore.password: ""
xpack.encryptedSavedObjects.encryptionKey: 'fhjskloppd678ehkdfdlliverpoolfcr'
xpack.security.session.idleTimeout: "30m"
xpack.security.session.lifespan: "8h"

With those changes in place, we should now be good to go.

Let's authenticate to Kibana using the username and password we setup for the "elastic" user. Once authenticated, you should create a few additional users based on their roles.

HTTPS Login Page


Once you authenticate successfully, you should now see the following.

Before we go, if you remember, when we started, we were able to see the clear text data crossing the wire. If we run tcpdump again, we don't see any data in the clear.

root@n3-monitoring:/etc/elasticsearch# tcpdump -nnti any host 10.0.0.1 and port 9200 -A -c 5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
IP 10.0.0.1.36344 > 10.0.0.1.9200: Flags [P.], seq 2879060748:2879061003, ack 1452891267, win 512, options [nop,nop,TS val 2724200057 ecr 2724168569], length 255
E..3y.@.@.>...........#.....V.\......~.....
._.y._.y.........@.../....Ls}..E...
.l.;........x.^..u..3S...A.,.O...2e.h...../.).*@.a15ou.u.^D...~.3..Hy\W'.../....D.ahL..H...q;...j....$..L.0<%J..._..k]..TMQj.B
m....n.id.O....5.....S...=1.Iq.|Ox...}.mOP......z@..L.&...&X.>....R.u'I.-U*!|.Z..._.Z.[%7=...K.6m...
IP 10.0.0.1.9200 > 10.0.0.1.36344: Flags [P.], seq 1:238, ack 255, win 512, options [nop,nop,TS val 2724200061 ecr 2724200057], length 237
E..!..@.@...........#...V.\..........l.....
._.}._.y..............{.m6!....ZK^...
.j..$-.......4(..\Y6...y=j.....T0v..J.yG..8.I.....'.+%w.6...y..z...r...*K..L......t..cc.Pk.\..RZ.%.N.....l...Zq......qw.W..!05."...{...I..g..)\(.H..........Q...H..<X...1...X4.W..).
.eoB...w..i.....[......+.W
IP 10.0.0.1.36344 > 10.0.0.1.9200: Flags [.], ack 238, win 511, options [nop,nop,TS val 2724200061 ecr 2724200061], length 0
E..4y.@.@.?...........#.....V.]p...........
._.}._.}
IP 10.0.0.1.36294 > 10.0.0.1.9200: Flags [.], ack 2500636561, win 512, options [nop,nop,TS val 2724200281 ecr 2724199268], length 0
E..43.@.@.............#.3.1................
._.Y._.d
IP 10.0.0.1.9200 > 10.0.0.1.36294: Flags [.], ack 1, win 512, options [nop,nop,TS val 2724200281 ecr 2724199268], length 0
E..4..@.@...........#.......3.1............

At this point, it is time to move on, as we were able to provide some basic security to Kibana and Elastic.

Posts in this series:

Security On The Cheap - Beginning Elastic Stack - Installing Elastic 7.9 on Ubuntu 20.04
Security On The Cheap - Beginning Elastic Stack - Installing Kibana 7.9 on Ubuntu 20.04
Security On The Cheap - Beginning Elastic Stack - Providing Basic Security to Elastic and Kibana 7.9 communication on Ubuntu 20.04
Security On The Cheap - Beginning Elastic - Installing and Providing Basic Security to Metricbeat - Elastic Stack 7.9 on Ubuntu 20.04
Security On The Cheap - Beginning Elastic - Installing and Providing Basic Security to Auditbeat - Elastic Stack 7.9 on Ubuntu 20.04
Security On The Cheap - Beginning Elastic - Installing and Providing Basic Security to Filebeat - Elastic Stack 7.9 on Ubuntu 20.04
Beginning Elastic - Installing, Configuring and Providing Basic Security to Packetbeat
Security On The Cheap - Beginning Elastic - Installing and Providing Basic Security to Winlogbeat


References:

https://www.elastic.co/blog/getting-started-with-elasticsearch-security
https://www.elastic.co/guide/en/elasticsearch/reference/7.8/configuring-tls.html#node-certificates
https://www.elastic.co/guide/en/elasticsearch/reference/current/get-started-built-in-users.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/get-started-kibana-user.html
https://www.youtube.com/watch?v=nMh1HWWe6B4&feature=youtu.be
https://www.elastic.co/guide/en/elasticsearch/reference/current/trb-security-sslhandshake.html

https://www.elastic.co/guide/en/kibana/7.9/using-kibana-with-security.html

No comments:

Post a Comment