In the first post, we installed Elasticsearch. In the second post, we installed Kibana. In this post, we now provide some basic security to the communication between the Elasticsearch and Kibana. Note, there is a lot more you can do to secure this environment, taking advantage of keystores, etc.
In generating the Certification Authority (CA) certificate, I choose to use PKCS#12 format. In this format, this file contains both the CA certificate and its private key. This may be a cause for concern in some environments.
Additionally, I created one certificate file to be used by each component. This means the various components need to be able to read the file. This obviously makes the data in the file accessible by those users. In this post, I am keeping it simple. If you are looking at thoroughly securing your environment, this post is something you can essentially build on if you wish.
If you are wondering why we need to secure the communication between Elastic and Kibana, here is a simple reason why.
root@securitynik-monitoring:~# tcpdump -nnti any host 10.0.0.1 and port 9200 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
IP 10.0.0.1.60100 > 10.0.0.1.9200: Flags [P.], seq 3727928959:3727929473, ack 22660352, win 9203, options [nop,nop,TS val 4105791837 ecr 4105786838], length 514
E..6.G@.@.!"..........#..3...Y....#........
..i]..U.POST /.reporting-*/_search HTTP/1.1
content-type: application/json
Host: 10.0.0.1:9200
Content-Length: 374
Connection: keep-alive
{"seq_no_primary_term":true,"_source":{"excludes":["output.content"]},"query":{"bool":{"filter":{"bool":{"minimum_should_match":1,"should":[{"term":{"status":"pending"}},{"bool":{"must":[{"term":{"status":"processing"}},{"range":{"process_expiration":{"lte":"2020-08-14T01:02:35.764Z"}}}]}}]}}}},"sort":[{"priority":{"order":"asc"}},{"created_at":{"order":"asc"}}],"size":1}
IP 10.0.0.1.9200 > 10.0.0.1.60100: Flags [P.], seq 1:247, ack 514, win 512, options [nop,nop,TS val 4105791839 ecr 4105791837], length 246
E..*..@.@...........#....Y...3.......u.....
..i_..i]HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 159
....As shown above, using tcpdump, we were able to sniff the traffic on the wire, thus gaining visibility into actual communication occurring on the network. At this point, if we can see the data, anyone else can.
With that out of the way, let's get to providing some basic security to this environment via Transport Layer Security (TLS).
First up, we modify our "elasticsearch.yml" file to include "xpack.security.enabled: true". Here is what my configuration looks like.
root@securitynik-monitoring:~# cat /etc/elasticsearch/elasticsearch.yml | grep "xpack.security.enabled: true" xpack.security.enabled: true
For TLS to work properly, we need a certificate. There are many ways to get certificates. However, for us, we will use the built in "elasticsearch-certutil" utility, to generate our own certification authority.
root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-certutil ca ca-dn securitynik.local This tool assists you in the generation of X.509 certificates and certificate signing requests for use with SSL/TLS in the Elastic stack. The 'ca' mode generates a new 'certificate authority'. This will create a new X.509 certificate and private key that can be used to sign certificate when running in 'cert' mode. Use the 'ca-dn' option if you wish to configure the 'distinguished name' of the certificate authority By default the 'ca' mode produces a single PKCS#12 output file which holds: * The CA certificate * The CA's private key If you elect to generate PEM format certificates (the -pem option), then the output will be a zip file containing individual files for the CA certificate and private key Please enter the desired output file [elastic-stack-ca.p12]: SecurityNik-CA.p12 Enter password for SecurityNik-CA.p12 :
Above, my CA cert is named "SecurityNik-CA.p12". Also you have the option to specify a password for the CA cert. I did not specify a password. In my example, the file was stored in:
root@securitynik-monitoring:~# ls /usr/share/elasticsearch/SecurityNik-CA.p12 -al -rw------- 1 root root 2527 Aug 13 21:49 /usr/share/elasticsearch/SecurityNik-CA.p12
With the CA certificate generated, let's now generate a certificate for our node at "10.0.0.1".
root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /usr/share/elasticsearch/SecurityNik-CA.p12 --days 1825 --dns monitoring,monitoring.securitynik.local --ip 10.0.0.1 --keysize 2048 --name 10.0.0.1 --out /usr/share/elasticsearch/10.0.0.1.p12 --pass "" --silent Enter password for CA (/usr/share/elasticsearch/SecurityNik-CA.p12) :
Below we see the file which was created.
root@securitynik-monitoring:~# ls -al /usr/share/elasticsearch/10.0.0.1.p12 -al -rw------- 1 root root 3529 Aug 13 22:23 /usr/share/elasticsearch/10.0.0.1.p12
Copy this certificate to the "/etc/elasticsearch/" folder.
root@securitynik-monitoring:~# cp /usr/share/elasticsearch/10.0.0.1.p12 /etc/elasticsearch/ -v '/usr/share/elasticsearch/10.0.0.1.p12' -> '/etc/elasticsearch/10.0.0.1.p12'
Next I changed the permission of the certificate so it is world readable. Not necessarily the best thing to do but this is for simplicity.
root@securitynik-monitoring:~# chmod 644 /etc/elasticsearch/10.0.0.1.p12root@securitynik-monitoring:~# ls -al /etc/elasticsearch/10.0.0.1.p12 -rw-r--r-- 1 root elasticsearch 3529 Aug 13 22:32 /etc/elasticsearch/10.0.0.1.p12
As our certificate is in PKCS#12 format, we will add the following lines to our "elasticsearch.yml".
xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: 10.0.0.1.p12 xpack.security.transport.ssl.truststore.path: 10.0.0.1.p12
Here is what my Elasticsearch configuration now looks like after those changes.
root@securitynik-monitoring:~# tail --lines 6 /etc/elasticsearch/elasticsearch.yml # Configuration added by Nik for security xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: "10.0.0.1.p12" xpack.security.transport.ssl.truststore.path: "10.0.0.1.p12"
Time to restart Elasticsearch and pray that everything works as expected.
root@securitynik-monitoring:~# systemctl status elasticsearch.service ● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-08-13 22:40:44 EDT; 3min 50s ago Docs: https://www.elastic.co Main PID: 24533 (java) Tasks: 61 (limit: 4563) Memory: 1.2G CGroup: /system.slice/elasticsearch.service ├─24533 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddres> └─24725 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller Aug 13 22:40:17 securitynik-monitoring systemd[1]: Starting Elasticsearch... Aug 13 22:40:44 securitynik-monitoring systemd[1]: Started Elasticsearch.
Above, all looks well. Looking at the network ports via "ss". Note you can also use "netstat".
root@securitynik-monitoring:~# ss --numeric --listen --tcp | grep 9200 LISTEN 0 4096 [::ffff:10.0.0.1]:9200 *:*
Awesome!
Next step, let's now generate a certificate for HTTP communication. For example, between Kibana and Elastic and ultimately between our browser and Kibana. Once again, we use "elasticsearch-certutil", this time with the "http" argument.
root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-certutil http --silent ## Elasticsearch HTTP Certificate Utility ## Do you wish to generate a Certificate Signing Request (CSR)? Generate a CSR? [y/N]N ## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate? Use an existing CA? [y/N]Y ## What is the path to your CA? CA Path: /usr/share/elasticsearch/SecurityNik-CA.p12 Password for SecurityNik-CA.p12: ## How long should your certificates be valid? For how long should your certificate be valid? [5y] ## Do you wish to generate one certificate per node? Generate a certificate per node? [y/N]N ## Which hostnames will be used to connect to your nodes? securitynik-monitoring securitynik-monitoring.securitynik.local You entered the following hostnames. - securitynik-monitoring - securitynik-monitoring.securitynik.local Is this correct [Y/n]y ## Which IP addresses will be used to connect to your nodes? 10.0.0.1 You entered the following IP addresses. - 10.0.0.1 Is this correct [Y/n]y ## Other certificate options Key Name: securitynik-monitoring Subject DN: CN=securitynik-monitoring Key Size: 2048 Do you wish to change any of these options? [y/N]n ## What password do you want for your private key(s)? Provide a password for the "http.p12" file: [<ENTER> for none] ## Where should we save the generated files? What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]
Confirming the file was successfully created.
root@securitynik-monitoring:~# ls /usr/share/elasticsearch/elasticsearch-ssl-http.zip -l -rw------- 1 root root 7334 Aug 13 23:22 /usr/share/elasticsearch/elasticsearch-ssl-http.zip
First we install "unzip". With unzip installed, we are now able to look into the zip file
root@securitynik-monitoring:/usr/share/elasticsearch# apt install unziproot@securitynik-monitoring:/usr/share/elasticsearch# unzip -l elasticsearch-ssl-http.zip Archive: elasticsearch-ssl-http.zip Length Date Time Name --------- ---------- ----- ---- 0 2020-08-13 23:22 elasticsearch/ 1091 2020-08-13 23:22 elasticsearch/README.txt 3499 2020-08-13 23:22 elasticsearch/http.p12 657 2020-08-13 23:22 elasticsearch/sample-elasticsearch.yml 0 2020-08-13 23:22 kibana/ 1306 2020-08-13 23:22 kibana/README.txt 1200 2020-08-13 23:22 kibana/elasticsearch-ca.pem 1056 2020-08-13 23:22 kibana/sample-kibana.yml --------- ------- 8809 8 files
Extracting the contents from the "elasticsearch-ssl-http.zip" into a folder named "certs" and verifying the extraction.
root@securitynik-monitoring:/usr/share/elasticsearch# unzip -d certs/ elasticsearch-ssl-http.zip Archive: elasticsearch-ssl-http.zip creating: certs/elasticsearch/ inflating: certs/elasticsearch/README.txt inflating: certs/elasticsearch/http.p12 inflating: certs/elasticsearch/sample-elasticsearch.yml creating: certs/kibana/ inflating: certs/kibana/README.txt inflating: certs/kibana/elasticsearch-ca.pem inflating: certs/kibana/sample-kibana.yml root@securitynik-monitoring:/usr/share/elasticsearch# ls certs/ elasticsearch kibana
Time to copy "certs/elasticsearch/http.p12" file to "/etc/elasticsearch/" folder.
root@securitynik-monitoring:~# cp /usr/share/elasticsearch/certs/elasticsearch/http.p12 /etc/elasticsearch/ -v '/usr/share/elasticsearch/certs/elasticsearch/http.p12' -> '/etc/elasticsearch/http.p12'
I then added the following lines to the bottom of my "elasticsearch.yml".
# This turns on SSL for the HTTP (Rest) interface xpack.security.http.ssl.enabled: true # This configures the keystore to use for SSL on HTTP xpack.security.http.ssl.keystore.path: "http.p12"
For consistency I also rename the "elasticsearch-ca.pem" file to "SecurityNik-CA.pem"
root@securitynik-monitoring:/usr/share/elasticsearch/certs# cd /etc/kibana/ root@securitynik-monitoring:/etc/kibana# mv elasticsearch-ca.pem SecurityNik-CA.pem root@securitynik-monitoring:/etc/kibana#
Next the file "kibana/SecurityNik-ca.pem" was copied to the "/etc/kibana" folder
root@securitynik-monitoring:/usr/share/elasticsearch/certs# cp /usr/share/elasticsearch/certs/kibana/SecurityNik-ca.pem /etc/kibana/ -v '/usr/share/elasticsearch/certs/kibana/SecurityNik-ca.pem' -> '/etc/kibana/SecurityNik-ca.pem'
Next I modified my "kibana.yml" file changing "elasticsearch.hosts: ["http://10.0.0.1:9200"]" to "elasticsearch.hosts: ["https://10.0.0.1:9200"]". Note the https. I also added "elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/SecurityNik-CA.pem" ]"
With these configurations now out of the way, let's restart both Elasticsearch and Kibana.
root@securitynik-monitoring:/etc/kibana# systemctl stop elasticsearch.service root@securitynik-monitoring:/etc/kibana# systemctl start elasticsearch.service root@securitynik-monitoring:/etc/kibana# systemctl stop kibana.service root@securitynik-monitoring:/etc/kibana# systemctl start kibana.service
Let's now configure Kibana so that we connect to it via HTTPS.
First I copy the "10.0.0.1.p12" node certificate which was previously created to the Kibana folder.
root@securitynik-monitoring:~# cp /etc/elasticsearch/10.0.0.1.p12 /etc/kibana/ -v '/etc/elasticsearch/10.0.0.1.p12' -> '/etc/kibana/10.0.0.1.p12'Once again, modifying "kibana.yml", we add the following lines.
server.ssl.keystore.path: "/etc/kibana/10.0.0.1.p12" server.ssl.enabled: true server.ssl.keystore.password: ""
Once completed, restarted Kibana
root@securitynik-monitoring:/etc/kibana# systemctl stop kibana.service root@securitynik-monitoring:/etc/kibana# systemctl start kibana.service
Before we login, let's setup the default users using "elasticsearch-setup-passwords".
root@securitynik-monitoring:~# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive --silent Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana_system]: Reenter password for [kibana_system]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]:
Now that the users are setup, the Kibana configuration file must now reflect the values for username and password for "kibana_system" account.
root@securitynik-monitoring:~# cat /etc/kibana/kibana.yml | grep "kibana_system" --after-context 1 elasticsearch.username: "kibana_system" elasticsearch.password: "WelcomeToSecurityNikElastic"
Below represents some of those changes I've made to the "kibana.yml"
root@securitynik-monitoring:~# tail --lines 8 /etc/kibana/kibana.yml # Below added by Nik server.ssl.enabled: true elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/SecurityNik-CA.pem"] server.ssl.keystore.path: "/etc/kibana/10.0.0.1.p12" server.ssl.keystore.password: "" xpack.encryptedSavedObjects.encryptionKey: 'fhjskloppd678ehkdfdlliverpoolfcr' xpack.security.session.idleTimeout: "30m" xpack.security.session.lifespan: "8h"
With those changes in place, we should now be good to go.
Let's authenticate to Kibana using the username and password we setup for the "elastic" user. Once authenticated, you should create a few additional users based on their roles.
Once you authenticate successfully, you should now see the following.
Before we go, if you remember, when we started, we were able to see the clear text data crossing the wire. If we run tcpdump again, we don't see any data in the clear.
root@n3-monitoring:/etc/elasticsearch# tcpdump -nnti any host 10.0.0.1 and port 9200 -A -c 5 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes IP 10.0.0.1.36344 > 10.0.0.1.9200: Flags [P.], seq 2879060748:2879061003, ack 1452891267, win 512, options [nop,nop,TS val 2724200057 ecr 2724168569], length 255 E..3y.@.@.>...........#.....V.\......~..... ._.y._.y.........@.../....Ls}..E... .l.;........x.^..u..3S...A.,.O...2e.h...../.).*@.a15ou.u.^D...~.3..Hy\W'.../....D.ahL..H...q;...j....$..L.0<%J..._..k]..TMQj.B m....n.id.O....5.....S...=1.Iq.|Ox...}.mOP......z@..L.&...&X.>....R.u'I.-U*!|.Z..._.Z.[%7=...K.6m... IP 10.0.0.1.9200 > 10.0.0.1.36344: Flags [P.], seq 1:238, ack 255, win 512, options [nop,nop,TS val 2724200061 ecr 2724200057], length 237 E..!..@.@...........#...V.\..........l..... ._.}._.y..............{.m6!....ZK^... .j..$-.......4(..\Y6...y=j.....T0v..J.yG..8.I.....'.+%w.6...y..z...r...*K..L......t..cc.Pk.\..RZ.%.N.....l...Zq......qw.W..!05."...{...I..g..)\(.H..........Q...H..<X...1...X4.W..). .eoB...w..i.....[......+.W IP 10.0.0.1.36344 > 10.0.0.1.9200: Flags [.], ack 238, win 511, options [nop,nop,TS val 2724200061 ecr 2724200061], length 0 E..4y.@.@.?...........#.....V.]p........... ._.}._.} IP 10.0.0.1.36294 > 10.0.0.1.9200: Flags [.], ack 2500636561, win 512, options [nop,nop,TS val 2724200281 ecr 2724199268], length 0 E..43.@.@.............#.3.1................ ._.Y._.d IP 10.0.0.1.9200 > 10.0.0.1.36294: Flags [.], ack 1, win 512, options [nop,nop,TS val 2724200281 ecr 2724199268], length 0 E..4..@.@...........#.......3.1............
At this point, it is time to move on, as we were able to provide some basic security to Kibana and Elastic.
Posts in this series:
Security On The Cheap - Beginning Elastic Stack - Installing Elastic 7.9 on Ubuntu 20.04
Security On The Cheap - Beginning Elastic Stack - Installing Kibana 7.9 on Ubuntu 20.04
Security On The Cheap - Beginning Elastic Stack - Providing Basic Security to Elastic and Kibana 7.9 communication on Ubuntu 20.04
Security On The Cheap - Beginning Elastic - Installing and Providing Basic Security to Metricbeat - Elastic Stack 7.9 on Ubuntu 20.04
Security On The Cheap - Beginning Elastic - Installing and Providing Basic Security to Auditbeat - Elastic Stack 7.9 on Ubuntu 20.04
Security On The Cheap - Beginning Elastic - Installing and Providing Basic Security to Filebeat - Elastic Stack 7.9 on Ubuntu 20.04
Beginning Elastic - Installing, Configuring and Providing Basic Security to Packetbeat
Security On The Cheap - Beginning Elastic - Installing and Providing Basic Security to Winlogbeat
References:
https://www.elastic.co/blog/getting-started-with-elasticsearch-security
https://www.elastic.co/guide/en/elasticsearch/reference/7.8/configuring-tls.html#node-certificates
https://www.elastic.co/guide/en/elasticsearch/reference/current/get-started-built-in-users.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/get-started-kibana-user.html
https://www.youtube.com/watch?v=nMh1HWWe6B4&feature=youtu.be
https://www.elastic.co/guide/en/elasticsearch/reference/current/trb-security-sslhandshake.html
https://www.elastic.co/guide/en/kibana/7.9/using-kibana-with-security.html
