Monday, October 13, 2014

Splunk Backup Script -

A while back I was working on a Splunk setup and needed to automate the backup and restore process. Since Splunk does not do this natively I thought I should whip-up something quickly. Currently going through the latest Splunk Admin guide (6.1.4) and noticed that there was still no option for automating the backup process. As a result I figured I should share this script as someone may either use it as it is or perfect it. Whichever works for you is fine with me.

The Backup Process
The script backups "/opt/splunk/etc" to the "/home/admin/s_backups" folder. Obviously you can change this. Maybe one of these days I will rewrite the script to allow you to choose your path. However, when this was done was to address a specific need.
It creates a log file of every backup which is done. The events are written to "/home/admin/s_backups/backup.log"

Sample backup log:
Sun Oct 12 21:25:48 2014: info:Backup /home/admin/s_backups/Sun_Oct_12_21_25_42_2014_BACKUP.tar.gz Completed successfully :root pid:19379 

Sample backup file "Sun_Oct_12_21_25_42_2014_BACKUP.tar.gz"

The Restore Process
The restore process restores the previous backup files by extracting the tar.gz file
It then replaces the existing "/opt/splunk/etc" folder
It then write a log entry similar to the backup entry

You will need to specify your mail relay server
You will need to specify a receiving email 
Once the backup/restore process is completed, an email is sent to the address previously specified

Hope this script helps someone else

#!/usr/bin/env python
# This script makes a backup of the splunk 'etc' folder located in /opt/splunk/etc
# which contains all the Splunk configuration. It does not backup any data
# Author Nik Alleyne, CISSP | GCIA | SFCA 
# Email < nikalleyne at gmail dot com >
# March 2014

# To run this script manually, simple put an argument after the script. Any argument would do
# eg ./ --backup
# another example  even ./ 0 will work also

import os
import shutil
import subprocess
import sys
import time
import tarfile

import smtplib
import email.utils
from email.mime.text import MIMEText

# Check the OS to ensure if is Linux
def check_os():
    if ( == 'posix' ) and ( os.uname()[0] == 'Linux' ):
        print('=' * 50)
        #print(' Supported OS ')
        print(' :: OS: %s :: Host: %s \n :: User: %s :: PID:%s \n :: Time: %s ' %(os.uname()[0],os.uname()[1],os.getlogin(),os.getpid(),time.ctime()))
        print('=' * 50)   
        print(' While this script may work on other platforrms \n it was designed for Linux.')

# Build the menu
def menu():
    print('=' * 50)
    print(' Welcome to the Splunk Backup Script ')
    print(' Powered by SecurityNik ')
    print('=' * 50)
    print(' 1. Backup The System Config')
    print(' 2. Restore System Config ')
    print(' 3. Press any other key to exit  ')
    choice = raw_input(' \n Enter Selection:  ')

    #Read the choice and process
    if choice == '1':
        #print(' Beginning the backup process ... ')
    elif choice == '2':
        #print(' \n Beginning the restore process ...')
        print(' \n Exiting ... ')

# Do the work for backing up the configs
def _backup():
    BACKUP_DIR = '/home/admin/s_backups'
    BACKUP_LOG = '/home/admin/s_backups/backup.log'
    BACKUP_SRC = '/opt/splunk/etc'
    #print(' Beginning the Backup Process ')
    print(' \n Checking for backup directory ... %s ' %BACKUP_DIR)

    if ( os.path.exists(BACKUP_DIR) ) and ( os.path.isdir(BACKUP_DIR) ):
        print(' Backup directory found ')
        print(' Backup directory not found ')
        print(' Creating backup directory ...  %s '%BACKUP_DIR)
            f = open(BACKUP_LOG,'w')
            print(' Backup directory successfully created ')
            print(' An error occurred while creating the directory ... ')
            print(' Maybe a file currectly exists with this name in the /home/admin directory')
            print(' Try creating the directory %s manually and rerun the script ' %(BACKUP_DIR))
    # Write Backup information to log file
    f = open(BACKUP_LOG,'a')
    f.write('%s: info:Backup Started by user:%s pid:%s \n'%(time.ctime(),os.getlogin(),os.getpid()))

    print('\n Gathering files for backup .... ')
    print(' The following files and directories will be backuped up ')
    for root, dirs, files in os.walk(BACKUP_SRC):
        for d in dirs:
            print('%s' %root)
            for f in files:
                print('%s' %(os.path.join(root,f)))

    # Let's tar and zip the files in the /opt/splunk/etc folder
        BACKUP = + '/'+ '_'.join('_'.join(time.ctime().split()).split(':')) + '_BACKUP.tar.gz','w:gz')
        BACKUP.add(BACKUP_SRC, arcname='etc')
        print(' \n\n Backup completed successfully \n Backup stored in %s ' %(

        f = open(BACKUP_LOG,'a')
        f.write('%s: info:Backup %s Completed successfully :%s pid:%s \n'%(time.ctime(),,os.getlogin(),os.getpid()))
        print(' An error occurred during the backup process. ')
        f = open(BACKUP_LOG,'a')
        f.write('%s: info:Backup %s ERROR!!! Backup not completed successfully :%s pid:%s \n'%(time.ctime(),,os.getlogin(),os.getpid()))


def _restore():
    i = 0
    BACKUPS = {}
    BACKUP_DIR = '/home/admin/s_backups'
    BACKUP_LOG = '/home/admin/s_backups/backup.log'
    RESTORE_DIR = '/opt/splunk/'
    print(' Beginning the Restore Process ')
    print(' Locating backup directory ')

    # Write Restore information to log file
    f = open(BACKUP_LOG,'a')
    f.write('%s: info:Restore Started by user:%s pid:%s \n'%(time.ctime(),os.getlogin(),os.getpid()))

    if ( os.path.exists(BACKUP_DIR) and os.path.isdir(BACKUP_DIR)):
        print(' Backup dir found %s \n' %BACKUP_DIR )
        print(' Could not locate backup dir %s' %BACKUP_DIR)
        print(' You can also manually use tar to extract the file ')
        print(' Exiting ')
    for tar_file in os.listdir(BACKUP_DIR):
        if tar_file.endswith('_BACKUP.tar.gz'):
            i = i + 1
            BACKUPS[i] = tar_file
#    print(BACKUPS)
    for bak_no, bak_file in BACKUPS.items():
        print(' %d : %s' %(bak_no,bak_file))
    restore_choice = raw_input(' Please select a backup number:')
    if BACKUPS.has_key(int(restore_choice)):
        print(' Preparing to restore  %s ' %(BACKUPS.get(int(restore_choice))))
        RESTORE_FILE =  BACKUPS.get(int(restore_choice))

        print(' Not a valid backup \n Exiting ... ')

    try:['/opt/splunk/bin/splunk', 'stop'])
        if os.path.exists(RESTORE_DIR +'/etc.OLD'):
            shutil.rmtree(RESTORE_DIR +'/etc.OLD')
        os.rename(RESTORE_DIR +'/etc', RESTORE_DIR + '/etc.OLD')
        RESTORE =, 'r:gz')

        f = open(BACKUP_LOG,'a')
        f.write('%s: info:Restore %s Completed successfully :%s pid:%s \n'%(time.ctime(),RESTORE_FILE,os.getlogin(),os.getpid()))
        print('\n\n Restore completed successfully ')['/opt/splunk/bin/splunk', 'start'])
        f = open(BACKUP_LOG,'a')
        f.write('%s: info:Restore %s Failed :%s pid:%s \n'%(time.ctime(),RESTORE_FILE,os.getlogin(),os.getpid()))
        print('\n\n Restore Failed to complete for %s ' %RESTORE_FILE)

def _mailer():
    BACKUP_LOG = '/home/admin/s_backups/backup.log'   
    SEND_FROM = 'SecurityNik Splunk <>'
    SEND_TO = 'Splunk Receiver <>'

    print(' Sending Mail ...')
    f = open(BACKUP_LOG)
    for line in f.readlines():

    msg = MIMEText(' Backup/Restore notification \n' + line + '\n Powered by SecurityNik')
    msg['From'] = SEND_FROM
    msg['To'] = SEND_TO
    msg['Subject'] = 'Splunk Backup/Restore Notification'

    _mailer_send = smtplib.SMTP('localhost')
    _mailer_send.sendmail(SEND_FROM, [SEND_TO], msg.as_string())


def main():'clear')

    if len(sys.argv) == 1:
        print(' Running in Automated Mode ... This mode works with cron. A cron job needs to be setup for this to be used')


if __name__ == '__main__':
MD5: 6598a5c8f6de293b2a032972847e7288
SHA1: 6e6a515ead39151873cb8b23a3d43c836f565cdc

My Top 5 Computer Future Security Issues

Data Breaches

“A breach is defined as an event in which an individual’s name plus a medical record and/or a financial record or debit card is potentially put at risk—either in electronic or paper format.” (Ponemon, 2014)

For the first half of 2014, there were a reported 1,331 incidents. As a result of these incidents, there were a reported 502 million records exposed (, 2014).  For the entire 2013, there were a reported 2308 incidents (, 2014). While it is generally accepted that sooner or later an organization will be breached, the frequency with which these occur and the number of records involved is what is astounding.

Interestingly, these breaches are not limited to a specific country and or industry. Figure 1 below shows the number of breaches experienced by countries while Figure 2 shows by Industry during the period May 2013 to May 2014.

Figure 1
Source: Pomenon Institue

Figure 2:

Source: Pomenon Institue 

Identity Theft/Fraud

Identify theft is referred to as the “preparatory stage of acquiring and collecting someone else’s personal information for criminal purposes” ( Identity Fraud on the other hand, is defined as “the unauthorized use of another person’s personal information to achieve illicit financial gain” (, 2014)

With the massive amount of breached records and the known proclivity for cyber criminals to ensure they are compensated for their illegal activities one can only conclude that sooner or later, these records will be on the blackmarket. Some sites from which credit card and other personal information can be bought are rescator.* (.cm, .la. and .so), kaddaf[dot]hk, octavian[dot]su and cheapdumps[dot]org (Krebs, 2013).

More importantly, It is reported that there was A New Identity Fraud Victim Every Two Seconds in 2013” (, 2014). Of greater importance, one in 3 people who received data breach notification letter were victims of identity fraud (, 2014).

The graph below shows millions of Identity Theft Victims

Figure 3:


Human Factor

Whether it is a user that has clicked on a link via a phishing email or an administrator that misconfigured a firewall, the human factor plays a tremendous role in the security threats paradigm. It is reported that 30% of all data breaches is a result of Human Error (Ponemon, 2014).

Figure 4:


Organizations such as the SANS Institute have recognize the importance of the role of humans in IT Security and thus have implemented programs based on “Securing the Human” (  It is through these programs that while everyone cannot be made an expert in IT security, everyone can at least be made knowledgeable about some of the dangers related to technology. Through its’ OUCH newsletter, SANS has also produced a free document which each explains a specific topic and the necessary actions people can take to protect themselves (, 2014)

Mobile/Wearable Malware

It is predicted that in 2015, 87% of connected devices sales will be tablets and smart phones (, 2013). This immediately implies that most of our online activities will be done via a smart phone and or tablet.  To be able to effectively use these devices, an operating system is required. It is estimated that Android owns 76% of this market with iOS at 14.4% as shown in the figure below.

Figure 5:

Source: MobileThinking

The threat comes not necessarily from these devices themselves but from the underlying OS which they used. According to the F-Secure, for Q1 2014, there were 275 threat families (malware) that run on Android, 1 for iPhone and 1 for Symbian (F-Secure, 2014). 

In addition, the advent of Smart Watches, google glass, fitness tracking bands and other wearables, makes for an even more interesting mobile future.

Internet of Things
As we continue to march towards the future, by biggest fears lies in what else we may choose to connect to the Internet.

The Internet of things is considered as a network of physical objects which are accessed through the Internet. Through the Internet of things, connections can be made between manufacturing floors, energy grids, healthcare facilities, transportations systems to the Internet (

It is reported at a staggering 30 Billion devices will connect wirelessly to the Internet of Everything in 2020 (, 2013).  If we contrast this with the world’s population which currently stands at 7.2 Billion (, we can conclude that each person will be responsible for at least 4 devices in 2020. Our rush to have everything inter-connected will provide us with a very interesting future.


(n.d.). Retrieved from
(n.d.). Retrieved from
(n.d.). Retrieved from
(n.d.). Retrieved from
(2013, 9 11). Retrieved from
(2013, 05 09). Retrieved from
(2014, 08 24). Retrieved from
(2014, 08 25). Retrieved from
(2014, 02 5). Retrieved from,pressRoomDetail
(2014, 08). Retrieved from
F-Secure. (2014). Mobile Threat Report Q1. F-Secure.
Krebs, B. (2013, 12 13). Retrieved from
Ponemon. (2014). 2014 Cost of Data Breach Study: Global ANalysis. Ponemon Institute.
ZIOBRO, P. (n.d.). Retrieved from