Monday, October 13, 2014

Splunk Backup Script -

A while back I was working on a Splunk setup and needed to automate the backup and restore process. Since Splunk does not do this natively I thought I should whip-up something quickly. Currently going through the latest Splunk Admin guide (6.1.4) and noticed that there was still no option for automating the backup process. As a result I figured I should share this script as someone may either use it as it is or perfect it. Whichever works for you is fine with me.

The Backup Process
The script backups "/opt/splunk/etc" to the "/home/admin/s_backups" folder. Obviously you can change this. Maybe one of these days I will rewrite the script to allow you to choose your path. However, when this was done was to address a specific need.
It creates a log file of every backup which is done. The events are written to "/home/admin/s_backups/backup.log"

Sample backup log:
Sun Oct 12 21:25:48 2014: info:Backup /home/admin/s_backups/Sun_Oct_12_21_25_42_2014_BACKUP.tar.gz Completed successfully :root pid:19379 

Sample backup file "Sun_Oct_12_21_25_42_2014_BACKUP.tar.gz"

The Restore Process
The restore process restores the previous backup files by extracting the tar.gz file
It then replaces the existing "/opt/splunk/etc" folder
It then write a log entry similar to the backup entry

You will need to specify your mail relay server
You will need to specify a receiving email 
Once the backup/restore process is completed, an email is sent to the address previously specified

Hope this script helps someone else

#!/usr/bin/env python
# This script makes a backup of the splunk 'etc' folder located in /opt/splunk/etc
# which contains all the Splunk configuration. It does not backup any data
# Author Nik Alleyne, CISSP | GCIA | SFCA 
# Email < nikalleyne at gmail dot com >
# March 2014

# To run this script manually, simple put an argument after the script. Any argument would do
# eg ./ --backup
# another example  even ./ 0 will work also

import os
import shutil
import subprocess
import sys
import time
import tarfile

import smtplib
import email.utils
from email.mime.text import MIMEText

# Check the OS to ensure if is Linux
def check_os():
    if ( == 'posix' ) and ( os.uname()[0] == 'Linux' ):
        print('=' * 50)
        #print(' Supported OS ')
        print(' :: OS: %s :: Host: %s \n :: User: %s :: PID:%s \n :: Time: %s ' %(os.uname()[0],os.uname()[1],os.getlogin(),os.getpid(),time.ctime()))
        print('=' * 50)   
        print(' While this script may work on other platforrms \n it was designed for Linux.')

# Build the menu
def menu():
    print('=' * 50)
    print(' Welcome to the Splunk Backup Script ')
    print(' Powered by SecurityNik ')
    print('=' * 50)
    print(' 1. Backup The System Config')
    print(' 2. Restore System Config ')
    print(' 3. Press any other key to exit  ')
    choice = raw_input(' \n Enter Selection:  ')

    #Read the choice and process
    if choice == '1':
        #print(' Beginning the backup process ... ')
    elif choice == '2':
        #print(' \n Beginning the restore process ...')
        print(' \n Exiting ... ')

# Do the work for backing up the configs
def _backup():
    BACKUP_DIR = '/home/admin/s_backups'
    BACKUP_LOG = '/home/admin/s_backups/backup.log'
    BACKUP_SRC = '/opt/splunk/etc'
    #print(' Beginning the Backup Process ')
    print(' \n Checking for backup directory ... %s ' %BACKUP_DIR)

    if ( os.path.exists(BACKUP_DIR) ) and ( os.path.isdir(BACKUP_DIR) ):
        print(' Backup directory found ')
        print(' Backup directory not found ')
        print(' Creating backup directory ...  %s '%BACKUP_DIR)
            f = open(BACKUP_LOG,'w')
            print(' Backup directory successfully created ')
            print(' An error occurred while creating the directory ... ')
            print(' Maybe a file currectly exists with this name in the /home/admin directory')
            print(' Try creating the directory %s manually and rerun the script ' %(BACKUP_DIR))
    # Write Backup information to log file
    f = open(BACKUP_LOG,'a')
    f.write('%s: info:Backup Started by user:%s pid:%s \n'%(time.ctime(),os.getlogin(),os.getpid()))

    print('\n Gathering files for backup .... ')
    print(' The following files and directories will be backuped up ')
    for root, dirs, files in os.walk(BACKUP_SRC):
        for d in dirs:
            print('%s' %root)
            for f in files:
                print('%s' %(os.path.join(root,f)))

    # Let's tar and zip the files in the /opt/splunk/etc folder
        BACKUP = + '/'+ '_'.join('_'.join(time.ctime().split()).split(':')) + '_BACKUP.tar.gz','w:gz')
        BACKUP.add(BACKUP_SRC, arcname='etc')
        print(' \n\n Backup completed successfully \n Backup stored in %s ' %(

        f = open(BACKUP_LOG,'a')
        f.write('%s: info:Backup %s Completed successfully :%s pid:%s \n'%(time.ctime(),,os.getlogin(),os.getpid()))
        print(' An error occurred during the backup process. ')
        f = open(BACKUP_LOG,'a')
        f.write('%s: info:Backup %s ERROR!!! Backup not completed successfully :%s pid:%s \n'%(time.ctime(),,os.getlogin(),os.getpid()))


def _restore():
    i = 0
    BACKUPS = {}
    BACKUP_DIR = '/home/admin/s_backups'
    BACKUP_LOG = '/home/admin/s_backups/backup.log'
    RESTORE_DIR = '/opt/splunk/'
    print(' Beginning the Restore Process ')
    print(' Locating backup directory ')

    # Write Restore information to log file
    f = open(BACKUP_LOG,'a')
    f.write('%s: info:Restore Started by user:%s pid:%s \n'%(time.ctime(),os.getlogin(),os.getpid()))

    if ( os.path.exists(BACKUP_DIR) and os.path.isdir(BACKUP_DIR)):
        print(' Backup dir found %s \n' %BACKUP_DIR )
        print(' Could not locate backup dir %s' %BACKUP_DIR)
        print(' You can also manually use tar to extract the file ')
        print(' Exiting ')
    for tar_file in os.listdir(BACKUP_DIR):
        if tar_file.endswith('_BACKUP.tar.gz'):
            i = i + 1
            BACKUPS[i] = tar_file
#    print(BACKUPS)
    for bak_no, bak_file in BACKUPS.items():
        print(' %d : %s' %(bak_no,bak_file))
    restore_choice = raw_input(' Please select a backup number:')
    if BACKUPS.has_key(int(restore_choice)):
        print(' Preparing to restore  %s ' %(BACKUPS.get(int(restore_choice))))
        RESTORE_FILE =  BACKUPS.get(int(restore_choice))

        print(' Not a valid backup \n Exiting ... ')

    try:['/opt/splunk/bin/splunk', 'stop'])
        if os.path.exists(RESTORE_DIR +'/etc.OLD'):
            shutil.rmtree(RESTORE_DIR +'/etc.OLD')
        os.rename(RESTORE_DIR +'/etc', RESTORE_DIR + '/etc.OLD')
        RESTORE =, 'r:gz')

        f = open(BACKUP_LOG,'a')
        f.write('%s: info:Restore %s Completed successfully :%s pid:%s \n'%(time.ctime(),RESTORE_FILE,os.getlogin(),os.getpid()))
        print('\n\n Restore completed successfully ')['/opt/splunk/bin/splunk', 'start'])
        f = open(BACKUP_LOG,'a')
        f.write('%s: info:Restore %s Failed :%s pid:%s \n'%(time.ctime(),RESTORE_FILE,os.getlogin(),os.getpid()))
        print('\n\n Restore Failed to complete for %s ' %RESTORE_FILE)

def _mailer():
    BACKUP_LOG = '/home/admin/s_backups/backup.log'   
    SEND_FROM = 'SecurityNik Splunk <>'
    SEND_TO = 'Splunk Receiver <>'

    print(' Sending Mail ...')
    f = open(BACKUP_LOG)
    for line in f.readlines():

    msg = MIMEText(' Backup/Restore notification \n' + line + '\n Powered by SecurityNik')
    msg['From'] = SEND_FROM
    msg['To'] = SEND_TO
    msg['Subject'] = 'Splunk Backup/Restore Notification'

    _mailer_send = smtplib.SMTP('localhost')
    _mailer_send.sendmail(SEND_FROM, [SEND_TO], msg.as_string())


def main():'clear')

    if len(sys.argv) == 1:
        print(' Running in Automated Mode ... This mode works with cron. A cron job needs to be setup for this to be used')


if __name__ == '__main__':
MD5: 6598a5c8f6de293b2a032972847e7288
SHA1: 6e6a515ead39151873cb8b23a3d43c836f565cdc


  1. brillant piece of information, I had come to know about your web-page from my friend hardkik, chennai,i have read atleast 9 posts of yours by now, and let me tell you, your webpage gives the best and the most interesting information. This is just the kind of information that i had been looking for, i'm already your rss reader now and i would regularly watch out for the new posts, once again hats off to you! Thanx a million once again, Regards,splunk training in hyderabad

  2. One could argue that testing small, well-defined sub-components (APIs) of huge TCP servers is the best way to go here. Unfortunately covering with fuzzing routines every parsing library and routine exposed to external users inside behemoths of a size of Apache HTTPD or ISC BIND would be a long and tiresome task. What is more, there's no guarantee of achieving 100% code coverage here, as identifying every piece of code which can touch/parse user-supplied HTTP headers, POST content, HT Serious Security Alarms