BIG SAVINGS - Free GIAC Exam or Scholarship Seat - SANS SEC 503 Mentor Session, Toronto, ON | Thu Sep 17 - Thu Nov 19, 2015

Even More reasons to register. Choose between a scholarship seat where you pay $2,700 (a saving of $1,460) limited time offer or a free exam worth over $1,000, when combined with early bird discount. Now that's some BIG saving for SANS training.

Use registration code "MenScholar15" when registering to receive your scholarship seat.

Use registration code "MGIAC15" when registering to receive your GIAC exam at no extra cost. If you would like to do the training but not the exam, we are also hear to accommodate you. You can receive a discount of $629 off the course tuition, in place of the GIAC exam.

Register early, to save $400 on early bird registration.

For further information please visit http://www.sans.org/mentor/class/sec503-toronto-17sep2015-nik-alleyne

The long and short term effects of the Target breach on the retail sector

About Target

Target Corporation was incorporated in Minnessota in 1902 and currently offers everyday essentials and fashionable differentiated merchandise at discounted prices. Target relies strongly on its technology infrastructure to deliver a preferred shopping experience. Prior to the first quarter of 2013, Target operated in the US credit card segment, providing credit to its Guests. Additionally, until January 15, 2015 Target also operated within the Canadian segment  (Target Corporation, 2015).

About the data breach

            During the fourth quarter of 2013, Target experienced a significant data breach which impacted its ability to handle customer enquiries resulting in weaker than expected sales along with the loss of certain payment card and guest information from its network (Target Corporation, 2015). The information which was taken included names, email addresses, phone numbers or mailing addresses and has affected up to 70 million individuals. Additionally the breach has impacted approximately 40 million credit and debit card accounts (corporate.target.com, n.d.).

Short term effects

            From Target’s perspective, as of January 31, 2015 the financial impact of the data breach was costly as it has incurred net cumulative breach expenses of $162 million. In addition, fraudsters and scam artists were using the opportunity to target Target’s customers via various attack vectors such as phishing, social engineering, smishing and pretexting. In addition, Target took a number of initiatives to enhance monitoring and logging, create application whitelisting of point-of-sale systems, implementation of enhanced segmentation, reviewed and limited vendor access while resetting passwords of 445,000 within its infrastructure while also broadening the use of two-factor authentication (corporate.target.com, 2014).  All these initiatives may be part of the $162 million Target spent as a result of the data breach.

Even though the above is directly related to Target, the short term effect on the industry has seen more companies allocate resources towards preventing, detecting and resolving cyber breaches. Cyber Security budgets have increased by 34 percent, with fifty percent of this going towards Security Incident and Event Management (SIEM) tools, forty eight percent to Endpoint Security, forty four percent going to Intrusion Prevention and Detection technologies and thirty eight percent towards encryption and tokenization (Ponemon, 2015). While the companies have also focused on technologies, Senior Management in general has shown extremely high concern to Cyber Security incidents (Ponemon, 2015).

While companies may be in a position to implement new processes and hire new people to deploy and or maintain the latest and or greatest technologies, the same cannot be readily said about individuals. In the short term individuals need to be vigilant, when receiving any communications purporting to be from Target and or any other breached entity. As stated above, fraudsters have chosen techniques such as phishing, social engineering, smishig and pretexting to dupe potential victims. Individuals may help to protect themselves by at a minimum, having all patches and relevant updates installed on their computer, while also having up-to-date antivirus or other anti-malware tools.

Long Term effects

In the long term, greater effort would be placed on the strategic importance of IT security to the operations of the companies. This would result in more focus being placed on the people and the process rather than on the technology. As an example, companies are now focusing on the creation of an Incident Response Team, training and awareness activities, measurement of data security effectiveness, policies and procedures along with specialized education for IT security staff (Ponemon, 2015).

In addition, as in the case with Target, companies may face court action from customers, card issuing banks, shareholders and or other individuals seeking relief (Target Corporation, 2015).

While it is expected that companies will continue to invest in people, processes and technologies, they will also look into buying insurance to cover any costs which may be associated with a data breach on its infrastructure. Target maintains a $100 million network security insurance coverage, above a $10 million deductible. It is believed that this coverage along with other insurance may reduce its exposure to cyber attacks (Target Corporation, 2015). In the next 5 years a larger amount of services will be conducted online. This increase in online activity will result in more companies looking to lower the risk from cyber attacks (ibisworld.com, 2014)

More importantly, as hacks similar to Target continues to occur, greater efforts will be made towards information sharing, so as to reduce the probability (and effects) of the breaches propagating beyond its initial target. In his Cyber Security Legislative Proposal, President Obama has included voluntary information sharing with Industry, States, and Local Government as one of the key methods for protecting the security of the United States (US) digital infrastructure (whitehouse.gov, 2011).

Finally and most importantly, governments can and may enact laws which controls how companies respond to data breaches. At present, a number of US states and some of its territories have enacted data breach laws, which dictates who must comply with the law, what constitutes a breach, requirements for notice along with exemptions (ncsl.org, 2015). At the federal level, the proposed Data Security and Breach Notification Act of 2015 requires entities that owns or possess data in electronic form containing personal information to notify affected individuals and a designated government entity (FTC,  Secret Service, FBI, Attorney General, etc ) no later than 30 days after the date of discovery of a breach. In the event of a breach, any individual who intentionally and willingly conceals the fact that a breach occurred and there is damage greater than $1000 may be fined and or imprisoned for not more than 5 years (congress.gov, 2015).

Conclusion

In the long term, hacks like target will cause both individuals and companies to place more attention on cyber security. Companies though will be the ones that ultimately need to make the investments to secure our personal data. The world is becoming more networked and big data is the ultimate objective for most of these companies. They more data is had, the more companies can predict your next move. However, how they protect this data will be the ultimate concern. Action by governments is needed to ensure that all companies respond consistently whenever a breach occurs.

References

congress.gov. (2015, 1 13). S.177 - Data Security and Breach Notification Act of 2015. Retrieved from https://www.congress.gov/bill/114th-congress/senate-bill/177/text

corporate.target.com. (2014, 04 29). updates on Target’s security and technology enhancements. Retrieved from corporate.target.com: https://corporate.target.com/discover/article/updates-on-Target-s-security-and-technology-enhanc

corporate.target.com. (n.d.). data breach FAQ. Retrieved from corporate.target.com: https://corporate.target.com/about/shopping-experience/payment-card-issue-faq

ibisworld.com. (2014, 08). Cyber Liability Insurance in the US: Market Research Report. Retrieved from ibisworld.com: http://www.ibisworld.com/industry/cyber-liability-insurance.html

ncsl.org. (2015, 12 1). Security Breach Notification Laws. Retrieved from http://www.ncsl.org/research/telecommunications-and-information-technology/security-breach-notification-laws.aspx

Ponemon. (2015). 2014: A Year of Mega Breaches. Ponemon Institute LLC.

Target Corporation. (2015). Annual Report . Minneapolis.

whitehouse.gov. (2011, May 12). FACT SHEET: Cybersecurity Legislative Proposal. Retrieved from https://www.whitehouse.gov/the-press-office/2011/05/12/fact-sheet-cybersecurity-legislative-proposal

QRadar - Threat Intel On the Cheap - Checking our environment for suspect IPs

In this our final post in this series, we will search our logs and flows to see if any of the suspected IPs.

P.S. Point to note is the quality of these list are dependent on the people who publish them. I give no warranty or am I vouching for these lists. These IPs and or domains should be used as a starting point of your investigation not the ultimate decision as to whether something good or bad has happened.

To search our logs let's do the following:
1. Login to QRadar
2. Select "Logs" or "Network Activity" tab
3. From the "Search" dropdown, select "New Search"
3. Select your "Time Range"
    Let's start with the "Last 7 Days"
4. In the "Search Parameters" choose "Reference Set"
5. In the "Value: Data Entry" select "Destination IP"
6. Operators select "Exists in any of"
7. From the "Reference Set" list, choose "SecurityNik_IP_Darklist"
8. Click the "+"
9. Click "Add Filter"
10. Click "Search"

Based on the search query we just entered, you would be able to see if any of the hosts in your environment has been or is communicating with any of the IPs in the SecurityNik_IP_Darklist.


To check for domains in the SecurityNik_DNS_Darklist, you would basically do the same steps above. However, you would chose a different reference set.

Have fun and don't forget the other posts in this series to ensure your reference set and rules are properly created.


1. The Code to download the Darklist
2. Verifying the Reference Set Creation
3. Writing the Common Rule to check for the IPs
4. Writing the Event Rule to check for the domains
5. Checking your environment for the malicious IPs and or domains.

QRadar - Threat Intelligence On The Cheap - Creating the rule to detect IPs in the SecurityNik_DNS_Darklist

Most of the information in the previous post can be used to develop the rule for detecting the malicious domains.

P.S. Point to note is the quality of these list are dependent on the people who publish them. I give no warranty or am I vouching for these lists. These IPs and or domains should be used as a starting point of your investigation not the ultimate decision as to whether something good or bad has happened.

Follow the steps below to create an Event rule:
 1.    Click "Offenses" tab
    2    Select "Rules"
    3.    Select "Action" menu
    4.    From the "Action" menu, select "New Event Rule"

Once the "Rule Wizard" opens, it should default to "Events"


Next you build your rule by selecting your criteria.
    1.    First in the search box, type "referen". This reduces the number of rule options to sort through
    2.    select the one which states "when any of these properties is contained in any of these reference set(s)"
    3.    Assign your rule a name
    4.    Click "these event properties".
    5.     When the window pops up, select the "domain"
    6.     Click "OK"
    7.    Click "these reference set(s)"
    8.    From the pop up window select the "SecurityNik_DNS_Darklist", then click add
    9.    Click "OK"
   
Selecting the "Rule Response"
    1.    Ensure that a new event gets created
    2.    you may set the severity, credibility and relevance of the new event which gets created
        Note that these setting may influence the values of an offense but does not specifically set the values on the offense
    3.    Create a new offense when the domain from the SecurityNik_DNS_Darklist reference set is seen in either an event
    4.    Send off an email once a detection is made
   
   
Verify the rule is as expected.

You should be good to go.

In the final post, we will run a query against the environment to see if any of the detected IPs (or you can do the domain) is found in our logs and or flows.


Have fun and don't forget the other posts in this series to ensure your reference set and rules are properly created.


1. The Code to download the Darklist
2. Verifying the Reference Set Creation
3. Writing the Common Rule to check for the IPs
4. Writing the Event Rule to check for the domains
5. Checking your environment for the malicious IPs and or domains.

QRadar - Threat Intelligence On The Cheap - Creating the rule to detect IPs in the SecurityNik_IP_Darklist

Now that we've downloaded our suspect IPs and domains with the SecurityNikThreatIntel script along with verifying our reference set creation, we will now create the relevant rules. Note in this post, I will focus primarily on the IP darklist rule creation. Basically the same steps can be used to create a rule to detect the malicious DNS. I will put a quick snapshot below on what needs to be done to get the rule to detect the suspect domains.


P.S. Point to note is the quality of these list are dependent on the people who publish them. I give no warranty or am I vouching for these lists. These IPs and or domains should be used as a starting point of your investigation not the ultimate decision as to whether something good or bad has happened.

Because we would like to detect the suspect IPs in either flows or events we will create a common rule.

Follow the steps below to create a common rule:
    1.    Click "Offenses" tab
    2    Select "Rules"
    3.    Select "Action" menu
    4.    From the "Action" menu, select "New Common Rule"




Once the "Rule Wizard" opens, it should default to "Events and Flows"
Next you build your rule by selecting your criteria.
    1.    First in the search box, type "referen". This reduces the number of rule options to sort through
    2.    select the one which states "when any of these properties is contained in any of these reference set(s)"
    3.    Assign your rule a name
    4.    Click "these event properties".
    5.     When the window pops up, select the "Source IP", click add, then select "Destination IP"
    6.     Click "OK"
    7.    Click "these reference set(s)"
    8.    From the pop up window select the "SecurityNik_IP_Darklist", then click add
    9.    Click "OK"

   
Selecting the "Rule Response"
    1.    Ensure that a new event gets created
    2.    you may set the severity, credibility and relevance of the new event which gets created
        Note that these setting may influence the values of an offense but does not specifically set the values on the offense
    3.    Create a new offense when the source or detination IP from the SecurityNik_IP_Darklist reference set is seen in either an event or flow
    4.    Send off an email once a detection is made

  
Verify the rule is as expected.

You should be good to go.


Have fun and don't forget the other posts in this series to ensure your reference set and rules are properly created.


1. The Code to download the Darklist
2. Verifying the Reference Set Creation
3. Writing the Common Rule to check for the IPs
4. Writing the Event Rule to check for the domains
5. Checking your environment for the malicious IPs and or domains.

 
 

QRadar - Threat Intelligence On The Cheap - Verifying Reference Set Creation

P.S. Point to note is the quality of these list are dependent on the people who publish them. I give no warranty or am I vouching for these lists. These IPs and or domains should be used as a starting point of your investigation not the ultimate decision as to whether something good or bad has happened. To verify that the reference set exists you can do the following.

    1.    Logon to QRadar
    2.    Select "Admin" Tab
    3.    Under the section "System Configuration" select "Reference Set Management"
    4.     From the Reference Set window, scroll until you find the 2 reference sets.
        a.    'SecurityNik_IP_Darklist'
        b.    'SecurityNik_DNS_Darklist'
    5.    You can now double click the list or click "Edit" to verify their contents.
   
Once you have verified that the reference sets have been successfully created and contains your IPs and or DNS Entries. You may now go ahead and create your rules.
Have fun and don't forget the other posts in this series to ensure your reference set and rules are properly created.

1. The Code to download the Darklist
2. Verifying the Reference Set Creation
3. Writing the Common Rule to check for the IPs
4. Writing the Event Rule to check for the domains
5. Checking your environment for the malicious IPs and or domains.

QRadar - Threat Intelligence on the Cheap - The Code

Ever thought of a way to get get threat intelligence information for your QRadar on the cheap. Here it is, I'm making your life and or job easier :-).

Basically this script goes out to a few websites and download lists of suspected bad IPs and Domains. From this list I then compile one list for IPs and one for DNS. 

Once I have the above lists, I then create 2 reference sets in QRadar to import the data.

Once the script runs for the first time, you will need to create your QRadar rules manually. This post addresses that issue.


P.S. Point to note is the quality of this list is dependent on the people who are putting it out. I give no warranty or am I vouching for the list. These IPs should be used as a starting point of your investigation, not the ultimate decision as to whether something good or bad has happened.

#!/usr/bin/env python
# This is code is designed to download list of known bad IPs and domains
# Once the lists have been downloaded, 2 reference sets are created
# 1 for IPs and 1 for domains
# Manual creation of QRadar rules are then done. These rules are then run against these
# list to identify known bad IPs and Domain
#
# SecurityNikThreatIntel.py v1.0
# Author: Nik Alleyne, CISSP|GCIH|A < nikalleyne at gmail.com >
# Date: 2015-02-25
# Disclaimer: In no way am I responsible for any damages which you may
# cause to your system by running this script.

from os import uname, path, system, remove, getcwd
from shutil import rmtree,copytree
from subprocess import call
from sys import exit
from time import sleep


# This function checks to see if this script is running on Linux.
def check_os():
    qRadar_path = '/opt/qradar/conf/'
    qRadar_ver = '/opt/qradar/bin/myver'

    print(' Checking OS ... ')
    if ( uname()[0] == 'Linux' ) or ( uname()[0] == 'linux'):
        #print(' Running on Linux ... ')
       
        if ( path.exists('/etc/system-release') and path.isfile('/etc/system-release') ):
            call(['cat', '/etc/system-release'])
        else:
            print('\n Looks like you are running Linux. ')
            print('\n However, I am unable to determine your version info. ')
       
        print(' \n Looking for an installed version of QRadar')
        if ( path.exists(qRadar_path) and ( path.isdir(qRadar_path)) ):
            print(' \n looks like you are running QRadar version ... ')
            call([qRadar_ver])
            print(' \n Good stuff ... \n Blast off =>>>>>>> ')
        else:
            print(' An installed version of QRadar was not found on your system ')
            print(' This script will not work for you, it was designed to be used on box running IBM QRadar ')
            print(' Exiting ... ')
            exit(0)
       
        sleep(2)
    else:
        print(' Running this is a waste of your time. ')
        print(' This script is SPECIFICALLY for QRadar ')
        exit(0)


# This function downloads a list of known bad IPs and
def grab_ip_list():
    ip_path = ''
    bad_ip_list =    ['http://malc0de.com/bl/IP_Blacklist.txt' ,
                     'http://talosintel.com/feeds/ip-filter.blf',
                    'http://malc0de.com/bl/IP_Blacklist.txt',
                    'http://www.malwaredomainlist.com/hostslist/ip.txt',
                    'https://zeustracker.abuse.ch/blocklist.php?download=badips' ,
                    'http://www.spamhaus.org/drop/drop.txt',
                    'http://www.spamhaus.org/drop/edrop.txt',
                    'http://www.spamhaus.org/drop/drop.lasso',
                    'http://www.okean.com/chinacidr.txt' ,
                    'http://myip.ms/files/blacklist/general/latest_blacklist.txt' ,
                    'http://myip.ms/files/blacklist/csf/latest_blacklist.txt' ,
                    'http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt' ,
                    'http://rules.emergingthreats.net/blockrules/compromised-ips.txt' ,  
                    'http://feeds.dshield.org/block.txt' ,
                    'http://feeds.dshield.org/top10-2.txt',
                    'http://www.dshield.org/feeds/topips.txt'
                    'https://feodotracker.abuse.ch/blocklist/?download=ipblocklist',
                    'https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist' ,
                    'https://zeustracker.abuse.ch/blocklist.php?download=badips' ,
                    ]


    # Check to see if ip_tmp/ folder exists - This folder stores the files a the first download.
    # Basically this will determine if its the first time the script is being run
    if ( path.exists('.ip_tmp/') and (path.isdir('.ip_tmp/')) ):
        ip_path = '.ip_tmp_path/'
    else:
        ip_path = '.ip_tmp/'

    try:
        print(' Preparing to download list of bad IP addresses ')
        for link in bad_ip_list:
            print(link)
            call(['wget', link, '--directory-prefix='+ip_path , '--tries=2', '--continue', '--timestamping', '--timeout=5', '--random-wait', '--no-proxy', '--inet4-only'])
            print(' \n  %s \n retrieved successfully \n' %link )
            sleep(2)
    except:
        print(' A problem occurred while downloading IP information from %s ' %link )
        print(' This link may be broken. Please copy the URL and paste into a browser to ensure it is accessible')
    else:
        # Looks like all went well
        print(' \n Looks like we have some baddddd IPs! ')



# This fuction download the list of malicious and or suspected domains
# DO NOT add entry to this list unless you are sure what you are doing
# These files are in different formats, thus may need to be manipulated the files individually

def grab_dns_list():
    dns_path = ''
    bad_dns_list =    [ 'http://www.joewein.net/dl/bl/dom-bl.txt',
                       'http://www.joewein.net/dl/bl/dom-bl-base.txt',
                       'http://mirror1.malwaredomains.com/files/immortal_domains.txt',
                       'http://mirror1.malwaredomains.com/files/dynamic_dns.txt',
                       'https://zeustracker.abuse.ch/blocklist.php?download=baddomains',
                       'http://www.malwaredomainlist.com/hostslist/hosts.txt',
                       'http://malc0de.com/bl/BOOT',
                       'http://malc0de.com/bl/ZONES'
                    ]

    if ( path.exists('.dns_tmp') and (path.isdir('.dns_tmp')) ):
        dns_path = '.dns_tmp_path'
    else:
        dns_path = '.dns_tmp'

    try:
        print(' Preparing to download list of bad Domain  ')
        for dns in bad_dns_list:
            print(dns)
            call(['wget', dns, '--directory-prefix='+dns_path , '--tries=2', '--continue', '--timestamping', '--timeout=5', '--random-wait', '--no-proxy', '--inet4-only'])
            print(' \n  %s \n retrieved successfully \n' %dns )
            sleep(2)
    except:
        print(' A problem occurred while downloading DNS information from %s ' %dns )
        print(' This link may be broken. Please copy the URL and paste into a browser to ensure it is accessible')
    else:
        # Looks like all went well
        print(' \n Looks like we have some baddddd domains! ')


# Checking the directories to see if the last run added new info
def compare_ip_dirs():
    print(' Checking if there is need for an update .... ')

    #first check to see if .ip_tmp_path exists
    if ( path.exists('.ip_tmp_path') and (path.isdir('.ip_tmp_path')) ):
        print(' Give me just a few seconds more')
        sleep(2)
  
        if ( int(path.getsize('.ip_tmp')) <= int(path.getsize('.ip_tmp_path')) ):
            print(' \n Looks like new content is available ')
            # copying new content in .ip_tmp_path to .ip_tmp
            try:
                rmtree('.ip_tmp')
                copytree('.ip_tmp_path','.ip_tmp')
            except:
                print(' Failed to copy new data ... ')
                print(' Exiting ... ')
                exit(0)
            else:
                print(' Successfully moved new data')
        else:
            print(' Nothing new was added ... ')
            print(' Exiting ... ')
            exit(0)
    else:
        print(' This is first run ... \n moving on ... ')

    sleep(2)


# Comparing the DNS folders to see if new content may have been added
def compare_dns_dirs():
    print(' Checking if there is need for an update .... ')
  
    #first check to see if .ip_tmp_path exists
    if ( path.exists('.ip_tmp_path') and (path.isdir('.ip_tmp_path')) ):
        print(' Give me just a few seconds more')
        sleep(2)
       
        if ( int(path.getsize('.ip_tmp')) <= int(path.getsize('.ip_tmp_path')) ):
            print(' \n Looks like new content is available ')
           
            # copying new content in .dns_tmp_path to .dns_tmp
            try:
                rmtree('.dns_tmp')
                copytree('.dns_tmp_path','.dns_tmp')
            except:
                print(' Failed to copy new data ... ')
                print(' Exiting ... ')
                exit(0)
            else:
                print(' Successfully moved new data')              
        else:
            print(' Nothing new was added ... ')
            print(' Exiting ... ')
            exit(0)
    else:
        print(' This is first run ... \n moving on ... ')
        sleep(2)


# Now that the files have been successfully downloaded, let's combine them all
def combine_ip_files():
    print(' \n Checking for .ip_tmp folder ... ')
    sleep(2)
    if ( path.exists('.ip_tmp') and path.isdir('.ip_tmp') ):
        print(' directory .ip_tmp/ found ')
        system('cat .ip_tmp/* | grep --perl-regexp --only-matching "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" | sort -i | uniq --unique --check-chars=15 > SecurityNikBadIPs.txt')
       
        if ( path.exists('SecurityNikBadIPs.txt') and path.isfile('SecurityNikBadIPs.txt') ):
            print(' Successfully created file SecurityNikBadIPs.txt ')
        else:
            print(' Unable to create SecurityNikBadIPs.txt file ')
            print(' The program will now exit ... Exiting ... ')
            exit(0)
    else:
        print(' \n ip_tmp/ directory not found ')
        print(' Unable to continue ... Exiting!')
        exit(0)


# This function manipulates the downloaded DNS files, so that all can be placed into one standard file
def combine_dns_files():
    print(' Combining DNS files ')
    if ( path.exists('.dns_tmp') and  path.isdir('.dns_tmp') ):
        print(' directory .dns_tmp/ found ')
        try:
            print(' Combining downloaded files into .... ')
            system('cat .dns_tmp/dom-bl.txt > .SecurityNikBadDomains.txt')
            system('cat .dns_tmp/dom-bl-base.txt >> .SecurityNikBadDomains.txt')
            system("cat .dns_tmp/hosts.txt | awk  '/127.0.0.1/ { print $2 }'  >> .SecurityNikBadDomains.txt")
            system('cat .dns_tmp/immortal_domains.txt | grep -i -P "This is a list|^$" -v >> SecurityNikBadDomains.txt')
            system('cat .dns_tmp/BOOT | grep -i PRIMARY | cut -f 2 -d " " | grep -i -v -P "ibm\.com" -v >> .SecurityNikBadDomains.txt')
            system('cat .dns_tmp/dynamic_dns.txt | grep -P -v "^#|^$" | cut -f 1 -s >> .SecurityNikBadDomains.txt')
            system('cat .dns_tmp/blocklist.php\?download\=baddomains | grep -P -v "^#|^$" >> .SecurityNikBadDomains.txt')
            system('cat .SecurityNikBadDomains.txt | sort -i | uniq --unique > SecurityNikBadDomains.txt')
       
        except:
            print(' Looks like an error occurred while combining the files')
            print(' Please retry later ... \n Exiting ... ')
            exit(0)
        else:
            print(' files successfully combined ')
            print(' A list of known bad domains can be found in SecurityNikBadDomains.txt')
            remove('.SecurityNikBadDomains.txt')

    else:
        print(' \n dns_tmp/ directory not found ')
        print(' The program will now exit ... Exiting ... ')
        exit(0)




# This function does all the work for the IP reference set
def verify_create_ip_reference_set():
    reference_set_name = 'SecurityNik_IP_Darklist'
    ip_txt = getcwd()+'/SecurityNikBadIPs.txt'
    rows = []
  
    print('Checking to see if the reference set %s already exists' %reference_set_name)
    f =open('.count.txt', 'w')
    call(["psql", "-U", "qradar", "--command=SELECT COUNT(*) FROM reference_data WHERE name='SecurityNik_IP_Darklist'"], stdout=f )
    f.close()

    # Resting ... I'm tired
    sleep(2)
  
    f = open('.count.txt', 'r')
       
    for line in f.readlines():
        rows.append(line.strip())
    #print(rows)
  
    if (rows[2].strip() != '0'):
        print(' Looks like reference set already exists \n ')
    else:
        print(' Reference Set %s not found ...  %reference_set_name ')
        print(' Looks like we will have to create this bad boy ...')
       
        try:  
            call(['/opt/qradar/bin/ReferenceSetUtil.sh', 'create', reference_set_name , 'IP'])
            print(' Successfully created reference set %s \n ' %reference_set_name )
            #print(' Looks like that went well ... ' )
        except:
            #This does not catch any java exception that may be created
            print(' Error occurred while creating reference set %s ' %reference_set)
            print(' You may create the reference set %s manually if needed ' %reference_set_name )
            exit(0)

    print(' Loading information into reference set %s ' %reference_set_name )
    try:  
        call(['/opt/qradar/bin/ReferenceSetUtil.sh', 'load', reference_set_name , ip_txt ])
        print(' \n You may need to verify that you have rules created to use %s ' %reference_set_name )
    except:
        print(' An error occurred while loading the reference set ... ')
        print(' Please retry later!')
        exit(0)
    remove('.count.txt')


# This function creates the DNS reference set
def verify_create_dns_reference_set():
    reference_set_name = 'SecurityNik_DNS_Darklist'
    dns_txt = getcwd()+'/SecurityNikBadDomains.txt'
    dns_rows = []
  
    print('Checking to see if the reference set %s already exists' %reference_set_name)
    f = open('.count.txt', 'w')
    call(["psql", "-U", "qradar", "--command=SELECT COUNT(*) FROM reference_data WHERE name='SecurityNik_DNS_Darklist'"], stdout=f )
    f.close()

    # Taking a nap ...
    sleep(2)

    f = open('.count.txt', 'r')
    for line in f.readlines():
        dns_rows.append(line.strip())
    #print(dns_rows)

    if (dns_rows[2].strip() != '0'):
        print(' Looks like reference set already exists \n ')
    else:
        print(' Reference Set %s not found ' %reference_set_name )
        print(' Looks like we will have to create this bad boy ...')
        try:
            call(['/opt/qradar/bin/ReferenceSetUtil.sh', 'create', reference_set_name , 'ALN'])
            print(' Successfully created reference set %s ' %reference_set_name )
           
            #print(' Looks like that went well ... ' )
        except:
            # This does not catch any java exception that may be created
            print(' Error occurred while creating reference set %s ' %reference_set)
            print(' You may create the reference set %s manually if needed ' %reference_set_name )
            exit(0)
               
    print(' Loading information into reference set %s ' %reference_set_name )
       
    try:
        call(['/opt/qradar/bin/ReferenceSetUtil.sh', 'load', reference_set_name , dns_txt ])
        print(' \n You may need to verify that you have rules created to use %s ' %reference_set_name )
    except:
        print(' An error occurred while loading the reference set ... ')
        print(' Please retry later!')
        exit(0)
    remove('.count.txt')



# Main Function
def main():
    #print('You are in the main part of the code')
    call('clear')
    check_os()

    # Let's work on the IP Reference Set
    grab_ip_list()
    compare_ip_dirs()
    combine_ip_files()
    verify_create_ip_reference_set()

    # Let's work on the DNS Reference Set
    grab_dns_list()
    compare_dns_dirs()
    combine_dns_files()
    verify_create_dns_reference_set()


if __name__ == "__main__":
    main()


# End of Script

As stated above, hopefully this script makes someone else happy.

Have fun and don't forget the other posts in this series to ensure your reference set and rules are properly created.


1. The Code to download the Darklist
2. Verifying the Reference Set Creation
3. Writing the Common Rule to check for the IPs
4. Writing the Event Rule to check for the domains
5. Checking your environment for the malicious IPs and or domains.


SecurityNikThreatIntel.py
This project can also be found on github at https://github.com/SecurityNik/QRadar---Threat-Intelligence-On-The-Cheap