IBM Qradar: How to import logs from an Amazon S3 compatible log source

Many vendors nowadays are using the Amazon S3 API as a method to access and download their logs. Cisco is an example of this, and they host their Cloud Web Security (CWS) product logs at vault.scansafe.com and use the Amazon S3 API to make the logs accessible to their users (Other vendors include Hitachi, EMC Vcloud, and many more).

IBM Qradar has added support for the Amazon S3 API as a log protocol to allow Qradar to download logs from AWS services such as CloudTrail, but we found out that the use of this protocol on Qradar is limited to downloading logs if they are stored on Amazon S3, and that we couldn’t use it in the case of products such as Cisco CWS where the logs are hosted on their own servers.

To add Cisco CWS as a log source for IBM Qradar, we used a manual python script to download the logs using the S3 API to a local directory on the Qradar console, and then configured Qradar to automatically import the logs from that local directory.

In this blog post, I will walk through the steps to allow you to add S3 compatible log destinations as log sources in Qradar. The overall steps are as follows:

1. Download the script to the Qradar server (console or log collector).

2. Install dependencies for the script and configure the script parameters.

3. Setup a cronjob to run the script on a recurrent basis.

4. Create a new log source in Qradar to pull the downloaded log files using SFTP.

Cisco support has a python script available to automatically pull logs from their servers for CWS (vault.scansafe.com). However, the script needed to be modified and a few features to be added before we could properly run it on Qradar (Original script can be requested from Cisco support here):

• Added statefulness to the script. Originally, the script would download all log files available every time it ran, but this meant it could be downloading many gigabytes of data on each run. Instead, we added the capability of saving the timestamp of the last file downloaded, and then only downloading recently created log files on subsequent runs.

• Cleanup: Once the log files are downloaded and processed by Qradar, there is no need to keep the files on the system. The script was modified so that it deletes files after a set number of hours.
• The script was written for a newer python version but Qradar has version 2.6 installed. We had to modify the ‘with .. as’ statements as they were not supported, and replaced them with manual os.open and gzip.open (and corresponding close) function calls.

The full script can be found at the bottom of this post.

Script setup and configuration

1. First, copy the script to Qradar using SCP. You can choose any directory you prefer (We placed it under /opt/qradar/bin/)

2. Install the boto python library (Boto is the AWS SDK provided by Amazon):

1. Download boto-2.42.0.tar.gz from https://pypi.python.org/pypi/boto (This is the older version but the script was written to use it, and so we didn’t modify it).

2. Copy the file to Qradar.

3. Uncompress the file using tar ('tar -xzvf boto-2.42.0.tar.gz').

4. Install boto by running ‘python setup.py install’ from the directory the files were extracted to.

3. Edit the script using vi and set the following values:

1. Endpoint: hostname for the endpoint hosting the logs (For Cisco CWS this would be vault.scansafe.com).

2. accessKey: Should be provided from the vendor

3. secretKey: Should be provided from the vendor

4. bucket: Should be provided from the vendor

5. localPath: Local path to where the files should be downloaded (We used /store/tmp/)

6. Hours: number of hours to keep the files after they were downloaded. The default is one hour.

4. Add execution permissions to the file ('chmod +x cws_script.py').

5. Configure a cronjob to run the script automatically. We set the script to run every 5 minutes, and as a rule you should set it to run more frequently than the interval at which Qradar is processing them. To configure cron to run the script every 5 minutes:

1. Edit the cronjob scheduled tasks by running ‘crontab -e’

2. Add the following line at the end:

*/5 * * * * /opt/qradar/bin/cws_script.py >> /var/log/cws-script.log 2>&1

Create a new log source in Qradar

1. From the Qradar Console go to Admin > Log Sources, and click Add.

2. Select Univeral DSM for the ‘Log Source Type’, and select ‘Log File’ for the protocol.

3. Choose ‘SFTP’ and enter the Qradar’s own IP address and enter user/password details.

4. Set the Remote Directory to the directory on Qradar to which the script downloads the log files.

5. For ‘File Pattern’ enter a regex that would match the files downloaded. For Cisco CWS logs, we used ‘.*txt’

6. Set the recurrence to specify the interval at which Qradar will import the logs. We used 15M so that the log files are processed every 15 minutes.

Once configured and saved, you can verify operations by going to the ‘Log Activity’ tab and setting the newly added log source as a filter and then viewing the logs as they are downloaded. You can also use the log file specified when configuring the cronjob to verify the script operations and if it is running properly.

Script

#!/usr/bin/env python

import gzip
import boto
import boto.s3.connection
import sys, os

from boto.s3.key import Key
from datetime import datetime

# Required parameters

accessKey = 'access key here'
secretKey = 'secret key here'
bucket = 'bucket id here'
localPath = '/store/tmp/logfiles' # local path to download files to
endpoint = 'vault.scansafe.com'
hours = 1  #number of hours to keep downloaded files


# Optional parameters
extractLogs = True   # set to True or False. If set to True, the script will also extract the log files from the downloaded .gz archives
consolidateLogs = False   # set to True or False. If True, will consolidate content of all .gz archives into a single file, <bucket-id>.log

if not localPath.endswith('/'):
        localPath = localPath + "/"

current_datetime = datetime.now()

print "======================================================="
print "Running at",current_datetime

# Get the date/time for the last downloaded file (stored in the file 'timestamp')

if os.path.exists(localPath+"timestamp"):
 last_download_timestamp = str(open(localPath+"timestamp",'r').read()).strip() 
 print "Last timestamp",last_download_timestamp
 last_download_timestamp = datetime.strptime(last_download_timestamp, '%Y-%m-%jT%H:%M:%S.000Z')
else:
 last_download_timestamp = ""


s3Conn = boto.connect_s3(accessKey, secretKey, host=endpoint)
myBucket = s3Conn.get_bucket(bucket, validate=False)

print "Connected to CWS backend infrastructure..."
print "Downloading log files to " + localPath + "\n"

for myKey in myBucket.list():
 if (last_download_timestamp == "" or last_download_timestamp < datetime.strptime(myKey.last_modified, '%Y-%m-%jT%H:%M:%S.000Z')):
  print "{name}\t{size}\t{modified}".format(
   name = myKey.name,
   size = myKey.size,
   modified = myKey.last_modified,
   )

  #save the timestamp of the last file read
  timestamp = open(localPath+"timestamp",'w')
  timestamp.write(myKey.last_modified)
  timestamp.close()


  fileName = os.path.basename(str(myKey.key))
  if not os.path.exists(localPath + fileName):

   myKey.get_contents_to_filename(localPath + fileName)

   if extractLogs:
    mode = 'w'
    extractedFilename = fileName[:-3]

    if consolidateLogs:
     extractedFilename = bucket + ".log"
     mode = 'a'

    extractedLog = os.path.join(localPath, extractedFilename)

    gzipFile = gzip.open(localPath + fileName, 'rb')
    print "{name} extracted to {log_file}".format(
     name = myKey.name,
     log_file = extractedFilename,
     )
    gzFile = gzipFile.read()
    unzippedFile = open(extractedLog, mode)
    for line in gzFile:
     unzippedFile.write(line)
    gzipFile.close()
    unzippedFile.close()

# Clean files older than number of hours specified 
dnldFiles = os.listdir(localPath)
for file in dnldFiles:
 if ((datetime.now() - datetime.fromtimestamp(os.path.getmtime(localPath+file))).seconds > (hours * 60 * 60)):
  print "deleting file",file
  os.remove(localPath + file)

print "\nLog files download complete.\n"

Active (Nmap) vs Passive (p0f) information gathering

In most cases, while attempting to learn about an environment, you may wish to either actively obtain information about the environment or passively do so.

Active information gathering means you are in one way or the other directly interacting with the system(s). An example of active information gathering is when a tool such as Nmap is used.

Passive information gathering on the other hand, means you passively “sit” and learn about the system(s) as information passes in your path. Note, these examples are specifically from a network perspective. Basically, you are not doing anything that will cause the remote system(s) to perform any response or any activity based on what you are doing.

Let’s look at this in practice.

Let’s “actively” learn about a remote system using Nmap.
From below we tell Nmap to target the host 10.0.0.15. The options we specify are as follows:
-Pn - no need to ping the host, assume it is online
-sS - perform a SYN scan
-p1-65535 - Scan all TCP port
-sV - determine service version numbers
10.0.0.15 - The host we are would like to scan

From the above we see Nmap learned that the host has TCP ports 135, 139 and 445 opened.

As we look at the Wireshark snapshot below, we can see a sample of the packets which were sent to and seen by host 10.0.0.15. This is what is meant by “active” information gathering, the target host was “touched" and there is (in this case) supporting evidence to show that it was.

Alternatively, as we perform “passive” information gathering we can simply start up “p0f” and let it listen on the interface. Note, for this to work properly, you would need to be in the path of the traffic. This can be inline or even seeing the traffic off a SPAN port.

Once again, let’s look at this in practice.
We will use p0f with the following options.
-f /usr/share/p0f/p0f.fp - specify the fingerprint file to use
-i eth0 - specify the interface to listen on
-p - put the interface in promiscuous mode

From above we see that p0f “learned” that the system 10.0.0.15 is running Windows NT kernel and this was learned via a SYN packet.

While the example above shows how we can use p0f to listen on an interface, we can actually run p0f against an existing .pcap file to learn about the environment.

To demonstrate this further, let’s borrow the attack-trace packet capture from honeynet project.

When running p0f against the pcap file, we can get even more creative. Let’s examine some of this creativity.

First up, let’s run p0f against our downloaded pcap. The main difference between the command line arguments when using p0f live vs reading a packet is the “-r” option.
-r attack-trace.pcap - use the attack-trace.pcap file as input.















From above we see a snapshot of the OS, client, server, etc from the pcap.

Let’s dig more. To find out the various OSes in the file, let’s run that through some additional tools, such as grep, awk, sort and uniq.
From the above, we see we have a count of all the different systems detected within the pcap.

The image below shows the various IPs. In this example, we introduce another Linux utility - “cut”.

Ok, let’s do one more before we go. Let’s use the same command above, but this time we will ask “cut” to use field 2 thus allowing us to see all the unique ports found in the file.


So there you go. In this post we looked at the difference between active and passive information gathering from the perspective of network reconnaissance.

Hope you enjoyed this post. If you did, please feel free to leave a comment.


References:honeynet project
Nmap
p0f

AWS Security: Automating Palo Alto security rules with AWS Lambda

With the increased adoption of IaaS cloud services such as Amazon Web Services (AWS) and Microsoft Azure, there is also a greater need for security controls in the cloud. Firewall and IPS vendors such as Palo Alto, Checkpoint, Fortinet have made available virtual instances of their products ready to run in these cloud environments. These tools can provide great advantages on top of the existing security controls inherent in the cloud platform. For example, security groups in AWS control traffic flow to/from IP addresses and ports, but adding a Palo Alto or Checkpoint gateway can provide more inspection capabilities and filtering, especially on the application level. In addition to firewall capabilities, these products can provide extra features such as intrusion prevention, URL filtering, and other features that are lacking with the native security controls.

However, there is still lots of manual configuration expected from the Security or Network administrators, such as configuring interfaces or network settings, firewall and threat prevention rules, and so on. But in a cloud environment where speed and flexibility are its main assets, waiting for a security administrator to put in a rule for a newly created instance can seem to be a step back. The vendors above are definitely aware of this, and have introduced features such as Palo Alto’s capability of reading AWS attributes (for example tags or instance IDs) and then using them in dynamic rules that get updated as changes occur in the cloud environment. For example, you can create a group with the setting to include anything with a certain tag, and then use this group in a security rule to allow Web traffic to it. The gateway will then know to allow web traffic to any new or existing instance that has this tag. (A full list of monitored AWS attributes can be found here).

But there are still more features that you might want to configure automatically that are not included by the vendor. The vendors mentioned above all have API interfaces available, and so combining that with the tools from Microsoft or Amazon, we can easily write small pieces of code to automate lots of these tasks. To demonstrate this, I wrote a lambda function that monitors AWS instance starts and stops, as well as security group updates, and then pushes (and deletes) rules to a Palo Alto gateway based on these changes.

In this post, I will go over the different components of my code, its design, and then show it in action. Finally i will list a number of notes and considerations as well as a link to download the code.

Components

Palo Alto

Palo Alto instances can be accessed from the Amazon marketplace. Palo Alto provides excellent documentation on how to set up a gateway in the AWS, and I would recommend to start here for the initial configuration. Another useful case study provided by Palo Alto is on how to configure and use dynamic address groups in rules, where the groups are based on AWS attributes.

For this setup, I had a Palo Alto gateway configured as an Internet gateway in AWS, and so all internet traffic from my instances was passing through it. I also had an elastic IP assigned to its management port, and my lambda function used this IP address to configure the gateway. Using a public IP address to configure the gateway is not the best option, and I would recommend using a private IP instead (I address some of the limitations this might introduce in the notes section below).

AWS CloudTrail/S3

CloudTrail is a service from AWS to log and store the history of all API calls made in an AWS environment. CloudTrail saves all the logs to AWS S3 which is another AWS service that provides object storage. CloudTrail has to be enabled so that we can monitor when changes are made that are relevant to our code, and then act based on that information.

More details on CloudTrail and S3 can be found here and here.

AWS Lambda

Lambda is a service from AWS that lets you upload code, and AWS will run it for you based on triggers you set up (such as triggers from other AWS services or external triggers), all this without having to run a dedicated instance for your code. In our case, we will use CloudTrail (S3) as a trigger for our code, so that whenever a new change is made in the environment, the code can scan the changes and determines if a corresponding change is required in the Palo Alto gateway.

Details on AWS Lambda can be found here.

Code Features/Design

Adding rules

The lambda function will monitor the following two events for adding new rules:

• StartInstances: Event indicating that a new instance was started
• AuthorizeSecurityGroupIngress: Event indicating that a new rule was adding to an existing security group

Once any of these two events is detected, the function will extract the relevant information for rules required to the instances affected, and add the corresponding rules.

Rules added have the name corresponding to the type of event that triggered them. If the rule added is because of an instance started, then it’s named ‘instanceId-#’ where # is increased with every rule added, but if it is due to a change in a security group, then the naming is ‘groupid-#’. The naming convention is used by the code to track the rules it added.

Unnecessary rules

Since the Palo Alto gateway is running as an internet gateway, there are many scenarios that are not relevant, and the code will try to filter out these events so that we don’t make any unnecessary changes to the Palo Alto gateway. The following scenarios would not introduce changes to the Palo Alto gateway:

• Instances that are started but that don’t use the Palo Alto as their internet gateway. For example, there can be multiple internet gateways configured and we're only concerns with instances that use the Palo Alto to reach the internet.
• In instances with multiple interfaces, the code checks all the interfaces, and only includes those that use the Palo Alto instance as an internet gateway.
• Security group rules that have a source from within the AWS VPC will be filtered out. The Palo Alto gateway in this instance is used as an internet gateway, and so traffic from within the VPC would not pass through it.
• Security group rules that reference other security groups as a source will also not be included. These rules imply that the traffic would be local to the VPC and so would not pass the internet gateway.

Also before adding a rule, a test is made to make sure traffic is not already allowed, and only after making sure that traffic is denied, we will add a new rule.

Rule location

The code will also only add rules at the bottom, so that the security administrator can create rules at the top of the rule base that would override anything added dynamically. This can be used to control the rules automatically added. Furthermore, we specify in the code the bottom most rule that the new rules have to go above, and we can use this to control the location of the rules so that certain rules always remain at the bottom (For example, our clean up rule).

Cleaning up

When instances are stopped or rules in security groups are removed, we want any rules that we added to be removed. To avoid removing any permanent rules added by the security administrator, the code will only remove rules that it added previously to the rule base (This is can be controlled using the naming). The following events are monitored as triggers:

• StopInstances: An instance was stopped.
• RevokeSecurityGroupIngress: A rule was removed from a security group.

Imported Modules

I tried not to import any modules that don’t come with a default installation of Python except when needed. The only exceptions are:

Boto3

Boto3 is the AWS SDK for python. Using boto3 we can make API calls to AWS to get relevant information that will help us gather the necessary details to read events from AWS and build the rules and changes we want to push to Palo Alto. More details on boto3 at https://boto3.readthedocs.io/en/latest/.

Paloalto.py

These are functions that I wrote to interface with the Palo Alto gateway. The functions include adding/deleting rules or objects, searching rules and getting details, and saving changes.

To download the latest version of this code, refer to the github link. You can also refer to this blog post which goes over the details for writing it.

Netaddr

I used the IPAddress and IPNetwork functions from netaddr to allow quick checks on IP addresses (For example, if an IP address belongs to a certain subnet).

Event Handling Logic

The main function in the code is the lambda_handler, AWS passes to it the event that triggered it, which in this case would be adding a new entry to S3 (by CloudTrail):

1. The event passed by AWS contains the location of the S3 object that has the new cloudtrail entries. Our first step is to extract the name and location of this file.
2. Second we have to retrieve the file using the S3 methods from boto3, and then uncompress it using gzip.
3. The contents are then parsed as json to allow us to read and extract properties easily.
4. Finally we iterate through all the records in the logs provided searching for any of the following events:

• StartInstances

1. Call event_StartInstances which returns the list of rules relevant to the Instance in the event. In this function, a list of instance Ids are extracted, and the following is performed for each instance.
1. First a list of relevant subnets is created. Relevant subnets are those that use the Palo Alto as their internet gateway.
2. Second, a list of all interfaces belonging to the instance is created along with the subnets each belongs to.
3. For each interface that belongs to a subnet in the relevant subnets, a list of security groups attached to it is compiled.
4. Finally, the rules of all security groups compiled are parsed through, and the relevant rules are added to a list to be sent back.
2. For each rule returned, first we have to convert the format to something that can be understood by Palo Alto. This means that we need to add zone definition, translate the destination port to a corresponding service and application, and specify the action for the rule. I used the aws_rules_to_pa function to convert the format, which in turn uses aws_to_pa_services to map port numbers to application and service combinations.
3. Once the format is changed, we can now test the existing rule base allows this traffic. If it is already allowed, then we move to the next rule in our list from the first Step, otherwise we add the rule on the Palo Alto gateway and move it to the proper location.
4. Finally, commit to save the changes on the Palo Alto.

• StopInstances

1. Call event_StopInstances to get a list of Instance Ids from the log event.
2. For each instance id, call paloalto_rule_findbyname to get a list of all existing rules added by earlier by our code.
3. Remove each rule returned.
4. Commit to save changes

• AuthorizeSecurityGroupIngress

1. Call event_AuthorizeSecurityGroupIngress to get a list of all rules to be added. (Similar function to event_StartInstances described above).
2. Convert each rule to Palo Alto format using aws_rules_to_pa.
3. Find if there are already existing rules that would allow the traffic for each rule, and discard any rule that has a match.
4. Add remaining rules and move them to the proper location.
5. Commit to save changes.

• RevokeSecurityGroupIngress

1. Call event_RevokeSecurityGroupIngress to get a list of relevant security rules to be removed.
2. Find all rules added by our code for this security group (using the rule names)
3. Compare the matching rules on the Palo Alto with the list of relevant rules from Step 1.
4. Remove rules that match both lists.
5. Commit to save changes.

In action

Setup

To run the code as is, the following will be required:

1. Palo Alto instance configured with a publicly accessible IP address for management.
2. Lambda function created with the following settings (For help configuration the lambda function, refer to this link, and in particular Using AWS Lambda with AWS CloudTrail):
1. Handler should be set to lambda.lambda_handler.
2. No VPC set. (If you would like to set a VPC, refer to the Notes section below for more details).
3. IAM role (with policy attached) to allow the lambda function access to query your S3 and EC2 resources.
4. Timeout value of 25 seconds.
5. Trigger set to the S3 bucket containing the CloudTrail logs.
6. Runtime set to ‘Python 2.7’
3. Finally you will need to upload the code as a zip file to your lambda function. Before doing so, there are some hardcoded variables that need to be set first (All of which are at the top of the lambda_handler function in lambda.py):
1. pa_ip: IP address of your Palo Alto gateway.
2. pa_key: Access key for the Palo Alto gateway (Refer to the Pan-OS XML API User guide for more details on this, and specifically this page).
3. pa_bottom_rule: Name of the rule which the lambda function would be adding on top of. This would usually be the clean up rule in your security policy.
4. pa_zone_untrust: Name of the outside zone configured on the Palo Alto gateway.
5. pa_zone_trust: Name of the inside zone configured on the Palo Alto gateway.
6. pa_sgp: name of security profile group in Palo Alto to be set on rules added.
7. igwId: Instance id of the Palo Alto gateway.

Runtime

In the following example, i had a simple setup of a 3 web server instances that use a Palo Alto instance as their internet gateway. I set the variables for my lambda function to point to my Palo Alto, provided the Access Key, etc.

I had a basic security rule base configured with 4 rules initially:

• Two rules for my web servers. One rule to allow access in (on tcp ports 80,81, and 8000), and one rule to allow access out from the web servers.
• One rule to deny any clear text authentication protocols such as ftp, telnet, etc.
• Finally a clean up rule so that all other dropped traffic is logged.

I set the ‘Clean up’ rule to be my bottom rule in the lambda function, so that all rules created would be added between rules 3 and 4.

Starting an Instance

I then started my 'web server 2' instance which had the IP address 172.20.200.225, and had the webserver_sg security group assigned, which allowed traffic from any internet source to destination ports 80 and 443:

Once the instance is started, you can see the Palo Alto rulebase updated with new rule #4:

Note that only one rule was added (for ssl - tcp port 443) since port 80 was already allowed by rule #2 in the rulebase.

Adding rules to a security group

Next I updated the security group ‘webserver_sg’ and added two new rules:

The lambda function adds two new rules with the security group id as the name:

Removing a rule from a security group

Finally, I removed one of the newly added rules from the security group (for port 22):

And the rulebase was updated accordingly:

Notes and considerations

• There might be a delay from the time of an event to the time the action is seen in the Palo Alto gateway. This is because AWS can have up to 5-15 minutes delay from the time an API call is made to the time it is logged in CloudTrail. I am not aware of an easy way to overcome this, other than configure the lambda function to run on a schedule (for example every 1 or 5 minutes), or moving the code to be run continuously on a host that has access to CloudTrail and can monitor it in real time.
• In my tests, I used a public IP address of the Palo Alto gateway to configure it. This was easier since I didn’t place my lambda function to run from within my VPC, and so it couldn’t access the private IP address. To have the lambda function access the Palo Alto gateway through a private IP address, the lambda function must be run from within the VPC, and with security groups assigned to allow it to access the private IP of the Palo Alto. Furthermore, running lambda from within the VPC might interfere with how it accesses S3 objects since those are accessed through the internet. The easiest way to get around this to have an endpoint created in the VPC for accessing S3 (See https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/).
• All ICMP rules from AWS are treated the same when pushed to Palo Alto (configured with the application ‘icmp’ that would allow all types of ICMP regardless of the configuration in the security groups). Modify the function aws_to_pa_services to introduce more granularity.
• Currently only inbound rules from the security groups are examined and added, but I will be adding support for outbound rule access as well.

Download

You can download the latest version of the code on github.

To use the code as is, you only need to upload the zip file to your lambda function. If you want to make modifications, you have to zip all files (lambda.py, paloalto.py, netaddr, and netaddr-0.7.18.dist-info).

I hope this has been helpful, and note that while the functionality described in this post should be fully functional, there are a number of other features that are in progress, and the github link will be updated as these features are completed.