QRadar - Extracting fields from Sourcefire events

As mentioned in the previous 2 posts in this series, we don't always get the fields parsed and displayed as we would like from the SIEM. In addition, as previously mentioned also, ensuring the payload is accessible is critical. In this post we will extract fields from Sourcefire logs.


Sample Payload Data:
DeviceType=Estreamer    DeviceAddress=10.0.0.1    CurrentTime=1406554255965    recordType=INTRUSION_EVENT_RECORD    recordLength=60    timestamp=28 Jun 2014 08:30:54    detectionEngineRef=2    eventId=458907    eventSecond=1406554254    eventMicrosecond=552218    rule.generatorId=1    rule.ruleId=16606    rule.ruleRevision=9    rule.renderedSignatureId=16606    rule.message=IPS Alert From SourceFire    rule.ruleUUID=2FEB420CE5684FC5A90610F645E268C7    rule.ruleRevisionUUID=2B1A93DA34E811E3B791848F69E36DD2    classification.classificationId=9    classification.name=attempted-user    classification.description=Attempted User Privilege Gain    classification.classificationUUID=9D0A6F5ECBA211D9925A005056040501    classification.classificationRevisionUUID=00000000000000000000000000000000    priority.priorityId=1    priority.name=high    sourceAddress=10.0.0.2    destinationAddress=10.0.0.3    sourcePortOrICMPType=49555    destinationPortOrICMPCode=443    ipProtocolId=6    impactFlags=00000000    impact=5    blocked=0 



The fields to be extracted in this case are:
• DeviceType 
• DeviceAddress 
• recordType 
• detectionEngineRef 
• eventId 
• rule.generatorId 
• rule.ruleId 
• rule.ruleRevision 
• rule.renderedSignatureId 
• rule.message 
• rule.ruleUUID 
• rule.ruleRevisionUUID 
• classification.classificationId 
• classification.name 
• classification.description 
• classification.classificationUUID 
• classification.classificationRevisionUUID 
• priority.priorityId 
• priority.name 
• sourceAddress 
• destinationAddress 
• sourcePortOrICMPType 
• destinationPortOrICMPCode 
• ipProtocolId 
• impactFlags 
• impact 
• blocked

Without further ado, let's "Extract Property".
As always the prroperty type will be "Regex Based"

New Property Name: DeviceType
Field Type: AlphaNumeric
Description: Device Type as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: DeviceType=([a-zA-Z]*) - Capture Group: 1
Enabled

New Property Name: DeviceAddress
Field Type: AlphaNumeric
Description: Device Address as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: DeviceAddress=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - Capture Group: 1
Enabled

New Property Name: RecordType
Field Type: AlphaNumeric
Description: Record Type as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: recordType=(.*?)(\s) - Capture Group: 1
Enabled

New Property Name: DetectionEngineRef
Field Type: Numeric
Description: Detection Refernece as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: detectionEngineRef=([0-9]*) - Capture Group: 1
Enabled

New Property Name: EventID
Field Type: Numeric
Description: Event ID as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: eventId=([0-9]*) - Capture Group: 1
Enabled


New Property Name: GID
Field Type: Numeric
Description: GeneratorID as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: GeneratorID=([0-9]*) - Capture Group: 1
Enabled


New Property Name: RuleID
Field Type: Numeric
Description: Rule ID as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: rule.ruleId=([0-9]*) - Capture Group: 1
Enabled


New Property Name: RuleRevision
Field Type: Numeric
Description: Rule Revision Number as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: rule.ruleId=([0-9]*) - Capture Group: 1
Enabled


New Property Name: RuleRenderedSignatureID
Field Type: Numeric
Description: Rule Rendered Signature ID Number as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: rule.renderedSignatureId=([0-9]*) - Capture Group: 1
Enabled


New Property Name: RuleUUID
Field Type: AlphaNumeric
Description: Rule UUID ID Number as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: rule.ruleUUID=([a-zA-Z0-9]*) - Capture Group: 1
Enabled

New Property Name: RuleRevisionUUID
Field Type: AlphaNumeric
Description: Rule Revision UUID ID Number as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: ruleRevisionUUID=([a-zA-Z0-9]*) - Capture Group: 1
Enabled


New Property Name: Classification
Field Type: AlphaNumeric
Description: Rule classification as extracted from SourceFire Payload
Log Source Type: Sourcefire Defense Center
Log Source: All
Category: High Level: Any
Low Level Category Any
Regex: classification.description=(.*)\sclassification.classificationUUID=(.*) - Capture Group: 1
Enabled

Regex Refernces:
http://www.autohotkey.com/docs/misc/RegEx-QuickRef.htm
https://www.tcl.tk/man/tcl8.5/tutorial/Tcl20.html
http://www.adobe.com/devnet/dreamweaver/articles/regular_expressions_pt1.html
http://www.rexegg.com/