SN08 Javier-Inclan Writing Rules

SN08: Primer: Writing Rules Not Meant to be Broken
Javier Inclan Worldwide Principal Instructor

September 2010
2010 ArcSight, Inc. All rights reserved. ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners. www.arcsight.com 2010 ArcSight Confidential 1
Agenda

Rules foundations Understanding rule components

Conditions Aggregation Actions Triggers
Mastering rules Additional rule features Troubleshooting rules Tuning rules
www.arcsight.com
2010 ArcSight Confidential
Rules Foundations
www.arcsight.com
Rules Foundations
First Things First: Rule Definition
What is a rule? Process who is running in the ArcSight ESM manager Evaluates incoming events looking for specific conditions and patterns. Based on these results it infers meaning about their significance and can initiate actions in response Applied to events during the correlation evaluation phase of the event lifecycle Rules are loaded by the ArcSight ESM correlation engine when ArcSight ESM starts up
www.arcsight.com 2010 ArcSight Confidential 4
Rules Foundations
Concepts for Configuring Rules
Constructed using aggregation and Boolean pattern matching within the CCE (AND, OR, NOT) Rules operate in the real time event stream Must be activated (save or linked into the real time rules folder) Moving rules in and out of the real-time rules folder triggers the correlation engine to reload the rules Rules can be scheduled to run at predefined intervals Scheduled rules do not have to be in the real-time rules group
2010 ArcSight Confidential Event Definition
Two additional editor conditions
Join Condition
and
or
not
plus same regular conditions

www.arcsight.com 5
Rules Foundations
What Rules Do
Incoming events are compared to conditions and aggregation settings of each enabled rule Event matches trigger pre-configured actions and a correlation event is generated by default
www.arcsight.com
Rules Foundations
What is a Correlation Event?
Correlation events become new events to be evaluated by the correlation engine
www.arcsight.com
Rules Foundations
Identifying Rules Types
Types of Rules
Simple Rules
Match one or more events against one set of conditions
Join Rules
Match more than one event against two or more sets of conditions
www.arcsight.com
Understanding Rule Components
www.arcsight.com
A rule definition is based on four components

Conditions Aggregation Actions Triggers
These components are distributed within three different tabs in the rules editor
2010 ArcSight Confidential 10
www.arcsight.com
What Events am I Looking For?

Conditions will define the set of events that I am looking for Drivers for define this set of conditions could be use case definitions, compliance or computer/network/device security business requirements Conditions are created using the CCE Conditions rely on Boolean Logic principles
www.arcsight.com
11
Defining Precise Conditions

What events am I looking for? Events who have met these conditions are going to be named matches Conditions can be loose attacker address inSubnet 192.168.1.0/24 or target address HasVulnerability xxxx Conditions can be well defined/precise attacker address=192.168.1.10 Which one is best for performance?
The devil is in the details (i.e put a = instead of a > and your rule conditions change, whether intentional or !=)
Use Categorization Fields in Rules

Develop logical framework for grouping resources Leverage ArcSight event categorization
Since devices do not utilize a common naming schema for events, ArcSight Connectors map individual signatures to a common taxonomy so that ArcSight ESM can later reason over those events Without categories: [ID contains 529 or 621] OR [login and failure and SSH] OR [login and failure and target port 23] With categories:
www.arcsight.com
13
Benefits of Network Modeling in Rules Conditions

Effective Rules DEPEND on ArcSight ESM product intelligence Enables content that can make informed decisions based on more detailed information
Asset model describes attributes of the assets
Vulnerabilities locations active lists asset categories Increases accuracy of ArcSight priority formula Identifies assets subject to compliance
www.arcsight.com
14
Benefits of Network Modeling in Rules Conditions

Enables you to build a business-oriented view of data assets/ranges zones networks customers ArcSight WITHOUT network, zone, asset modeling, categorization and vulnerability information will produce more false positives and background chatter than a mis-configured IDS (aka OPEN THE FLOOD GATES) How is a vulnerability scanner throwing out traffic on port 23 to 100 servers and analyzing a response differ from a CiscoWorks server using port 23 to push IOS upgrades to 100 switches?
www.arcsight.com
15
Aggregation or Aggravation?

Not a mis-spelling; we did not say aggravation Aggravation might be a symptom if timing parameters and number of events within a specified time frame arent well understood Do you want to aggregate on unique or identical fields? Before rolling your rule out to production TEST IT in development, QA or with simulated events fed from a test connector
www.arcsight.com
Defining Aggregation
Rule Aggregation Sets required number of event matches within a specified timeframe Time frame set here is known as time window expiration Matches only if specified field or fields is unique amongst evaluated events Matches only if specified field or fields is identical amongst evaluated events Values from fields listed in aggregation settings will be carried from base events to correlation events
Rule: Defining Aggregation
What fields to aggregate on?

Generally: event name, attacker/target Hostname/Address/FQDN/Domain Name/User Name/Zone Resource Non-aggregated fields cant be used in dashboards and reports
!
www.arcsight.com
Tip For MSSPs: Aggregate on Customer Resource to ensure events from the same IP address are really from the same machine
Aggregation impacts memory, as aggregation matches are counted and tracked

Do not aggregate over long periods of time; instead use an active list Limit the set of aggregated values
18
Defining Aggregation
Use to limit the amount of rule firing for repeat events, or to set thresholds that define certain scenarios

This specifies number of matches (threshold) in specified amount of time by the rule Example five failed login attempts in two minutes may signify a brute force
www.arcsight.com
19
Advanced Aggregation

There are four time-evaluation criteria that can affect eventoccurrence aggregation and rule-triggering You can apply these to rules through the aggregation tab and the statement panel of the conditions tab
1. Time Frame establishes the time span for occurrence aggregation
Event-occurrence aggregation is always controlled by time frame This is the amount of time that qualifying events for all aliases will be retained in memory for evaluation and is based on manager receipt-time
2. Global Expiration global expiration applies to an entire rule
3. Alias Expiration an alias expiration applies to a single alias within a rule

This is the amount of time that a qualifying event for this alias will be retained in memory for evaluation and is based on manager receipt-time
4. Matching Time matching time creates a time-proximity comparison for multiple-alias rules and is based on events' actual creation times
Now, We Are Ready for the Action!
Once the rule conditions are met and we meet the threshold requirements set in aggregation, its time to take action! When a rule fires, an action will be taken based on the trigger that you set You can select single or multiple triggers Why is my rule firing at weird times? Why is my rule not firing? (lets look a little closer at timing and triggers)
www.arcsight.com
Types of Available Rule Actions

A rule can trigger any combination of the following actions

Set event field Send to open view operations Send notification Execute command Execute connecter command Export to external system Create new case Add to existing case Add to active list Remove from active list Add to session list Remove from session list
www.arcsight.com
Defining Triggers Rule Action Triggers

Three types of rule action triggers are available 1. Event triggers act on individual events On first event On every event On subsequent events 2. Threshold triggers act on groups of events that satisfy the time frame requirements On first threshold On every threshold On subsequent thresholds 3. Timing triggers act on timing of events On Time Unit triggers on a specified unit of time after a threshold is met On Time Window Expiration (TWE) triggers after the time frame expires without meeting the number of matches requirement
First Threshold
Threshold condition: five matches within two minutes

Threshold condition reached Action takes place Threshold time window resets
1st Threshold 1st Threshold
1 2 3
Matches
(2 Minutes)
12
3
Time Reset
Time
60 sec
180 sec
240 sec
24
www.arcsight.com
Every Threshold

Every time the threshold is met Action takes place Continues until TWE
1st Threshold 2nd Threshold
2 3
5 6 7 8 9 10 11
12
13
14
(2 Minutes)
Time
60 sec
100 sec TWE
www.arcsight.com
25
Subsequent Threshold

After the first threshold is met Waits for second threshold to be met Action takes place at subsequent thresholds Continues until TWE
1st Threshold (No Action) Subsequent Threshold TWE
12
3 4
5 6 7 8 9 10 11 12 13 14 15
16 17 18
Subsequent Threshold
Time
www.arcsight.com
(2 Minutes)
60 sec
100 sec
160 sec
26
On Time Unit
Threshold condition: five matches within two minutes with a 30 seconds time unit

Initial threshold is met Action takes place every time the time unit elapses Continues to take action until TWE
1st Threshold
1 2 3
5
30 sec Time Unit
6 7
8
TWE
30 sec Time Unit
Time
www.arcsight.com
60 sec 90 sec 120 sec

Time Window Expiration


Initial threshold is met Waits until TWE Action takes place

1st Threshold
12 3
5 6
(2 Minutes)
Time
TWE 180 sec
60 sec
www.arcsight.com
28
Correlation Events Created by Rules

What fields to set in correlation events?

These are the fields that will be set in the correlation event Make sure you dont create a feedback loop (rules firing on themselves)
!
www.arcsight.com
Tip: Agent Severity Use low for informational rules that have indirect consequence Use medium to very high for rules of direct consequence
29
Dont Break Your Rule with Excessive Actions
What to do when conditions and thresholds have been met?

Create a new event, create a case, etc
Use on first event or on first threshold to avoid excessive rule firing due to heavy attack traffic
www.arcsight.com
30
Dont Break Your Rule with Excessive Actions

To add all rule firings to a single case, use on subsequent events A solution to handle long running continuous attacks would be to define following triggers
On first threshold will notify start of attack On time unit will periodically notify that the attack is still going on On time window expiration will notify end of attack
www.arcsight.com
31
Mastering Rules
www.arcsight.com
32
Mastering Rules
Know the business conditions, requirements or use case

Thats how you start to build a rule!
Rule development
science (boolean, timing, action definitions, etc) art, so keep rules conditions as simple and precise as possible
Know our ArcSight event SCHEMA

thats what you have to work with! (the fields and the output of those fields)
www.arcsight.com
33
Mastering Rules
Create Multiple Simple Rules Instead of One Complex

Break down the use case requirements by listening for key words
Define organizations ArcSight network topology

Network modeling
Track all user logins, from where and to what device

Rule / session List
Track all user logouts, from where and from what device
Rule / session List
www.arcsight.com
34
Mastering Rules
Document Your Rules
Long after youve forgotten (maybe six months to a year down the road) when you need to review what you were thinking. A best business practice when developing rules is to DOCUMENT the use case, business requirements and details of how the rule was developed on the NOTES tab Possible topics to note: Who requested the rule, who are the stakeholders, original date and time of testing vs. deployment, etc
www.arcsight.com
Mastering Rules
Use Stock Content and Solutions Foundations
ArcSight provides MANY solutions foundation and stock content rules to facilitate out of box functionality upon installation If you need to get your bearings this is a good place to start Remember, any rules enabled in the real time rules folder are LIVE
www.arcsight.com
36
Additional Rule Features
www.arcsight.com
37
Automatic rule disabling

ArcSight automatically disables improperly written rules that would produce excessive or meaningless events
Clearing rule actions

In a grid view, select a correlation event Right-click and choose correlation options Clear rule actions to clear all actions associated with this rule
Showing rule errors

If rules have errors, the rule icon ( ) changes to indicate it In the rules resource tree, right-click the rule-error icon and choose show error The error appears in a dialog box
www.arcsight.com
38
Automatic Rule Disabling
Rule disabling factor operation Alias matches if an alias is defined, this is the number of events matching that alias and is independent of other defined aliases in the same rule Partial matches if more than one alias is defined, the number of events matching the aliases defined before the current one, and for the current one, and for their join condition (if present) Generated events counts the number of correlation events generated Base event counts number of base events used to generate correlation events Time unit counts number of time units (minutes) that passed since the rule activated Above values for rule disabling may be adjusted for your enterprise ArcSight ESM will disable a rule if the rule exceeds the configured limits on number of rules triggered per minute or the ratio of base events to triggered rules and is defined in server.defaults.properties file on the manager
www.arcsight.com
Troubleshooting and Tuning
www.arcsight.com
40
Rules: Troubleshooting
What do you do when the check engine light comes on in your car? Apply the same methodology Break components down into their most basic form (dont digest the entire conditions tab, take it one line or maybe one statement at a time)
Is the data your looking for actually available? Start back at the basics (RAW logs from the device prior to hitting our connector and being normalized)
Was the rule imported via an ARB? If so, was it done on the same revision of ArcSight ESM? Were resource IDs exported into the ARB?
Has the rule completed? (partial matching rule?)

www.arcsight.com
My Rule is Broken!
How would I know? What Clues Do I look for?
www.arcsight.com
42
Check your Condition Logic First
What are your rules dependant on?

Active/session lists, asset/network modeling, variables, etc?
How do you know? Check out https://localhost:8443 Resource management and rules to look at details:
www.arcsight.com
43
Identifying Attacks
If a rule is defined to identify the following attacks, it will excessively fire: Denial of Service or Distributed Denial of Service Attack IDS / SIM / SIEM / ESM Smoke Screening akaCopperfield/Angel magic If rule trigger is activated on EVERY EVENT or EVERY THRESHOLD, it may lead to excessive firing What would this look like?
www.arcsight.com
44
Potential Issues Related with Timing

Timing is very sensitive in rules firing End time field is a key player during correlation phase. Network latency could lead to potential issues during correlation:
Verify start time, end time, agent receipt time and manager receipt time values 1-2 min off could be an indicator of network latency
Poor bandwidth or high EPS could produce same results Did something recently change that could effect the arrival of events into the connector? Anything more could trip an exception error "DCERPC pipe is no longer open" reported in server.log check the following:
Changed behavior of A/V or HIPS which now blocks remote pipes Changed network behavior after a Patch (those do get tested first, right? ;> ) Has your OS stopped allowing remote pipe comms? (ieWindows Firewall or IPTABLES) Domain Admin recently tightened access policy or net admin threw a new ACL/rule
www.arcsight.com
45
Rules: Troubleshooting

Is your rule recursive ? Starting in Arcsight ESM 4.5.1, rules that trigger themselves recursively will Automatically be disabled temporarily, then re-enabled (aka rule bouncing) Has your rule trigger exceeded the max. # of correlated alerts per min. limit? You would see an error as seen below in your server.log file:
[2009-07-30 10:21:59,750][ERROR][default.com.arcsight.rulesengine.actionengine.ActionCommandHandler][onSingl eEvent]
Too many pending actions 1000, not adding more .... This is set in server.default.properties as:
#number of correlated alerts per rule per minute rules.max.fan-out.time-unit.ratio=1000 Remember persisted settings must be set in server.properties

To reduce excessive firing, consider using ON FIRST and TU/TWE triggers Monitor your rules engine via rcsight ESM dashboards or status monitoring web page
www.arcsight.com
46
Limit Partial Match Storage Using Time Constraints

This condition occurs when using join rules and an event matches one alias Partial matches for a rule are stored in memory for the specified time window To limit memory consumption Limit the aggregation time frame Use active lists to correlate information from events spaced far in time
!
www.arcsight.com
Tip: Partial Matches Can be monitored using the Rules Status dashboard in ArcSight Administration
47
Tuning Rules
ArcSight ESM comes with a dashboard that can enable you to view the statistics of the rules within your environment The following data monitors are included:
Partial matching Top firing rules Recent fired rules Rules engine internal stats Rule error logs
www.arcsight.com
48
More Information?

Rules arent something we expect you to be a subject matter expert in by attending this workshop or by attending 3-5 day classes 4.5.1 user guide; chapter 13: Rules Authoring 4.5.1 system content reference guide Talk through your rules
Engineering 101: If you cant explain the process, you dont understand the process
ArcSight Protect 724

Content sharing and ARBs how are your colleagues writing rules?
Review summary for SQL look http://en.wikipedia.org/wiki/De_ Morgan's_laws
www.arcsight.com
49
Your Feedback Builds a Better Conference!

Text to 32075 (USA & Canada) or 447786204951 (Non-USA) Type ARCS <space> 08 and the letter to each response SMS body example: ARCS 08ae*your comments Excellent Good Fair Rate the speaker Rate the content
Poor
a e
b f
c g
d h
Please provide comments: (*) enter any comments/feedback Download session replays after the conference: https://protect724.arcsight.com/community/protect10/sessions
ArcSight, Inc. Corporate Headquarters: 1 888 415 ARST EMEA Headquarters: +44 (0)844 745 2068 Asia Pac Headquarters: +65 6248 4795 www.arcsight.com
www.arcsight.com
51

SN08 Javier-Inclan Writing Rules

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SN08 Javier-Inclan Writing Rules

Uploaded by

Copyright:

Available Formats

SN08: Primer: Writing Rules Not Meant to be Broken

Javier Inclan Worldwide Principal Instructor

Rules foundations Understanding rule components

Mastering rules Additional rule features Troubleshooting rules Tuning rules

2010 ArcSight Confidential

2010 ArcSight Confidential

First Things First: Rule Definition

Concepts for Configuring Rules

Two additional editor conditions

plus same regular conditions

2010 ArcSight Confidential

What is a Correlation Event?

Correlation events become new events to be evaluated by the correlation engine

2010 ArcSight Confidential

Identifying Rules Types

2010 ArcSight Confidential

Understanding Rule Components

2010 ArcSight Confidential

Understanding Rule Components

A rule definition is based on four components

Understanding Rule Components

What Events am I Looking For?

2010 ArcSight Confidential

Understanding Rule Components

Defining Precise Conditions

Understanding Rule Components

Use Categorization Fields in Rules

2010 ArcSight Confidential

Understanding Rule Components

Benefits of Network Modeling in Rules Conditions

2010 ArcSight Confidential

Understanding Rule Components

Benefits of Network Modeling in Rules Conditions

2010 ArcSight Confidential

Understanding Rule Components

Understanding Rule Components

Understanding Rule Components

Rule: Defining Aggregation

What fields to aggregate on?

Aggregation impacts memory, as aggregation matches are counted and tracked

2010 ArcSight Confidential

Understanding Rule Components

2010 ArcSight Confidential

Understanding Rule Components

2. Global Expiration global expiration applies to an entire rule

3. Alias Expiration an alias expiration applies to a single alias within a rule

Understanding Rule Components

Now, We Are Ready for the Action!

Understanding Rule Components

Types of Available Rule Actions

Understanding Rule Components

Defining Triggers Rule Action Triggers

Understanding Rule Components

Understanding Rule Components

100 sec TWE

2010 ArcSight Confidential

Understanding Rule Components

Understanding Rule Components

30 sec Time Unit

60 sec 90 sec 120 sec

Understanding Rule Components

Time Window Expiration

Initial threshold is met Waits until TWE Action takes place

TWE 180 sec