During the past few months, I’ve spent a good deal of my blogging time sharing methods to build composite rules that will broaden your detection coverage. During this mini-series, I’ve shared many examples including how we can use detection metadata, variables from the detections, rule labels and risk scores to build these composite rules.

In these examples, we’ve continually used a matching time window of a few hours. Today, that’s going to change. In the past we might have built our rules to observe and aggregate events with  a time range between one minute and 48 hours. Now, we are going to broaden that time window to 14 days! Let’s take a look at how this wider detection window can be leveraged in Google Security Operations (SecOps).

Rule Aggregation - match

To make sure everyone is up to speed, we are going to start by going back into the archives to a blog I wrote when I was getting up to speed on YARA-L around multi-event rules. If you aren’t familiar with the match section of a YARA-L rule, I’d strongly suggest taking five minutes to read this blog. If you can’t wait and just want the tl;dr version, the match section of a YARA-L rule serves as a method to aggregate (or group) events by one or more values over a defined time window and all of these events with that commonality within the time window will be grouped together in a detection.

Rule Types

Rules are divided broadly into two general categories, single event rules and multi-event rules.  Single event rules are a bit of a misnomer, as they aren’t always a single event. In fact, they can use match variables to aggregate similar events. Why? Oftentimes it’s simpler to group similar events together to be presented to the analyst or downstream systems as single detection, rather than two detections.

A great example of this is a mimikatz detection (or some other exe that should not be running on your system) triggering on both the EDR and the Windows Event Logs. We don’t want two distinct detections, so we use a match variable to aggregate by the hostname within a small time window, five minutes, and bundle these two events as a single detection with two log types.

Single event rules can generate alerts if desired, but in the world of composite rules, single event rules are generally used as detections that can be linked with other detections. 

The other category is multi-event. Multi-event rules cover a number of different types of rules including:

• Composite rules - Rules made up of multiple detections
• Rules that aggregate the same types of events where a threshold is applied in the condition statement - More than 10 failed logins in a 5 minute period
• Rules that join disparate UDM events together - Detect user creation, process launch and user deletion within a specific time window 
• Rules that use row matching with data tables
• Rules that are joined to the entity graph - Detect DNS queries to domains with low prevalence

The most obvious place where a broader detection window can be advantageous are in these multi-event examples, particularly in the first few examples where we need additional time to assess the activities of a user or actions taking place on a system where two days may not be sufficient.

Rules of the Road

With the extension of the detection window, there are minimal changes that a detection engineer needs to be concerned with. The syntax for the match window remains the same where we specify one or more match variables over a time window. That time window range is now 1 minute to 14 days. The time value can be denoted as m (minute), h (hour), and d (day).

A rule to identify events with a common hostname value over a fourteen day period would declare the $hostname placeholder variable in the events section and then use it in the match section of the rule.

match:
   $hostname over 14d

With this broader detection window, it’s important to understand that when the match window extends beyond two days (48 hours), the frequency that the rule is run will be adjusted to a 24 hour frequency versus a one hour run frequency.

In fact, if we modify an existing rule and extend the match window out beyond two days (48 hours), we will be presented with the following prompt and Google SecOps will change the run frequency of the rule for us and save the rule.  

Similarly, if we create a new multi-event rule and specify a match window greater than two days, the system will automatically set the run frequency for us. Here in the rule editor, we can click to see different rule attributes and notice how in the Run Frequency, 24 hours has been selected and the other frequencies are disabled and greyed out. Shortening the match window will allow this to be modified if desired.

Broader Detection Windows At Work

Now that we’ve covered the theory behind this, let’s put it to work. Below is our rule to assess the accumulated risk for a user. In this example, there are a few key fields highlighted. The first is in the events section which is where we defined the placeholder variable of $user. This variable is then used in the match section of our rule where we are aggregating the detections that contain a common outcome variable of principal_user_userid together and we are grouping these values over a 14 day window.

rule composite_cumulative_risk_score_threshold_exceeded_user {

 meta:
   author = "Google Cloud Security"
   description = "Detects a userid that exceeds a risk score threshold based on all rules"
   severity = "High"
   priority = "High"
   type = "composite"

 events:
   $detect_prod.detection.detection.outcomes["principal_user_userid"] = $user
   $detect_prod.detection.detection.outcomes["principal_user_userid"] != "SYSTEM"

 match:
   $user over 14d

 outcome:
   $risk_score = 60
   $uniq_detection_count = count_distinct($detect_prod.detection.detection.rule_id)
   $total_detection_count = count($detect_prod.detection.detection.rule_id)
   $rules_triggered = array_distinct($detect_prod.detection.detection.rule_name)
   // sum of the risk score to measure against the threshold
   $cumulative_risk_score = sum($detect_prod.detection.detection.risk_score)

 condition:
   $detect_prod and $cumulative_risk_score >= 2000
}

The values in the outcome section are also being aggregated and calculated over that same time window. One of those values, the $cumulative_risk_score, is used in our condition section and requires this risk score over 14 days to exceed 2000 for the rule to trigger.

When we test our rule, notice that our cumulative risk score has exceeded 2000 for two users. We have some additional detection metrics around the total number of detections and unique detections triggered during that two week window. One final thing to point out is that while the detection triggered on August 1, as identified in the test detection itself, when the test detection is expanded and the underlying detections are shown, we can see that some of our detections occurred 10 days prior.

It’s important to note that when running a test rule, the time range of the test must be the same or more than the size of the match window. Because test rules are limited to 14 days currently, make sure you use the full window when testing a match variable that is 14 days in length.

Similarly, with retrohunts, the time range for the retrohunt must be the same or greater than the size of the match window.

Single event rules and Long Detection Windows

We’ve focused most of this discussion on multi-event rules, but you might be wondering if we could use these extended windows for single event rules, that is those single event rules where we are aggregating values. The answer is yes you could, but I’m not sure what benefit you would achieve by doing this. Because threshold rules are treated as multi-event rules, extending the detection window for aggregation mainly will result in getting a chunk of events over a longer time delivered as a single detection. This might be useful if you want to capture some subset of events for periodic analysis but holding the window open for a long time will also delay when it is actually created.  

With that, here are a couple of tips to keep in mind:

• Broadening the detection window is really for multi-event or composite rules that require a longer time to detect low and slow types of activities
• Extending the match window to more than 48 hours will force the run frequency to 24 hours
• When testing or running retrohunts, make sure the time window is at equal or greater than the match window in the rule
• A rule will trigger when the match window closes which means that if we exceed a threshold in 5 minutes but our threshold is set to 14 days, we won’t see the rule trigger until the window closes, so plan accordingly

The extension of the match window to 14 days opens up additional use cases that detection engineers can leverage with Google SecOps. We’d love to hear how you are using this! Whether you are using composite rules, thresholds or comparing disparate events, this broader window provides the ability to find suspicious activity and act on it.