Welcome to part two of this blog series focused on building a CI/CD pipeline to manage detection rules in Google Security Operations (SecOps). In part one, I provided an overview of the principles and benefits of managing detection rules as code and a typical detection engineering workflow used by security teams. I configured a CI/CD pipeline in GitHub Actions to pull the latest version of my existing rules via Google SecOps' API and commit them to a GitHub repo.

In this post, I’ll demonstrate how to create and modify detection rules via Google SecOps' API. As a reminder, our Content Manager tool that I'm using for this project can be found on GitHub. Let’s go!

Creating a New Rule

At this point, we have a GitHub Actions workflow that runs on a schedule to pull the latest version of rules from Google SecOps and commit changes to the GitHub repo as code. Now I’ll demonstrate how to create a new rule in the GitHub repo and push the changes to Google SecOps.

As I mentioned in part one, we have a collection of community rules for Google SecOps that can be tested and tailored to fit your organization’s environment. For this demonstration, I’ve copied a rule from the GitHub repo into the rules directory of my project. I’ve also added an entry for the rule in the config file and specified that I want the rule to be enabled and generate alerts. The images of the GitHub pull request below show these proposed changes.

Note, an error will occur if the enabled or alerting option is missing for a rule in the config file.

The security team will discuss the changes in the pull request and collaborate on the changes. The rule logic may be challenged or even the basis for the rule. Remember that conflict can occur at this stage of the process as you welcome a respectful and productive review of team member perspectives.

Testing for Issues

It’s crucial to have a set of tests that execute when changes are being made to detection rules. Once the proposed changes are committed to GitHub in a branch other than the main branch, another GitHub Actions workflow (Run Tests) is executed and does the following:

• Tests that all rule files load successfully.


• Verifies that all rules in the rules directory are valid YARA-L rules. This is accomplished using the verifyRuleText API method.


• Tests for invalid rule configuration. For example, an archived rule cannot be enabled.


• Executes unit tests for the code in this project.

The images below show what it looks like when the pipeline job fails. A rule verification error occurred for the Okta rule because an invalid UDM field name was used, “udm.metadata.product_namee”. Oops. This should be changed to “udm.metadata.product_name”.

Additional Notes on Testing

Mature Detection & Response programs will also have tests to trigger their detection rules on a regular basis (and/or when changes are being made) and validate that alerts were generated. This gives the security team confidence that their logging, detection, and alerting capabilities are working. I'll be writing more about this important topic in an upcoming blog post.

Note that it’s possible to configure protection for the main branch in GitHub/GitLab so that changes cannot be merged into it unless the tests pass and approval has been obtained. This provides more change control over your detection rules.

Pushing Rule Changes to Google SecOps

Once the proposed changes in the pull request have been reviewed and approved they can be merged into the main branch of the project. After the changes are merged, another GitHub Actions workflow (Update Remote Content) runs to push the rule updates to Google SecOps.

The output from the pipeline job below shows that our new rule was created, enabled, and is enabled for alerting.

Finally, the pipeline job pulls the latest version of all rules from Google SecOps and commits them to the repo. This is done to retrieve rule configuration and metadata from Google SecOps such as the rule_id, revision_id, and revision_create_time. The image below shows an example of the rule config file being updated automatically with the latest rule data from Google SecOps.

Modifying Rules

How about if someone needs to modify a rule? This can be done in the rules editor within Google SecOps' UI or via the API. The process for modifying a rule via the GitHub repo is not radically different from how we created a rule earlier.

Let’s say that we want to modify the severity and priority for the Okta rule that was created earlier and also disable alerting for the rule. A Detection Engineer can create a new branch and pull request in the GitHub repo with the proposed changes.

Once the tests pass and approval is obtained, the changes are merged into the main branch and the (Update Remote Content) workflow pushes the rule updates to Google SecOps.

Modifying rules using Google SecOps' Rules Editor

While we’re on the subject of modifying rules in Google SecOps, I’d like to mention a couple of features of the UI that I think are particularly useful to practitioners. First, YARA-L syntax verification happens automatically in the rules editor so authors can easily identify and fix any issues. The following images show what it looks like when a rule compilation error occurs due to a mistyped UDM field name.

Another useful feature of the rules editor enables users to view and manage the version history for a rule. When a user modifies and saves a rule in Google SecOps, a new “rule version" is created.

Many SIEM platforms and security tools do not save the previous state of a rule after it is modified by a user. Google SecOps maintains the version history for detection rules as they’re modified by the security team, which makes it easy to track changes to a rule over time in the UI, compare rule versions, and revert to an earlier version if needed. This functionality is also available via the API.

Wrap up

That’s a wrap for this blog series where we learned how to build a CI/CD pipeline to manage detection rules in Google SecOps — from soup to nuts!

Here’s a summary of what was covered:

• Principles and benefits of managing detection rules as code as well as an example detection engineering workflow used by security teams.


• Configuring a CI/CD pipeline GitHub Actions to pull existing detection rules via Google SecOps' API and commit them to a GitHub repo.


• Creating and modifying detection rules via Google SecOps' API


• The importance of testing as it relates to detection engineering.

If you’re interested in learning more about Detection Engineering and Detection-as-Code, please refer to the “additional reading” section in part one.

As a reminder, the Content Manager tool used in this project can be found in the chronicle/detection-rules repo on GitHub.

Please feel free to reach out on the Google Cloud Security Community with any questions or feedback.

Acknowledgement

Special thanks to the following people for providing me with valuable feedback on this project: James Black, Serhat Gülbetekin, Dave Herrald, Venkata Koppaka, Vishwanath Mantha, Christopher Martin, and John Stoner.