After onboarding a new log type to Google Security Operations (SecOps), the next step for a Detection Engineer is usually to explore the various events and fields that have been parsed and plan how to use the data for searches, detections, and investigations.

But what if a specific field is missing? The data exists in the raw log, but it’s not being parsed into a UDM field for ease of use in your searches, rules, and dashboards. Until recently, the solution was to learn how to develop a custom parser extension to manually extract the field into a UDM field.

I’ll show you how to solve this problem in just a few minutes using the Parser Extension Generator in SecOps Labs, which leverages natural language to build the parser for you.

SecOps Labs is a dedicated space in Google SecOps that provides early access to AI pilots. It’s where customers can experiment with applying AI to their security operations functions.

The Problem: A Missing Piece of the Puzzle

Let’s look at a real-world example. I’m ingesting GitHub Enterprise audit logs in Google SecOps. I’m particularly interested in the value of the “secrets_passed” field when a GitHub Actions workflow is started. This field contains the names of the secrets that were provided to the GitHub Actions job and I want to present this information in alerts that are generated by my GitHub rules.

The image above shows the presence of the “secrets_passed” field in the raw log event that was ingested into Google SecOps. The image below shows the UDM event that was created by the default parser for GitHub log events. Note the absence of the “secrets_passed” field in this UDM event.

The Solution: The SecOps Labs Parser Generator

After grabbing a copy of the raw log event, I head over to SecOps Labs and open the Parser Extension lab.

To generate a parser extension, we need to provide the following information:

1. The log type that we want to create the parser extension for. The log type is “GITHUB” in this example.
2. A copy of the raw log event that was ingested into Google SecOps that contains the fields and values that we want to parse. I copied the raw log event pictured earlier in this post.
3. A natural language instruction describing what the parser extension should do.

I’ve included a copy of my natural language instruction below for reference.

Create a parser extension that parses the "secrets_passed" field into the UDM field additional.fields["secrets_passed"]. The values from the "secrets_passed" field should be parsed into a comma-separated string like this: "SECRET1, SECRET2, SECRET3"

The parser extension code is returned. You can find a copy of it at the end of this post. We can test it by navigating to SIEM Settings - Parsers and creating a new parser extension for the “GITHUB” log type.

Testing the Parser Extension

To test the parser extension, I paste the raw log event into the box on the left and the generated parser extension code on the right. Clicking “Preview UDM Output” runs the parser extension against the raw log event and shows the parsed UDM event.

Remember, the Parser Extension Generator is an experimental feature. You should always review the parser logic and perform testing & validation against your logs before submitting it.

It worked! The secret names were parsed into the UDM field, additional.fields[“secrets_passed”]

Clicking “Validate” runs the parser extension against all my existing GitHub events that have been ingested in Google SecOps. The results, including any parsing errors, show up once validation is complete.

The results below show that several parsing errors occurred. In the “Failed Events” tab, I’ll go ahead and click the “Use This Log” button to figure out why the parser extension failed to parse this event.

I see the problem with the parser extension’s logic. The final step in the parser extension is to copy the “event” field to the UDM event. The “event” field is only being created when the “secrets_passed” field is present in the raw GitHub log event. This field was present in the sample raw log event that I provided in SecOps Labs, but it isn’t present in all GitHub events, so we’ll tweak the parser extension and validate it again.

I’ve updated the parser extension to check if the “event” field is populated before attempting to copy it to the UDM event.

Validating the parser extension again tells us that all of my raw GitHub raw log events were parsed successfully. The parser extension is now ready to be submitted and used for all future GitHub events ingested into Google SecOps. This will allow me to use the additional.fields[“secrets_passed”] field in my searches, rules, and dashboards going forward.

Final Thoughts

The Parser Extension Generator in SecOps Labs provides a quick way to parse fields of interest from raw log events to UDM format, unlocking more data for searches, rules, and dashboards. Validating new parser extension logic against your existing events in Google SecOps is a crucial (and required) step to catch any errors before submitting changes.

We encourage you to take a look at the other experiments in SecOps Labs and provide feedback within Google SecOps.

Appendix

Parser Extension Example

filter {
  # Initialize temporary variables used for string concatenation
  mutate {
    replace => {
      "event" => ""
      "secrets_passed_string" => ""           # To build the comma-separated string
      "temp_secret" => ""                    # Temporary variable for each secret during iteration
      "is_first_secret" => "true"            # Flag to handle comma placement correctly
      "_json_parse_error" => "false"         # Initialize boolean for JSON parsing error handling
    }
  }

  # Parse the incoming log message as JSON.
  # The array_function => "split_columns" ensures that array elements can be accessed by index (e.g., secrets_passed.0).
  # Even with this function, the original array field (secrets_passed) can typically still be iterated over.
  json {
    source => "message"
    array_function => "split_columns"
    on_error => "_json_parse_error"
  }

  # Proceed only if JSON parsing was successful
  if ![_json_parse_error] {
    # Iterate over the 'secrets_passed' array from the parsed JSON.
    # The loop will not execute if 'secrets_passed' does not exist or is empty.
    for index, secret in secrets_passed {
      # Assign the current secret element to a temporary variable
      mutate {
        replace => {
          "temp_secret" => "%{secret}"
        }
      }

      # Concatenate secrets into a comma-separated string.
      # The 'is_first_secret' flag ensures that a comma is only added between elements, not before the first one.
      if [is_first_secret] == "true" {
        mutate {
          replace => {
            "secrets_passed_string" => "%{temp_secret}"
            "is_first_secret" => "false" # Set flag to false after processing the first element
          }
        }
      } else {
        mutate {
          replace => {
            "secrets_passed_string" => "%{secrets_passed_string}, %{temp_secret}"
          }
        }
      }
    }

    # After the loop, if the 'secrets_passed_string' has been populated, map it to UDM.
    if [secrets_passed_string] != "" {
      mutate {
        replace => {
          "additional_secrets_field.key" => "secrets_passed"
          "additional_secrets_field.value.string_value" => "%{secrets_passed_string}"
        }
      }
      # Merge the newly created custom additional field into the event's additional.fields.
      mutate {
        merge => {
          "event.idm.read_only_udm.additional.fields" => "additional_secrets_field"
        }
      }
    }
  }

  # Final merge to output the UDM event
  if [event] != "" {
    mutate {
      merge => {
        "@output" => "event"
      }
    }
  }
}