I hope the first part of this guide was useful and thorough. Now in this section we will focus on more advanced operators and cases, including how to handle repeated fields and more advanced nested structures. The last section will have a “capstone” example that utilizes what was discussed in the previous section .
We will use the same log sample described in Security Operations: Deep Dive into UDM Parsing - Part 1.
Advanced Tokenization:
Conditionals
Condition on Simple String Fields |
| Task: Print Tokens if $.event_type = “user_activity” |
|
| Snippet from statedump output: all tokens are printed |
| 1. Equivalent to ; $← flatten($.message) If $.event_type == "user_activity" : Print All Fields 2. Bracket Notation for Field Names: When referencing field names within a conditional statement, you must enclose them in square brackets [...]. For example, to check the value of the event_type field, you would use [event_type] 3. Case-Sensitive Comparison Values: The values used in conditional comparisons are case-sensitive. This means that "user_activity" is considered different from "User_Activity". Ensure that you use the correct case when specifying comparison values. |
Condition on Composite Field Values |
| Task: Print Tokens if $.user.username = “johndoe” and his $.user.id = 12345 |
|
| Snippet from statedump output: all tokens are printed |
| 1. Equivalent to ; $← flatten($.message) If $.user.username == "johndoe" and $.user.id == 12345 Print All Fields 2. GoStash conditional statements can be used to evaluate conditions based on values within nested objects or subfields. • Bracket and Dot Notation for Nested Fields: To refer to a field within a nested object, use square brackets and dot notation. For example, to access the username field within the user object, use [user].[username]. This allows you to target specific values within complex JSON structures. • Support for Non-String Field Arguments: GoStash conditionals support comparisons with various data types, not just strings. You can use conditionals to compare values of integers, booleans, and other data types. • Support for Non-String Field Arguments: GoStash conditionals support comparisons with various data types, not just strings. You can use conditionals to compare values of integers, booleans, and other data types. • Operators: String operators are different from the numerical ones ; a. Numerical : • == (Equal to) • != (Not equal to) • < (Less than) • > (Greater than) • <= (Less than or equal to) • >= (Greater than or equal to) b. String : Will be discussed in the next section. |
Condition on a String Field using Regex and Else-if |
| Task: Print Tokens if $.user.username field is LIKE the regex pattern r”alexi|jo” |
|
| Snippet from statedump output: all tokens are printed |
| 1. Equivalent to ; $← flatten($.message) If $.user.name matches regex (r"alexi|joe") Print All Fields Else: Do Nothing Also equivalent to ; If $.user.name contains ("alexi" OR "joe") Print All Fields 2. Regular Expression Matching: To use regular expressions in string conditionals, replace the standard comparison operator == with the regex matching operator =~ (LIKE Operator) with the same syntax as string matching. The regex non equality (Not LIKE Operator) is !~ . 3. String Operators:a. == (Equal to, case-sensitive) b. != (Not equal to, case-sensitive) c. =~ (Regular expression match) d. !~ (Regular expression does not match) Regular expressions can be used to achieve "contains" functionality. 4. Else Clause: The Else clause provides a way to specify an alternative action to be taken when the If condition evaluates to false. It acts as a catch-all case, ensuring that some action is performed regardless of whether the condition is met. if [token] == "value1" { <code block 1> } else if [token] == "value2" { <code block 2> } else { <code block 3> } |
Multiple Conditions with Logical Operators AND/OR |
| Task: Raise a flag if $.user.username field is LIKE the regex pattern r”alexi|jo” OR $.user.id < 12345 |
|
| Snippet from statedump output: all tokens are printed |
| 1. You can use multiple conditions to filter or process your data, even if those conditions involve different types of information. You can also combine these conditions using 'and', 'or', and other logical operators . |
Conditions using Short-hand Notation |
| Task: Raise a flag if $.user.vip is True |
|
| Snippet from statedump output: all tokens are printed |
| 1. For boolean values (true or false), Chronicle SIEM provides a shorthand notation. 2. if [user][profile][VIP] == true is equivalent to if [user][profile][VIP]. For negation: ![user][profile][VIP] |
Advanced String Tokens Manipulations
Concatenate String Fields |
| Task: Concatenate the username, user id and event type into a single field |
|
| Snippet from statedump output: … "myConcatenatedField": "johndoe-user_activity", …. |
| 1. Equivalent to ; $← flatten($.message) $.myUserName ← $.user.username $.myEventType ← $.event_type $.user.profile.VIP ← bool($.user.profile.VIP) $.myVIP ← $.user.profile.VIP $.user.id ←string($.user.id) $.myUserId ←$.user.id $.myConcatenatedField ← concat ( $.myUserName, "-", $.myUserId, "-", $.myEventType ) 2. We can only combine text fields. To include 'VIP' and 'user ID' in the combined text, we have to convert them to strings. |
Concatenate String Fields of a Composite Field Dynamically |
| Task: Concatenate all the string fields under composite field $.system into a single field |
|
| Snippet from statedump output: … "myConcatField": "server-001|192.168.1.100", …. |
| 1. Equivalent to ; $← flatten($.message) $.myConcatField ← "" For index_, field_ in $.system{} : If $.myConcatField == "" : # Evaluated as True for the first element $.myConcatField ← field_ Else : $.myConcatField ← concat( $.myConcatField, "|", field_ ) 2. A placeholder field is initialized, the loop then iterates through each sub-field within the $.system{} field. In each iteration, the value of the current sub-field is concatenated (joined) to the existing value of the $.myConcatField. This effectively accumulates the values from all the sub-fields into a single field Recall “Loop through Composite Fields” in Part 1, "map" keyword was used in the loop since "system" is a composite field. |
Concatenate String Fields of a Repeated Fields Dynamically |
| Task: Concatenate all the string fields under repeated field $.tags into a single field |
|
| Snippet from statedump output: … "myConcatField": "login_logs|dev", …. |
| 1. Equivalent to ; $← flatten($.message) For index_, field_ in $.Tags{}[] : If index_ == 0 : $.myConcatField ← field_ Else : $.myConcatField ← concat( $.myConcatField, "|", field_ ) 2. In the case of looping through repeated fields, notice that the index variable "index_" is actually numerical, so we did not need to initialize a placeholder here unlike the case of composite field looping. 3. When looping through a repeated field like Tags, you don't need to use the map keyword. 4. If you do use map in the loop, even with a repeated field, you must change how you reference the index. Instead of [index_] == 0, you'll need to use [index_] == "0" (comparing to the string '0'). |
Part 2.2 will cover Advanced Loops, Manipulating the Schema Hierarchy and Sub-fields and Handling Repeated Fields.