Skip to main content

Miniature Examples

We will avoid using complex loops in this set of examples to focus on demonstrating the mapping algorithm.

We will use the UDM field list https://cloud.google.com/chronicle/docs/reference/udm-field-list as reference for field types.


 

Recipe1: Placeholder without Subfields + Merge + Clear

Used to quickly map several JSON objects into a hierarchy with multiple repeated fields.

 

Map: $.system.ip_address to UDM

UDM Schema: A candidate UDM field will be $.event.idm.read_only_udm.observer.ip

UDM Schema Analysis:  event(Repeated, composite) → idm(composite) → read_only_udm (composite) → observer(composite, non-repeated) → ip(Repeated, atomic string in the form of an IP Address).

AD_4nXcTeEuWDqUlF4eZAYZtOlx3H6VS4T4Zt560YfinqrH8G_hepAYeb-ZUIm_l6B77SwQnrYX_HqzESHy1gA09L-RaWb5xBts9qKnDuSfWZfeGa0G5wjFd7ufSR-U8Ng01URGYu4O-1g?key=D6S2Pz4xypBqnhbqU9jG-8gv

Recipe:

  1. 1. Setup the main block
filter {
json {  source => "message"  array_function => "split_columns"}
…
statedump {label => "end"}
}

  1. 2. Fill in the mandatory “metadata.event_type” field with “GENERIC_EVENT”;

mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}

  1. 3. Analyze the UDM schema $.event.idm.read_only_udm.observer.ip from the right ip to the left event ;

    1. a. The Last field “ip” is a repeated atomic string (Repeated Strings) ⇒ So the placeholder field to capture the input value won’t require any sub-fields ⇒

                      mutate {replace => {"temp" => "%{system.ip_address}"}} #temp ⇔ ip(repeated)

                        So the placeholder variable represents the first repeated field in the left                                  direction, i.e. temp ⇔ ip (Repeated)

  1.              b. Moving to the left ; there are no repeated fields till “event”(Repeated Composite) ⇒ The “merge” operator variable will extend from the rightmost repeated field ip to the leftmost field event , so to map a value to “event.idm.read_only_udm.observer.ip” ⇒ mutate {merge => {"event.idm.read_only_udm.observer.ip" => "temp"}}

  2.              c. Afterwards the placeholder field must be cleared ⇒ 

                               mutate {replace => {"temp" => ""}}

  1. 4. Use the Recipe ;

    1. a. Capture the IP field using a placeholder variable “temp” without any subfields ;

mutate {replace => {"temp" => "%{system.ip_address}"}}

  1.              b. Extend the merge variable to the topmost repeated level from “ip”                 up to “event” ;

mutate {merge => {"event.idm.read_only_udm.observer.ip" => "temp"}}

  1.              c. Clear placeholder ;

mutate {replace => {"temp" => ""}}

  1.              d. Write the events to the parser output, clear the event variable and                  add “statedump” for debugging;

mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}
statedump {label => "end"}
..}

  1.              e. Check UDM Output;

AD_4nXdLwCvfOHDEJQBtmOgWitoeGg_Fiej50Y1-S7lN8WzXx4eYrv5_8GExvB855CRTOuZJJIQbVlID5N5wzNTyCZzQ8QzgaV6Pm7kvaoORpyQ3qTvys5eEcGBeZz_zauV3TGZLSJd1qMI?key=D6S2Pz4xypBqnhbqU9jG-8gv

 
filter {
json {  source => "message"  array_function => "split_columns"}
mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}


mutate {replace => {"temp" => "%{system.ip_address}"}}
mutate {merge => {"event.idm.read_only_udm.observer.ip" => "temp"}}
mutate {replace => {"temp" => ""}}


mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}


statedump {label => "end"}
}

 

"@output": t

    {

      "idm": {

        "read_only_udm": {

          "metadata": {

            "event_type": "GENERIC_EVENT"

          },

          "observer": {

            "ip":

              "192.168.1.100"

            ]

          }

        }

      }

    }

  ],

 

 

Recipe2 : Placeholder with Subfield + Merge + Clear
 

Map: $.system.ip_address to UDM

UDM Schema: A candidate UDM field will be $.event.idm.read_only_udm.observer.ip

UDM Schema Analysis:  event(Repeated) → idm(composite) → read_only_udm (composite) → observer(composite, non-repeated) → security_result (Repeated, composite) → rule_set (non-repeated atomic string)

AD_4nXdRdR7Q6jMDMQ6Do-3-C3dynN_nabaFzX7pLMIxjwPTWKPchWSgee-pLhbbRo5j35vv1Qcyuv-C9PWtXWjGgjLkARaGclvCXLkzVJ7exX0WAog_X13-e6h5GF0Sh0Qa6iNIvNz96OE?key=D6S2Pz4xypBqnhbqU9jG-8gv

Recipe:

  1. 1. Setup the main block

filter {
json {  source => "message"  array_function => "split_columns"}
…
statedump {label => "end"}
}

  1. 2. Fill in the “metadata.event_type” mandatory field;

mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}

  1. 3. Analyzing the UDM schema $.event.idm.read_only_udm.observer.security_result.rule_set from right to left:

    1. a. The 'rule_set' field is a non-repeated atomic string. Moving leftward, the first repeated field encountered is 'security_result' ⇒ Consequently, our temporary placeholder temp will represent a single instance of 'security_result'. To accommodate the 'rule_set' field within this structure, temp will have a sub-field named 'rule_set'.

                             mutate {replace => {"temp.rule_set" => "%{system.ip_address}"}}

                       So here temp ⇔ security_result  and temp.rule_set ⇔ security_result.rule_set

 

  1.             b. Moving up there are repeated fields “security_result” and “event”                  Repeated Composite), We already mapped “security_result” with                        temp, then “merge” operator variable to the topmost level will extend                from the placeholder temp to the leftmost repeated field event

                             mutate {merge => {"event.idm.read_only_udm.observer.security_result"                                    => "temp"}}

                             mutate {replace => {"temp" => ""}}

  1. 4. Use the Recipe ;

  1.           i. Use a Placeholder variable “temp” to capture the first repeated field               “security_result” along with the non-repeated sub-field “ip_address” ;

mutate {replace => {"temp.rule_set" => "%{system.ip_address}"}}

  1.           ii. Extend the merge variable from the first repeated field to the                          leftmost repeated field, i.e. from “security_result” to “event” ;

mutate {merge => {"event.idm.read_only_udm.observer.security_result" => "temp"}}

  1.           iii. Clear placeholder “temp” ;

mutate {replace => {"temp" => ""}}

  1.           iv. Write the events to the parser output, clear the event variable and                 add “statedump” for debugging;

mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}
statedump {label => "end"}
}

  1.           v. Check UDM Output;

AD_4nXdLdD9E7_nQILV5O4zKWAFMiPg6vbG2s4AjaCyUDD3bppXeUdjfZrhmzXMoSKK8kGlzoTv3IQaWvaDHSSzba9TZwxXTcW_-wdUMKml8-JPr6T6tjuv53tUHuWaXzl6ruef9x8gaupI?key=D6S2Pz4xypBqnhbqU9jG-8gv

 
filter {
json {  source => "message"  array_function => "split_columns"}
mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}


mutate {replace => {"temp.rule_set" => "%{system.ip_address}"}}
mutate {merge => {"event.idm.read_only_udm.observer.security_result" => "temp"}}
mutate {replace => {"temp" => ""}}


mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}
statedump {label => "end"}
}

 

  "@output": g

    {

      "idm": {

        "read_only_udm": {

          "metadata": {

            "event_type": "GENERIC_EVENT"

          },

          "observer": {

            "security_result":

              {

                "rule_set": "192.168.1.100"

              }

            ]

          }

        }

      }

    }

  ],

Extension: Adding More Sub-fields

Map: In addition to $.system.ip_address, map also $.system.hostname to UDM

UDM Schema: Suggested Mapping 

event.idm.read_only_udm.observer.security_result.rule_set ← $.system.ip_address

event.idm.read_only_udm.observer.security_result.summary ← $.system.hostname

 

UDM Schema Analysis:  event(Repeated) → idm(composite) → read_only_udm (composite) → observer(composite, non-repeated) → security_result (Repeated, composite) → rule_set, summary (both are non-repeated atomic string)

AD_4nXepEtKB95pJnCoY5SEWxvhVC0lXsLxzUkJGJIzzp0zkPA_Ryv5w84mJ17OpORLsDQMKnoGq_QYR9FaNbhzAgFP8tjfXCQWYZ2P_XLXYZAmaoPfmovx4yAMMFbb09F_SzXqSg2liFi8?key=D6S2Pz4xypBqnhbqU9jG-8gv

Recipe:

  1. 1. Following the exact same steps in Recipe2, but we are just going to add an additional sub-field in the first step;

mutate {replace => {"temp.rule_set" => "%{system.ip_address}"}}
mutate {replace => {"temp.summary" => "%{system.hostname}"}}
 

So here temp ⇔ security_result , temp.rule_set ⇔ security_result.rule_set, temp.summary ⇔ security_result.summary

 

Check UDM Output;

AD_4nXfVPKL1Q4CxCo3Ch41ad0JfPErpUqQl6OzUqOhXBZc5OhesjV4krj32d6EvEFHdN77AZ8SQBzylMLTLJ3yWrNuCglCfBpiAjVRDdIhQOid-DCpKdi_jQqI8Nu4Mob_uqODC32q4uQs?key=D6S2Pz4xypBqnhbqU9jG-8gv

 

 

filter {
json {  source => "message"  array_function => "split_columns"}
mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}


mutate {replace => {"temp.rule_set" => "%{system.ip_address}"}}
mutate {replace => {"temp.summary" => "%{system.hostname}"}}
mutate {merge => {"event.idm.read_only_udm.observer.security_result" => "temp"}}
mutate {replace => {"temp" => ""}}


mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}
statedump {label => "end"}
}

 

"@output": p

    {

      "idm": {

        "read_only_udm": {

          "metadata": {

            "event_type": "GENERIC_EVENT"

          },

          "observer": {

            "security_result": "

              {

                "rule_set": "192.168.1.100",

                "summary": "server-001"

              }

            ]

          }

        }

      }

    }

  ],



 

 

 

Recipe3 : Temp Placeholder with Multiple Subfields + Multiple Merge + Clear
 

Map: $.system.ip_address to UDM

UDM Schema: A candidate UDM field will be $.event.idm.read_only_udm.intermediary.security_result.rule_set , summary

UDM Schema Analysis:  event(Repeated) → idm(composite) → read_only_udm (composite) → intermediary(Repeated, composite) → security_result (Repeated, composite) → rule_set (non-repeated atomic string)

 

The difference between this example and Recipe2 is that here the UDM schema contains another repeated field “intermediary” unlike “observer” used earlier which is not repeated.

Recipe:

  1. 1. Setup the main block

filter {
json {  source => "message"  array_function => "split_columns"}
…
statedump {label => "end"}
}

Fill in the “metadata.event_type” mandatory field;

mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}

  1. 2. Analyzing the UDM schema $.event.idm.read_only_udm.intermediary.security_result.rule_set from right to left looking for repeated fields ;

    1. a. The repeated fields are security_result, intermediary then from right to left we have few non-repeated fields up to event repeated field ⇒ we will need 2 placeholder variables for security_result and intermediary then link them ;

      1.  i. temp1 for security_result : done as in previous examples

                                   mutate {replace => {"temp.rule_set" => "%{system.ip_address}"}}

                                   mutate {replace => {"temp.summary" => "%{system.hostname}"}}

                                   mutate {merge =>                                                                                                                                  {"event.idm.read_only_udm.intermediary.security_result" => "temp"}}

                                   mutate {replace => {"temp" => ""}}

 

  1.                           ii. temp2 for intermediary : in this case, to link the repeated                                  field security_result to intermediary  ; rename is used to add a                           placeholder temp2 representing intermediary  on top of                                       security_result

                                   mutate {rename =>                                                                                                                                {"event.idm.read_only_udm.intermediary.security_result" =>                                                       "temp2.security_result"}}

 

                                   Here temp2 ⇔ intermediary, Afterwards merge temp2 into UDM till                                          event sameway as temp1

                                   mutate {merge => {"event.idm.read_only_udm.intermediary" =>                                                  "temp2"}}

                                   for placeholders; temp1 ⇔ security_result  and temp2 ⇔ intermediary

 

                                   This forms the second block;

                                   mutate {rename =>                                                                                                                               {"event.idm.read_only_udm.intermediary.security_result" =>                                                       "temp2.security_result"}}

                                   mutate {merge => {"event.idm.read_only_udm.intermediary" =>                                                  "temp2"}}

                                   mutate {replace => {"temp2" => ""}}



 

  1. 5. Use the Recipe ;

  1. Placeholder capture with sub-field for both same-level non-repeating fields  ;

mutate {replace => {"temp.rule_set" => "%{system.ip_address}"}}
mutate {replace => {"temp.summary" => "%{system.hostname}"}}

  1. Extend the merge variable to the topmost non-repeated level, here we could not extend it beyond “security_result” since “intermediary” is also repeated ;

mutate {merge => {"security_result" => "temp"}}

  1. Clear the “temp” placeholder ;

mutate {replace => {"temp" => ""}}

  1. Use the “rename” trick to start a new hierarchy from “security_result” ;

mutate {rename => {"security_result" => "temp2.security_result"}}
 

  1. Construct the 2nd “merge” variable to cover “intermediary”, and since we have no more uppermost repeated fields ⇒ Extend to the topmost level ;

mutate {merge => {"event.idm.read_only_udm.intermediary" => "temp2"}}

  1. Clear the 2nd Placeholder variable ;

mutate {replace => {"temp2" => ""}}

  1. Write the events to the parser output, clear the event variable and add “statedump” for debugging;

mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}
statedump {label => "end"}
}

Check UDM Output;

AD_4nXdzJVyeyVagO0jG9_OAhcIzVbe8MYUY_I0fBwtnr89Jk7-eBze9oA7za6yVIW0PhNSC6H4TyWobSqP57CZGUXHVufx4mwJo62P8tIJ835-5MyiMGdAcQNSJfTHSMf0PxAbU_r-Y5Qo?key=D6S2Pz4xypBqnhbqU9jG-8gv

 
filter {
json {  source => "message"  array_function => "split_columns"}
mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}


mutate {replace => {"temp.rule_set" => "%{system.ip_address}"}}
mutate {replace => {"temp.summary" => "%{system.hostname}"}}
mutate {merge => {"event.idm.read_only_udm.intermediary.security_result" => "temp"}}
mutate {replace => {"temp" => ""}}


mutate {rename => {"event.idm.read_only_udm.intermediary.security_result" => "temp2.security_result"}}
mutate {merge => {"event.idm.read_only_udm.intermediary" => "temp2"}}
mutate {replace => {"temp2" => ""}}


mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}
statedump {label => "end"}
}

 

 
#Another Version
filter {
json {  source => "message"  array_function => "split_columns"}
mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}


mutate {replace => {"temp.rule_set" => "%{system.ip_address}"}}
mutate {replace => {"temp.summary" => "%{system.hostname}"}}
mutate {merge => {"security_result" => "temp"}}
mutate {replace => {"temp" => ""}}




mutate {rename => {"security_result" => "temp2.security_result"}}
mutate {merge => {"event.idm.read_only_udm.intermediary" => "temp2"}}
mutate {replace => {"temp2" => ""}}




mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}
statedump {label => "end"}
}

 

"@output": t

    {

      "idm": {

        "read_only_udm": {

          "intermediary": s

            {},

            {

              "security_result": t

                {

                  "rule_set": "192.168.1.100",

                  "summary": "server-001"

                }

              ]

            }

          ],

          "metadata": {

            "event_type": "GENERIC_EVENT"

          }

        }

      }

    }

  ],


 

Recipe4 : Temp Placeholder with Multiple Subfields + Multiple Merge + Clear + Intersecting Branches
 

Map: $.system.ip_address and $.system.hostname UDM

UDM Schema: Candidate UDM fields will be; 

  1. a) $.event.idm.read_only_udm.intermediary.ip

  2. b) $.event.idm.read_only_udm.intermediary.security_result.summary

  3. c) $.event.idm.read_only_udm.intermediary.security_result.rule_set

 

UDM Schema Analysis:  

  1. a) event(Repeated) → idm(composite) → read_only_udm (composite) → intermediary(Repeated, composite) → ip (Repeated, atomic string)

  2. b) event(Repeated) → idm(composite) → read_only_udm (composite) → intermediary(Repeated, composite) → security_result (Repeated, composite) → rule_set (non-repeated atomic string)

 

Unlike Recipe3, here the UDM schema contains another repeated field “ip” in addition to “intermediary” and “security_result”.
 

Recipe:

  1. 1. Setup the main block

filter {
json {  source => "message"  array_function => "split_columns"}
…
statedump {label => "end"}
}

  1. 2. Fill in the “metadata.event_type” mandatory field;

mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}

  1. 3. Analyze the target schema from the right to the left for both paths for the repeated fields; 

                   $.event.idm.read_only_udm.intermediary.security_result.summary

                   $.event.idm.read_only_udm.intermediary.security_result.rule_set

                   $.event.idm.read_only_udm.intermediary.ip

 

  1. a. Start with the farthest rightmost field  ;

    1. i. “rule_set” is not repeated ⇒ no need for placeholder

  2. b. Check for other fields on the same level (having the same parent field) 

    1. i. Only 1 field exists under the same parent security_result ; 

      1. 1. “summary” is a non-repeated field ⇒ no need for placeholder

    2. ii. Since “summary” and “rule_set” are Non-repeating sharing the same parent field ⇒  Use summary and rule_set as sub-fields for their parent field ⇒ parent.summary and parent.rule_set

  3. c. Next field to the left ; “security_result” is repeated+composite ⇒ We will need to use a placeholder + Merge.

    1. i. Then temp_security_result ⇔ security_result

    2. ii. From (b.ii) ⇒then temp_security_result .summary and temp_security_result .rule_set  will represent the sub-fields

  4. d. No more field on the same level under security_result, so start mapping security_result same way as in previous recipes

  1. 4. Mapping is done for the first 3 variables ;

                   mutate {replace => {"temp_security_result.rule_set" =>

                   "%{system.ip_address}"}}

                   mutate {replace => {"temp_security_result.summary" =>

                   "%{system.hostname}"}}

                   mutate {merge => {"security_result" => "temp_security_result"}}

                   mutate {replace => {"temp_security_result" => ""}}

 

  1. 5. As in 3b ; Check for other fields in the same level of security_result 

    1. a. ip field is in the same level as  security_result ⇒ We need to map ip before going further to the left.

    2. b. ip is a repeated field ⇒ Map it using a placeholder + merge as before ;

                   mutate {replace => {"temp_ip" => "%{system.ip_address}"}}

                   mutate {merge => {"ip" => "temp_ip"}}

                   mutate {replace => {"temp_ip" => ""}}

 

               So far we mapped 3 fields; Security_result.summary, security_result.rule_set, ip

  1. 6. Moving left;  

    1. a. “intermediary” field is the parent field of the 3 mapped fields ⇒ we need to link the 3 fields into intermediary, i.e. all 3 fields should be sub-fields to intermediary.

  2.             b. intermediary is a repeated field ⇒ it will require a placeholder                        variable + Merge

                       Linking variables is done using rename to assign a common parent field for                          ip and security_result, and since this parent field is repeated ⇒ use a                                     placeholder variable temp_intermediary

                       mutate {rename => {"ip" => "temp_intermediary.ip"}}

                       mutate {rename => {"security_result" => "temp_intermediary.security_result"}}

  1.            c. The rest of code goes the same way for handling repeated fields ;

                       mutate {merge => {"event.idm.read_only_udm.intermediary" =>                                                 "temp_intermediary"}}

                       mutate {replace => {"temp_intermediary" => ""}}

 

In short : Follow the same algorithm for handling repeated variables but connect the variables at the common fields using rename. 

Here the common field between both paths is event{}.idm.read_only_udm{}.intermediaryl]{}, so use the “temp_intermediary” placeholder for “intermediary;]{}” to link both branches. 

 

Use the Recipe ;

  1. a. For the first branch $.event.idm.read_only_udm.intermediary.security_result.rule_set,summary ; Capture the tokens into sub-fields of the first rightmost repeated field security_result via its placeholder temp_security_result and extending it using the sub-fields temp_security_result .summary and temp_security_result.rule_set , then clear the placeholder variable;

mutate {replace => {"temp_security_result.rule_set" => "%{system.ip_address}"}}
mutate {replace => {"temp_security_result.summary" => "%{system.hostname}"}}
mutate {merge => {"security_result" => "temp_security_result"}}


mutate {replace => {"temp_security_result" => ""}}

  1. b. Use rename to connect “security_result” with the next rightmost repeated field intermediary using another placeholder temp_intermediary  ;

mutate {rename => {"security_result" => "temp_intermediary.security_result"}}
 

  1. c. For the second branch $.event.idm.read_only_udm.intermediary.ip ⇒ ip field is the rightmost repeated field so its placeholder is temp_ip without sub-fields :

mutate {replace => {"temp_ip" => "%{system.ip_address}"}}
mutate {merge => {"ip" => "temp_ip"}}
mutate {replace => {"temp_ip" => ""}}

 

  1. d. Use rename to connect “ip” with the next rightmost repeated field intermediary using the placeholder temp_intermediary  ;

mutate {rename => {"security_result" => "temp_intermediary.security_result"}}
 

  1. e. Construct the 2nd “merge” variable to cover “intermediary”, and since we have no more uppermost repeated fields till event ⇒ Extend to the topmost field event ;

mutate {merge => {"event.idm.read_only_udm.intermediary" => "temp_intermediary"}}
mutate {replace => {"temp_intermediary" => ""}}

  1. f. Write the events to the parser output, clear the event variable and add “statedump” for debugging;

mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}
statedump {label => "end"}
 

Check UDM Output;

AD_4nXdXaQV2jN_v-NHACEYXijyuBx8TQ4hHnlg44IIlZIbBOFNzs931owpdllVmdaJnPAt9pdFzJ8q09c23wxEnDMc4iNSm_9KsQoZFnxpcJgcEs8RPZjJKUXEx3MFfbWhouwVIcigvvos?key=D6S2Pz4xypBqnhbqU9jG-8gv

 
filter {
json {  source => "message"  array_function => "split_columns"}
mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}


#Map security_result.*
mutate {replace => {"temp_security_result.rule_set" => "%{system.ip_address}"}}
mutate {replace => {"temp_security_result.summary" => "%{system.hostname}"}}
mutate {merge => {"security_result" => "temp_security_result"}}
mutate {replace => {"temp_security_result" => ""}}


#Link security_result.* to *.intermediary.security_result
mutate {rename => {"security_result" => "temp_intermediary.security_result"}}


#Map ip
mutate {replace => {"temp_ip" => "%{system.ip_address}"}}
mutate {merge => {"ip" => "temp_ip"}}
mutate {replace => {"temp_ip" => ""}}


#Link ip to *.intermediary.ip
mutate {rename => {"ip" => "temp_intermediary.ip"}}


#Map event.idm.read_only_udm.intermediary
mutate {merge => {"event.idm.read_only_udm.intermediary" => "temp_intermediary"}}
mutate {replace => {"temp_intermediary" => ""}}




mutate {merge => { "@output" => "event" }}
mutate {replace => {"event" => ""}}
statedump {label => "end"}
}

 

"@output": /

    {

      "idm": {

        "read_only_udm": {

          "intermediary": a

            {

              "ip": F

                "192.168.1.100"

              ],

              "security_result": 9

                {

                  "rule_set": "192.168.1.100",

                  "summary": "server-001"

                }

              ]

            }

          ],

          "metadata": {

            "event_type": "GENERIC_EVENT"

          }

        }

      }

    }

  ],


 

 

Capstone UDM Mapping Example

Constructing a List of Complex Composite JSON Objects Across Multiple Repeated Fields in UDM Schema

 

In this example we will go in detail explaining a more complex case where the same log message is going to be parsed into multiple events.

 

In this example, let us ;

  1. Use the “GENERIC_EVENT” event type. 

  2. Map the “actions” objects fields (session id, action, query, targetIP) in a suitable sub-field of “SecurityResults” fields.

  3. Generate one event for each user-session , so based on the log we have, the parser should generate 2 events, one per session.

 

We start our analysis to build the parser ;

 

  1. The topmost level for UDM schema is “whatever.idm.read_only_udm”, so any field in the target schema will have this field as a parent.

It is commonly used to use “event.idm.read_only_udm” , but in general you could replace “event” with any placeholder name.

Each $.event instance is an event generated from the parser. So we expect “event1” and “event2” parent nodes, OR Loop through the sessions to generate a different session event for each session.

 

For i0, event0 in $user{}.session{}p]:

$.event…something ← some fields

Write $.event to the Parser output

Clear $.event 

 

  1. Starting with the mandatory field “Metadata.event_type” https://cloud.google.com/chronicle/docs/unified-data-model/udm-usage#metadataevent_type , so our first field will be ;

 

$.event.idm.read_only_udm.metadata.event_type

More information about this field in https://cloud.google.com/chronicle/docs/reference/udm-field-list#metadata

AD_4nXe5tHJ_lr0p15MO3y4s9Ig35TRqhR5A57GgkOLD0wqqwFqGq7edhK7L8xCsFQUpf3vB_jWyxWzILC4iDKIxpVtcCEfHQUWvd-4XJyFQBGvtb7DaRfdWCtSFB5zwhQN8ojawK-v7dnw?key=D6S2Pz4xypBqnhbqU9jG-8gv

This field type is a custom type, called “Metadata.EventType”.

Clicking on the type field link, we find more info about the field, indicating some “Enum Value”.

Enumerated fields like metadata.event_type can only take certain values.

We will choose the value “GENERIC_EVENT” for this field in the parser.

 

  1. Include the optional vendor/product fields in our parser ;

Encoding: Case-sensitive, alphanumeric string, punctuation allowed , so we can set any arbitrary vendor value like “myVendor” 

 

  • event.idm.read_only_udm.metadata.product_name

https://cloud.google.com/chronicle/docs/unified-data-model/udm-usage#metadataproduct_name

Similarly we set its value to “myProduct”

 

  1. We start tracking the “security_result” field https://cloud.google.com/chronicle/docs/unified-data-model/udm-usage#result-metadata and https://cloud.google.com/chronicle/docs/reference/udm-field-list#udm_event_data_model

This field was chosen because it is “Repeated”

AD_4nXcBgBTRMJ3cvvnS88R4SW6M3LNt-4l6nLZj45a8sMvebv2SpPXmZ72IJj7n6yO_Nfw31Upaq4OA6Pqzb-x2MykoN7rhp6-WEkw8UQoGdSKXSB1MC_IdwsD8D8c3XcazUTK9DdRfCmU?key=D6S2Pz4xypBqnhbqU9jG-8gv

The “security_result” is the parent field name, its schema/data type/object type is “SecurityResult”

We pick a subfield field “category_details” listed under “security_result” by looking up the “SecurityResult” custom object https://cloud.google.com/chronicle/docs/reference/udm-field-list#securityresult

AD_4nXeKthW4kJTfXZEW6nTETGi5bUP8yGGyzk4cJ9xVSm8ujk7OjWPJE8LoGyxC-w8rKvwpNXJSJmfY2tFy5_e4UoGmTZxgYqVCoud0qXMTKj4h_J40r2pZZ4S7TQVCxnvhf5sM--7NN4A?key=D6S2Pz4xypBqnhbqU9jG-8gv

So the full field name will be ; 

  • event.idm.read_only_udm.metadata.securityResult.category_details

 

Applying the algorithm ;

  • Target UDM Schema ; 

  1. event (Composite).idm (Composite).read_only_udm (Composite).security_result (Repeated Composite).category_details (Repeated String)

Modeled as ;

event{}.idm{}.read_only_udm{}.security_result{}/].category_details/]

 

  1. event (composite).idm (composite).read_only_udm (composite).metadata( Composite).event_type(string)

Modeled as ;

event{}.idm{}.read_only_udm{}.metadata{}.event_type{}

 

  • Input Log fields Schema Mappings  ;

  1. user{}.sessions{}o] → event{} : Repeated Field to Composite, so it will require a loop with event{} inside the loop.

 

for i0, v0 in user.sessions :

event{} ← v0

 

  1. user{}.sessions{}_].actions{}t].action_type → event{}.idm{}.read_only_udm{}.security_result{}u].category_detailsl]

 

  • Start off with the mandatory “metadata.event_type” and the optional “metadata.vendor_name”/“metadata.vendor_name” ⇒ Mapping the “event_type” field early facilitates the troubleshooting/debugging.

Since we need to map these fields per-event AND we have 1 event per user session ⇒ these mappings will be inside the outer $.user.sessions{}p] loop ;

for i0_, v0_ in user.sessions {
    mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}
    mutate {replace => {"event.idm.read_only_udm.metadata.vendor_name" => "myVendor"}}
    mutate {replace => {"event.idm.read_only_udm.metadata.product_name" => "myProduct"}}

 

  • Looking at the Input Schema, use Top-Down approach to start from the root of the input schema to reach the required data type;

  1. “action_type” is a primitive data type (string).

  2.  To reach “action_type” in the input schema, there are 2 repeated fields in the path “user{}.sessions{}Z]” and “user{}.sessions{}Z].actions{}h]” ⇒ We need 2 loops.

for i0_, v0_ in user.sessions {
    for i1_, v1_ in v0_.actions {

  1. Switching to the target schema ;

  1.  We need to map “action_type” into a repeated field that is not composite “category_detailsf]” 

I.e. ;

..some parent fields…category_details>] ← v1_.action_type

⇒ Need to use the recipe of temp placeholder + “merge” inside the second loop.

  1. How far can we extend the hierarchy of the “category_details” ? ⇒ Move upwards till the first Repeated field 

Moving up from “category_details” : We stop at the first repeated field up which is “security_result” ⇒ We cannot hierarchize the “merge” variable further.

  1. Construct the placeholder variable with no extended hierarchy ;

“temp.category_details”

  1. Use the recipe of the temp placeholder + “merge” + clear placeholder +   ;

for i0_, v0_ in user.sessions {
    for i1_, v1_ in v0_.actions {
        mutate {replace => {"temp.category_details" => "%{v1_.action_type}"}}
        mutate {merge => {"category_details" => "temp.category_details"}}
        mutate {replace => {"temp" => ""}}
        statedump {label => "inside_v0.actions_loop"}
    }

  1. Verify using the last “statedump” inside the loop ;

We can see:  {.."category_details": v"login"],..} on the first run, {.."category_details": "login",”search”],..} on the second, and {.."category_details": l"login",”search”,”logout”],..} on the 3rd run.

  1. Now we finished the mapping of ;

category_detailse] ← v1_.action_type

What we are missing is the mapping of the parent of category_details>], which is event{}.idm{}.read_only_udm{}.security_result{}g].

 

  1. To move one level up in the target UDM schema, we exit the inner loop of i1_,v1_ .

  2. Now we start mapping the parent event{}.idm{}.read_only_udm{}.security_result{}v] : 

    1. Move upwards from security_result{}d] till the first repeated field ⇒ There are no further repeated fields upwards. Great ! So we can extend the placeholder up till the topmost field event{} ⇒ Use the “rename” recipe with “merge” to introduce a temporary hierarchy with a dummy placeholder.

 

  1. Introduce a dummy placeholder upper level using rename to “category_details”;

    mutate {rename => {"category_details" => "temp.category_details"}}

Now we have constructed ; 

temp{}.category_detailsl]

  1. Append this placeholder to the repeated field “security_result{}_]”, and since “security_result{}a]” does not have any further repeated parent fields ⇒ We can extend the repeated field name upwards to the desired level.

    mutate {merge => {"event.idm.read_only_udm.security_result" => "temp"}}

This eliminates (technically sandwiches) the “temp” dummy level.

 

  1. Do not forget to clear the dummy placeholder ; 

    mutate {replace => { "temp" => "" }}

 

F. Now we have constructed the full path ;

event{}.idm{}.read_only_udm{}.security_result{}p].category_details{}

 

"event": {

    "idm": {

      "read_only_udm": {

        "metadata": {

          "event_type": "GENERIC_EVENT",

          "product_name": "myProduct",

          "vendor_name": "myVendor"

        },

        "security_result": _

          {

            "category_details": g

              "login",

              "search",

              "logout"

            ]

          },

          {

            "category_details": /

              "login"

            ]

          }

        ]

      }

    }

  },

 

  1. Write the existing UDM schema through its root node “$.event” using this special “merge” statement. Make sure this “merge” is at the end of the outer loop since we need a single event per session : 

for i0_, v0_ in user.sessions {
………………    
    mutate {merge => { "@output" => "event" }}
}

 

  1. After writing the $.event json to the output, do not forget to clear the $.event object at the end of the loop to clear the schema for the following second session event

for i0_, v0_ in user.sessions {
……………….    
    mutate {merge => { "@output" => "event" }}
    mutate {replace => { "event" => "" }}
}

  1. Add “statedump” print statements at the end of each loop and the end of the parser.

    statedump {label => "end.user.sessions__loop"}
}


statedump {label => "end"}
}

 

  1. If the events are written successfully ; on the left hand side of the UI, you will see the “UDM Output” Tab enabled with what will be the UDM events ;

 

AD_4nXdX18QqAGIPhyO_KaJnpeWkNkw6XSaF1Gf6-zULYh7k_iWezMcH8J1u1_D5VFotmbNgvQD7iNbYrQBJ_CH79AfPpdXjZjo5Cpb-1mHYv5cAO7O4MKgCy6QSNf1stgV1qkl-sxopC4A?key=D6S2Pz4xypBqnhbqU9jG-8gv

 

  1. Statedump should show the special “:@output” Schema that will be serialized by the parser to the UDM stream ;

AD_4nXefGK0nER3HgoBxAyRDdH5xe9PwVFUAxcllVA-ZlEv961BFyu-tI2umA1AdEOv92xKk9FVcmArxKnn8PBZ9PFXbcbrebAhVKZs8wC2UHQ73Ndgj92ExWOjrmZdOCaCcEgtWF3AcMA?key=D6S2Pz4xypBqnhbqU9jG-8gv


 

Capstone Example Code

 
filter {


json {  source => "message"  array_function => "split_columns"}




for i0_, v0_ in user.sessions {
    mutate {replace => {"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"}}
    mutate {replace => {"event.idm.read_only_udm.metadata.vendor_name" => "myVendor"}}
    mutate {replace => {"event.idm.read_only_udm.metadata.product_name" => "myProduct"}}


    for i1_, v1_ in v0_.actions {
        mutate {replace => {"temp.category_details" => "%{v1_.action_type}"}}
        mutate {merge => {"category_details" => "temp.category_details"}}
        mutate {replace => {"temp" => ""}}
        statedump {label => "end.v0_.actions__loop"}
    }


    mutate {rename => {"category_details" => "temp.category_details"}}
    mutate {merge => {"event.idm.read_only_udm.security_result" => "temp"}}
    mutate {replace => { "temp" => "" }}

    mutate {merge => { "@output" => "event" }}
    mutate {replace => { "event" => "" }}
    statedump {label => "end.user.sessions__loop"}
}


statedump {label => "end"}
}

 

AD_4nXdfEE_xG4RFzGlW36E90ywI2GZS9OPSMB3pdXhYXuc64P2sOVSl3BhSZvoRqgOzmieN9l_YEaAlJOd_3pM7ZH8V4DdYiWF3nuS57vbviC1XbwVvRHKEy2ZZm3lDT-pefCZmiWAjfQ?key=D6S2Pz4xypBqnhbqU9jG-8gv
 

First Event ; 

Internal State (label=end.user.sessions__loop):

 

 "@output": U

    {

      "idm": {

        "read_only_udm": {

          "metadata": {

            "event_type": "GENERIC_EVENT",

            "product_name": "myProduct",

            "vendor_name": "myVendor"

          },

          "security_result": A

            {

              "category_details": M

                "login",

                "search",

                "logout"

              ]

            }

          ]

        }

      }

    }

  ],

 

Second Event ; 

Internal State (label=end.user.sessions__loop):

 

   {

      "idm": {

        "read_only_udm": {

          "metadata": {

            "event_type": "GENERIC_EVENT",

            "product_name": "myProduct",

            "vendor_name": "myVendor"

          },

          "security_result": n

            {

              "category_details": Z

                "login"

              ]

            }

          ]

        }

      }

    }

  ],

 

Parser Extensions

Parser extensions can be viewed as main parser add-ons to map extra fields or overwrite fields parsed by the main fields.

Example Use Cases : 

  1. VMWare error logs are parsed but we need to extract the error trace ID in a different field.

  2. Events are parsed as metadata.event_type = "GENERIC_EVENT" but we need to properly categorize them.

The main criteria for a Parser Extensions ;

  1. The parser extension is attached to a main parser.

  2. The main parser must be able to parse the main log message investigated, the extension will only add to or overwrite the fields generated by the main parser.

  3. Samples of logs are needed as an input to the SIEM to validate the parser extension.

 

Conclusion

So far we covered the repeated fields and how to systematically map fields to the UDM format. In future versions we will cover more advanced topics for bulk logs analysis and automation.

 

Thank you for taking the time to review this guide!

Be the first to reply!

Reply


Terms & ConditionsCookie settings
Privacy Policy Terms of Service Gen AI Policy Community Guidelines Cookie Settings

© 2025 Google. All rights reserved.