Skip to main content
Question

Custom Parser nested JSON

  • March 4, 2026
  • 0 replies
  • 9 views

thaishfmarques

I’m creating a parser based on a Webhook feed that returns a nested JSON. Due to the way the JSON was designed and how it is received in SecOps, when I use array_function => "split_columns", I end up losing an important field, which appears overwritten in the statedump.

My problem is that I’m unable to map it correctly without using split_columns.

This is how the fields looks like in the sample

 

And this is how it looks when I use split_columns

 

filter {
mutate {
gsub => ["message", "[\\r\\n\\t]*", ""]
}
mutate {
replace => {
"token_metadata.vendor_name" => "X"
"token_metadata.product_name" => "One"
}
}

json {
source => "message"
on_error => "not_json"
# array_function => "split_columns"
}
if [not_json] { drop { tag => "TAG_MALFORMED_ENCODING" } }


mutate {
replace => {
"token_metadata.event_type" => "GENERIC_EVENT"
}
}

mutate {
rename => {
"token_metadata" => "udm_event.idm.read_only_udm.metadata"
}
on_error => "rename_failure"
}

mutate {
merge => { "@output" => "udm_event" }
}
statedump {}
}

That’s my initial template, after that I started to map this specific email field

    ######## EMAIL ########

grok { match => { "message" => [ "\\\"user\\\":\\s*\\\"(?P<detection_email>%{EMAILADDRESS})\\\"" ] } }

if [detection_email] == "" {
mutate {
replace => {
"udm_event.idm.read_only_udm.principal.email" => "NOT_FOUND"
}
}
}
else {
mutate {
replace => {
"udm_event.idm.read_only_udm.principal.email" => "%{detection_email}"
}
}
}
######## EMAIL ########

This actually works but my issue is when I try to put this in a for loop and the fields do not map anymore

PS: I try with grok because I haven’t been successfull in using JSON field without split_columns.

The for loop I tried was 
for detection in collectionData.detections {}

 

That’s my statedump


Internal State (label=):

{
"@collectionTimestamp": {
"nanos": 0,
"seconds": 1772642927
},
"@createTimestamp": {
"nanos": 0,
"seconds": 1772642927
},
"@enableCbnForLoop": true,
"@onErrorCount": 0,
"@output": [
{
"idm": {
"read_only_udm": {
"metadata": {
"event_type": "GENERIC_EVENT",
"product_name": "One",
"vendor_name": "X"
}
}
}
}
],
"@timestamp": {
"nanos": 0,
"seconds": 1772642927
},
"@timezone": "",
"collectionData": {
"detections": [
{
"access.url": "xxxx",
"user": "zzzzz"
}
]
}
}

 

Since I am relatively new in Google SecOps Parsing, I am having a very hard time to make this for loop work