I’m creating a parser based on a Webhook feed that returns a nested JSON. Due to the way the JSON was designed and how it is received in SecOps, when I use array_function => "split_columns", I end up losing an important field, which appears overwritten in the statedump.
My problem is that I’m unable to map it correctly without using split_columns.


filter {
mutate {
gsub => ["message", "[\\r\\n\\t]*", ""]
}
mutate {
replace => {
"token_metadata.vendor_name" => "X"
"token_metadata.product_name" => "One"
}
}
json {
source => "message"
on_error => "not_json"
# array_function => "split_columns"
}
if [not_json] { drop { tag => "TAG_MALFORMED_ENCODING" } }
mutate {
replace => {
"token_metadata.event_type" => "GENERIC_EVENT"
}
}
mutate {
rename => {
"token_metadata" => "udm_event.idm.read_only_udm.metadata"
}
on_error => "rename_failure"
}
mutate {
merge => { "@output" => "udm_event" }
}
statedump {}
}That’s my initial template, after that I started to map this specific email field
######## EMAIL ########
grok { match => { "message" => [ "\\\"user\\\":\\s*\\\"(?P<detection_email>%{EMAILADDRESS})\\\"" ] } }
if [detection_email] == "" {
mutate {
replace => {
"udm_event.idm.read_only_udm.principal.email" => "NOT_FOUND"
}
}
}
else {
mutate {
replace => {
"udm_event.idm.read_only_udm.principal.email" => "%{detection_email}"
}
}
}
######## EMAIL ########This actually works but my issue is when I try to put this in a for loop and the fields do not map anymore
PS: I try with grok because I haven’t been successfull in using JSON field without split_columns.
The for loop I tried was
for detection in collectionData.detections {}
That’s my statedump
Internal State (label=):
{
"@collectionTimestamp": {
"nanos": 0,
"seconds": 1772642927
},
"@createTimestamp": {
"nanos": 0,
"seconds": 1772642927
},
"@enableCbnForLoop": true,
"@onErrorCount": 0,
"@output": [
{
"idm": {
"read_only_udm": {
"metadata": {
"event_type": "GENERIC_EVENT",
"product_name": "One",
"vendor_name": "X"
}
}
}
}
],
"@timestamp": {
"nanos": 0,
"seconds": 1772642927
},
"@timezone": "",
"collectionData": {
"detections": [
{
"access.url": "xxxx",
"user": "zzzzz"
}
]
}
}
Since I am relatively new in Google SecOps Parsing, I am having a very hard time to make this for loop work