Skip to main content
Solved

Converting data from json log into Chronicle UDM events

  • November 30, 2023
  • 3 replies
  • 37 views
N
Forum|alt.badge.img+4

Hello everyone!
I am creating a custom parser for json logs and I need to convert domains_list into principal.user.attribute.labels udm event.

 

"activity_metadata": {
    "settings_new_value": {
      "block_type": "whitelist",
      "domains_list": [
        "test.com",
        "attack.mitre.org"
      ]
    }, 

I converted block_type already, using the following code

 

#Get block_type
  mutate {
    replace => {
      "block_type.value" => "%{activity_metadata.settings_new_value.block_type}"
    }
    on_error => "block_type_not_set"
  }
  if ![block_type_not_set] {
    mutate {
      replace => {
        "block_type.key" => "block_type"
      }
    }
    mutate {
      merge => {
        "token_principal.user.attribute.labels" => "block_type"
      }
    }
  }
 
 
However, I can't catch [ "test.com", "attack.mitre.org" ] from domains_list , as it the statedump gives an error
"domains_list_not_set": true,
and doesn't parse the data 
 
Here is the code I was using: 
 
#Get domains_list
  mutate {
    replace => {
      "domains_list.value" => "%{activity_metadata.settings_new_value.block_type.domains_list}"
    }
    on_error => "domains_list_not_set"
  }
  if ![domains_list_not_set] {
    mutate {
      replace => {
        "domains_list.key" => "domains_list"
      }
    }
    mutate {
      merge => {
        "token_principal.user.attribute.labels" => "domains_list"
      }
    }
  }
 
 
Can you please help me to convert this piece of data inside [] into udm?

 Do I need to use regex to do that?

 

Thank you in advance!

Best answer by ion_

I'm not sure if you can write an array directly like that, but I do know you can iterate over arrays. I've solved something similar and modified the code to suit your example. I've left the statedump in so you can make sure each iteration is working correctly

for index, _domain in activity_metadata.settings_new_value.domains_list {
statedump {label=>"asd"}
mutate {
replace => {
"var_domain_label.key" => "domain_list"
"var_domain_label.value" => "%{_domain}"
}
}
 
mutate {
merge => {
"target.resource.attribute.labels" => "var_domain_label"
}
}
mutate {
replace => {
"var_domain_label" => ""
}
}
}

Notes:
1). You do need to clear the variable on each iteration otherwise you'll get weird referencing issues
2). I haven't encountered issues with multiple labels with the same name, but more testing might prove the opposite

Lemme know if that makes sense, you'll have to just copy paste your variables names over if you're happy


Ray_a

3 replies

I
Forum|alt.badge.img+6
  • ion_Bronze 2
  • Bronze 2
  • Answer
  • November 30, 2023

I'm not sure if you can write an array directly like that, but I do know you can iterate over arrays. I've solved something similar and modified the code to suit your example. I've left the statedump in so you can make sure each iteration is working correctly

for index, _domain in activity_metadata.settings_new_value.domains_list {
statedump {label=>"asd"}
mutate {
replace => {
"var_domain_label.key" => "domain_list"
"var_domain_label.value" => "%{_domain}"
}
}
 
mutate {
merge => {
"target.resource.attribute.labels" => "var_domain_label"
}
}
mutate {
replace => {
"var_domain_label" => ""
}
}
}

Notes:
1). You do need to clear the variable on each iteration otherwise you'll get weird referencing issues
2). I haven't encountered issues with multiple labels with the same name, but more testing might prove the opposite

Lemme know if that makes sense, you'll have to just copy paste your variables names over if you're happy


Ray_a
cmmartin_google
N
M
Forum|alt.badge.img+4
  • MustacheBronze 3
  • Bronze 3
  • December 1, 2023

Ah, the infamous nested arrays within logstash/chronicle implementations.

Nicely done @ion_ . I can confirm that works in labels: 

target.resource.attribute.labels[0].key"domain_list" target.resource.attribute.labels[0].value"test.com" target.resource.attribute.labels[1].key"domain_list" target.resource.attribute.labels[1].value"attack.mitre.org"

This is the output from the custom parser preview.  

Ray_a
N
N
Forum|alt.badge.img+4
  • NastyaSBronze 2
  • Author
  • Bronze 2
  • December 4, 2023

I'm not sure if you can write an array directly like that, but I do know you can iterate over arrays. I've solved something similar and modified the code to suit your example. I've left the statedump in so you can make sure each iteration is working correctly

for index, _domain in activity_metadata.settings_new_value.domains_list {
statedump {label=>"asd"}
mutate {
replace => {
"var_domain_label.key" => "domain_list"
"var_domain_label.value" => "%{_domain}"
}
}
 
mutate {
merge => {
"target.resource.attribute.labels" => "var_domain_label"
}
}
mutate {
replace => {
"var_domain_label" => ""
}
}
}

Notes:
1). You do need to clear the variable on each iteration otherwise you'll get weird referencing issues
2). I haven't encountered issues with multiple labels with the same name, but more testing might prove the opposite

Lemme know if that makes sense, you'll have to just copy paste your variables names over if you're happy



This code also worked perfectly for me, thank you!🙂

In my case, I just used different labels 

#Get domains_list
    for index, _domain in activity_metadata.settings_new_value.domains_list {
    statedump {label=>"asd"}
    mutate {
    replace => {
    "domains_list.key" => "domains_list"
    "domains_list.value" => "%{_domain}"
    }
    }
   
    mutate {
    merge => {
    "token_principal.user.attribute.labels" => "domains_list"
    }
    }
    mutate {
    replace => {
    "domains_list" => ""
    }
    }
    }

  mutate {
    replace => {
      "domains_list.value" => "%{activity_metadata.settings_new_value.block_type.domains_list}"
    }
    on_error => "domains_list_not_set"
  }
  if ![domains_list_not_set] {
    mutate {
      replace => {
        "domains_list.key" => "domains_list"
      }
    }
    mutate {
      merge => {
        "token_principal.user.attribute.labels" => "domains_list"
      }
    }
  }

and having this output:
 
principal.user.attribute.labels[5].key"domains_list"
principal.user.attribute.labels[5].value"test.com"
principal.user.attribute.labels[6].key"domains_list"
principal.user.attribute.labels[6].value"attack.mitre.org"
Terms & ConditionsCookie settingsAccessibility statement
Privacy Policy Terms of Service Gen AI Policy Community Guidelines Cookie Settings

© 2025 Google. All rights reserved.