Hi All,
I have already created a custom parser that works well for JSON object coming through logs. However, I now need to update the parser to support JSON arrays. but I'm unsure how to loop through a JSON array.
Sample log: [{"test": "testing","hostname":"hostname1"},{"test":"testing","hostname":"hostname2"}]
Any guidance or assistance on how to resolve this issue would be greatly appreciated.
Thank you!
Details on using "for" loops for looping through a JSON array are available here: https://cloud.google.com/chronicle/docs/reference/parser-syntax#for_loop
-mike
One of the most important things to do when looping through a "for" loop is to initialize the variables you use within the loop to null at the beginning of the loop.
for index,record in records {
mutate {
replace => {
"type_label" => ""
"Id_label" => ""
}
}
mutate {
convert => {
"index" => "string"
}
on_error => "index_conversion_failed"
}
mutate {
replace => {
"type_label.key" => "type_%{index}"
"type_label.value" => "%{record.attributes.type}"
}
on_error => "type_not_found"
}
In this example we are initializing the type_label to null. If you reuse the variable without initializing it to null then you will get unpredictable results - every time you run the parser you may see different output. Be VERY CAREFUL with this.
Details on using "for" loops for looping through a JSON array are available here: https://cloud.google.com/chronicle/docs/reference/parser-syntax#for_loop
-mike
@mikewilusz Thanks for your answer, but that's not what I'm looking for. I know how to loop through a JSON array if it exists inside a JSON object. The issue here is that the ingested log is not a single JSON object; instead, it is an array of objects or you can assume this as batch of logs into single log. So, now I need to loop through the array of objects and create multiple events from a single ingested log.
Can you share a sample log?
Can you share a sample log?
@sudeep_singh I have updated post. You will find sample log in description.
One of the most important things to do when looping through a "for" loop is to initialize the variables you use within the loop to null at the beginning of the loop.
for index,record in records {
mutate {
replace => {
"type_label" => ""
"Id_label" => ""
}
}
mutate {
convert => {
"index" => "string"
}
on_error => "index_conversion_failed"
}
mutate {
replace => {
"type_label.key" => "type_%{index}"
"type_label.value" => "%{record.attributes.type}"
}
on_error => "type_not_found"
}
In this example we are initializing the type_label to null. If you reuse the variable without initializing it to null then you will get unpredictable results - every time you run the parser you may see different output. Be VERY CAREFUL with this.
@rajukg11Thanks for the answer. but, that's not something I am looking for. I have updated sample log in post description. Please have a look. that' might be helpful for you to understand my problem.
You have to put the entire parser in your looping logic so it creates a UDM event for each event in your array.
You have to wrap the log with a dummy variable and then parse it with json in this case. I changed the log as this: {"x":[{"test": "testing","hostname":"hostname1"},{"test":"testing","hostname":"hostname2"}] } and use the x in the for loop. The following code is not changing the log itself, and for that you have to add mutate gsub.
filter {
json {
source => "message"
array_function => "split_columns"
on_error => "not_a_json"
}
statedump{}
if ![not_a_json] {
for msg in x {
mutate {
replace => {
"event" => ""
}
}
mutate {
replace => {
"event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
"event.idm.read_only_udm.principal.hostname" => "%{msg.hostname}"
}
}
mutate {
merge => {
"@output" => "event"
}
on_error => "event_generation_failure"
}
}
}
}
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.