Skip to main content

What is the best way of a parsing extension to ignore when a field doesn't exist within the raw log?

  • July 31, 2023
  • 4 replies
  • 44 views

mokatsu
Forum|alt.badge.img+6

Hello, what is the best way when creating a parsing extension to ignore when a field doesnt exist within the raw log? I tried

if [field][field2] - throws: expression must evaluate to bool, instead got map[string]interface
if [field][field2] != "" - throws error: field.field2 not found in state data
if [field][field2] or [field][field2] != "" - cannot be used with the logical operator '||', it is not a bool

4 replies

Forum|alt.badge.img+6
  • Bronze 2
  • July 31, 2023

I’ve been using a similar approach to how the normal parsers do it. E.g. setting it to “” in the top part of the parser. Then relying on != in the if statement

The below is hopefully a really basic instructive example:

filter {
mutate {
replace => {
"message.important.field" => ""
}
}

if [message] != "" {
json {
source => "message"
array_function => "split_columns"
on_error => "not_json"
}
}
if [message][important][field] != "" {
mutate {
replace => {
"DO STUFF HERE"
}

}
mutate {
merge => {
"@output" => "event"
}
}
}
}


Forum|alt.badge.img+6
  • Bronze 2
  • July 31, 2023

Note merging the event happens in the if statement. If you leave it outside of the if statement then event will be empty when you try to merge it into output. It feels like a hack, but I haven’t seen best practice guidance for this


mokatsu
Forum|alt.badge.img+6
  • Author
  • Bronze 5
  • July 31, 2023

That might be my issue, my merge is outside the if statement. Will try this thank you.


mokatsu
Forum|alt.badge.img+6
  • Author
  • Bronze 5
  • July 31, 2023

Thank you. That fixed the issue I was having.