Skip to main content

I encounter some problems today on working for a parser, I created a custom parser for meet my customer needs and added some additional fields.


I've got this pattern:



"<%{INT}>%{SYSLOGTIMESTAMP} %{DATA:hostname} %{DATA}: %{IPV4:ip} %{DATA} %{DATA:username} \\\\[(?P<time>[^\\]]+)\\\\] \\\\\\"%{DATA:method} %{DATA:target} %{DATA:proto}\\/%{DATA:proto_version} %{INT:response} (-|%{INT:bytes}) \\\\\\"%{DATA:referer}\\\\\\" \\\\\\"%{GREEDYDATA:useragent}\\\\\\" %{INT:custom_field_1} %{INT:custom_field_2} %{DATA:jsession_id} (?<public_ip>(?:%{IPV4})(?:,\\s*%{IPV4})*)%{NOTSPACE:referral_url}"

 

That matches this log:
"09:04:08 sbobo-tsf-ml01 apache_access_log: 10.x.x.x - - [02/May/2025:09:04:00 +0000] "POST /sbobo/sismon/ClexEpromSet.jsp?MN=NODE_6074&MPN=NODE_6050 HTTP/1.1" 200 8604 <URL Removed by Staff> "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36" 0 9780 34FSD5G433G4634FDSFSG454F4DWEDF5.as-test-ml01 79.x.x.x, 147.x.x.x  sbobo.net"

The part that i'm focused on is the last part "34FSD5G433G4634FDSFSG454F4DWEDF5.as-test-ml01 79.x.x.x, 147.x.x.x  sbobo.net", I created different additional fields and I want both ip 79.x.x.x and 147.x.x.x to be in an array under the additional field "public_ip".

Right now only the first IP will be saved in the public_ip field. I declared the public_ip additional field like this:


    # public_ip

    if [public_ip] != "" {

      mutate {

        replace => {

        "public_ip_label.value.string_value" => "%{public_ip}"

        }

        on_error => "public_ip_empty"

      }

      if ![public_ip_empty] {

      mutate {

        replace => {

        "public_ip_label.key" => "public_ip"

        }

      }

      mutate {

        merge => {

        "event.idm.read_only_udm.additional.fields" => "public_ip_label"

        }

        on_error => "public_ip_label_empty"

        }

      }

    }
I tried to use the split function but it didn't work.
What am I missing?

Thanks in advance!



Great question and one that took me a while to think about! So something to keep in mind, and I hope this is what you're describing as the issue, is right now you're saying you want the additional field to be an array, but right in your example you're setting it as a string:   "public_ip_label.value.string_value" => "%{public_ip}"


Additional allows you to choose the data type, using the docs we link to from our UDM documentation: https://protobuf.dev/reference/protobuf/google.protobuf/


If you look at this you can see where we get "string_value" and what you're actually looking for is "list_value." I searched our parsers and see others using this so it's valid. If you want an example you can go look at the OCSF parser 🙂


-mike


 

Here is the code that worked for me:

filter {

json {
source => "message"
array_function => "split_columns"
}

mutate {
replace => {
"additional_field.key" => "hits"
"udm.metadata.event_type" => "GENERIC_EVENT"
}
}

for hit in hits {
mutate {
replace => {
"value.string_value" => "%{hit}"
}
}

mutate {
merge => {
"list_value.values" => "value"
}
}
mutate {
remove_field => ["value"]
}

}

mutate {
rename =>{
"list_value" => "additional_field.value.list_value"
}
}


mutate {
merge => {
"udm.additional.fields" => "additional_field"
}
}

mutate {
rename => {
"udm" => "event.idm.read_only_udm"
}
}

mutate {
merge => {
"@output" => "event"
}
}




}

 

Reply


Terms & ConditionsCookie settings
Privacy Policy Terms of Service Gen AI Policy Community Guidelines Cookie Settings

© 2025 Google. All rights reserved.