Parser extension not creating additional.fields

Hi ​@MusicMule - 

Firstly, apologies if this shows up as a duplicate post. I tried answering earlier and it seems the post removed itself? 

Anywho, you’re close! There are just a few syntax things to refine. 

Firstly, additional.fields[key] is not valid field syntax, because additional is what’s known as a google.protobuf.Struct in UDM. This means that it needs an explicit type and is not a repeated K/V. Other fields in UDM, like labels, are K/V so this can be confusing. 

Secondly, dot syntax =/= brackets, so “ExperimentalFeatures.mcp” is taken as a literal, which isn’t what’s intended here. 

Lastly, and most importantly, parser extensions do not inherit the base/custom parser’s intermediate fields. This extension runs against the original raw log + the UDM the parser spits out, meaning you need to re-parse the JSON yourself here to access it in the way you want. 

To put this all together, start your extension with the JSON capture like this:

json {
  source => "message"
  array_function => "split_columns"
  on_error => "not_json"
}

if ![not_json] {
  mutate {
    replace => {
      "mcp_status" => ""
    }
  }

Then you can check your field existence as expected:

if [ExperimentalFeatures][mcp] != "" {
  mutate {
    convert => {
      "[ExperimentalFeatures][mcp]" => "string"
    }
    on_error => "_mcp_conversion_failed"
  }
  mutate {
    replace => {
      "mcp_status" => "%{[ExperimentalFeatures][mcp]}"
    }
  }
}

Then finally, assuming mcp_status != “”, you can just set the field. Note, however, that it’d need to set the value extracted, e.g. ...additional.fields.mcp.string_value. 

If desired, you could also extract everything as K/V via a loop, in case you wanted to dynamically pull out, say, anything under ExperimentalFeatures. The docs on that are here.

Let me know if this helps / if you have any other questions! 

Hi ​@MusicMule 

There are two main reasons your extension isn't populating `additional.fields`:

### 1. `additional.fields` is a Repeated Key-Value Object

In Google SecOps UDM, `additional.fields` is a repeated message (list of key-value pairs). You cannot assign to it directly via `fields[key]` and `fields[value]`. Instead, the canonical CBN syntax requires creating a temporary object with `.key` and `.value.string_value`, and then using `merge` to append it to the UDM field:

```ruby

mutate {

replace => {

"temp_mcp.key" => "mcp"

"temp_mcp.value.string_value" => "%{[ExperimentalFeatures][mcp]}"

}

}

mutate {

merge => {

"event.idm.read_only_udm.additional.fields" => "temp_mcp"

}

}

2. Boolean false Handling in Conditionals

In your log sample, "mcp": false is a boolean, not a string. In Logstash/CBN, conditional checks against a boolean false can evaluate as falsy, causing the if block to be skipped. You should explicitly check for booleans or convert it to a string first.

✅ Working Solution

Here is the revised, working parser extension code:

ruby

filter {

# Note: If the base parser did NOT already parse ExperimentalFeatures from the raw log,

# uncomment the json block below to extract it from "message":

# json {

# source => "message"

# target => "parsed_json"

# }

# Check if the field exists (handles boolean false/true or string values)

if [ExperimentalFeatures][mcp] in [true, false] or [ExperimentalFeatures][mcp] != "" {

# 1. Convert boolean to string so it stores cleanly as "false" or "true"

mutate {

convert => {

"[ExperimentalFeatures][mcp]" => "string"

}

}

# 2. Build the temporary KeyValuePair structure

mutate {

replace => {

"mcp_field.key" => "mcp"

"mcp_field.value.string_value" => "%{[ExperimentalFeatures][mcp]}"

}

}

# 3. Merge into additional.fields array

mutate {

merge => {

"event.idm.read_only_udm.additional.fields" => "mcp_field"

}

}

}

}

Verification

Once parsed, in UDM Search you will be able to query this field directly as expected:

yara

additional.fields["mcp"] = "false"

Assuming this is your whole parser extension and not just a snippet, you are missing a few important steps, mainly the JSON extraction and the merge of your state variables into your output.

Keep in mind that a parser extension has virtually no knowledge of the original parser. I like to imagine it is just a second parser running after the first and overwriting/appending the resultant UDM event from the first parser. As such, everything you do in your extension must operate on the raw log contents themselves and not any of the state from the original parser.

Generally, you can copy the parts you need from the original parser into your extension and then build on top of them. I have annotated the parser with the sections you need and a few tips below. Using that plus the original parser for inspiration should allow you to build your desired parser extension.

filter {
  mutate {
    replace => {
      "ExperimentalFeatures.mcp" => ""
    }
  }

  # TODO: Add JSON extraction

  if [ExperimentalFeatures][mcp] != "" and [ExperimentalFeatures][mcp] != "NULLPLACEHOLDERVALUE" {
    mutate {
      add_field => {
        "[@metadata][mcp_val]" => "%{[ExperimentalFeatures][mcp]}"
      }
    }
    mutate {
      add_field => {
        "event.idm.read_only_udm.additional.fields[key]" => "mcp"
        "event.idm.read_only_udm.additional.fields[value]" => "%{[@metadata][mcp_val]}"
      }
    }
  }

  # TIP: Use statedump before output to see intermediate state before final output
  # Remove before validating
  statedump{}

  # TODO: Add output merge
}

As an additional data point, the following is a code snippet I use often to map data into additional fields and you should be able to largely copy the structure, modifying names for your use-case. With these two pieces, you should be most of the way there and the only other potential issue I see arising is errors from “if” conditions when fields are empty or do not exist in your original log. The fix for that is a matter of nested “if”s and “on_error” handling, but if you get tripped up, share another code snippet, sanitized log sample, and error message and I can help further.
 

filter {
  ...
  # Example: Map additional field(s)
  # This example takes the full log message and puts it in an additional field called "full_log"
  mutate {
    replace => {
      "additional_full_log.key" => "full_log"
      "additional_full_log.value.string_value" => "%{message}"
    }
    on_error => "_full_log_replace_error"
  }
  if ![_full_log_replace_error] {
    mutate  {
      merge => {
        "event.idm.read_only_udm.additional.fields" => "additional_full_log"
      }
      on_error => "_full_log_merge_error"
    }
  }
  ...
}

If you run into any issues along the way, let me know and I’d be happy to provide additional troubleshooting steps.