Office 365 Parser extension to extract array of mail_ids from InternetmessageID

Hi ssuluguri,

I tested this in a lab environment against the default OFFICE_365 parser to verify how UDM handles multiple message IDs. In the Google SecOps Unified Data Model field list, network.email.mail_id is defined as a single scalar string rather than a repeated field (https://docs.cloud.google.com/chronicle/docs/reference/udm-field-list). In lab testing, assigning an array directly to network.email.mail_id in a parser extension fails UDM schema validation.

To store all extracted message IDs as an array that you can search and write rules against, your parser extension can initialize the source variable to an empty string before json parsing (so non email OFFICE_365 events without that field do not fail the conditional check), strip any spaces around commas, split the raw InternetMessageId string, and write each item into a leaf level repeated UDM field such as target.resource.attribute.labels as well as a list under additional.fields["mail_ids"] (https://docs.cloud.google.com/chronicle/docs/event-processing/using-parser-extensions). Here is a code snippet example tested against both matching email records and non email sign in logs using a comma separated ExchangeDetails.InternetMessageId field (you can adjust the field name in all five places, including the bracket form inside the if condition, to Item.InternetMessageId or InternetMessageId to match your raw Office 365 operation):

filter {
  mutate {
    replace => { "ExchangeDetails.InternetMessageId" => "" }
  }
  json {
    source => "message"
    array_function => "split_columns"
    on_error => "not_json"
  }
  if ![not_json] and [ExchangeDetails][InternetMessageId] != "" {
    mutate {
      gsub => [ "ExchangeDetails.InternetMessageId", " ", "" ]
      on_error => "gsub_err"
    }
    mutate {
      split => { source => "ExchangeDetails.InternetMessageId" separator => "," }
      on_error => "split_err"
    }
    if ![split_err] {
      for mid in ExchangeDetails.InternetMessageId {
        mutate { replace => { "value_array" => "" } }
        mutate { replace => { "value_array.string_value" => "%{mid}" } }
        mutate { merge => { "mail_ids_entry.value.list_value.values" => "value_array" } }
        mutate { replace => { "lbl_item" => "" } }
        mutate { replace => { "lbl_item.key" => "mail_id" "lbl_item.value" => "%{mid}" } }
        mutate { merge => { "event.idm.read_only_udm.target.resource.attribute.labels" => "lbl_item" } }
      }
      mutate { replace => { "mail_ids_entry.key" => "mail_ids" } }
      mutate { merge => { "event.idm.read_only_udm.additional.fields" => "mail_ids_entry" } }
      mutate { merge => { "@output" => "event" } }
    }
  }
}
 

When you save the parser extension in the Extension panel, ensure (Repeated Fields) is switched from the default (Replace Values) to (Append Values) so your extracted mail_id entries are appended alongside any labels the default OFFICE_365 parser already writes into target.resource.attribute.labels instead of overwriting them. Once active, each individual message ID can be queried in UDM Search using target.resource.attribute.labels["mail_id"] = "<message_id>", matched in YARA L rules using any $e.target.resource.attribute.labels.value = "<message_id>", and viewed as a JSON list under additional.fields["mail_ids"]. Please keep in mind that this is a sample parser extension provided as an example that we will not be supporting or maintaining, so your team will want to test and maintain it for your environment.

Thanks, Ben