Skip to main content

Hello,

 

I'm trying to parse the 'rule instance name' from CrowdStrike logs, which are in JSON format. I've created a parser extension to extract the data, but I'm encountering the following error.

 

generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter mutate (8) failed: merge failure: merge source field \"event\" must not be empty (try using replace to provide the value before calling merge)"

 

ParserExtention:

 

filter {

 

 mutate {

    gsub => ["message", "[\\r\\n\\t]*", ""]

  }

   

mutate {

  replace => {

    "var_rule_instance_nam" => ""

  }

}

 

    json {

        on_error => "not_json"

        source => "message"

        array_function => "split_columns"

    }

 

  mutate {

      convert => {

        "rule_instance_name" => "string"

      }

      on_error => "convert_error"

    }

 

  mutate {

      replace => {

        "var_rule_instance_name.value" => "%{rule_instance_name}"

      }

      on_error => "no_rule_instance_name"

    }

    if ![no_rule_instance_name] and [rule_instance_name]!= "" {

      mutate {

        replace => {

          "var_rule_instance_name.key" => "rule_instance_name"

        }

      }

      mutate {

        merge => {

          "security_result.detection_fields" => "var_rule_instance_name"

        }

      }

    }

 

     mutate {

    merge => {

      "@output" => "event"

    }

  }

  }

Nice work on the parser extension! Extensions work like standard parsers and still require the creation and merging of a UDM event. It looks like you’re creating security_results, but never subsequently merging those into a UDM event. If you want an example to reference, here’s a parser extension I wrote that operates on JSON data: https://github.com/pilot006/google-secops-parser-extension-gcp-model-armor/blob/main/gcp_cloudaudit_model_armor_extension.conf

 

-mike


Nice work on the parser extension! Extensions work like standard parsers and still require the creation and merging of a UDM event. It looks like you’re creating security_results, but never subsequently merging those into a UDM event. If you want an example to reference, here’s a parser extension I wrote that operates on JSON data: https://github.com/pilot006/google-secops-parser-extension-gcp-model-armor/blob/main/gcp_cloudaudit_model_armor_extension.conf

 

-mike

 

Hi Mike,

 

I referred to your parser and made a few modifications to the extension. After implementing following changes my extention passed validation.

 

*****change*****

mutate {

        merge => {

          "udm_event.idm.read_only_udm.additional.fields" => "var_rule_instance_name"

        }

      }

 

Below are the Community Posts helped me to correct errors

Google SecOps Parsing Questions | Community

Custom Parser Received non-slice or non-array raw output for repeated field" | Community

 

 

Although I successfully passed validation and parsed the data into the UDM field additional.fields, but failed to parse it into security_result.detection_fields
I attempted several changes, but validation failed each time. Out of curiosity, I’d like to understand why the data passed validation for additional.fields but failed for security_result.detection_fields.

 

Below are the error for each changes i made.

 

 mutate {
        merge => {
          "udm_event.idm.read_only_udm.security_result" => "var_rule_instance_name"
        }
      }

 

generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: failed to convert raw output to events: failed to convert raw message 0: field \"idm\": index 0: recursive rawDataToProto failed: field \"read_only_udm\": index 0: recursive rawDataToProto failed: field \"security_result\": index 0: recursive rawDataToProto failed: field \"value\": no descriptor found"

 

 

mutate {
        merge => {
          "udm_event.idm.read_only_udm.security_result.detection_fields" => "var_rule_instance_name"
        }
      }

 

generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: failed to convert raw output to events: failed to convert raw message 0: field \"idm\": index 0: recursive rawDataToProto failed: field \"read_only_udm\": index 0: recursive rawDataToProto failed: field \"security_result\": failed to make strategy: received non-slice or non-array raw output for repeated field"