Skip to main content

I'm building a custom parser and below is a snippet for which I'm facing a issue. This particular field is not always present in all events. It's gives values as False or will not be available altogether.

JSON:
"body

{

custom

{

security

{

blocked = "FALSE"

}}"

Below expression is not only extracting the value but gives the error message when validating the parser - generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter conditional (9) failed: failed to evaluate expression: generic::invalid_argument: \\"body.custom.security.blocked\\" not found in state data"

"if [body][custom][security][blocked] =~ "FALSE" {
  mutate {
    merge => {
   "event.idm.read_only_udm.security_result" => "ALLOW"
    }
  }
  }"

Adding on error statement also doesn't make the trick.

 

You'll want to initialize a variable, named after the JSON field, at the top of the parser. You'll set this to be an empty string, and then after you JSON extract it will populate if the field exists. With this, you can then test whether the string is empty or not

# initialize the token mutate { replace => { "body.custom.security.blocked" => "" } } # extract json json { source => "message" } # check the string to see if it's empty if [body][custom][security][blocked] != "" { # your parsing logic }

-mike


Hi @mikewilusz , I have already initialized the variable already in the parser. We have few events without this field name itself and during validation the parser build is failed with those events as examples.


@Aswin_Asokan I am not sure if you are still needing help. 

There are two options of UDM fields that will would be a good fit for raw log value ALLOW. 
 Security result avaible udm fields.

 

action     SecurityResult.Action     repeated     Actions taken for this event.
action_details     string         The detail of the action taken as provided by the vendor.


security_result.Action is an enumerated field. 
It can only equal one of the values listed here: Security_result.action enumerated values. 

Using security_result.action requires some more logic. I believe the error you are getting is with the long JSON path to the value. I have seen this happen when path to the JSON raw log value long. I think this might happen when it is 4 plus. I could be wrong about this information. I just know I have seen this several times in the past.

Below I am using a mutate replace to handle error handling.
If body.custom.security.blocked is not blank. The on_error will be false. In this example if ![zerror][REPLACE_SR_action_details]. The ! is the logic that is used to see if the on_error is false. To check if I on_error is false remove the !. 
I convert the action_result to lowercase. Then use the if statement to see if action_result is similar to any of the enumerated fields. When using security_result.action you will more then likely to modified the code to work with the raw log value and get it close to one of the enumerated values. 

At the end I merge x_action_res into  

mutatae {
replace => {
"action_res" => ""
}
}

mutatae {
replace => {
"action_result" => "%{body.custom.security.blocked}"
}
on_error => "zerror.REPLACE_sr_action_details"
}

if ![zerror][REPLACE_SR_action_details] {
mutate {
lowercase => [ "action_result" ]
}
# Replace Action result
if [action_result] == "failed" { mutate { replace => { "action_res" => "FAIL"}}}
else if [action_result] == "success" { mutate { replace => { "action_res" => "ALLOW"}}}
else if [action_result] == "block" { mutate { replace => { "action_res" => "BLOCK"}}}
else if [action_result] == "quarantine" { mutate { replace => { "action_res" => "QUARANTINE"}}}
else if [action_result] == "challenge" { mutate { replace => { "action_res" => "CHALLENGE"}}}
else if [action_result] == "allow_with_mods" { mutate { replace => { "action_res" => "ALLOW_WITH_MODIFICATION"}}}
else { mutate { replace => { "action_res" => "UNKNOWN_ACTION"}}}
}

if [action_res] != "" {
mutate {
merge => {
"x_security_result.action" => "action_res"
}
}
}




How to use action_details
If the below code the body.custom.security.blocked value is being mapped to security_result.action_details. No need to use an if statement to see if it is blank or contains a value. The on_error will allow the parser to fail gracefully and move on to the next part of the parsing logic. 

mutate {
replace => {
"security_result.action_details" => "%{body,custom.security.blocked}"
}
on_error => "zerror.REPLACE_SR_action_details"
}