Can you try initializing domain in the first mutate?

Doc - https://cloud.google.com/chronicle/docs/event-processing/parser-tips-troubleshooting#:~:text=%22generic%3A%3Ainvalid_argument%3A%20pipeline%20failed%3A%20filter%20conditional%20(4)%20failed%3A%20failed%20to%20evaluate%20expression%3A%20generic%3A%3Ainvalid_argument%3A%20%22product%22%20not%20found%20in%20state%20data%22

Hi Rmoos,

The 'Not Found State Data' error typically occurs when the fields are initially not mapped as null

Try mapping the domain field as “domain” => “” and then try validating.

 

Thanks for the response.

If I add:

mutate {

    replace => {

    "domain" => ""

    }

}    

Then I am getting the following:

generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::internal: pipeline failed: filter grok (2) failed: field \"domain\" already exists in data and is not overwriteable"

Please share me your parser code and ample log I will try it in my lab environment and let you know.

I have tested and the URL field is getting parsed, please refer the screenshot.

 

filter {

mutate {

    replace => {

        "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"

    "URL" => ""

    }

}    

if

 grok {

        match => {

            "message" => "URL=%{URIHOST:domain}"]

        }

    }

}

# Map domain

if

    mutate {

        replace => {

             "event.idm.read_only_udm.target.url" => "%{domain}"

                    }

            on_error => "invalid.target.url"

                }

            }

mutate {

    merge => {

        "@output" => "event"

        }

    }

}

Thanks for the response. This is pretty much the way I originally had the parser syntax. The issue is when I try an validate, it fails because the log is different from what I wrote the parser against.

Error: generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter conditional (3) failed: failed to evaluate expression: generic::invalid_argument: \"domain\" not found in state data
"Log: Jul 30 15:03:13 test-01-abc-gw AB_FLOW: AB_FLOW_SESSION_CREATE: session created 1.1.1.1/57720->2.2.2.2/443 0x0 junos-https 10.10.10.10/17381->8.8.8.8/443 0x0 source rule from-test-to-public-rule N/A N/A 6 Allow-Workload-Zone-Access-To-Internet ABC-Workload-Zone SL-PUBLIC [removed by moderator] N/A(N/A) ae0.917 UNKNOWN UNKNOWN UNKNOWN N/A N/A -1 N/A N/A N/A Off root N/A N/A N/A N/A

So the question is how do I get the Parser Extension I wrote to parse when it sees a log like this:

Jul 28 17:58:46 test-01-abc-gw AB_UTM: WEBFILTER_URL_PERMITTED: WebFilter: ACTION="URL Permitted" source-zone="EPT-Workload-Zone" destination-zone="SL-PUBLIC" 1.1.1.1(35806)->2.2.2.2(443) SESSION_ID=  removed by moderator] APPLICATION="SSL" NESTED-APPLICATION="UNKNOWN" CATEGORY="EPT-Cluster-Web-Profile" REASON="BY_USER_DEFINED" PROFILE="EPT-Cluster-Web-Profile" URL=test.connect.abc.com username N/A roles N/A application-sub-category Encryption urlcategory-risk 0

BUT then continue to use the existing, pre-built default JUNO_Firewall parser for all other logs that it sees., such as below.

"Log: Jul 30 15:03:13 test-01-abc-gw AB_FLOW: AB_FLOW_SESSION_CREATE: session created 1.1.1.1/57720->2.2.2.2/443 0x0 junos-https 10.10.10.10/17381->8.8.8.8/443 0x0 source rule from-test-to-public-rule N/A N/A 6 Allow-Workload-Zone-Access-To-Internet ABC-Workload-Zone SL-PUBLIC removed by moderator] N/A(N/A) ae0.917 UNKNOWN UNKNOWN UNKNOWN N/A N/A -1 N/A N/A N/A Off root N/A N/A N/A N/A

I was able to figure it out.  I had to add: overwrite => ;"domain"] in the GROK statement and then it works.  So below is the final code.

filter {

mutate {
    replace => {
        "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
        "URL" => ""
        "domain" => ""
    }
}    

if >message] =~ /\RT_UTM:\/ {
 grok {
        match => {
            "message" => g"URL=%{URIHOST:domain}"]
        }
        overwrite => g"domain"]
    }
}

# Map domain
if >domain] !="" {
    mutate {
        replace => {
             "event.idm.read_only_udm.target.url" => "%{domain}"
                    }
            on_error => "invalid.target.url"
                }
            }

mutate {
    merge => {
        "@output" => "event"
        }

   }
}