Skip to main content
Solved

Parser Extension and not found in state data

  • July 29, 2025
  • 7 replies
  • 107 views
Rmoss
Forum|alt.badge.img+3

Good Day,

I am trying to write a parser extension for JUNIPER FIREWALL.  I am trying to map the field URL to target.url but when I try and validate the parser it fails with “failed to evaluate expression: generic::invalid_argument: \"domain\" not found in state data".  How do I get it to only match the extension when it sees this specific log type that the prebuilt parser doesn’t fully parse out.

 

The raw log is as follows:

Jul 28 17:58:46 test-01-abc-gw AB_UTM: WEBFILTER_URL_PERMITTED: WebFilter: ACTION="URL Permitted" source-zone="EPT-Workload-Zone" destination-zone="SL-PUBLIC" 1.1.1.1(35806)->2.2.2.2(443) SESSION_ID=206160582269 APPLICATION="SSL" NESTED-APPLICATION="UNKNOWN" CATEGORY="EPT-Cluster-Web-Profile" REASON="BY_USER_DEFINED" PROFILE="EPT-Cluster-Web-Profile" URL=test.connect.abc.com username N/A roles N/A application-sub-category Encryption urlcategory-risk 0

 

Parser Syntax is as follows:

filter {

mutate {
    replace => {
    "URL" => ""
    }
}    

if [message] =~ /\AB_UTM:\/ {
 grok {
        match => {
            "message" => ["URL=%{URIHOST:domain}"]
        }

    }
}

# Map domain
if [domain] !="" {
    mutate {
        replace => {
             "event.idm.read_only_udm.target.url" => "%{domain}"
                    }
            on_error => "invalid.target.url"
                }
            }

mutate {
    merge => {
        "@output" => "event"
        }
    }

}

 

 

Best answer by Rmoss

I was able to figure it out.  I had to add: overwrite => ["domain"] in the GROK statement and then it works.  So below is the final code.

 

filter {

mutate {
    replace => {
        "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
        "URL" => ""
        "domain" => ""
    }
}    

if [message] =~ /\RT_UTM:\/ {
 grok {
        match => {
            "message" => ["URL=%{URIHOST:domain}"]
        }
        overwrite => ["domain"]
    }
}

# Map domain
if [domain] !="" {
    mutate {
        replace => {
             "event.idm.read_only_udm.target.url" => "%{domain}"
                    }
            on_error => "invalid.target.url"
                }
            }

mutate {
    merge => {
        "@output" => "event"
        }

   }
}

 

7 replies

cmorris
Staff
Forum|alt.badge.img+10
  • cmorris
  • Staff
  • July 30, 2025

Can you try initializing domain in the first mutate?

 

Doc - https://cloud.google.com/chronicle/docs/event-processing/parser-tips-troubleshooting#:~:text=%22generic%3A%3Ainvalid_argument%3A%20pipeline%20failed%3A%20filter%20conditional%20(4)%20failed%3A%20failed%20to%20evaluate%20expression%3A%20generic%3A%3Ainvalid_argument%3A%20%22product%22%20not%20found%20in%20state%20data%22

manoj610
Forum|alt.badge.img+4
  • manoj610
  • New Member
  • July 30, 2025

Hi Rmoos,

The 'Not Found State Data' error typically occurs when the fields are initially not mapped as null

Try mapping the domain field as “domain” => “” and then try validating.

 

Rmoss
Forum|alt.badge.img+3
  • Rmoss
  • Author
  • New Member
  • July 30, 2025

Thanks for the response.

If I add:

mutate {

    replace => {

    "domain" => ""

    }

}    

 

Then I am getting the following:

generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::internal: pipeline failed: filter grok (2) failed: field \"domain\" already exists in data and is not overwriteable"

manoj610
Forum|alt.badge.img+4
  • manoj610
  • New Member
  • July 30, 2025

Please share me your parser code and ample log I will try it in my lab environment and let you know.

manoj610
Forum|alt.badge.img+4
  • manoj610
  • New Member
  • July 30, 2025

I have tested and the URL field is getting parsed, please refer the screenshot.

 

filter {

 

mutate {

    replace => {

        "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"

    "URL" => ""

    }

}    

 

if [message] =~ /\AB_UTM:\/ {

 grok {

        match => {

            "message" => ["URL=%{URIHOST:domain}"]

        }

 

    }

}

 

# Map domain

if [domain] !="" {

    mutate {

        replace => {

             "event.idm.read_only_udm.target.url" => "%{domain}"

                    }

            on_error => "invalid.target.url"

                }

            }

 

mutate {

    merge => {

        "@output" => "event"

        }

    }

 

}

 

 

Rmoss
Forum|alt.badge.img+3
  • Rmoss
  • Author
  • New Member
  • July 30, 2025

Thanks for the response. This is pretty much the way I originally had the parser syntax. The issue is when I try an validate, it fails because the log is different from what I wrote the parser against.

Error: generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter conditional (3) failed: failed to evaluate expression: generic::invalid_argument: \"domain\" not found in state data
"Log: Jul 30 15:03:13 test-01-abc-gw AB_FLOW: AB_FLOW_SESSION_CREATE: session created 1.1.1.1/57720->2.2.2.2/443 0x0 junos-https 10.10.10.10/17381->8.8.8.8/443 0x0 source rule from-test-to-public-rule N/A N/A 6 Allow-Workload-Zone-Access-To-Internet ABC-Workload-Zone SL-PUBLIC [removed by moderator] N/A(N/A) ae0.917 UNKNOWN UNKNOWN UNKNOWN N/A N/A -1 N/A N/A N/A Off root N/A N/A N/A N/A

 

So the question is how do I get the Parser Extension I wrote to parse when it sees a log like this:

Jul 28 17:58:46 test-01-abc-gw AB_UTM: WEBFILTER_URL_PERMITTED: WebFilter: ACTION="URL Permitted" source-zone="EPT-Workload-Zone" destination-zone="SL-PUBLIC" 1.1.1.1(35806)->2.2.2.2(443) SESSION_ID= [removed by moderator] APPLICATION="SSL" NESTED-APPLICATION="UNKNOWN" CATEGORY="EPT-Cluster-Web-Profile" REASON="BY_USER_DEFINED" PROFILE="EPT-Cluster-Web-Profile" URL=test.connect.abc.com username N/A roles N/A application-sub-category Encryption urlcategory-risk 0

BUT then continue to use the existing, pre-built default JUNO_Firewall parser for all other logs that it sees., such as below.

"Log: Jul 30 15:03:13 test-01-abc-gw AB_FLOW: AB_FLOW_SESSION_CREATE: session created 1.1.1.1/57720->2.2.2.2/443 0x0 junos-https 10.10.10.10/17381->8.8.8.8/443 0x0 source rule from-test-to-public-rule N/A N/A 6 Allow-Workload-Zone-Access-To-Internet ABC-Workload-Zone SL-PUBLIC [removed by moderator] N/A(N/A) ae0.917 UNKNOWN UNKNOWN UNKNOWN N/A N/A -1 N/A N/A N/A Off root N/A N/A N/A N/A

 

 

Rmoss
Forum|alt.badge.img+3
  • Rmoss
  • Author
  • New Member
  • Answer
  • July 30, 2025

I was able to figure it out.  I had to add: overwrite => ["domain"] in the GROK statement and then it works.  So below is the final code.

 

filter {

mutate {
    replace => {
        "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
        "URL" => ""
        "domain" => ""
    }
}    

if [message] =~ /\RT_UTM:\/ {
 grok {
        match => {
            "message" => ["URL=%{URIHOST:domain}"]
        }
        overwrite => ["domain"]
    }
}

# Map domain
if [domain] !="" {
    mutate {
        replace => {
             "event.idm.read_only_udm.target.url" => "%{domain}"
                    }
            on_error => "invalid.target.url"
                }
            }

mutate {
    merge => {
        "@output" => "event"
        }

   }
}

 

Terms & ConditionsCookie settingsAccessibility statement
Privacy Policy Terms of Service Gen AI Policy Community Guidelines Cookie Settings

© 2025 Google. All rights reserved.