Hi GCSoNist,

Some additional conditional checks might help with this issue, and might look something like:

if [detail] and [detail][clientReferenceInformation] and [detail][clientReferenceInformation][code] != "" { mutate { replace => { "clientrefcode_field.key" => "ClientReferenceCode" "clientrefcode_field.value.string_value" => "%{detail.clientReferenceInformation.code}" } } mutate { merge => { "udm_event.idm.read_only_udm.additional.fields" => "clientrefcode_field" } } }

Hope this helps.

Thanks for the reply @Ben_T, I added in those additional checks but im still getting the same error:
generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter conditional (15) failed: failed to evaluate expression: generic::invalid_argument: \\"detail.clientReferenceInformaton\\" not found in state data"

I'm not sure why the fields arent being skipped since I initialize them at the beginning to be empty. I thought since they are empty it would just skip over them if the field is not in the raw log, is that not the case?

Just to add some additional context I just tried this raw log:
{
"title": "ERROR_SSP_LIBRARIES_EXT",
"type": "Error",
"date": "6/17/2025",
"time": "7:56 pm",
"user": "889990431 USERNAME",
"scripttype": "SSP Application",
"detail": {
"message": "Newsletter.ServiceController.Site missing ReCaptcha.Configuration.Model"
},
"name.script": "SuiteCommerce Advanced - Dev 2020.1"
}

And got no errors - I think it might be something with how I am using "detail" in the initialization but I am not 100% for sure.
Here is the UDM I got:

I think the root of the issue is that you're pre-creating empty placeholder fields for nested data at the start of your parser:
#Remove this "detail.clientReferenceInformation.code" => ""
#Remove this "detail.clientReferenceInformation.partner.solutionId" => ""

When a log comes in where detail is a simple number (like 201) instead of a complex object, your JSON parser correctly overwrites detail to be 201. However, your parser still thinks there might be a detail.clientReferenceInformation field because you created it initially. When it tries to check [detail][clientReferenceInformation], it finds that detail is now a number, not an object, so it can't look inside which is causing the error.

If you let your json parser create the 'detail' structure as it comes in your log then I think the condition checks you added previously will skip processing those parts when 'detail' isn't an object of in the cases when those nested fields are not present. Here's some sample code that may help:

filter {
mutate {
replace => {
"udm_event.idm.read_only_udm.metadata.vendor_name" => "Net Suite"
"udm_event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"

"title" => ""
"type" => ""
"user" => ""
"scripttype" => ""
#Remove this "detail.clientReferenceInformation.code" => ""
#Remove this "detail.clientReferenceInformation.partner.solutionId" => ""
# ... remove all other "detail.*" initializations ...
}
}

# Parse JSON - this will correctly create 'detail' as 201 or as an object
json {
source => "message"
on_error => "not_json"
}

statedump{label=TheStart}

# Placeholder for your existing title, type, user, scripttype handling

# Keep suggested conditional checks here
# Handle client reference information code
if [detail] and [detail][clientReferenceInformation] and [detail][clientReferenceInformation][code] != "" {
mutate {
replace => {
"clientrefcode_field.key" => "ClientReferenceCode"
"clientrefcode_field.value.string_value" => "%{detail.clientReferenceInformation.code}"
}
}
mutate {
merge => {
"udm_event.idm.read_only_udm.additional.fields" => "clientrefcode_field"
}
}
}

# Handle client solution ID
if [detail] and [detail][clientReferenceInformation] and [detail][clientReferenceInformation][partner] and [detail][clientReferenceInformation][partner][solutionId] != "" {
mutate {
replace => {
"solutionid_field.key" => "ClientSolutionId"
"solutionid_field.value.string_value" => "%{detail.clientReferenceInformation.partner.solutionId}"
}
}
mutate {
merge => {
"udm_event.idm.read_only_udm.additional.fields" => "solutionid_field"
}
}
}

# ... (continue with similar checks for all other detail fields) ...

statedump{label=TheEnd}

mutate {
merge => {
"@output" => "udm_event"
}
}
}

Hey @Ben_T, I removed the fields from the initialization and added the additional conditional checks and i'm still getting the generic error:
 generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter conditional (15) failed: failed to evaluate expression: generic::invalid_argument: \\"detail.clientReferenceInformation\\" not found in state data"

It doesnt seem to like my nested JSON and it doesnt look like it is creating the detail structure.

Hey @Ben_T any other ideas that I could try? I've removed all the initialized fields and i'm still getting the same generic error message about fields not be in the state data.

Looking at this further, additional conditional checks might be required for cases where dtail is not an object(integer).  I would try removing any lines that initialize detail fields that are nested like: detail.clientReferenceInformation.code. You could then add conditional checks to verify the existence of each parent level.
Something like: if [detail] and [detail][firstLevel] and [detail][firstLevel][secondLevel].

Then validate, using both log types where detail is an integer and another where it's an object.

Hope this helps.

Hey @Ben_T I found what seems to be a hacky way around what was happening with nested JSON. I initialized the field names and then used a mutate replace on the initialized field with the value from the log. Idk if that is proper way but I also added some error handling just in case.

 I still have a lot to add but so far its working pretty nice! 

Excellent, glad to see the progress!