Skip to main content

Hi there,

I am writing a custom parser for Microsoft Azure Resource logs. For the below raw log of category “AppServiceConsoleLogs”, I am getting the above error, even though the required field as per the error is mapped in the state data.

AppServiceConsoleLogs raw log:

{"_time": [removed by moderator] .752,"time":"2025-09-21T01:50:08.8341792Z","resultDescription":"\u0445601b[32m[2025-09-23T07:50:08.833] [INFO] Job API - \u00UID: V920","resourceId":"/SUBSCRIPTIONS/03D-99C8-4JQ25-965150-E09FC4SDFD6EB34/RESOURCEGROUPS/ABCD_EFGH_RG/PROVIDERS/MICROSOFT.WEB/SITES/ABCDEFGH","level":"Informational","operationName":"Microsoft.Web/sites/log","containerId":"e85f5rv6tc17_abcdefhj","location":"Central ","category":"AppServiceConsoleLogs","EventStampType":"Stamp","EventPrimaryStampName":"wawfryus-prod-pn1-011","EventStampName":"waws-progffjjd-pn1-011","Host":"pl0sdlwk001B8Y","EventIpAddress":"10.20.1.201","records":true}

And for the other type of log of category “AppServiceHTTPLogs” where records.0.Host is actually present in the raw log there is no such error as you can see in the screenshot below.

AppServiceHTTPLogs raw log:

{"records":[{"time":"2025-09-18T08:06:05.1848010Z","EventTime":"2025-09-18T08:06:05.1848010Z","resourceId":"/SUBSCRIPTIONS/ [removed by moderator] -99C8-4f45-9350-E09FC4D6EB34/RESOURCEGROUPS/ABCDEFG_RG/PROVIDERS/MICROSOFT.WEB/SITES/ABCDEFGH","properties":"{\"CsHost\":\"abc.com\",\"CIp\":\"192.0.0.1\",\"SPort\":\"80\",\"CsUriStem\":\"\\/\",\"CsUriQuery\":\"\",\"CsMethod\":\"GET\",\"TimeTaken\":4,\"ScStatus\":\"200\",\"Result\":\"Success\",\"CsBytes\":\"795\",\"ScBytes\":\"4835\",\"UserAgent\":\"AlwaysOn\",\"Cookie\":\"ARRAffinity: 1f71335hggx3446hfhgds4653ba; \",\"CsUsername\":\"\",\"Referer\":\"\",\"ComputerName\":\"asd56fgfcgh\",\"Protocol\":\"HTTP\\/1.1\"}","category":"AppServiceHTTPLogs","EventStampType":"Stamp","EventPrimaryStampName":"wrru-5676-sdfhg","EventStampName":"wreyt-32w465-wewr","Host":"abcdefgh12345","EventIpAddress":"10.20.0.201"}],"_time": [removed by moderator] .595,"sourcetype":"Azure_Prod_AB_CIDE"}

Can anyone please provide any insight on this or please correct me If am going wrong somewhere.

https://github.com/Riti0302/Microsoft-Azure-Resource-Custom-Parser-in-Progress/blob/main/Custom%20parser-in%20progress → parser code

The JSON structure for your two log categories is different:

For AppServiceConsoleLogs Host is at the top level:
 

{
"_time": "[removed by moderator] .752",
"time": "2025-09-21T01:50:08.8341792Z",
"resultDescription": "х601b[32m[2025-09-23T07:50:08.833] [INFO] Job API - UID: V920",
"resourceId": "/SUBSCRIPTIONS/03D-99C8-4JQ25-965150-E09FC4SDFD6EB34/RESOURCEGROUPS/ABCD_EFGH_RG/PROVIDERS/MICROSOFT.WEB/SITES/ABCDEFGH",
"level": "Informational",
"operationName": "Microsoft.Web/sites/log",
"containerId": "e85f5rv6tc17_abcdefhj",
"location": "Central ",
"category": "AppServiceConsoleLogs",
"EventStampType": "Stamp",
"EventPrimaryStampName": "wawfryus-prod-pn1-011",
"EventStampName": "waws-progffjjd-pn1-011",
"Host": "pl0sdlwk001B8Y",
"EventIpAddress": "10.20.1.201",
"records": true
}


For AppServiceHTTPLogs Host is nested under Records:

{
"records": [
{
"time": "2025-09-18T08:06:05.1848010Z",
"EventTime": "2025-09-18T08:06:05.1848010Z",
"resourceId": "/SUBSCRIPTIONS/ [removed by moderator] -99C8-4f45-9350-E09FC4D6EB34/RESOURCEGROUPS/ABCDEFG_RG/PROVIDERS/MICROSOFT.WEB/SITES/ABCDEFGH",
"properties": {
"CsHost": "abc.com",
"CIp": "192.0.0.1",
"SPort": "80",
"CsUriStem": "/",
"CsUriQuery": "",
"CsMethod": "GET",
"TimeTaken": 4,
"ScStatus": "200",
"Result": "Success",
"CsBytes": "795",
"ScBytes": "4835",
"UserAgent": "AlwaysOn",
"Cookie": "ARRAffinity: 1f71335hggx3446hfhgds4653ba; ",
"CsUsername": "",
"Referer": "",
"ComputerName": "asd56fgfcgh",
"Protocol": "HTTP/1.1"
},
"category": "AppServiceHTTPLogs",
"EventStampType": "Stamp",
"EventPrimaryStampName": "wrru-5676-sdfhg",
"EventStampName": "wreyt-32w465-wewr",
"Host": "abcdefgh12345",
"EventIpAddress": "10.20.0.201"
}
],
"_time": "[removed by moderator] .595",
"sourcetype": "Azure_Prod_AB_CIDE"
}


To resolve this, you need to introduce conditional logic into your parser based on the category of the log. The parser should check the category first and then use the correct path to extract the hostname.


Hi ​@_Ritika13 , in addition to ​@kentphelps  suggestion, you could use this probe trick with replace ;

mutate {replace => {"probe"=>"%{records.0.time}"} on_error=> "_error.missingRepeatedField.records" } #probe for records repeated field existence using a known subfield

if ![_error][missingRepeatedField][records] { #records repeated field exists
...
}

else { #records repeated field does not exist, it is possible that the field does not exist at all or exists as a boolean type -as in your sample logs- or as any other data type.
}

This probe trick was mentioned in the adoption guide Deep Dive into UDM Part 3.


Adding “Host” in the statedata fixed the error.

And I don’t see the error mentioned above in the query, possibly a bug? Who knows!.
Let me know if this works. Thanks!

Attaching the screenshots of the output for the logs given above.
 


 

 


Hi ​@kentphelps 

 

Thank you for the reply. 

I actually tried with the conditional logic for parsing two types of logs (one starting with records.0 and other one starting with _time) based on the condition if EventName ==“ DiagnosticsLogs”.

Like you mentioned I also tried with checking the category first and then executing the code below, but even that gave me the same error. 

 

https://github.com/Riti0302/Microsoft-Azure-Resource-Custom-Parser-in-Progress/blob/main/Custom%20parser-in%20progress → CODE LINK of the custom parser.

 

The parser works for the above log starting with records.0. correctly.

https://github.com/Riti0302/Microsoft-Azure-Resource-Custom-Parser-in-Progress/blob/main/records.0%20raw%20log  → RAW LOG starting with records.0

 

But it starts throwing an error for the below kind of logs starting with _time and having the same EventName “DiagnosticsLogs”.

 

Microsoft-Azure-Resource-Custom-Parser-in-Progress/_time raw log at main · Riti0302/Microsoft-Azure-Resource-Custom-Parser-in-Progress → RAW LOG starting with _time

 

I’m quite confused about how to resolve this issue. Thank you for your time though.

 


Hi ​@AbdElHafez ,

Thank you for your response. I’ll explore this option and let you know if I make any progress.


@_Ritika13 Your condition variables are not correct.

If you removed all the code from line 931 onwards and do statedump ; you won’t find any records.0 value, so it the conditional will generate an error because the field does not exist, if you are looking for the existence of a field then you need to use probes/flags, not !=”” conditions.