Skip to main content
Solved

Json parsing inside Message

  • May 20, 2024
  • 4 replies
  • 22 views
A
  • Anonymous

Hello Team, 
we are trying to parse fields from json log format, but there are nested fields  Within the "Message".

we user the json filter and other different steps.

All the fields inside the "Message" are not being parsed, Please chek the sample log for reference.
{"EventReceivedTime": "202*-0*-** 21:44:45","SourceModuleName": "in_json_log","SourceModuleType": "im_file","Message": ",{\\"_id\\":\\"66**c7d23afb**6e6*0f2\\",\\"serialNum\\":0,\\"acknowledged\\":***,\\"time\\":\\"202*-0*-**T14:14:04.657Z\\",\\"hostname\\":\\"e-d**-g**l-cluster-normal--7-mm.q-gcp-00-d-g-24-02.181862379",\\"fqdn\\":\\"\\",\\"containerName\\":\\"bu\\",\\"containerID\\":\\"2a91f881c8075c4f34db5*****3d6e\\",\\"imageName\\":\\"d***.io/google/cloud-:l\\",
 please suggest a way to parse the fields insite Message.

Best answer by bsalvatore

Hi,

This is the real message received in the SIEM (escaped character, comma character before { , etc.)? Because if the original message like the following:

{ "EventReceivedTime": "202*-0*-** 21:44:45", "SourceModuleName": "in_json_log", "SourceModuleType": "im_file", "Message": { "_id": "66**c7d23afb**6e6*0f2", "serialNum": 0, "acknowledged":"***", "time": "202*-0*-**T14:14:04.657Z", "hostname": "e-d**-g**l-cluster-normal--7-mm.q-gcp-00-d-g-24-02.181862379", "fqdn": "", "containerName": "bu", "containerID": "2a91f881c8075c4f34db5*****3d6e", "imageName": "d***.io/google/cloud-:l" } }

 you can easy extract it using the json command:

json { source => "message" array_function => "split_columns" on_error => "not_json" }

 and the parser automatically create the id, serialNum, hostname, etc. variable.

4 replies

bsalvatore
Forum|alt.badge.img+6
  • bsalvatoreBronze 1
  • Bronze 1
  • Answer
  • May 20, 2024

Hi,

This is the real message received in the SIEM (escaped character, comma character before { , etc.)? Because if the original message like the following:

{ "EventReceivedTime": "202*-0*-** 21:44:45", "SourceModuleName": "in_json_log", "SourceModuleType": "im_file", "Message": { "_id": "66**c7d23afb**6e6*0f2", "serialNum": 0, "acknowledged":"***", "time": "202*-0*-**T14:14:04.657Z", "hostname": "e-d**-g**l-cluster-normal--7-mm.q-gcp-00-d-g-24-02.181862379", "fqdn": "", "containerName": "bu", "containerID": "2a91f881c8075c4f34db5*****3d6e", "imageName": "d***.io/google/cloud-:l" } }

 you can easy extract it using the json command:

json { source => "message" array_function => "split_columns" on_error => "not_json" }

 and the parser automatically create the id, serialNum, hostname, etc. variable.

Aravind3
A
  • Anonymous
  • May 21, 2024

Hi,

This is the real message received in the SIEM (escaped character, comma character before { , etc.)? Because if the original message like the following:

{ "EventReceivedTime": "202*-0*-** 21:44:45", "SourceModuleName": "in_json_log", "SourceModuleType": "im_file", "Message": { "_id": "66**c7d23afb**6e6*0f2", "serialNum": 0, "acknowledged":"***", "time": "202*-0*-**T14:14:04.657Z", "hostname": "e-d**-g**l-cluster-normal--7-mm.q-gcp-00-d-g-24-02.181862379", "fqdn": "", "containerName": "bu", "containerID": "2a91f881c8075c4f34db5*****3d6e", "imageName": "d***.io/google/cloud-:l" } }

 you can easy extract it using the json command:

json { source => "message" array_function => "split_columns" on_error => "not_json" }

 and the parser automatically create the id, serialNum, hostname, etc. variable.


Hi bsalvatore,

I tried the json filter, it dosent work for the fileds inside Message.

{
    "EventReceivedTime": "202*-0*-** 21:44:45",
    "SourceModuleName": "in_json_log",
    "SourceModuleType": "im_file",

These fields will work with the json filter 

but the fields inide Message will not get parsed.

"Message": {
        "_id": "66**c7d23afb**6e6*0f2",
        "serialNum": 0,
        "acknowledged":"***",
        "time": "202*-0*-**T14:14:04.657Z",
        "hostname": "e-d**-g**l-cluster-normal--7-mm.q-gcp-00-d-g-24-02.181862379",
        "fqdn": "",
        "containerName": "bu",
        "containerID": "2a91f881c8075c4f34db5*****3d6e",
        "imageName": "d***.io/google/cloud-:l"
    }

 

Aravind3
bsalvatore
Forum|alt.badge.img+6
  • bsalvatoreBronze 1
  • Bronze 1
  • May 21, 2024

have you tried to use the statedump {} command to debug the message? If the raw log is correct, you should view like the screenshot in the debug view

In this case you can simply access to data using the following block (I suggest you to check if a text are mapped):

if [Message][containerID] != "" { mutate { replace => { "event.idm.read_only_udm...." => "%{Message.containerID}" } } }

 

A
  • Anonymous
  • May 22, 2024

have you tried to use the statedump {} command to debug the message? If the raw log is correct, you should view like the screenshot in the debug view

In this case you can simply access to data using the following block (I suggest you to check if a text are mapped):

if [Message][containerID] != "" { mutate { replace => { "event.idm.read_only_udm...." => "%{Message.containerID}" } } }

 


I have user the statedump, the fields are getting mapped.

I will use the above syantx, hopefully it works 

Terms & ConditionsCookie settingsAccessibility statement
Privacy Policy Terms of Service Gen AI Policy Community Guidelines Cookie Settings

© 2025 Google. All rights reserved.