Assistance Needed with Custom Parser Issues in Google Security Operations

Hi all,

I am new to Google Security Operations and am trying to create a custom parser but facing issues in two cases. Can anyone help me with this?

1. Problem:

I am attempting to parse the following data to UDM, but I am unable to loop through this data and print it, as it does not have a parent object.
sample input
[
{
"info": "STRING"
},
{
"info": "STRING"
}
]

2. Problem:

I have data that contains coordinates, and I am trying to map it to the region_coordinates field, which has a map type structure.

Sample input:

"location": { "coordinates": [ -84.6350721252, 33.8159259083, 0.0 ], "type": "Point" },

Code I tried:

The following code is not producing any response, and when I try with some other code, it throws an error for mapping:

for index, phoneNumber in location.coordinates {
mutate {
convert => {
"index" => "string"
"phoneNumber" => "string"
}
}
mutate {
replace => {
"phoneNumber_label" => ""
}
}
mutate {
replace => {
"phoneNumber_label.value" => "%{phoneNumber}"
}
on_error => "phoneNumber_invalid"
}
if ![phoneNumber_invalid] {
mutate {
merge => {
"event.idm.read_only_udm.principal.location.region_coordinates" => "phoneNumber_label"
}
on_error => "phoneNumber_label_merge_failed"
}
}
statedump {}
}

Any guidance or help would be greatly appreciated.
@cmmartin_google can you help in this problem .

Page 1 / 1

What error is it throwing, specifically?

hi @Mustache ,
this is the error i am getting for problem 1.
generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter json (0) failed: invalid JSON: json: cannot unmarshal array into Go value of type map[string]interface {}"generic::unknown: pipeline.ParseLogEntry failed: LOG_PARSING_CBN_ERROR: "generic::invalid_argument: pipeline failed: filter json (0) failed: invalid JSON: json: cannot unmarshal array into Go value of type map[string]interface {}"

can you guide me.

hi @cmmartin_google can you help me with the above problem that i am facing.
Thanks.

One tip about GSO parsers is that it's based off logstash. When you run into issues you can Google the error and logstash. Just a friendly tip.

It appears that you may not be extracting the JSON from the message prior to parsing it. Just a hunch without seeing the entire parser.

To do this you need a code block as your 2nd step after your placeholder variables.

Instead of walking through it, check out cmmartins layout of what a JSON parser should look like to understand what I mean by the JSON extraction.

It's this part:

json {

source => "message"

array_function => "split_columns"

}

https://www.googlecloudcommunity.com/gc/SIEM-Forum/Chronicle-Parser-JSON/m-p/669258/highlight/true#M273

hi @cmmartin_google can you help me with the above problem that i am facing.
Thanks.

Here is a working example that may help.

filter {



    json {

        source => "message"

        array_function => "split_columns"

    }



    mutate {

        copy => {

            "_lat" => "location.coordinates.1"

            "_long" => "location.coordinates.0"

        }

    }

    mutate {

        rename => {

            "_lat" => "event1.idm.read_only_udm.principal.location.region_coordinates.latitude"

            "_long" => "event1.idm.read_only_udm.principal.location.region_coordinates.longitude"

        }    

    }



    mutate {

        replace => {

            "event1.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"

        }

    }



    mutate {

        merge => {

            "@output" => "event1"

        }

    }



}

You may need to use

Thanks @cmmartin_google ,
can you provide any sample solution for the problem 1
if i the same code in array as given bellow i get th[
{
"info": "STRING"
},
{
"info": "STRING"
}
]

Thanks,
Harshwardhan R Patil

Thanks @cmmartin_google ,
can you provide any sample solution for the problem 1
if i the same code in array as given bellow i get th[
{
"info": "STRING"
},
{
"info": "STRING"
}
]

Thanks,
Harshwardhan R Patil

I tried the log sample below, but also get the same error. If this is the entire log then I think you will need to either split it in advance, or else use gsub to change the out brackets. If this is part of a larger log then you can use the JSON Loop functionality in CBN.

[

  {

    "info": "STRING"

  },

  {

    "info": "STRING"

  }

]

The json i have is in this format only. And as this doesn't have parent so facing issue for loop the data. Got completely stuck because of this. Is there any way to get loop this data.

I would explore using a GROK statement and capture the values using Regular Expressions against the entire message field.

That would be very helpful if you can provide solution for this.

Any guidance or help would be greatly appreciated.

One tip about GSO parsers is that it's based off logstash. When you run into issues you can Google the error and logstash. Just a friendly tip.

It appears that you may not be extracting the JSON from the message prior to parsing it. Just a hunch without seeing the entire parser.

To do this you need a code block as your 2nd step after your placeholder variables.

Instead of walking through it, check out cmmartins layout of what a JSON parser should look like to understand what I mean by the JSON extraction.

It's this part:

json {

source => "message"

array_function => "split_columns"

}

https://www.googlecloudcommunity.com/gc/SIEM-Forum/Chronicle-Parser-JSON/m-p/669258/highlight/true#M273

I try that it throw same error.

hi @cmmartin_google did you able to find any solution for this .
Thanks.

I know this is laaate... but to solve this problem is simple: just wrap the log with gsub and add a label (I added "events") to the log itself and then simply call json to parse. Use events to loop through it.

{"events":[

  {

    "info": "STRING"

  },

  {

    "info": "STRING"

  }

]

}

Reply

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded