Assistance Needed for Parsing JSON Array in Google Security Operations

Hi @harshwardhan16  

You can loop for this hope this will resolve your issue .

Hi @matthewnichols / @deeshu 

Can you help me in this problem i try some method but nothing work. 

Any guidance or help on how to resolve this issue would be greatly appreciated.

Thanks 

Hi @harshwardhan16 Apologies, I've been out of office the last week. Let me find someone that might be able to help answer your question. 

What does your custom parser look like?

also, you can use a <statedump> on the 2nd to last line of your parser config to see where the problem lies.  

Thanks

hi @dnehoda ,
I try code similer to this 

filter {

json {
source => "message_1"
array_function => "split_columns"
on_error => "not_in_json_format"
}
mutate {
replace => {
"event1.idm.read_only_udm.metadata.product_name" => "Upstream"
"event1.idm.read_only_udm.metadata.vendor_name" => "Upstream"
"event1.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"
"event1.idm.read_only_udm.metadata.product_version" => "%{json_data.source}"
}
}
#TODO(evalute a more specific UDM event type)

mutate {
merge => {
"@output" => "event1"
}
}
}

error is that given data is not a json format. 

Hi @harshwardhan16 ,

The input message is a list not a JSON, that is why the parser is throwing away an error because the json clause expects a json object, the split clause is for json values that are lists within the root json object, not for a list input.

Before I modify your parser, do you need your parser to be able to handle -or expect it to receive- list input like that ? and do you need a single event or multi-event output ?

I modified my last post, this should be fine, you could assign the tokens of "v" to other fields instead of having all of them in "metadata.product_version" ;

filter {

     

mutate {

        gsub => [

            "message","\\\\[","",

               ]

        on_error => "subError"

    }

mutate {

        gsub => [

            "message","\\\\]","",

               ]

        on_error => "subError"

    }

mutate {

        gsub => [

            "message","\\\\n","",

               ]

        on_error => "subError"

    }

mutate {

        gsub => [

            "message","\\\\},\\\\{","}|{",

               ]

        on_error => "subError"

    }

    mutate {

        split => {

        source => "message"

        separator => "|"

        target => "messageSplit"

        }

    }

    for k,v in messageSplit {

   mutate {

     convert => {

       "k" => "string"

     }

   }

    mutate {

    replace => {

    "event" => ""

    }

    }

    json {

    source => "v"

    array_function => "split_columns"

    }

    mutate {

    replace => {

    "event.idm.read_only_udm.metadata.product_name" => "Upstream"

    "event.idm.read_only_udm.metadata.vendor_name" => "Upstream"

    "event.idm.read_only_udm.metadata.event_type" => "GENERIC_EVENT"

    "event.idm.read_only_udm.metadata.product_version" => "%{v}"

    }

    }

    mutate {

    merge => {

        "@output" => "event"

    }

    }

statedump {

    "label" => "afterLoop"}

    }

}