+6Hello,
We have configured two separate export filters to send GCP Cloud Audit Logs from two different GCP organizations to a single Chronicle instance.
Is there a way to add a field, label, or tag to these logs to identify the originating organization?
Thank you in advance for your support!
Best answer by cmmartin_google
Unfortunately the native GCP ingestion method does not allow for tagging (ingestion labels).
I have written about this before, and using your own pipeline means you can use ingestion labels and tag data accordingly - https://medium.com/@thatsiemguy/multi-tenant-google-cloud-log-collection-with-pubsub-push-5f4bf9775c84 - however, this does mean you incur the additional costs of using PubSub or GCS.
Parser Extensions can support custom Ingestion Labels, but the challenge with GCP Logs is they don't consistently include the Org ID. Example Parser Extension below for reference, but note this is not going to solve your issue as is, and you would need to find a unique key so you could apply a label (which as I say, from past experience GCP logs don't include this for the most part).
Otherwise, this is an area to request a Feature Request that the native GCP Ingestion requires the ability to support custom Ingestion Labels (e.g., like Feed Management)
filter {
mutate {
replace => {
"severity" => ""
"protoPayload.authenticationInfo.principalSubject" => ""
}
}
json {
on_error => "not_json"
source => "message"
array_function => "split_columns"
}
if [not_json] {
drop{
tag => "TAG_UNSUPPORTED"
}
} else {
#1 - Override Namespaces
mutate {
replace => {
"event1.idm.read_only_udm.principal.namespace" => "TMO"
}
}
mutate {
replace => {
"event1.idm.read_only_udm.target.namespace" => "TMO"
}
}
mutate {
replace => {
"event1.idm.read_only_udm.src.namespace" => "TMO"
}
}
#2 - Custom Ingestion Keys
if [severity] != "" {
mutate {
replace => {
"_ingestion_labels.key" = "severity"
"_ingestion_labels.value" = "%{severity}"
}
}
mutate {
merge => {
"event1.idm.read_only_udm.metadata.ingestion_labels" => "_ingestion_labels"
}
}
}
mutate {
merge => {
"@output" => "event1"
}
}
# google.identity.sts.SecurityTokenService.WebSignIn
if [protoPayload][authenticationInfo][principalSubject] != "" {
mutate {
replace => {
"_additional_principalSubject.key" => "authenticationInfo_principalSubject"
"_additional_principalSubject.value.string_value" => "%{protoPayload.authenticationInfo.principalSubject}"
}
}
mutate {
merge => {
"event1.idm.read_only_udm.additional.fields" => "_additional_principalSubject"
}
}
}
}
}
+11Unfortunately the native GCP ingestion method does not allow for tagging (ingestion labels).
I have written about this before, and using your own pipeline means you can use ingestion labels and tag data accordingly - https://medium.com/@thatsiemguy/multi-tenant-google-cloud-log-collection-with-pubsub-push-5f4bf9775c84 - however, this does mean you incur the additional costs of using PubSub or GCS.
Parser Extensions can support custom Ingestion Labels, but the challenge with GCP Logs is they don't consistently include the Org ID. Example Parser Extension below for reference, but note this is not going to solve your issue as is, and you would need to find a unique key so you could apply a label (which as I say, from past experience GCP logs don't include this for the most part).
Otherwise, this is an area to request a Feature Request that the native GCP Ingestion requires the ability to support custom Ingestion Labels (e.g., like Feed Management)
filter {
mutate {
replace => {
"severity" => ""
"protoPayload.authenticationInfo.principalSubject" => ""
}
}
json {
on_error => "not_json"
source => "message"
array_function => "split_columns"
}
if [not_json] {
drop{
tag => "TAG_UNSUPPORTED"
}
} else {
#1 - Override Namespaces
mutate {
replace => {
"event1.idm.read_only_udm.principal.namespace" => "TMO"
}
}
mutate {
replace => {
"event1.idm.read_only_udm.target.namespace" => "TMO"
}
}
mutate {
replace => {
"event1.idm.read_only_udm.src.namespace" => "TMO"
}
}
#2 - Custom Ingestion Keys
if [severity] != "" {
mutate {
replace => {
"_ingestion_labels.key" = "severity"
"_ingestion_labels.value" = "%{severity}"
}
}
mutate {
merge => {
"event1.idm.read_only_udm.metadata.ingestion_labels" => "_ingestion_labels"
}
}
}
mutate {
merge => {
"@output" => "event1"
}
}
# google.identity.sts.SecurityTokenService.WebSignIn
if [protoPayload][authenticationInfo][principalSubject] != "" {
mutate {
replace => {
"_additional_principalSubject.key" => "authenticationInfo_principalSubject"
"_additional_principalSubject.value.string_value" => "%{protoPayload.authenticationInfo.principalSubject}"
}
}
mutate {
merge => {
"event1.idm.read_only_udm.additional.fields" => "_additional_principalSubject"
}
}
}
}
}
+6Unfortunately the native GCP ingestion method does not allow for tagging (ingestion labels).
I have written about this before, and using your own pipeline means you can use ingestion labels and tag data accordingly - https://medium.com/@thatsiemguy/multi-tenant-google-cloud-log-collection-with-pubsub-push-5f4bf9775c84 - however, this does mean you incur the additional costs of using PubSub or GCS.
Parser Extensions can support custom Ingestion Labels, but the challenge with GCP Logs is they don't consistently include the Org ID. Example Parser Extension below for reference, but note this is not going to solve your issue as is, and you would need to find a unique key so you could apply a label (which as I say, from past experience GCP logs don't include this for the most part).
Otherwise, this is an area to request a Feature Request that the native GCP Ingestion requires the ability to support custom Ingestion Labels (e.g., like Feed Management)
filter {
mutate {
replace => {
"severity" => ""
"protoPayload.authenticationInfo.principalSubject" => ""
}
}
json {
on_error => "not_json"
source => "message"
array_function => "split_columns"
}
if [not_json] {
drop{
tag => "TAG_UNSUPPORTED"
}
} else {
#1 - Override Namespaces
mutate {
replace => {
"event1.idm.read_only_udm.principal.namespace" => "TMO"
}
}
mutate {
replace => {
"event1.idm.read_only_udm.target.namespace" => "TMO"
}
}
mutate {
replace => {
"event1.idm.read_only_udm.src.namespace" => "TMO"
}
}
#2 - Custom Ingestion Keys
if [severity] != "" {
mutate {
replace => {
"_ingestion_labels.key" = "severity"
"_ingestion_labels.value" = "%{severity}"
}
}
mutate {
merge => {
"event1.idm.read_only_udm.metadata.ingestion_labels" => "_ingestion_labels"
}
}
}
mutate {
merge => {
"@output" => "event1"
}
}
# google.identity.sts.SecurityTokenService.WebSignIn
if [protoPayload][authenticationInfo][principalSubject] != "" {
mutate {
replace => {
"_additional_principalSubject.key" => "authenticationInfo_principalSubject"
"_additional_principalSubject.value.string_value" => "%{protoPayload.authenticationInfo.principalSubject}"
}
}
mutate {
merge => {
"event1.idm.read_only_udm.additional.fields" => "_additional_principalSubject"
}
}
}
}
}
Thank you very much for the reply @cmmartin_google !
I’ve noticed that most GCP audit logs, if not all, include the logName field, which typically follows structures such as:
Would it be possible to create a parser extension that extracts the organization ID directly from logName values starting with organizations/...?
Additionally, for logName values starting with projects/..., could the parser assign the organization ID based on a naming convention, such as the one I showcased above with org1 and org2 as values that the project id starts with ?
Would appreciate any support !
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.