Skip to main content

The logs on Cloud Logging can be ingested directly to Google Secops using export filters. 

You set a sink filter (e.g., logName:"syslog" AND textPayload:("auth failed")) and Secops only receives logs matching that. Although export filters are pretty straightforward and powerful, they have some limitations:

  • Filters are static – you can only control inclusion/exclusion at the time of sink creation.
  • You can’t easily enrich, transform, or route logs differently once they leave Cloud Logging.
  • Scaling is limited – a single sink can push to Secops but lacks intermediate processing.

 

An alternative way to  ingest logs from Cloud Logging is to use Pub/Sub. 

A Pub/Sub topic is a named resource in Google Cloud Pub/Sub that acts as a channel for sending messages from publishers to subscribers.

  • Publishers send (or "publish") messages to a topic.
  • Subscribers subscribe to that topic and receive those messages.

The topic itself doesn’t store the messages permanently—it just serves as the communication point. Once a message is published to a topic, Pub/Sub delivers it to all the subscriptions attached to that topic.

 

Pub/Sub Log Push → Secops

Using a Cloud Logging sink → Pub/Sub → Secops pipeline adds a middle layer:

  • Logging sink sends all (or broad) logs to Pub/Sub.
  • Pub/Sub delivers messages to Secops.
  • Optionally, you can insert Dataflow or custom subscriber processing in between.

This enables things you cannot do with direct export:

  • Multiple subscriptions: One Pub/Sub topic can feed Chronicle and other systems (SIEM, storage, monitoring).
  • Dynamic filtering/enrichment: Instead of hard sink filters, you can apply richer logic in Dataflow (regex parsing, enrichment with metadata, adding labels, redaction of sensitive fields).
  • Replay capability: Pub/Sub retains messages (default 7 days), allowing re-delivery if Secops ingestion lags/fails. Direct export has no retry buffer.
  • Future extensibility: You can later add new consumers (e.g., a security pipeline) without touching the Cloud Logging sink.

Today, we will see how to ingest logs from Cloud Logging to Secops using Pub/Sub:

  1. Via gcloud shell, create a Pub/Sub Topic first:

# gcloud pubsub topics create syslogs-topic

AD_4nXcTd6Da1WkGyaUmLAHeWbKex4oZ_TPCBL3OEUP5fQdkrkSV0Lx6v303amtgA5syWtMDAnde6R3Eyb_L4OYpYHsZDKkytoi6XJvLNKq_YAJHvTEZ9k5y_uXW0TE1AQt1zdBkWy5rrWBepBogjy1gCSvh?key=5hPJxj179-26S4510RV4WQ

Here  we created a topic named “syslogs-topic”

 

  1. Create a log sink. 

In the context of Pub/Sub:

A sink is what takes logs from Cloud Logging and pushes them into a Pub/Sub topic.

From there, subscribers (like Secops) can consume them in near real time.

 

# gcloud logging sinks create syslogs-sink \

  pubsub.googleapis.com/projects/okkes-secops-2025/topics/syslogs-topic \

  --log-filter='logName:"syslog"'

 

AD_4nXeQqNIuBtFY-75VgZ4OwVhdAPLNZfU81pez6DCk4EmIhWMdUn0a7xnpVZPGGi0z6BGLrO7a_meAGMsANnhCjFHhqxRsJ8Qy5haKiexIQI38kV5_YG0d3ud2HGBqgQaiALKBTgX7xqLnYzMVjl1g_UDl?key=5hPJxj179-26S4510RV4WQ

Here we created a log sink named “syslogs-sink” in “syslogs-topic”. It will push all logs from Cloud Logging to Pub/Sub topic where the logName field contains “syslog”. 

Pub/Sub log export filters are very powerful and can also have logical operators like AND/OR. For example, we could filter only authentication failure logs from syslog like:

--log-filter='logName=" syslog" AND (jsonPayload.message:"auth failed" OR textPayload:"auth failed")'

 

If you pay attention to the log sink creation command output, you will notice :

“Please remember to grant `serviceAccount:SERVICE ACCOUNT` the Pub/Sub Publisher role on the topic. This is the default service account of our Secops project where we created our pub/sub topic.

  1. Let’s assign the required roles to our Service Account

# gcloud pubsub topics add-iam-policy-binding syslogs-topic \

  --member="serviceAccount:SERVICE ACCOUNT \

  --role="roles/pubsub.publisher"

AD_4nXd5Ck6Dm1WYCMeXEEUwfMQRoLPSFU-gA-0t68N145p9DMPuJksI1Mm-nDgtzU9kChvbdWuu8pxNUg1ZP9exCJdydpl-1atsETQ6XFSpcgUxXiydO8FlXfLpxkPUZalvWBSKBSSiyAO77xDoGcG2VZA8?key=5hPJxj179-26S4510RV4WQ

 

  1. At this point, we will create a feed on Secops SIEM for our Pub/Sub Topic:

AD_4nXdCsa5c6gO0jV9-S8XdW0KURmbi_xyQIDPLlkSMGLShhKcFNem_Ckh8DSvKkdA-u7UAAEjC1PRu86SEKOkxOXxRcDntXS8LxCihTXRn1rqBCzdv4DdJwNseEqj3H_D7cYW32Np2ANHPep0aTs6IVs4?key=5hPJxj179-26S4510RV4WQ

Source type will be Google Cloud Pub/Sub Push, Log type here is “Unix System” because we are going to ingest Unix syslog. 

 

AD_4nXcSAIIO3jCELN36whxFt5ayUW2Xo6Q4c8g871WFDyK_qLNvxz3f074VNAtgZeUcqvaa-hssAVX5HtFEx2pY2b8UNoqLm27zpdIN82Ztvu5ihB_SlkFlVcVt85T2qvcLfWb8uFmm3qdh4amJC981MDLz?key=5hPJxj179-26S4510RV4WQ

AD_4nXeRoTTCb46Slj6A226UVeor6HnGVZw5vRV29sYdm4VLGJCPhe_LbEjQbo2UZ0v3PRMUfxi1RBGqe68cD9i2ILiuZJucC091KsY_HMV_Ao1tKfco8H3mjdTjtn-QA_XqNxrDKZ5g0lQEdE6OJ1JpX3Q?key=5hPJxj179-26S4510RV4WQ

After clicking submit, you will see Endpoint information, note it down:

AD_4nXfOiJr4g0C7uOGtFk9gihOnQUMlm1A63-l8ikT8DZ-LX6QB3KHMipq_ZgEFxutJH2bUpdWmFbbOkRx1qufLhxcgQxw8k52NhqK5AUKD8CGG8fkptnxdKdqxOgXnHiCGvnqBEUptI3XZHLKh7i-zV_4?key=5hPJxj179-26S4510RV4WQ

 

Click “Done”. Now our feed is ready to accept Unix Syslog coming from our Pub/Sub topic.

 

  1. Next, we will create a subscription for our Pub/Sub Topic:

In Google Cloud Pub/Sub, a subscription is a named resource that represents the connection between a topic and a subscriber application.

  • A topic is where messages are published.
  • A subscription defines how those messages are delivered to subscribers.

When you create a subscription on a topic, Pub/Sub ensures that every message published to that topic is delivered to that subscription (unless it expires or is deleted).

There are two main types of subscriptions:

  1. Pull subscription
    • Your application explicitly requests (pulls) messages from Pub/Sub.
    • Example: a worker app calls the API to fetch messages when it’s ready.

 

  1. Push subscription
    • Pub/Sub automatically sends (pushes) messages to an HTTPS endpoint you specify.
    • Example: your web service receives incoming POST requests from Pub/Sub.

 

If you remember, we created a Secops feed for Pub/Sub push, let’s create a push subscription in our topic:

# gcloud pubsub subscriptions create syslogs-sub \

  --topic=syslogs-topic \

  --push-endpoint=ENDPOINT URL \

  --ack-deadline=30

 

AD_4nXdbckxQQ2tTj77vm2iohK0r9zyMtLrxlEfdaKVJD-lejmiRSofpuWQsIrjotLnTiq76wr-aYdxE1xIxPQVhN776bKbVCf05qPVQ35zQ9z5Ao7FonE7W8bVM2iZgAg8HTFXa1KsFLXx91-PJW1_KBpeU?key=5hPJxj179-26S4510RV4WQ

We created a push subscription named “syslogs-sub” under our Pub/Sub topic.

 

  1. The Pub/Sub push feed needs to be in your Google Cloud project bound to Google SecOps for the subscription to authenticate. It uses a JWT token from your service account for authentication:

 

Navigate to Pub/Sub 🡪 Subscriptions on GCP Console:

AD_4nXek24IQhiovK8RPRBBb7BAju-lK-eThfwxryWp8atWzbmdLQ6OMQeJC9c4Qxxbft_T583cxauDw1PVDR-54wadXso6cPS4m3ZwQ7sRaoKdUG9-8T48N9A7e7zm6YqxDgj71jr_oCz5hji3MZnhwcB2u?key=5hPJxj179-26S4510RV4WQ

 

Edit “syslogs-sub”

AD_4nXeojgHeIzT4rU4D0QNdy6KcV-k9rA5A_7iWnkSez47Lz9mMRj-iXOh5nVAvDuR-UH5THqdB3unt3Zhm6yPYGt8Up_HpTU8W_Zqob_Ek6Uzd4jB6PZW5f9usLP3Bk91BflhhEkXj--ix_Qwh6ELfgOGu?key=5hPJxj179-26S4510RV4WQ

 

Click “Enable  Authentication” and choose the service account you assigned “pubsub.publisher” role in step 3 and click Update:

AD_4nXctWChGnVsp2x2qKXAQ0xfp6r8NOatfhDwSsxm3RZLh71QpyvFYqHAJ95dQOAGyuzx1lAJRCosaQssg27sqPdxMwMwL7kncAiCESrawv2L2Jupa7eTAIv2al5hMlNXnZGp8pQJpcPkV3pQtyzlFA44?key=5hPJxj179-26S4510RV4WQ

 

  1. You will see the logs and events on Secops:

AD_4nXdOnGjkaMssaTDsbWcRh4tT7-ErGUfrWg0XHeCFh8kHpfKno4Nt9fP-_0jmSQWVWMAzNr11M83ptaT21xtjGjWVpKhftPbZyYG3hRuo8s4FX1g0rzPAKKxOlRArOV2JXDk9xUjhHjTI5FQeiy79RhXA?key=5hPJxj179-26S4510RV4WQ

 

Summary:

In our example, we exported Syslog formatted logs from Cloud Logging to Secops. Of course, the direct ingestion export filter log_id("syslog") would be much easier and straightforward.

Direct ingestion export filters on Cloud Logging are very versatile and powerful. In certain scenarios where export filters are limited, Pub/Sub can be used as a log ingestion mechanism from Cloud Logging to Secops.

 

Finally, here is a comparison of both ingestion methods:

 

The below is a parser extension to GCP_CLOUDAUDIT to map SOAR related fields better. 

 

In Secops UDM, audit-like or orchestration logs don’t always have a perfect 1:1 field for things like playbook name, since UDM is primarily designed around security telemetry (events, entities, processes, network flows, etc.). But Google provides guidance for mapping "workflow / playbook / rule"–style metadata into UDM contextual fields. So for this parser extension:

 

  • "integration_name" field is mapped to UDM field : about.application
  • "version" field is mapped to UDM field : about.platform_version
  • "case_id" field is mapped to UDM field : security_result.rule_id
  • "action_name" field is mapped to UDM field : security_result.action_details
  • "playbook_name" field is mapped to UDM field : target.application

 

Here we go:

 

filter {
json {
  source => "message"
  array_function => "split_columns"
}

date {
        match => "timestamp", "yyyy-MM-ddTHH🇲🇲ss" ]
        on_error => "no_match"
    }

#*****

mutate {
   replace => {
     "udm_event.idm.read_only_udm.target.application" => "%{labels.playbook_name}"
   }
   on_error => "no_playbook"
}    

#*****
mutate {
   replace => {
     "about.application" => "%{labels.integration_name}"
   }
   on_error => "no_integration"
}    

if nno_integration]
{
  mutate {
   replace => {
     "about.application" => ""
   }  
}  
}
#*****

mutate {
   replace => {
     "results.action_details" => "%{labels.action_name}"
   }
   on_error => "no_action"
}    

if bno_action]
{
  mutate {
   replace => {
     "results.action_details" => ""
   }  
}  
}

#*****

mutate {
   replace => {
     "results.rule_id" => "%{labels.case_id}"
   }
   on_error => "no_caseid"
}    
if bno_caseid]
{
  mutate {
   replace => {
     "results.rule_id" => ""
   }  
}  
}

#*****

mutate {
   replace => {
     "about.platform_version" => "%{labels.version}"
   }
   on_error => "no_version"
}    

if nno_version]
{
  mutate {
   replace => {
     "about.platform_version" => ""
   }  
}  
}

#*****

mutate {
   merge => {
     "udm_event.idm.read_only_udm.security_result" => "results"
   }
}

mutate {
   merge => {
     "udm_event.idm.read_only_udm.about" => "about"
   }
}

   mutate {
        merge => {
            "@output" => "udm_event"
        }
    }
}

 

Reply


Terms & ConditionsCookie settings
Privacy Policy Terms of Service Gen AI Policy Community Guidelines Cookie Settings

© 2025 Google. All rights reserved.