Author:Β hzmndt@google.com
Category:Β Google Security Operations / SIEM & SOAR Analytics
Reading Time:Β 8 min
Β
Introduction
As SOC teams mature, combining detection engineering, incident response, and business intelligence is critical for executive dashboards, metric tracking (MTTD/MTTR), and correlation with non-security business data.
Google Security Operations provides direct access to your data inΒ Google Cloud BigQuery. However, there is an architectural distinction you need to understand betweenΒ SIEM telemetryΒ andΒ SOAR operational data:
- SIEM Telemetry (UDM Events, Detections, IoCs, Entities):Β Handled by the modernΒ Advanced BigQuery ExportΒ streaming architecture.
- SOAR Operational Data (Cases, Alerts, Playbooks, SLAs):Β Currently not included in Advanced BigQuery Export and still accessed viaΒ Legacy BigQuery / BYOBQ (
siemplify_search_everything_db).
This guide covers how both pipelines work, the datasets available in each, how to gain access, and practical use cases for analyzing your SOAR data in BigQuery.
π Part 1: Advanced BigQuery Export (SIEM Data)
Advanced BigQuery ExportΒ is a streaming pipeline for Google SecOpsΒ Enterprise PlusΒ customers.
Instead of requiring customers to maintain heavy ingestion jobs, Google manages the storage and streaming pipeline via the BigQuery Storage Write API in a dedicated tenant project. Customers receive aΒ read-only Linked DatasetΒ namedΒ secops_linked_datalakeΒ directly inside their own Google Cloud Bring-Your-Own-Project (BYOP).
Β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Google SecOps Managed Tenant β
β ββββββββββββββββ Storage Write API ββββββββββββββ β
β β Ingestion & β βββββββββββββββββββββΊ β Managed BQ β β
β β Normalizationβ (Fine-grained DML) β Storage β β
β ββββββββββββββββ βββββββ¬βββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββΌββββββββββ
β Analytics Hub /
β Linked Dataset
βββββββββββββββββββββββββββββββββββββββββββββββββΌββββββββββ
β Customer BYOP Project β β
β βΌ β
β `secops_linked_datalake`
β (events, rule_detections,
β ioc_matches, entity_graph)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Key Highlights
- Near Real-Time Data Freshness:Β Telemetry streams in withinΒ < 5β10 minutesΒ of ingestion.
- Predictable Cost:Β Google absorbs all ingestion, streaming insert, and storage costs. You only pay for BigQuery compute queries you run in your BYOP.
- Automated Deduplication:Β Uses Fine-Grained DML (FGDML) merges to handle late-arriving and re-enriched events in place.
Available SIEM Linked Datasets & Freshness
TheΒ secops_linked_datalakeΒ dataset exposes the following tables/views:
| Table / Dataset Name | Description | Target Freshness | Deduplication Unique Identifier |
|---|---|---|---|
events | Normalized security events in the Unified Data Model (UDM) schema. | < 5 minutes | idΒ (String representation) |
rule_detections | Detections generated by Google SecOps YARA-L detection engine rules. | < 5 minutes | detection.id |
ioc_matches | Indicator of Compromise (IoC) matches against UDM events (global & customer feeds). | < 5 minutes | Composite key:Β day_bucket_seconds,Β feed_log_type,Β ioc_type,Β ioc_value |
entity_graph | Contextual data about entities (users, assets) and their relationships. | ~4 hoursΒ (Batch) | Composite key:Β partition_day,Β metadata.product_entity_id,Β metadata.event_metadata.id |
ingestion_metrics | Statistics on log ingestion volume, produced events, and unparsed errors. | ~5 minutes | None (Append-only time-series) |
entity_enum_value_to_name_mapping | Maps numeric enum values to human-readable strings for Entity Graph. | Static / Reference | None |
udm_enum_value_to_name_mapping | Maps numeric enum values to human-readable strings for UDM events. | Static / Reference | None |
πΒ Quick Verification Query:
Βsql
SELECT
metadata.event_timestamp,
metadata.product_name,
metadata.event_type,
principal.ip,
target.ip
FROM
`YOUR_PROJECT_ID.secops_linked_datalake.events`
WHERE
metadata.event_timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
LIMIT 10;
