Author: Matan Hayun, Product Manager, Google SecOps
Eliminate ingestion blind spots, verify log freshness, and isolate pipeline issues with a unified control plane.
| TL;DR: The Google SecOps Data Health Hub (DHH) is now live in Public Preview! DHH delivers a centralized, real-time control plane for log ingestion monitoring, parser normalization tracking, and automated anomaly detection. Get instant pipeline visibility, actionable status badges, and direct one-click remediation pathways.
🎬 Prefer watching over reading? Catch a quick high-level overview video of the Health Hub in action here: 👇 |
The Ingestion Challenge: Ending Pipeline Blind Spots
Security detections depend strictly on data freshness and complete log coverage. When a rule stops firing, security teams face immediate questions: Are logs arriving? Did a vendor update break parser normalization? Is an API collector experiencing lag?
Historically, answering these questions required hopping between fragmented dashboards, checking timestamps manually, or opening support tickets. The Data Health Hub replaces this manual guesswork with continuous, automated pipeline observability across the ingestion lifecycle.
What is the Data Health Hub?
The Data Health Hub provides a single, integrated command center built on three core capabilities:
-
Real-Time Transparency: Tracks live ingestion volumes, collection health, and normalization success across all feeds and log types.
-
Actionable Diagnostics: Automatically flags volume drops, spikes, or parsing errors with direct links to edit feeds or inspect active parsers.
-
Contextual Integration: Embeds health badges (mini-maps) across SecOps workflows, including Rules, Search, and Content Hub.
At its core, DHH features an automated Irregularity-Detection Engine that calculates z-score statistical baselines against a personalized 15-day rolling average, catching true anomalies while ignoring normal seasonal fluctuations.
Key Operational Use Cases: Accelerating SOC Operations
Whether you are a security engineer troubleshooting an active alert or a SOC manager auditing overall coverage, DHH streamlines core operational workflows:
-
Rapid Troubleshooting & Self-Remediation: When detections stop firing, analysts check DHH to see if raw logs dropped or parsing failed, using direct links to Edit Feed or View Parser to self-remediate immediately.
-
Post-Onboarding Feed Verification: Engineers onboarding new feeds (e.g., CrowdStrike or GCP Audit) instantly confirm that events are flowing and normalizing into UDM correctly.
-
Proactive Pipeline Maintenance: SOC managers audit health weekly to spot ballooning sources, monitor quota thresholds, evaluate parsing errors, and catch silent host outages before they impact detection coverage.
Core Workspace: Dashboard & Deep-Dive Analytics
DHH combines high-level system summaries with granular deep-dive views across three main visual components:
1. System Summary & Metrics Overview
Top-level metric cards surface Total Data Sources, Total Parsers, Overall System Status, active anomaly counts, and more.
2. Dual High-Density Health Tables
-
Health Status by Data Source: Monitors raw log streams, collector feeds, and status badges (Healthy, Warning, Critical).
-
Health Status by Parser: Tracks log parsing success rates. When errors or anomalies occur, DHH pinpoints the primary cause—such as configuration issues, regex errors, or filter drops—and provides clear, actionable remediation guidance.
3. Precise Deep-Dive Analytics & Metrics Workspace
Clicking 'Dashboard' on any row opens a comprehensive deep-dive workspace. Far beyond basic uptime counters, the Deep-Dive Dashboard delivers rich, multi-dimensional analytics to diagnose complex pipeline behaviors:
-
Ingestion & Throughput Volumetrics: High-resolution time-series charts displaying volume trends, feed run history and unexpected traffic spikes or drops.
-
Parsing & Normalization Ratios: Detailed breakdowns of parsed vs. unparsed events, error code distributions, and field-level mapping success rates.
-
Granular Latency Tracking: Correlated curves for SecOps Ingestion Latency (gateway-to-database write) and end-to-end Source-to-Ingestion Latency, filterable by Collector ID.
Summary of Latency Metrics Available in Deep-Dive Dashboard:
Metric Name | Pipeline Stage / Calculation | Operational Purpose |
| SecOps Ingestion Latency | Collection-to-Ingestion (T_ingestion - T_collection) | Measures internal SecOps gateway-to-database processing speed. Flat low lines indicate optimal SecOps backend performance. |
| Source-to-Ingestion Latency | Event-to-Ingestion (T_ingestion - T_event) | The total end-to-end delay represents the real-world age of the data when it finally becomes available for parsing . Spikes alongside flat SecOps latency point to external network or agent delays. |
| 📺 Stay Tuned! We are working on an upcoming series of community blog posts and video walkthroughs to help you better understand how the Data Health Hub can benefit your security operations! Less searching, more SecOps. |
For full documentation, visit the Google SecOps Documentation Portal and join the conversation in our Google Cloud Security Community Forums.
