Skip to main content
Question

Building Autonomous Telemetry Agents in Google SecOps: Deep Dive into Our Chronicle UDM Search Agent, Defender KQL & Entra Loops

  • Nov 15, 2026 2:18 PM UTC
  • 0 replies
  • 9 views

mccrilb
Forum|alt.badge.img+12

Hi everyone,

As security operations scale, one of the biggest bottlenecks in triage and incident response is the mechanical grind of gathering forensic telemetry. When an alert fires, analysts and playbooks are forced to manually pivot into Chronicle SIEM, craft UDM queries, sift through hundreds of events, identify rare outliers, pivot to endpoint EDR (like Defender XDR), and cross-check cloud identity logs in Entra ID.

Over the past few months, we designed and deployed an Autonomous Multi-Agent Investigation Architecture inside Google SecOps SOAR, powered natively by Gemini SecOps.

In this post, I want to share how our Autonomous Chronicle UDM Search Agent works under the hood, how we expanded this pattern to Microsoft Defender XDR and Entra ID, how we leverage the different Gemini SecOps Personas, and how the combination of Authoritative Data Models and DFIR Skills keeps the system deterministic and grounded.

1. The Core Engine: The Autonomous Chronicle UDM Search Agent

Rather than relying on static, hardcoded playbook blocks with rigid query templates, our UDM Search Agent operates as an autonomous forensic investigator:

A. Dynamic Query Synthesis (Zero Python Reasoning Gates)
The agent receives a high-level investigation directive from upstream triage (for example: Audit child processes spawned by powershell.exe and verify whether external outbound network connections occurred around 14:22 UTC). Instead of using brittle regexes or hardcoded string rules in Python, the agent prompts Gemini SecOps referencing our authoritative UDM schema to synthesize syntactically valid, strongly scoped Chronicle UDM queries dynamically.

B. Adaptive Aggregation & The Saturation Cap (Pivot / Table View)
In enterprise environments, a broad query can return thousands of noisy events. Our agent enforces an automatic 25-event saturation cap:

  • Direct Pluck: If the query returns fewer than 25 events, the agent plucks high-signal forensic evidence directly (target processes, file hashes, remote network endpoints, user identity).
  • Adaptive Pivot on Saturation: If total events reach or exceed 25, the agent automatically pivots to an Aggregated Table Search (grouping by target entity over time, ordered by frequency). Gemini SecOps evaluates the resulting table view to isolate 1-off outliers and rare long-tail telemetry (prioritizing external routable traffic over known internal subnets), synthesizes a surgical micro-query targeting the outlier, and extracts the exact evidence needed.

C. The "Pure Evidence Gatekeeper" Invariant (The Lab vs. The Judge)
A key lesson from production: telemetry gatherers must never act as the judge.
Our UDM Search Agent acts strictly as an expert forensic witness. It extracts and summarizes factual telemetry (events found, process lineage, connection endpoints). It is explicitly forbidden from issuing overall case verdicts or threat classifications (such as marking an unexecuted phishing payload as "Benign" simply because network sockets were empty). Threat adjudication happens downstream once all evidence is gathered.

2. Expanding to Endpoint & Identity: Defender XDR & Entra ID Loops

Adversary activity rarely stays in a single log source. Using the same architectural framework, we built companion agents to provide 360-degree visibility across endpoints and identity:

1. Autonomous Defender KQL Hunting Loop (Microsoft Defender XDR)

  • Scope: Advanced hunting across 25 verified, compiler-grounded hunting tables (DeviceProcessEvents, DeviceNetworkEvents, UrlClickEvents, CloudAppEvents, EmailEvents, etc.).
  • Dynamic KQL Generation: Generates multi-table KQL joins while preserving strict endpoint scoping (DeviceName).
  • Lineage & In-Memory Forensics: Reconstructs dual-process trees (InitiatingProcess vs. TargetProcess) and recursively decodes Base64 encoded PowerShell commands in memory.

2. Autonomous Entra Forensic Loop (Microsoft Entra ID)

  • Scope: Cloud authentication and sign-in forensics bridging Entra ID and Active Directory Domain Controllers (AADSignInEventsBeta, IdentityLogonEvents, IdentityDirectoryEvents).
  • Cloud Boundary Awareness: Dynamically routes queries based on network boundaries—omitting private RFC1918 subnets from cloud sign-in logs and querying domain controller logon events for internal authentications.
  • Identity Anomaly Detection: Identifies anomalous sign-ins, conditional access policy evaluations, and impossible travel patterns.

3. How We Leverage the 5 Gemini SecOps Personas

Google SecOps Gemini includes distinct operational personas. Rather than treating Gemini as a generic one-size-fits-all model, our agents explicitly engage the right persona for each phase:

  • Hunter (Threat Hunter & Detection Engineer): Used in UDM and KQL query formulation. Decomposes triggering YARA-L detection logic, identifies key forensic pivot fields, analyzes attack tradecraft, and plans targeted hunt paths.
  • SOAR (Security Automation Engineer): Used in action architecture and orchestration. Generates resilient Python actions, manages API connection pooling, enforces exponential backoff (handling transient HTTP 429/503 errors), and guarantees session cleanup.
  • Analyst (Security Analyst / Triage & IR): Used in telemetry plucking and forensic fact-finding. Evaluates process lineages, timeline sequences, and factual evidence without confirmation bias.
  • CTI (Threat Intelligence Analyst): Used in threat intelligence grounding. Evaluates extracted hashes, domains, and IPs against Mandiant Frontline Threat Intelligence and VirusTotal multi-engine telemetry to attribute threat actors, campaigns, and malware families.
  • Advisor (Unified Security Advisor): Used in containment gating and peer review. Evaluates systemic blast radius (e.g., Domain Controller vs. clinical workstation) and conducts independent peer review to prevent false auto-containment actions.

4. The Foundation: Data Models and Modular Skills

To ensure our agents remain completely deterministic and never hallucinate syntax or fields, we built our architecture on two pillars:

A. Authoritative Data Models (Principles in Prompts, Entities in Dictionaries)
A core rule we enforce: Prompts contain principles; data models contain entities. We never inject long, brittle lists into prompts (e.g., no hardcoded lists of executables or flags). Instead, we supply strongly typed data models:

  • UDM Schema Vocabulary: Strongly typed Chronicle UDM field paths across 15 domains (Execution, Disk/Registry, Identity/Auth, Network, SaaS, and Governance).
  • Ingestion Source Mappings: Verified enterprise log source mappings (CrowdStrike EDR, Windows Event Logs, Sysmon, Abnormal Security, Defender ATP).
  • KQL & Entra Schema Vocabularies: 25 verified hunting tables, field contracts, and relational syntax rules.
  • Enterprise Environment Models: Codified internal CIDRs, corporate domains, and vulnerability scanner subnets so the model knows what is internal vs. external.

B. Modular DFIR Skills
Skills act as domain-specific playbooks and forensic procedures that guide agent behaviors:

  • Encoded PowerShell Decoding: Multi-layer recursive unwrapping of Base64 wrappers, Shannon entropy analysis, and script sandboxing.
  • CMDB & Change Control Grounding: Validating whether an observed administrative command is tied to an active ServiceNow Change Request.
  • Lateral Movement Context: Correlating source-to-target host identity, administrative sessions, and network boundaries.
  • Chronicle Rule Health Auditing: Monitoring rule runtime execution errors, scan gaps, and outcome variable limits.

5. Key Takeaways & Community Discussion

  1. Keep Python Mechanical, Keep AI Cognitive: We use Python strictly for parameter extraction, API transport with exponential backoff, defanging, and data plucking. All semantic reasoning and query formulation belongs to Gemini SecOps.
  2. Never Mock Production Telemetry: Testing against live Chronicle Alpha endpoints and real tenant telemetry exposed API syntax nuances and schema drift that synthetic tests completely hide.
  3. Evidence Over Conclusion: Separating evidence collection (UDM/KQL loops) from final case adjudication eliminated false benign dismissals and gave our SOC complete auditability.

We’d love to hear from the community:

  • How are you approaching autonomous or AI-assisted UDM search in your environments?
  • Are you integrating Chronicle UDM queries with endpoint EDR queries (like Defender or CrowdStrike) inside your SOAR playbooks?
  • What guardrails have you found most effective when allowing LLMs to synthesize hunting queries?

Looking forward to your thoughts and discussion!