Tuesday's Tip of the Week - Who Watches the Watchmen? Detecting When Your Detections Break

October 20, 2026

Audience: Detection engineers, SecOps platform admins, SOC leads

Summary: Detection pipelines fail silently. A rule gets disabled, a log source stops sending, a parser stops filling in fields, or a rule hits a runtime error. None of these raises an alert by default. This briefing covers how to detect each failure, the SecOps feature that catches it, and the rules, alert policies, and dashboard queries to set up.

1. Failure modes and what catches them

2. Approach 1: Monitor detection changes using SecOps audit logs

SecOps writes Cloud Audit Logs to the Google Cloud project bound to your instance. The service name is chronicle.googleapis.com and the resource type is audited_resource. Write operations, such as creating rules or changing deployments, are Admin Activity logs. These are always on and can’t be disabled.

Which audited method does what:

How the GCP_CLOUDAUDIT parser maps these events (from Google’s field mapping reference and changelog):

• protoPayload.methodName → metadata.product_event_type
• protoPayload.authenticationInfo.principalEmail → principal.user.email_addresses
• protoPayload.status.message → metadata.description
• Log severity ERROR → security_result.severity = "ERROR"
• For UpdateRuleDeployment:
  • Request: target.resource.attribute.labels keys req_rule_deployment_name and req_rule_deployment_alerting
  • Response: keys res_enabled, res_archived, and res_execution_state

Before you build anything:

1. Ingest the audit logs. Bring the bound project’s Cloud Audit Logs into SecOps as GCP_CLOUDAUDIT. Admin Activity logs are all you need here. Google warns that Data Access logs can be very high-volume, so filter them out unless you need them.
2. Run a test change. Disable and then re-enable a test rule. Then confirm in UDM search that the event shows res_enabled and the other labels above.

Rule 1: detection rule disabled

rule secops_detection_rule_disabled {  meta:    author = "SecOps Team"    description = "A custom detection rule was disabled"    severity = "Medium"  events:    $e.metadata.log_type = "GCP_CLOUDAUDIT"    $e.metadata.product_event_type = /RuleService\.UpdateRuleDeployment$/    $e.target.resource.attribute.labels["res_enabled"] != "true"    $e.principal.user.email_addresses = $actor  match:    $actor over 1h  outcome:    $change_count = count_distinct($e.metadata.id)    $rule_deployments = array_distinct($e.target.resource.attribute.labels["req_rule_deployment_name"])  condition:    $e}

res_enabled is the rule’s state after the change. The rule checks != "true" rather than = "false" because audit logs often leave out fields that are false. If the response doesn’t contain enabled, the rule is disabled. $rule_deployments shows the rule’s resource ID (…/rules/ru_…), not its display name.

Rule 2: any change to how detections are deployed

rule secops_detection_deployment_changed {  meta:    author = "SecOps Team"    description = "A rule or curated rule set deployment was changed (enabled, disabled, alerting, archive)"    severity = "Low"  events:    $e.metadata.log_type = "GCP_CLOUDAUDIT"    $e.metadata.product_event_type = /(RuleService\.UpdateRuleDeployment|RuleService\.ModifyRules|CuratedRuleService\.BatchUpdateCuratedRuleSetDeployments)$/    $e.principal.user.email_addresses = $actor  match:    $actor over 1h  outcome:    $change_count = count_distinct($e.metadata.id)    $methods = array_distinct($e.metadata.product_event_type)    $rule_deployments = array_distinct($e.target.resource.attribute.labels["req_rule_deployment_name"])    $alerting_requested = array_distinct($e.target.resource.attribute.labels["req_rule_deployment_alerting"])    $enabled_after = array_distinct($e.target.resource.attribute.labels["res_enabled"])  condition:    $e}

This rule makes every deployment change visible to the team, including alerting turned off and curated rule sets changed. Those can’t be isolated reliably from the documented fields alone. Deployment changes are rare in most environments, so the volume stays low. Exclude your CI/CD service account and approved admins with a data table, for example not $actor in %approved_rule_admins.email.

Dashboard queries (Events data source)

Detection changes per day, by method:

metadata.log_type = "GCP_CLOUDAUDIT"metadata.product_event_type = /(RuleService\.(CreateRule|UpdateRule|DeleteRule|UpdateRuleDeployment|ModifyRules)|CuratedRuleService\.BatchUpdateCuratedRuleSetDeployments)$/$method = metadata.product_event_type$date = timestamp.get_date(metadata.event_timestamp.seconds)match:  $date, $methodoutcome:  $change_count = count(metadata.id)order:  $date desc

Who changed which rule deployment:

metadata.log_type = "GCP_CLOUDAUDIT"metadata.product_event_type = /RuleService\.UpdateRuleDeployment$/principal.user.email_addresses != ""$actor = principal.user.email_addresses$rule_deployment = target.resource.attribute.labels["req_rule_deployment_name"]$enabled_after = target.resource.attribute.labels["res_enabled"]$date = timestamp.get_date(metadata.event_timestamp.seconds)match:  $date, $actor, $rule_deployment, $enabled_afterorder:  $date desc

$enabled_after will be empty when the rule was disabled, for the same reason as Rule 1.

Failed rule changes, for example syntax errors from CI/CD:

metadata.log_type = "GCP_CLOUDAUDIT"metadata.product_event_type = /RuleService\./security_result.severity = "ERROR"$method = metadata.product_event_type$error = metadata.description$date = timestamp.get_date(metadata.event_timestamp.seconds)match:  $date, $method, $erroroutcome:  $failure_count = count(metadata.id)order:  $date desc

Google now puts the rule’s display name at the start of these error messages, for example rule "Suspicious PowerShell Execution": compiling rule: syntax error….

Add an alert that doesn’t depend on SecOps: If SecOps stops ingesting audit logs, both rules and all three charts go quiet with it. Back them up with a log-based alert in Cloud Logging and Cloud Monitoring, which works even when SecOps ingestion doesn’t:

protoPayload.serviceName="chronicle.googleapis.com"protoPayload.methodName=~".*RuleService.UpdateRuleDeployment.*"

Also include GCP_CLOUDAUDIT in the silent-source alerts in Approach 2.

3. Approach 2: Ingestion and parsing health

Alerts in Cloud Monitoring

SecOps publishes ingestion metrics to Cloud Monitoring under the Chronicle Collector metric category. Despite the name, these metrics cover every ingestion path: collection agents, feeds, and direct API ingestion.

Forwarder deprecation: The legacy SecOps forwarder reaches end of life in January 2027, and Bindplane and OpenTelemetry-based collection agents replace it. Build new silent-source monitoring for agents, and retire forwarder-specific alert policies as you migrate.

Silent log type or collector:

1. Create a policy on Chronicle Collector → Ingestion → Total ingested log count.
2. Under Transform data, set the rolling window function to mean and choose how to group. Group by log_type to get one alert per log type, or by collector_id for one per collector. With no grouping, Cloud Monitoring raises a separate alert for every individual source.
3. Choose Metric absence as the condition. The alert fires after roughly the rolling window plus the trigger absence time. Google’s sample sets these to a combined 60 minutes.
4. Scope it to critical sources with filters on log_type, collector_id, or ingestion_source.
5. For high-availability pairs, filter to both members and set Alert trigger to All time series violate. That way it only fires when every member stops sending.

Silent collection agent: Google’s sample policy uses Chronicle Collector → Agent → Exporter Accepted Spans Count, grouped by collector_id, with a metric-absence condition.

Host-level silence: If you manage agents through Bindplane, Bindplane has its own silent host monitoring. Use it to catch individual hosts that stop sending, which per-log-type metrics can miss when other hosts keep the volume up.

Parser failures: Create a threshold policy on Chronicle Collector → Normalizer (Total record count or Total event count). Filter on the state label (failed_parsing, failed_validation, or failed_indexing) and on log_type.

Health Hub (built in, nothing to configure): Health Hub uses a 15-day statistical baseline to flag data sources and parsers as Irregular or Failed. For example, it flags a parser when its error share rises by 5 percentage points or more over the previous day. Its Set Up Alerts link opens Cloud Monitoring for that source.

Dashboard queries (Ingestion metrics data source)

Last log received per log type, stalest first:

ingestion.component = "Ingestion API"$Log_Type = ingestion.log_typematch:  $Log_Typeoutcome:  $Recent_Ingestion_Time = timestamp.get_timestamp(max(ingestion.end_time), "%F %T")  $Total_Log_Volume = math.round(sum(ingestion.log_volume) / (1000 * 1000 * 1000), 2)order:  $Recent_Ingestion_Time asc

Last log received per collector and log type (catches a dead agent even when other agents still send the same log type):

ingestion.component = "Ingestion API"$Collector = ingestion.collector_id$Log_Type = ingestion.log_typematch:  $Collector, $Log_Typeoutcome:  $Last_Log_Received = timestamp.get_timestamp(max(ingestion.end_time), "%F %T")order:  $Last_Log_Received asc

Parsing errors per log type:

ingestion.component = "Normalizer"ingestion.state = "failed_parsing"$Log_Type = ingestion.log_typematch:  $Log_Typeoutcome:  $Parsing_Errors = sum(ingestion.log_count)order:  $Parsing_Errors desc

Notes on the ingestion queries:

• Missing log types: A log type or collector that sent nothing during the whole time range won’t appear, so compare against your list of expected sources.
• Approximate counts: Ingestion metrics are sampled at intervals, so treat the numbers as indicators, not exact counts.
• Access: If your data access scope uses a custom label, ingestion metrics show no data at all.

Dashboard query: when a parser stops filling in a field (Events data source)

This doesn’t show up as a parsing failure, because the event parses fine and the field is simply empty. To catch it, track how often the fields your rules depend on are filled in. For example, for process command lines from EDR:

metadata.log_type = "YOUR_EDR_LOG_TYPE"metadata.event_type = "PROCESS_LAUNCH"$date = timestamp.get_date(metadata.event_timestamp.seconds)match:  $dateoutcome:  $total = count(metadata.id)  $missing_cmdline = sum(if(target.process.command_line = "", 1, 0))order:  $date asc

Replace YOUR_EDR_LOG_TYPE with your log type. If $missing_cmdline suddenly rises as a share of $total, every rule that relies on command lines is blind, even though it still runs without error.

4. Approach 3: Rule runtime errors

SecOps reports two kinds of rule errors:

• Compilation errors are found when you save or test a rule, and you can’t save a rule that has one.
• Runtime errors only appear when the rule actually runs: during Run test, live execution, or a retrohunt. For a live or retrohunt run, the error appears as a link on that rule’s Detections page. There’s no central list of every rule’s errors, and no dashboard data source for them.

Common runtime errors (from Google’s troubleshooting guide):

Covering the gap without a central list:

1. Test before deploying: Run test every rule before deploying it, and again after any parser change to a log type it depends on.
2. Chart failed rule changes: Use the failed rule changes query in Approach 1.
3. Spot rules that stop firing: A rule hitting runtime errors usually stops producing detections, so Section 5’s monitoring catches it.

5. The silent failure problem: monitor rule detection counts

A rule that stops firing means either the threat went away or the detection broke. You can only tell the difference by watching each rule’s detection count over time.

Daily detections per rule (Detections data source):

$date = timestamp.get_date(detection.created_time.seconds)$rule_name = detection.detection.rule_namematch:  $date, $rule_nameoutcome:  $detection_count = count_distinct(detection.id)order:  $date asc

A day with no detections returns no row for that rule, not a zero. A rule that normally fires 10 times a day and then disappears from the results needs investigating.

Last detection for each enabled rule (Rules data source):

$status = rules.live_status$status = "ENABLED"$rule_name = rules.display_name$last_detection = timestamp.get_timestamp(rules.latest_detection_time.seconds, "%F %T")match:  $rule_name, $last_detectionorder:  $last_detection asc

Rules that have been quiet longest sort to the top. A rule that has never fired shows 1970-01-01 00:00:00. Set this chart’s time range to the maximum, because the Rules data source applies the time range to each rule’s creation date.

Weekly triage for a rule that went quiet:

1. Was it changed or disabled? Check the Approach 1 rules and the “who changed which rule deployment” chart.
2. Did its log source stop? Check Health Hub and the “last log received” charts.
3. Did a field stop being filled in? Run the field coverage query for the fields the rule uses.
4. Is it hitting runtime errors? Look for the error link on the rule’s Detections page.
5. Did the threat actually go away? Only conclude this after ruling out 1–4.

6. Action items

• Ingest SecOps Admin Activity audit logs (GCP_CLOUDAUDIT). Make a test disable and enable, and confirm the labels in UDM search.
• Deploy Rule 1 and Rule 2, plus an out-of-band log-based alert in Cloud Logging
• Create Cloud Monitoring metric-absence policies for critical log types and collection agents, including GCP_CLOUDAUDIT. Retire forwarder-specific policies as you migrate to Bindplane.
• Create Normalizer failed_parsing threshold alerts for critical log types
• Build the dashboard: detection changes, failed rule changes, last log received (by log type and by collector), parsing errors, field coverage, detections per rule, and last detection per rule
• Review the dashboard weekly, and make Run test a required step for rule changes and after parser changes