Author: Darren Davis
Back in the early days of Google SecOps, when defining how we would evaluate various normalized log fields against a unified dataset, we determined that a highly structured, specialized query language derived from the YARA language would be the best course of action. This language, YARA-L 2.0, has transformed into the modern-day detection and analytical engine.
Before diving into the syntax, it is important to understand the execution architecture. Internally, YARA-L is not executed directly as written; rather, it gets compiled to then run against databased UDM fields. Through this compilation, the engine evaluates massive volumes of telemetry, meaning that properly architected YARA-L code is the definitive step in turning a flood of disparate logs into actionable intelligence.
What are the components of YARA-L 2.0?
The YARA-L rule structure is broken down into distinct, named sections: meta, events, match, outcome, and condition. We will introduce the properties from what is considered foundational to advanced.
The Anti-Pattern: A Poorly Optimized Rule
Before demonstrating proper operational mechanisms, we must examine what happens when YARA-L is written without optimization.
The Syntax:
rule poorly_optimized_regex_nightmare {
meta:
author = "Google Cloud Security"
description = "A poorly constructed rule that will cause performance degradation"
events:
re.regex($event.target.process.command_line, `.*whoami.*`) nocase or
re.regex($event.target.file.full_path, `.*cmd\.exe.*`) nocase
condition:
$event
}
- Operational Mechanism: This rule attempts to find execution of whoami or cmd.exe by scanning the command line or file path of every incoming event.
- Why this is detrimental: Because this rule lacks an enumerated field (such as metadata.event_type) at the very top of the events section, the engine cannot "fail-fast." It is forced to execute computationally expensive regular expressions and the nocase modifier against every single log in the platform—including DNS queries, firewall blocks, and cloud audit logs. This leads to severe system performance degradation.
Phase 1: Optimized Single-Event Rules
The single-event rule is designed specifically for logs where a solitary action dictates a threat. The most fundamental best practice involves the "instantiation" of enumerated fields to quickly discard irrelevant data.
Data Tables vs. Static Detection Rules
A critical evolution in YARA-L is the introduction of Data Tables, which are multicolumn data constructs that act as lookup tables. Instead of creating dozens of separate, static detection rules to account for various malicious domains, restricted IPs, or sensitive roles, you should leverage Data Tables. This dramatically reduces your rule count and simplifies SOC maintenance.
1. Process Execution: PROCESS_LAUNCH
This filter is designed specifically for tracking endpoint behavior.
The Syntax:
rule optimized_process_launch_whoami {
meta:
author = "Google Cloud Security"
description = "Detects precise execution of whoami command"
severity = "Low"
events:
// Fail-fast using the enumerated event type
$process.metadata.event_type = "PROCESS_LAUNCH"
// Exact string match is computationally cheap and highly accurate
$process.target.process.command_line = "whoami"
condition:
$process
}
- Why this is necessary: By evaluating $process.metadata.event_type = "PROCESS_LAUNCH" first, the rule instantaneously discards all non-process logs, making the subsequent string match lightning fast.
2. Network Activity: NETWORK_DNS with Data Tables
This filter relies on the aforementioned Data Table schema to precisely locate malicious indicators without requiring dozens of OR statements.
The Syntax:
rule optimized_network_dns_data_table {
meta:
author = "Google Cloud Security"
description = "Detects DNS lookups to bad domains using a Data Table"
severity = "High"
events:
$dns.metadata.event_type = "NETWORK_DNS"
$dns.network.dns.questions.name = $queried_name
// Column-based comparison against a Data Table
$queried_name in %malicious_domains_table.domain_name
outcome:
$risk_score = 75
$domains_queried = $queried_name
condition:
$dns
}
- Operational Mechanism: The in %table_name.column_name syntax performs a column-based comparison. It isolates the domain_name column within the malicious_domains_table Data Table and checks if the event's queried name exists within it.
3. File Activity: SCAN_FILE
This rule utilizes the outcome section to conditionally suppress alerts based on the security_result array.
The Syntax:
rule soc_optimized_scan_file_dynamic_risk {
meta:
author = "Google Cloud Security"
description = "Detects malicious files but suppresses if successfully blocked"
severity = "Critical"
events:
$event.metadata.event_type = "SCAN_FILE"
$event.security_result.category = "SOFTWARE_MALICIOUS"
$event.security_result.action = $secAction
$event.target.file.full_path = $targetfilepath
$event.security_result.threat_name = $threatname
outcome:
$risk_score = max(100 - if($secAction = "QUARANTINE", 50, 0) - if($secAction = "BLOCK", 70, 0))
$action_taken = array_distinct($secAction)
$threat = array_distinct($threatname)
$file_path = $targetfilepath
condition:
$event and $risk_score > 49
}
- Critical Detail on Outcomes: The outcome section permits the generation of up to 20 contextual variables. Here, the $risk_score is mathematically manipulated. The condition ensures that if the action was BLOCK, the score falls below 50, and the event is summarily dropped, explicitly preventing SOC alert fatigue.
4. User Activity: USER_RESOURCE_UPDATE_PERMISSIONS
The Syntax:
rule soc_optimized_gcp_sensitive_role_add {
meta:
author = "Google Cloud Security"
description = "Detects addition of sensitive GCP roles using Data Tables"
severity = "High"
events:
$iam.metadata.event_type = "USER_RESOURCE_UPDATE_PERMISSIONS"
$iam.metadata.product_event_type = "SetIamPolicy"
$iam.principal.user.attribute.roles.description = "ADD"
$iam.principal.user.attribute.roles.name = $role
// Leveraging Data Tables instead of static rule logic
$role in %gcp_sensitive_roles_table.role_name
$iam.principal.user.userid = $actor
$iam.target.user.userid = $recipient
outcome:
$risk_score = 85
$granted_by_user = $actor
$granted_to_user = $recipient
$sensitive_role_assigned = $role
condition:
$iam
}
Phase 2: Multi-Event Rules (Event Correlation)
Once we extract and isolate the data, we can begin to join disparate events. The non-negotiable step for multi-event rules is the inclusion of the match section, which groups events over a strict time boundary ranging from 1 minute to 48 hours.
Critical Requirement: Whenever you use the outcome section in a multi-event rule, every outcome variable must be encapsulated within an aggregation function (such as max(), count(), or array_distinct()).
1. Discovery to External Connection
The Syntax:
rule soc_multievent_discovery_to_external_connection {
meta:
author = "Google Cloud Security"
description = "Detects discovery commands followed by an external outbound network connection"
events:
$proc.metadata.event_type = "PROCESS_LAUNCH"
$proc.target.process.command_line = /whoami|netstat \-an/ nocase
$proc.principal.hostname = $hostname
$proc.principal.user.userid = $user
$net.metadata.event_type = "NETWORK_CONNECTION"
$net.principal.hostname = $hostname
$net.principal.user.userid = $user
not net.ip_in_range_cidr($net.target.ip, "10.0.0.0/8")
// Sliding Window Enforcement
$proc.metadata.event_timestamp.seconds < $net.metadata.event_timestamp.seconds
match:
$hostname, $user over 30m
outcome:
$risk_score = max(75)
$external_ips_accessed = array_distinct($net.target.ip)
$commands_executed = array_distinct($proc.target.process.command_line)
condition:
$proc and $net
}
- Operational Mechanism: The logic requires that a process event ($proc) and a network event ($net) share identical placeholder variables ($hostname and $user). The chronological sequence is rigidly enforced by comparing their metadata timestamps ($proc.metadata.event_timestamp.seconds < `$net.metadata.event_timestamp.seconds). Keep in mind that for each normalized timestamp (event, ingest, and create) you have seconds and nanos that you must use for this comparison.
2. DNS Lookup to Malicious Domain followed by HTTP Traffic
The Syntax:
rule soc_multievent_dns_and_http_to_malicious_domain {
meta:
author = "Google Cloud Security"
description = "DNS resolution of a malicious domain followed by actual HTTP traffic"
events:
$dns.metadata.event_type = "NETWORK_DNS"
$dns.network.dns.questions.name = $bad_domain
// Using Data Tables for multi-dimensional matching
$bad_domain in %malicious_domains_table.domain_name
$dns.principal.hostname = $hostname
$http.metadata.event_type = "NETWORK_HTTP"
$http.target.hostname = $bad_domain
$http.principal.hostname = $hostname
match:
$hostname, $bad_domain over 10m
outcome:
$risk_score = max(90)
$malicious_urls_accessed = array_distinct($http.target.url)
condition:
$dns and $http
}
- Why this is necessary: DNS lookups alone generate noise due to browser pre-fetching. By requiring the presence of an identical domain in an HTTP log shortly after, we confirm successful malicious communication.
3. Malicious File Allowed -> File Executes
The Syntax:
rule soc_multievent_allowed_malware_execution {
meta:
author = "Google Cloud Security"
description = "A file flagged as malicious was allowed by AV and subsequently executed"
events:
$scan.metadata.event_type = "SCAN_FILE"
$scan.security_result.category = "SOFTWARE_MALICIOUS"
$scan.security_result.action = "ALLOW"
$scan.target.file.full_path = $filepath
$scan.principal.hostname = $hostname
$exec.metadata.event_type = "PROCESS_LAUNCH"
$exec.target.process.file.full_path = $filepath
$exec.principal.hostname = $hostname
match:
$hostname, $filepath over 1h
outcome:
$risk_score = max(100)
$threat_identified = array_distinct($scan.security_result.threat_name)
condition:
$scan and $exec
}
4. Temporary Account Creation & Usage
The Syntax:
rule soc_multievent_temporary_account_abuse {
meta:
author = "Google Cloud Security"
description = "Account created, used to login, and deleted within 4 hours"
events:
$create.metadata.event_type = "USER_CREATION"
$create.target.user.userid = $target_user
$create.metadata.event_timestamp.seconds < $login.metadata.event_timestamp.seconds
$login.metadata.event_type = "USER_LOGIN"
$login.target.user.userid = $target_user
$login.metadata.event_timestamp.seconds < $delete.metadata.event_timestamp.seconds
$delete.metadata.event_type = "USER_DELETION"
$delete.target.user.userid = $target_user
match:
$target_user over 4h
outcome:
$risk_score = max(85)
$account_created_by = array_distinct($create.principal.user.userid)
condition:
$create and $login and $delete
}
Phase 3: Composite Detections (Risk-Based Alerting)
The final architectural pattern is the implementation of Composite Detections. Instead of executing highly complex arrays of logic, you parse behavior into foundational Input Rules (Producers) that feed a Composite Rule (Consumer). This drastically reduces alert fatigue by summing mathematical risk prior to alerting.
1. Producer Rules
These rules evaluate raw data and append risk scores, but do not necessarily trigger direct SOC alerts.
Producer 1: Discovery Command
rule producer_process_discovery_whoami {
meta:
author = "Google Cloud Security"
rule_labels = { "risk_aggregation" : "true" }
events:
$process.metadata.event_type = "PROCESS_LAUNCH"
$process.target.process.command_line = "whoami"
$process.principal.hostname = $host
outcome:
$risk_score = 35
$hostname = $host
condition:
$process
}
Producer 2: Malicious DNS (Leveraging Data Tables)
rule producer_network_dns_malicious {
meta:
author = "Google Cloud Security"
rule_labels = { "risk_aggregation" : "true" }
events:
$dns.metadata.event_type = "NETWORK_DNS"
$dns.network.dns.questions.name = $queried_name
// Data Table replacement for reference lists
$queried_name in %malicious_domains_table.domain_name
$dns.principal.hostname = $host
outcome:
$risk_score = 75
$hostname = $host
condition:
$dns
}
2. The Consumer Rule
This rule monitors the RULE_DETECTION schema generated by the producers.
The Syntax:
rule consumer_composite_host_risk_exceeded {
meta:
author = "Google Cloud Security"
description = "Alerts when a host breaches a risk threshold of 90"
severity = "Critical"
events:
$d.metadata.event_type = "RULE_DETECTION"
// Validates detection depth to prevent feedback loops
$d.detection.detection.detection_depth = 0
$d.detection.rule_labels.key = "risk_aggregation"
$d.detection.rule_labels.value = "true"
$d.detection.outcomes["hostname"] = $hostname
$d.detection.risk_score = $risk
match:
$hostname over 2h
outcome:
$total_risk_score = sum($risk)
$contributing_rules = array_distinct($d.detection.rule_name)
condition:
$d and $total_risk_score >= 90
}
Critical Requirement: The variable utilized to join events in the match section (e.g., $hostname) must be explicitly defined in the outcome section of all contributing Producer rules, and the values must be identical.
Phase 4: Optimizing Entity Context (Derived vs. The Entity Graph)
As your detection engineering matures, you will inevitably need to build rules that rely on contextual information rather than just raw telemetry. For example, you may want a rule that only alerts if the targeted user is in the Finance department, or if a launched file hash has a low prevalence in your environment.
When working with context in Google SecOps, you have two primary methods: utilizing Derived Entity Context (Native Enrichment) or Directly Leveraging the Entity Context Graph (ECG). Understanding the difference—and why one is computationally superior—is critical for SOC optimization.
1. Derived Entity Context (Native Enrichment)
Entity Data is associated with the event at the exact time the event occurred. Google SecOps is designed so that a significant amount of this contextualization is natively absorbed into the event stream during ingestion.
When you leverage derived entity context, you are querying the enriched context fields directly on the event itself without needing to look up external data.
The Syntax (Optimized Derived Context):
rule soc_optimized_derived_entity_context {
meta:
author = "Google Cloud Security"
description = "Detects suspicious activity specifically targeting Finance users"
events:
$login.metadata.event_type = "USER_LOGIN"
// Leveraging the natively enriched, derived entity data directly on the event
$login.target.user.department = "Finance"
condition:
$login
}
- Why this is optimal: This remains a Single-Event Rule. Because the LDAP data (Finance department) was absorbed into the event stream natively, the YARA-L engine evaluates this in milliseconds. It does not have to perform any cross-referencing or joins.
2. Direct Leverage of the Entity Context Graph (ECG)
The Entity Context Graph (ECG) is a massive, secondary dataset that contains contextual data for assets, IPs, domains, users, file hashes, prevalence metrics, and global threat intelligence.
If the data you need (such as rolling prevalence or external Threat Intel) is not natively stamped on the event, you must directly leverage the ECG. This requires writing a Multi-Event Rule where you join your $event to the $graph.
The Syntax (Direct ECG Leverage):
rule soc_multievent_direct_ecg_leverage {
meta:
author = "Google Cloud Security"
description = "Detects process launches of low prevalence executables"
events:
$e.metadata.event_type = "PROCESS_LAUNCH"
$e.target.process.file.sha256 = $hash
$e.principal.hostname = $hostname
// 1. You MUST filter the millions of rows in the ECG first
$g.graph.metadata.entity_type = "FILE"
$g.graph.metadata.source_type = "DERIVED_CONTEXT"
// 2. Joining the event to the ECG
$g.graph.entity.file.sha256 = $hash
// 3. Evaluating the graph context
$g.graph.entity.file.prevalence.day_max < 5
match:
$hostname over 1d
condition:
$e and $g
}
The Verdict: Why Derived Entity Context is Better
Whenever possible, Derived Entity Context should always be chosen over Direct Leverage of the ECG.
Directly querying the ECG requires the rules engine to perform a complex join against a database containing millions of rows of contextual data. This consumes vastly more memory and processing power. While the ECG is incredibly powerful for advanced use cases like tracking prevalence metrics, using it for basic context checks creates unnecessary multi-event correlation overhead.
The Optimization Rule for the ECG: If you must directly leverage the Entity Context Graph because the data is not natively enriched on the event, you must filter the graph data first. By hardcoding the $g.graph.metadata.entity_type (e.g., "FILE" or "USER") and $g.graph.metadata.source_type into your events section, you instantly reduce the millions of rows in the entity graph down to a much smaller subset before the engine attempts to execute the join, ensuring your rule executes efficiently.
Conclusion
Mastering YARA-L 2.0 is the definitive step in transforming an overwhelming flood of disparate log data into structured, actionable intelligence. By systematically applying the principles of Fail-Fast Filtering, Event Correlation, and Composite Risk Scoring, you gain total control over your detection pipeline.
Recommended Next Steps
To ensure a robust parsing and detection strategy within Google SecOps, adhere to the following practical path forward:
- Prioritize Structure: Always attempt to use strictly enumerated fields (like metadata.event_type) at the very top of your events section. Reserve the use of Regular Expressions for unstructured text where no other option exists.
- Leverage Data Tables: Rigorously utilize Data Tables instead of creating redundant detection rules for specific IPs, Users, or Domains. They offer multi-dimensional matching and profoundly simplify management.
- Use Derived Context Where Possible: Check if your desired context fields are already natively mapped to your event streams before invoking the heavy Entity Context Graph.
- Implement Safety Checks: Ensure the use of outcome suppression, verifying fields exist before evaluating them, and dropping events that fall below an acceptable $risk_score threshold.
Commit to Testing: Remember that the rule creation process is only complete when you perform validation using the "Test Rule" functionality against historical data prior to enabling live alerting.
