Google Secops offers (and no way to opt-out) automatic rule replays to handle late arriving data, which is an useful feature to tackle delayed events that can happen quite often especially in large environments with multiple sources shipping from Cloud, on-prem or hybrid.
A side effect of this feature is the impact on relative functions / functionalities like
timestamp.current_seconds()/timestamp.now()
%<data_table>.write_row()
current_seconds() is a volatile function that evaluates always to the moment of the task compute.
If the rule is replayed (i.e. because late arrival events), then current_seconds() = the moment the rule is replayed / rescheduled and not the original execution time.
This impacts directly on those rules leveraging current_seconds() as a relative anchor to calculate a time difference, e.g. in those use cases to calculate silent sources, or based on $age to calculate MTTD or similar bread and butter SOC metrics, where statements like below can be observed:
outcome:
$max_diff = timestamp.current_seconds() - $max_event_time
condition: $e and $max_diff > 600 // or similaror
$age = max(timestamp.current_seconds() - case.create_time.seconds)/86400or
$age = (timestamp.current_seconds() - max(case.update_time.seconds))/86400or similar variations.
If a late-arriving log triggers a replay 5 hours later, the math will always result in a True condition for that period. This leads to "ghost alerts" that are actually false positives from late arrival re-processing, and could potentially generate false positive storms in delayed high-volume sources.
write_row() writes the query results to data table, used often for state-tracking.
If a replay occurs, it may write stale status back into the table, overwriting a more recent entry written by a real-time execution.
The source of truth (the data table) becomes non-deterministic. We cannot trust if the data in the table represents the latest event or the latest processed event.
In some situations we might require priotize current data over complete data, and getting the latest current_seconds() is already great to have.
In other situations we might require prioritize completeness over recency, requiring the original rule execution time (not the latest current_seconds() from the replay).
Google SecOps is moving heavily toward "Detection-as-Code," and for code to be reliable, it must be idempotent (running it twice should produce the same result). Current Rule Replay behavior breaks idempotency when current_seconds() or write_row() are involved.
I am aware the workaround culture is to handle these gaps externally, via GCP Cloud monitoring, in bigquery or even BYOS (Bring Your Own SIEM) on top of Google Secops API.
But relying on external tools for core SIEM logic is a significant friction point. I am looking for Google SecOps to provide the native building-blocks necessary to make it a fully self-contained detection platform. Relying on externals adds operational overhead and architectural complexity. The product’s value is maximized when detection capabilities are native; it shouldn't function primarily as a data lake that requires customers to engineer their own logic externally to fill basic gaps.
Please community admin can you forward these feature requests to Google PMs:
Allow the user to adapt to each situation, to be able to choose between the original execution time or current_seconds():
- implement a new funtion: timestamp.original_execution_time() excluding rule replays, to allow the logic to remain "anchored" to the window it is originally evaluating. If it is replaying from 2:00 PM to 3:00 PM, the function should return 3:00 PM, even if it’s currently 8:00 PM.
- modify the current_seconds() adding an optional argument: current_seconds(origina_execution_time), to allow use the original rule scheduling/execution time, same effect as the 1st option.
- expose the window.end_time of the execution to the user. Assuming the detection engine window of the original execution is preserved during the replay, then the user should be able to choose window.end_time instead of current_seconds() to escape from the current time relativity.
A safety switch, to provide an immediate "opt-out" for rules that aren't compatible with late-arriving data logic.
- implement a boolean filter for the rules: to optionally exclude the rule replays from the results; prevents replays from corrupting Data Tables.
condition: $e and !is_rule_replayor
options: run_on_replay: false // If false, this rule ignores the Rule Replay trigger
