day 2 issue: collateral damage from rule replay relativity

Question

Google Secops offers (and no way to opt-out) automatic rule replays to handle late arriving data, which is an useful feature to tackle delayed events that can happen quite often especially in large environments with multiple sources shipping from Cloud, on-prem or hybrid.

A side effect of this feature is the impact on relative functions / functionalities like

timestamp.current_seconds() / timestamp.now()

%<data_table>.write_row()

current_seconds() is a volatile function that evaluates always to the moment of the task compute.
If the rule is replayed (i.e. because late arrival events), then current_seconds() = the moment the rule is replayed / rescheduled and not the original execution time.
This impacts directly on those rules leveraging current_seconds() as a relative anchor to calculate a time difference, e.g. in those use cases to calculate silent sources, or based on $age to calculate MTTD or similar bread and butter SOC metrics, where statements like below can be observed:

outcome:
$max_diff = timestamp.current_seconds() - $max_event_time

condition: $e and $max_diff > 600 // or similar

or

$age = max(timestamp.current_seconds() - case.create_time.seconds)/86400

or

$age = (timestamp.current_seconds() - max(case.update_time.seconds))/86400

or similar variations.

If a late-arriving log triggers a replay 5 hours later, the math will always result in a True condition for that period. This leads to "ghost alerts" that are actually false positives from late arrival re-processing, and could potentially generate false positive storms in delayed high-volume sources.

write_row() writes the query results to data table, used often for state-tracking.
If a replay occurs, it may write stale status back into the table, overwriting a more recent entry written by a real-time execution.
The source of truth (the data table) becomes non-deterministic. We cannot trust if the data in the table represents the latest event or the latest processed event.

In some situations we might require priotize current data over complete data, and getting the latest current_seconds() is already great to have.
In other situations we might require prioritize completeness over recency, requiring the original rule execution time (not the latest current_seconds() from the replay).

Google SecOps is moving heavily toward "Detection-as-Code," and for code to be reliable, it must be idempotent (running it twice should produce the same result). Current Rule Replay behavior breaks idempotency when current_seconds() or write_row() are involved.

I am aware the workaround culture is to handle these gaps externally, via GCP Cloud monitoring, in bigquery or even BYOS (Bring Your Own SIEM) on top of Google Secops API.
But relying on external tools for core SIEM logic is a significant friction point. I am looking for Google SecOps to provide the native building-blocks necessary to make it a fully self-contained detection platform. Relying on externals adds operational overhead and architectural complexity. The product’s value is maximized when detection capabilities are native; it shouldn't function primarily as a data lake that requires customers to engineer their own logic externally to fill basic gaps.

Please community admin can you forward these feature requests to Google PMs:

Allow the user to adapt to each situation, to be able to choose between the original execution time or current_seconds():
- implement a new funtion: timestamp.original_execution_time() excluding rule replays, to allow the logic to remain "anchored" to the window it is originally evaluating. If it is replaying from 2:00 PM to 3:00 PM, the function should return 3:00 PM, even if it’s currently 8:00 PM.
- modify the current_seconds() adding an optional argument: current_seconds(origina_execution_time), to allow use the original rule scheduling/execution time, same effect as the 1st option.
- expose the window.end_time of the execution to the user. Assuming the detection engine window of the original execution is preserved during the replay, then the user should be able to choose window.end_time instead of current_seconds() to escape from the current time relativity.

A safety switch, to provide an immediate "opt-out" for rules that aren't compatible with late-arriving data logic.
- implement a boolean filter for the rules: to optionally exclude the rule replays from the results; prevents replays from corrupting Data Tables.

condition: $e and !is_rule_replay

or

options: run_on_replay: false  // If false, this rule ignores the Rule Replay trigger

kylechamplin · Answer

​@hliu- I really appreciate the thorough writeup - I work on the PM team for SecOps and wanted to first ack the requests and also offer a few things that might help with this particular scenario, that you can use today.1. For scenarios where you want to know, after the detection object is created, if it came from a replay or the original rule execution, we started annotating detection objectswith that info. The enum field is called “detection.detection_timing_details” and you can pick from three values to filter on:DETECTION_TIMING_DETAILS_UNSPECIFIED - this is the default and would have an “accurate”current_seconds() valueDETECTION_TIMING_DETAILS_REPROCESSING - this is when a detection is generated by a reprocessing runDETECTION_TIMING_DETAILS_RETROHUNT - this is when you run a rule as a retrohunt (and would of course be the most sensitive to techniques like timestamp.current_seconds() - $max_event_time)This only works in the context of a “composite” detention where your data source is actually the detections table, but very useful forthose use-cases. That said, it’s an effective way to also better filter on your MTTD calculations as you can have an MTTD that excludes “late arriving data” cases.2. We are in Private Preview (PrPr) for “customizable rule schedules” - this will allow you to turn off the second rule replay. We’re still leaving the 4 hour replay on for now, but are looking to introduce a user controllable setting for that as well - for the very good reasons you highlight! For more info on how this feature works, we have docs up:https://docs.cloud.google.com/chronicle/docs/detection/set-customized-scheduleHappy to add you to the PrPr.3. Lastly, I do want to push a little bit on the statement:“Google SecOps is moving heavily toward "Detection-as-Code," and for code to be reliable, it must be idempotent (running it twice should produce the same result). Current Rule Replay behavior breaks idempotency when current_seconds() or write_row() are involved.”In this case the function call itself for currenttime() can’t be made “idempotent” on its own, by design it returns a different value for each execution. While I definitely get the value of more tools for our users to handle late arriving data (and I think you recognize this when you ask for an`"opt-out" for rules that aren't compatible with late-arriving data logic`)our philosophy around replays was that it solved a toil problem (automating replays means our customers don’t have to manually rerun rules when a data sources is down a few hours), and it also provides signal to detection engineering teams about late arriving data that may be endemic to a class of detections - we just did a bad job labeling it such that it was easy to distinguish, but we’re doing better.Again, really appreciate the thorough writeup and deep thinking on this, and would love to bring you in on the PrPr for schedules.

Sign up

Login with SSO

Login to the community

Login with SSO