+2I updated a rule, and a few minutes later I received a new alert triggered by an event from the past. The event has an ingestion timestamp from two days ago.
Is this expected behavior? Does updating a rule automatically trigger a retroactive run/reprocessing of previously ingested events, or could this be a bug?
Best answer by thineth_dasun
Hi @gabriel.marchi,
Yes, this is expected behavior in Google SecOps Detection Engine under specific conditions, and it is generally not a bug.
Here is why this happens when you update a detection rule:
Google SecOps evaluates events based on their Event Timestamp (metadata.event_timestamp), whereas execution is bound to when the logs arrive in the system. If an event occurred two days ago but was only ingested recently (due to forwarder lag, network outages, or batch uploads), updating or saving a rule forces the Detection Engine to process all newly available/unprocessed events in that pipeline window.
When you update and save a YARA-L rule:
The engine re-initializes execution state for the rule.
If the event was ingested within the system's retroactive lookback window (or while the rule was previously disabled/modified), the newly saved rule logic will evaluate those late-arrived logs.
Check the event's metadata in your UDM Search:
metadata.event_timestamp: When the activity actually occurred (e.g., 2 days ago).
metadata.ingested_timestamp: When Chronicle actually received the log.
If ingested_timestamp is recent (or right around when you saved the rule), the alert was rightfully triggered by the late ingestion pipeline rather than a full system backtest replay.
Hope this clarifies how the rule engine handles delayed logs!
+5Hi,
Without further context about your rule, I cannot tell whether this is caused by the Detection Engine of Google SecOps, your rule logic, your rule type, or ingestion latency.
There are quite a few mechanisms that aren’t obvious for the general SecOps user unless you have scrutinized the docs (which is not an easy task given they have been moving all the information recently). You may want to take a look at Automatic true-up stages in the Schedule Rule Runs Google SecOps documentation.
Basically, multi-event rules by default run a few more times in the background to account for late arriving data due to enrichment lag or ingestion latency.
Also, Google SecOps recently introduced Customized Schedules for rules. It is in Pre-GA though, but it may help you address your own use cases better.
+6Hi @gabriel.marchi,
Yes, this is expected behavior in Google SecOps Detection Engine under specific conditions, and it is generally not a bug.
Here is why this happens when you update a detection rule:
Google SecOps evaluates events based on their Event Timestamp (metadata.event_timestamp), whereas execution is bound to when the logs arrive in the system. If an event occurred two days ago but was only ingested recently (due to forwarder lag, network outages, or batch uploads), updating or saving a rule forces the Detection Engine to process all newly available/unprocessed events in that pipeline window.
When you update and save a YARA-L rule:
The engine re-initializes execution state for the rule.
If the event was ingested within the system's retroactive lookback window (or while the rule was previously disabled/modified), the newly saved rule logic will evaluate those late-arrived logs.
Check the event's metadata in your UDM Search:
metadata.event_timestamp: When the activity actually occurred (e.g., 2 days ago).
metadata.ingested_timestamp: When Chronicle actually received the log.
If ingested_timestamp is recent (or right around when you saved the rule), the alert was rightfully triggered by the late ingestion pipeline rather than a full system backtest replay.
Hope this clarifies how the rule engine handles delayed logs!
+14Hi
Is there anything that hints towards a detection delay (for the detection id) - https://docs.cloud.google.com/chronicle/docs/detection/detection-delays
Kind Regards,
Ayman
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
