silent log source detection rule

Your assessment is correct, a YARA-L rule can only evaluate a log source that appears within the lookback window, so it cannot reliably detect a source that has been completely silent or never ingested data. A better approach is to maintain an expected-source inventory or heartbeat event and use YARA-L to detect missing heartbeats, while tracking recovery separately to avoid suppression hiding a second outage. Without that external state, a YARA-L-only solution will always have blind spots.

Thanks, Aleksei. That confirms my understanding. I agree that the reliable design is an external expected-source inventory or heartbeat mechanism, with YARA-L used only to detect missing expected heartbeats and recovery tracked independently. A YARA-L-only rule cannot reliably detect a source that never appears within the lookback window.

The match window can be extended a bit beyond 24h. Example below.

There are few ways to suppress alerts: throttling, built-in secops rule exclusion, exclusion by reference lists / data tables, SOAR alert grouping. It ultimately depends on finding the right balance adapted to your processes and timings to handle the alert.
 

rule unusual_log_volume_drop {
  meta:
    description = "Detects drops in log volume compared to daily average and weekday seasonality, or no events on last hour"
  events:
    $e.metadata.log_type = $log_type
    $e.metadata.log_type != ""
    //$e.metadata.log_type = /it might be interesting to group logtypes by volume envelope similarity/

    optimization.sample_rate($e.metadata.id, 1, 1000)
    //fine tune sampling rate according to the dataset volume.
    //remove the sampling if the dataset is small enough to avoid sampling
  match:
    $log_type over 8d
  outcome:
    $total_events_count = count($e.metadata.id)
    $max_event_time = max($e.metadata.event_timestamp.seconds)
    $curr_time = timestamp.now()
    $seconds_since_last_event = $curr_time - $max_event_time
    $average_events_per_day = $total_events_count / 8
    $events_in_last_24h = sum(if($e.metadata.event_timestamp.seconds > $curr_time - 60*60*24, 1, 0))
    $ratio_last24h_to_avgday = $events_in_last_24h / $average_events_per_day
    $same_day_lastweek = sum(if($e.metadata.event_timestamp.seconds > $curr_time - 60*60*24*8 AND $e.metadata.event_timestamp.seconds < $curr_time - 60*60*24*7, 1, 0))
    $ratio_weekday_seasonality = $events_in_last_24h / $same_day_lastweek
  condition:
    $e AND
    //making sure the sampled event volume is significant. Adapt as needed.
    $total_events_count > 1000 AND

    //check if there were events $same_day_lastweek
    $same_day_lastweek > 0 AND

    //fine tune as needed. Assuming same weekday would have similar ratios (closer to 1) than the comparison against the average. 
    ($seconds_since_last_event > 1800 OR ($ratio_last24h_to_avgday < 0.2 AND $ratio_weekday_seasonality < 0.5))
}

These kind of detections are best to cover drops / silence from sources with an expected envelope or baseline, E.g. traffic logs (usual expected camel envelope on weekdays and drops on weekends)
For irregular source volumes, it’s best to use heartbeat as mentioned by others. E.g. findings or alerting type of events.

Now I see the exact issue. hliu your approach is useful, but the example has an important correctness risk.

optimization.sample_rate(...) is unsafe for this part:

$max_event_time =

  max($e.metadata.event_timestamp.seconds)

Because the newest event may be excluded by sampling, $seconds_since_last_event can be artificially inflated and generate a false silence alert.

Also:

$seconds_since_last_event > 1800 OR ...

makes every matched log type a generic 30-minute silence detector, even when that source normally reports less frequently. The eight-day window extends historical visibility, but it still does not solve sources that never appeared—or remained silent beyond that window.

@hilu Extending the match window and comparing recent volume against a weekday baseline is useful for detecting degradation or partial silence.

One concern with the example is that max(event_timestamp) is calculated from the sampled event set. With optimization.sample_rate() enabled, the actual newest event may be excluded, which could inflate seconds_since_last_event and create false positives. For freshness or last-seen logic, I would avoid sampling that event population.

Also, seconds_since_last_event > 1800 OR ... may alert on naturally low-frequency sources regardless of their expected cadence. I would gate that threshold using a per-source expected interval or require supporting baseline deviation.

This looks like a good complementary volume-drop rule, while an external inventory or heartbeat still appears necessary for never-seen sources and silence extending beyond the rule window.

And that is only if I'm seeing this right 🔥

Use an unsampled, ingestion-time-based YARA-L rule for source silence and significant volume degradation.

rule unusual_log_volume_drop {

  meta:

    description = "Detects log-source silence or significant volume degradation"

    author = "whathehack81"

  events:

    $e.metadata.log_type = $log_type

    $e.metadata.log_type != ""

  match:

    $log_type over 8d

  outcome:

    $total_events_count = count($e.metadata.id)

    // Ingestion time reflects pipeline freshness more reliably

    // than source-controlled event timestamps.

    $last_ingestion_time =

      max($e.metadata.ingested_timestamp.seconds)

    $current_time = timestamp.current_seconds()

    $seconds_since_last_event =

      $current_time - $last_ingestion_time

    $events_in_last_24h =

      sum(

        if(

          $e.metadata.ingested_timestamp.seconds >

            $current_time - 86400,

          1,

          0

        )

      )

    $same_day_last_week =

      sum(

        if(

          $e.metadata.ingested_timestamp.seconds >

            $current_time - 691200

          and

          $e.metadata.ingested_timestamp.seconds <=

            $current_time - 604800,

          1,

          0

        )

      )

  condition:

    $e

    and $total_events_count > 1000

    and (

      // No ingestion for one hour.

      $seconds_since_last_event > 3600

      or

      (

        // Current 24-hour volume is below 20% of the

        // eight-day daily average:

        //

        // last24h < (total / 8) * 0.2

        // last24h * 40 < total

        $events_in_last_24h * 40 < $total_events_count

        and

        // Current volume is below 50% of the corresponding

        // 24-hour window from the previous week.

        $same_day_last_week > 0

        and $events_in_last_24h * 2 < $same_day_last_week

      )

    )

}

Improvements

Removes optimization.sample_rate(), preventing the newest event from being sampled out.

Uses metadata.ingested_timestamp instead of source event time.

Avoids division and divide-by-zero conditions through equivalent integer comparisons.

Allows silence detection independently of the weekday baseline.

Requires both baseline comparisons before declaring volume degradation.

Run it as an hourly scheduled rule and validate these cases:
Test
Expected
Last ingestion under 1 hour
No silence alert
Last ingestion over 1 hour
Silence alert
Last 24h below 20% average and 50% prior-week volume
Degradation alert
Low current volume but no prior-week baseline
No degradation alert
Source absent for more than eight days
Not detected

The final limitation remains: a source with zero events inside the eight-day window cannot create a $log_type match group. That case still requires an external source inventory or heartbeat event.

I wasn’t trying to come up with the perfect query to cover all cases, but provide some ideas and example. Hopefully the rule comments in my previous post are clear enough for others to fine tune adapting it to each case :)

In sources shipping large data volumes, without sampling the rule might run into time-out.
For me it is finding the right balance adapted to each ingestion envelope.


But for those posts based on AI, perhaps the intention is another.

rule unusual_log_volume_drop {
  meta:
    description = "Detects drops in log volume compared to daily average and weekday seasonality, or no events on last hour"
  events:
    $e.metadata.log_type = $log_type
    $e.metadata.log_type != ""
    //$e.metadata.log_type = /it might be interesting to group logtypes by volume envelope similarity/

    optimization.sample_rate($e.metadata.id, 1, 1000)
    //fine tune sampling rate according to the dataset volume.
    //remove the sampling if the dataset is small enough to avoid sampling
  match:
    $log_type over 8d
  outcome:
    $total_events_count = count($e.metadata.id)
    $max_event_time = max($e.metadata.event_timestamp.seconds)
    $curr_time = timestamp.now()
    $seconds_since_last_event = $curr_time - $max_event_time
    $average_events_per_day = $total_events_count / 8
    $events_in_last_24h = sum(if($e.metadata.event_timestamp.seconds > $curr_time - 60*60*24, 1, 0))
    $ratio_last24h_to_avgday = $events_in_last_24h / $average_events_per_day
    $same_day_lastweek = sum(if($e.metadata.event_timestamp.seconds > $curr_time - 60*60*24*8 AND $e.metadata.event_timestamp.seconds < $curr_time - 60*60*24*7, 1, 0))
    $ratio_weekday_seasonality = $events_in_last_24h / $same_day_lastweek
  condition:
    $e AND
    //making sure the sampled event volume is significant. Adapt as needed.
    $total_events_count > 1000 AND

    //check if there were events $same_day_lastweek
    $same_day_lastweek > 0 AND

    //fine tune as needed. Assuming same weekday would have similar ratios (closer to 1) than the comparison against the average. 
    ($seconds_since_last_event > 1800 OR ($ratio_last24h_to_avgday < 0.2 AND $ratio_weekday_seasonality < 0.5))
}

For time travelers that might be looking into this from the future,

please ignore my previous example: rule unusual_log_volume_drop

As it’d eventually produce false positives caused by the relativity of timestamp.now() or timestimap.current_seconds() in rule replays, described in detail here:
 

At this moment I am not aware of any reliable way to build these detections within the Secops SIEM itself.

Hopefully the product team could implement the feature requests described in the link.