A problem with how data is being fed into your SecOps SIEM could cause a backlog, leading to delayed alerts and a sudden burst when the backlog clears. Also the server blackouts might be related to this data ingestion process.
You may want to open a support case to troubleshoot this behavior further.
Hello, if I understand the scenario correctly, you are receiving the logs delayed by almost 8 hrs (difference between the create/ingest time and even time). And your time window is 5 min. In this case the real time execution will not pich up these delayed logs. We have clean up jobs that run to catch up all the logs we received late. Hence the detections you are seeing in a burst. Fix the underlying issue of delayed data ingestion.
Sounds like a tough issue. The server blackouts could be causing delays in rule execution. I’d suggest checking the server logs to see if outages are affecting things. Also, double-check the rule conditions to make sure nothing unexpected is triggering it. If the overflow of alerts is an issue, adding rate-limiting might help prevent the system from getting overwhelmed. Hope that points you in the right direction!
Hi,
This will certainly require some more information. But first and foremost, this is a rule looking for user logins to a UNIX system. If any of these are internet facing they should be locked down to known IP's /subnets only. Where are these unix devices located, what part of your network?
The other part is these blackouts. If your devices lose power and come back online there still shouldn't be any discrepancy as to when the events occur. They cant occur if they don't have power.
The question now becomes, as @rajukg11 mentions, what is the underlying issue - how is this data being sent to SecOps. Is this using some kind of feed or using some kind of syslog server that sends the data. If there is an 8 hour delay from when the event occurs to when the event is ingested there is most likely a resource constrain somewhere. This is where the blackout may come into play. When the devices affected by power outages all come back on once this could be causing a large pipeline problem.
