Hi
If we have by any reason have duplicate logs for example a log source like 0365 (emails) , while building a rule how can i remove deduplication of events .
You should check which data you can use to identify the duplicate log entries and group them together using a match variable.
My suggestions for match variables would be: metadata.id or metadata.product_log_id
rule ruleName {
events:
$mail.metadata.log_type = "OFFICE_365"
// other rule logic
$mail.metadata.id = $id
// $mail.metadata.product_log_id = $id
match:
$id over 10m
condition:
$mail
}
You should check which data you can use to identify the duplicate log entries and group them together using a match variable.
My suggestions for match variables would be: metadata.id or metadata.product_log_id
rule ruleName {
events:
$mail.metadata.log_type = "OFFICE_365"
// other rule logic
$mail.metadata.id = $id
// $mail.metadata.product_log_id = $id
match:
$id over 10m
condition:
$mail
}
Hi @rahul7514,
To add onto @JensW, I would suggest where possible to use the value that a unique identifier from the underlying event maps that value - namely in 'metadata.product_log_id'. This value usually holds a unique identifier (Encodes a vendor-specific event identifier to uniquely identify the event (a GUID). Users might use this identifier to search the vendor's proprietary console for the event in question.)[1]. The reason why I recommend using 'metadata.product_log_id', over 'metadata.id' is that at the point of event parsing, Chronicle adds 'metadata.id' to each event, regardless of whether it's duplicated. If you send 5 of the same alerts to Chronicle, they will all contain a unique 'metadata.id'. However if you send the 5 same alerts to Chronicle, and they include a unique log ID identifier, and if it's mapped to 'metadata.product_log_id', you can differentiate duplicates that way.
If the underlying log source does not contain a unique identifier for an event, other ways to do this would be to match over certain values which with almost certainty could not occur at the same time and be unique events.
[1] -
Kind Regards,
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.