Skip to main content

Hi team,

I am building a dashboard to show the top detections. For this, I am creating a table view with columns for detection title and counts. It's pretty simple—I have used the detection title and the product log id with count measure to achieve this. However, I am facing an issue.

Let's say, for product log id = 1, the title is abc. After some time, a new record gets ingested with the same product log id = 1 but with the title xyz. With the current approach, I get two rows: abc, 1 and xyz, 1. However, I only need xyz, 1, as it is the latest record.

In summary, I want to deduplicate the product log id by selecting only the most recently ingested record for each ID. Is this possible? If yes, please let me know how.

Thanks

cc: jstoner

 

Here is an example of what I believe you are driving for and hopefully the couple of ideas here will get you going in the direction you are driving...


The following query will give me a listing of the rule_id, rule_name, date and detection count. So in my results you can see I have a large number of dns_query_first_seen_past_day rules that fired. The rule id is a good way field to use in case the rule_name in detection gets duplicated in another rule btw.


 



$rule_id = detection.detection.rule_id
$rule_name = detection.detection.rule_name
$date = timestamp.get_timestamp(detection.detection_time.seconds, "%F %T")
match:
$rule_id, $rule_name, $date
outcome:
$d_count = count($rule_id)
order:
$date asc


A snippet of my results are above. I think this is what you are describing, in general with the exception that i don't have the same rule name on two different rules, but this is why i tend to prefer to use rule_id as a good value and then bring the name along for the human readable part.


 For the second part of the question,  let's revamp our query slightly. This time, we are moving the date out of the filtering statement and the match section and we are going to group on that same rule_id and rule_name but now we are going to return the max value of the detection by time and then use the get_timestamp function to make it nicely formatted. We will still count the number of detection in the past month by the rule_id and rule_name combination, but now in our results we have a single row for each rule_id/rule_name combo with the count of detections and the max detection time which would be the last time that rule triggered.


 



$rule_id = detection.detection.rule_id
$rule_name = detection.detection.rule_name
match:
$rule_id, $rule_name
outcome:
$d_count = count($rule_id)
$date = timestamp.get_timestamp(max(detection.detection_time.seconds), "%F %T")
order:
$date asc


 



Hope this helps! 


 



Thanks @jstoner 

It seems that the above solution is for the Preview Dashboards, but I am looking for the normal Dashboards.


Hi @GaurangPatel,

The below should work, import this into the Looker based dashboard solution:

lookml:
- dashboard: latest_record
description: ""
elements:
- col: 0
column_limit: 50
conditional_formatting_include_nulls: false
conditional_formatting_include_totals: false
defaults_version: 1
enable_conditional_formatting: false
explore: rule_detections_connector
fields:
- rule_detections__detection.rule_id
- rule_detections__detection.rule_name
- rule_detections.committimestamp_max_second
- rule_detections__detection.count
header_font_size: 12
header_text_alignment: left
height: 7
hide_row_totals: false
hide_totals: false
limit: 500
limit_displayed_rows: false
model: scn
name: Untitled
row: 0
rows_font_size: 12
show_row_numbers: true
show_view_names: false
size_to_fit: true
sorts:
- rule_detections.committimestamp_max_second desc 0
table_theme: white
title: Untitled
transpose: false
truncate_text: true
type: looker_grid
width: 18
layout: newspaper
title: Latest Record
metadata:
exported_at: "2025-02-11T04:18:40-08:00"
file_fingerprint: "125426485073724995716803761615960826218"
looker_version: 24.18.128
version: "1"


Kind Regards,



Ayman


Reply