Skip to main content

Hi team,

I am building a dashboard to show the top detections. For this, I am creating a table view with columns for detection title and counts. It's pretty simple—I have used the detection title and the product log id with count measure to achieve this. However, I am facing an issue.

Let's say, for product log id = 1, the title is abc. After some time, a new record gets ingested with the same product log id = 1 but with the title xyz. With the current approach, I get two rows: abc, 1 and xyz, 1. However, I only need xyz, 1, as it is the latest record.

In summary, I want to deduplicate the product log id by selecting only the most recently ingested record for each ID. Is this possible? If yes, please let me know how.

Thanks

cc: jstoner

 

Here is an example of what I believe you are driving for and hopefully the couple of ideas here will get you going in the direction you are driving...

The following query will give me a listing of the rule_id, rule_name, date and detection count. So in my results you can see I have a large number of dns_query_first_seen_past_day rules that fired. The rule id is a good way field to use in case the rule_name in detection gets duplicated in another rule btw.

 

$rule_id = detection.detection.rule_id
$rule_name = detection.detection.rule_name
$date = timestamp.get_timestamp(detection.detection_time.seconds, "%F %T")
match:
$rule_id, $rule_name, $date
outcome:
$d_count = count($rule_id)
order:
$date asc

A snippet of my results are above. I think this is what you are describing, in general with the exception that i don't have the same rule name on two different rules, but this is why i tend to prefer to use rule_id as a good value and then bring the name along for the human readable part.

 For the second part of the question,  let's revamp our query slightly. This time, we are moving the date out of the filtering statement and the match section and we are going to group on that same rule_id and rule_name but now we are going to return the max value of the detection by time and then use the get_timestamp function to make it nicely formatted. We will still count the number of detection in the past month by the rule_id and rule_name combination, but now in our results we have a single row for each rule_id/rule_name combo with the count of detections and the max detection time which would be the last time that rule triggered.

 

$rule_id = detection.detection.rule_id
$rule_name = detection.detection.rule_name
match:
$rule_id, $rule_name
outcome:
$d_count = count($rule_id)
$date = timestamp.get_timestamp(max(detection.detection_time.seconds), "%F %T")
order:
$date asc

 

Hope this helps! 

 


Thanks @jstoner 

It seems that the above solution is for the Preview Dashboards, but I am looking for the normal Dashboards.


Hi @GaurangPatel,

The below should work, import this into the Looker based dashboard solution:

lookml: - dashboard: latest_record description: "" elements: - col: 0 column_limit: 50 conditional_formatting_include_nulls: false conditional_formatting_include_totals: false defaults_version: 1 enable_conditional_formatting: false explore: rule_detections_connector fields: - rule_detections__detection.rule_id - rule_detections__detection.rule_name - rule_detections.committimestamp_max_second - rule_detections__detection.count header_font_size: 12 header_text_alignment: left height: 7 hide_row_totals: false hide_totals: false limit: 500 limit_displayed_rows: false model: scn name: Untitled row: 0 rows_font_size: 12 show_row_numbers: true show_view_names: false size_to_fit: true sorts: - rule_detections.committimestamp_max_second desc 0 table_theme: white title: Untitled transpose: false truncate_text: true type: looker_grid width: 18 layout: newspaper title: Latest Record metadata: exported_at: "2025-02-11T04:18:40-08:00" file_fingerprint: "125426485073724995716803761615960826218" looker_version: 24.18.128 version: "1"


Kind Regards,



Ayman


Reply