Discrepancy between dashboard filter for graph.metadata..interval.start_time and Local Time Filters inside edit mode of panel.

TLDR: Use the ‘global time filter’ at the dashboard level and apply it to your Total IOCs widget then the results should match.

Explanation:

Theres a few differences between events and entities, and the timepicker for search vs the timepicker for dashboards that combine to cause this behavior.

Time matching applies differently for entities than they do for events. With events the search timerange looks for  metadata.event_timestamp to be inside the range you specify; but when searching for entities the search time filter will match entities where their validity interval (the time between graph.metadata.interval.start_time & graph.metadata.interval.end_time) OVERLAPS with the search interval.  At first that may not seem terribly different, but when creating IOC entities (anything with a graph.metadata.threat section)  if you do not specify the interval start & end times in the entity context record they will default to 1970-01-01 and 9999-12-31.  Then the graph maintains a separate instance of the entity for each day (UTC midnight to UTC midnight) that entity is in its validity interval. So when searching for an entity you will see multiple entries for each entity, one for each day, running 00:00:00 to 00:00:00.

Next, the relative timerange picker for dashboards does not apply the time filter the same way the that search does.  In search when you click the ‘last hour’ time range quick select you can easily see that it sets the end time to now (matching to the minute) then subtracts one hour for the start time, and for last 24 hours the start time is exactly (to the minute) 24 hours ago.

In dashboards when using the absolute timerange you get the same behavior as search; however, when using the relative timerange, the timerange is actually anchored to the start of that interval.  If you run a dashboard for ‘last 1 hour’ at 14:35, the start for the dashboard timerange will be 14:00,  if you run a dashboard for ‘last 1 day’ at 14:35 the start for the dashboard timerange will be 00:00.

The other behavior is something I had not noticed previously but a little bit of testing confirmed;  the normal dashboard time filter (global or in the dashboard editor) is an inclusive match, so setting a time range for past 1 days will match all the way down to 00:00:00.000, but when you build a custom filter for time that becomes an exclusive match,  so would require the timestamp to be greater than 00.000.

So it is likely your IOC are all overlapping with the last 24h and show up when you use the time picker built into the editor, but that when you explicitly specify the dashboard level filter to reference the interval.start_time it doesn’t match because your IOCs intervals are all starting at 00:00:00 and are not included in the results which is looking for greater than 00:00:00

If you enable the global time filter for this chart you should see consistent data since the match method will align between the chart edit window.