Skip to main content

Building a connector which will ingest data from a source where there is no explicit unique event id

  • Feb 22, 2022 12:44 PM UTC
  • 1 reply
  • 13 views

Hi,
I'm currently building a connector which will ingest data from a source where there is no explicit unique event id. What's the best strategy for storing and evaluating whether an event has been seen before? I'm thinking I should concat a couple of unique values such as timestamp, IP url etc, and then hash them. The use the hash value to store and check to see if the event has been processed before, Though I'm not sure if this is a bit overkill? Any recommendations would be appreciated.

1 reply

  • Feb 24, 2022 10:18 AM UTC

thanks for the input, went with an MD5 of the strings.