Skip to main content
Solved

Best way to export events from broad queries

  • December 12, 2024
  • 2 replies
  • 68 views
W
Forum|alt.badge.img+1

Hi,

What is the best way to export large amounts of results from (very) broad queries? It seems that the SecOps UI only shows and allows the exporting of 1,000,000 events when very broad queries are executed. I want to export a large amount of parsed event data (more than 20 million individual events) and analyse the data through pandas on python. 

I understand that I can do this through the SecOps API or through BigQuery, but are there any costs incurred through these processes? Would there be any more efficient methods that I could try out?

Thank you very much.

Best answer by gkush

Hi Wonjulee,

As part of the shared resource of the SecOps SaaS, SecOps limits your queries, whether UI based or API-based, to 1M results in a UDM Search. This doesn't mean that aggregate information gets cut off at 1M records. Suppose you wanted a count of all DNS lookups done across all endpoints for a month?  In most places that would reach past 1M results, and with Stats Search you could get the aggregate count, but not the full set of data.

Since you said you wanted the records themselves for analysis, you could try to break your query up into chunks and align based on timestamps.  That would be something you could try in a script - get a set, figure out the timestamp, keep looking backwards from the last returned result.

Depending on your license tier, another option might be the BigQuery data lake?  You should be able to run your SQL-based query on the UDM fields you want and then pull the data out.  

2 replies

gkush
Staff
Forum|alt.badge.img+5
  • gkush
  • Staff
  • Answer
  • December 12, 2024

Hi Wonjulee,

As part of the shared resource of the SecOps SaaS, SecOps limits your queries, whether UI based or API-based, to 1M results in a UDM Search. This doesn't mean that aggregate information gets cut off at 1M records. Suppose you wanted a count of all DNS lookups done across all endpoints for a month?  In most places that would reach past 1M results, and with Stats Search you could get the aggregate count, but not the full set of data.

Since you said you wanted the records themselves for analysis, you could try to break your query up into chunks and align based on timestamps.  That would be something you could try in a script - get a set, figure out the timestamp, keep looking backwards from the last returned result.

Depending on your license tier, another option might be the BigQuery data lake?  You should be able to run your SQL-based query on the UDM fields you want and then pull the data out.  

    AymanC
    Forum|alt.badge.img+13
    • AymanCBronze 5
    • Bronze 5
    • December 12, 2024

    Hi @wonjulee,

    It may be worth looking into the following API endpoint: https://cloud.google.com/chronicle/docs/reference/rest/v1alpha/projects.locations.instances.legacy/legacyFetchUdmSearchCsv

    I've not personally tried this, but our Customer Success Team has previously suggested using it to export a large amount of data.

    Kind Regards,

    Ayman

      Terms & ConditionsCookie settingsAccessibility statement
      Privacy Policy Terms of Service Gen AI Policy Community Guidelines Cookie Settings

      © 2025 Google. All rights reserved.