Hello Everyone,While working with custom parsers and entity ingestion in Google SecOps, I have come across two related issues around Entity visibility and UDM field availability. I am outlining them below: 1. Delay in Entity Visibility I created a custom parser and ingested entities. The logs appear immediately in Raw Log Search, but the corresponding entities only become visible in UDM Search and Native Dashboards after almost a full day. I ensured that timestamp fields such as graph.metadata.interval.start_time and graph.metadata.collected_timestampare set to the current time. Interestingly: Entities ingested using the v2/entities:batchCreate endpoint (USER/ASSET) appear within minutes. However, when trying to ingest FILE type entities via the same endpoint, I encounter a 400 error.400 { "error": { "code": 400, "message": "Request contains an invalid argument.", "status": "INVALID_ARGUMENT" } } 👉 Question: Why is there such a long delay for entities from custom parsers, and are there any best practices to ensure they appear in UDM Search and dashboards in near real-time?Not sure why i am getting 400 while ingesting the `FILE` entitites using the v2/entities:batchCreate endpoint.2. Missing UDM Fields in Raw Log Search For logs where we generate both Events and Entities, the parsed UDM fields are not visible in Raw Log Search. These UDM fields, however, appear correctly in Native Dashboards. For other logs where only Entities are parsed, UDM fields display as expected in Raw Log Search. 👉 Question: Is this the expected behavior, or is there a way to configure things so that parsed UDM fields are also visible in Raw Log Search when both Events and Entities are created?<UDM parsing not showing><UDM parsing showing>

Delay in Entity Visibility and Missing UDM Fields in Raw Log Search

Hello Everyone,

While working with custom parsers and entity ingestion in Google SecOps, I have come across two related issues around Entity visibility and UDM field availability. I am outlining them below:

1. Delay in Entity Visibility

I created a custom parser and ingested entities.
The logs appear immediately in Raw Log Search, but the corresponding entities only become visible in UDM Search and Native Dashboards after almost a full day.
I ensured that timestamp fields such as graph.metadata.interval.start_time and graph.metadata.collected_timestampare set to the current time.

Interestingly:

Entities ingested using the v2/entities:batchCreate endpoint (USER/ASSET) appear within minutes.
However, when trying to ingest FILE type entities via the same endpoint, I encounter a 400 error.
400 { "error": { "code": 400, "message": "Request contains an invalid argument.", "status": "INVALID_ARGUMENT" } }

👉 Question: Why is there such a long delay for entities from custom parsers, and are there any best practices to ensure they appear in UDM Search and dashboards in near real-time?

Not sure why i am getting 400 while ingesting the `FILE` entitites using the v2/entities:batchCreate endpoint.

2. Missing UDM Fields in Raw Log Search

For logs where we generate both Events and Entities, the parsed UDM fields are not visible in Raw Log Search.
These UDM fields, however, appear correctly in Native Dashboards.
For other logs where only Entities are parsed, UDM fields display as expected in Raw Log Search.

👉 Question: Is this the expected behavior, or is there a way to configure things so that parsed UDM fields are also visible in Raw Log Search when both Events and Entities are created?

Page 1 / 1

When creating a custom parser in Google SecOps, optimizing for ingestion time is crucial for efficient operations, cost management, and timely threat detection. The core principle is to do only what's necessary, as efficiently as possible, and as early as possible in the data pipeline. Here are a couple of blogs that may be of some help here:
Data Ingestion Challenges in SecOps and How to Overcome Them
A Step-by-Step Guide to the Data Ingestion Process

It looks like there is an issue tracker for the delay here - https://issuetracker.google.com/issues/417951239?pli=1. If you’re seeing more significant delays, closer to a day, I would open a case for that.

Thank you for the insights and discussion regarding entity ingestion and UDM behavior in Google SecOps. Based on our observations and forum guidance:

Delay in Entity Visibility:
- IOC Entities ingested via parsers may take up to a day to appear in UDM Search and native dashboards, even when timestamp fields (graph.metadata.interval.start_time, graph.metadata.collected_timestamp) are correctly set. This behavior is partially a known limitation, and long delays for parser ingestions have been logged in the issue tracker (Issue 417951239).
Missing UDM Fields in Raw Log Search:
- When both Events and Entities are generated from the same log, parsed UDM fields may not appear in Raw Log Search, although they display correctly in UDM Search and dashboards. This is expected behavior in the current platform design and reflects how Raw Log Search indexes combined Event+Entity logs.

1. Delay in Entity Visibility

2. Missing UDM Fields in Raw Log Search

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded