Rapid7 Feed Ingestion

I don’t know this log type, is it event data, or entity/context data?
 

If Entity Context data, do you see raw data with:
graph.metadata.source_type = “ENTITY_CONTEXT”

Hi ​@SoarAndy , when I run a search for my two namespaces for Rapid7, I do see ENTITY_CONTEXT. On each event I see the tab on the right for the RAW_LOG.

Below is a sample of an expected Rapid7 API call to pull back details on an asset and the vulnerabilities on it. This puts them into 3 buckets: same (no change between comparison times), new (newly added vulnerabilities since comparison time) and remediated (removed vulnerabilities since comparison time).

https://help.rapid7.com/insightvm/en-us/api/integrations.html#tag/Asset/operation/getIntegrationAsset

For every single RAW_LOG entry going back 3 weeks the values of my events are “new”:[],, “remediated”:[], and there is no “same” (which it’s possible to ignore and not pull back in the API.

It also seems that the default parser from Google isn’t accounting for New, Same or Remediated.

But I believe it isn’t even pulling that additional vulnerability information. Based on all of this, I don’t see how anyone could use this addon as it doesn’t seem to be configured to pull the actual vulnerability findings.

Is there somewhere I can see the exact execution of the API calls to see what is being requested?
 

Thanks for the examples.

Looking at the parser and our internal sample data, I **think** that the sample data the Engineering teams were given had empty arrays for new/remediated, so that data wasn’t known. We do however extract lots more, so I think it’s fair to say it still has value to other use cases.

1 - I will log an internal request to ask Engineering to review a new log and add in more data to our standard parser. 

2 - Until that’s released (I have no control over prioritisation), I suggest looking at a parser extention.  You can either:

• Hard: Write the parser in the UI
• Easy: Use the AI labs parser extention tool (image below).  I pasted in the JSON from rapid 7, added a user description, I then saw the code and the extended output.  I think you could play with this to get the value you need then paste the code into the parser library?

HTH

Andy

Thanks ​@SoarAndy , I appreciate you looking into it.

In regards to your 2nd point, if I am not seeing those arrays (new, remediated) populated in any of the raw logs for the events, that would mean that that data is just not collected right?

Even if I wrote a parser extension, there wouldn’t be the data for it to populate the new fields, right?

The raw log in the event is the complete data received, correct?

Yes rawlog is what we collect.  If the data is not there I suggest looking at API endpoint versions, maybe API key permissions, licencing/features enabled, check the Entity does actually have data to create the key etc.

HTH

Andy
 

Thanks Andy,

I checked the API endpoints and those are what I am using in different workflows, so those are fine. 

But the problem I see is that this feed is not configured to pull those fields, it is only pulling the assets and the vulnerability definitions. There should be an additional pull done after pulling hosts where it checks for new/remediated vulnerabilities and then ingest those. 

I have confirmed on several systems they are generating the events with new/remediated in our legacy SIEM ingestion but not in the GSO ingestion.

While this feed has some value if this data is not ingested currently (which doesn’t seem to be in the raw logs I see), then this isn’t a feed which is collecting what it says it is, vulnerability findings.
 

1. Is there somewhere I can see the exact API calls of a feed to verify what is pulled?
2. Is there a ticket I can make on my end to track this?

I do recommend a support ticket, that way you can prioritise this enhancement and it gives you a better way to provide sample logs (redacted) to make sure any improvement, I have submitted the ticket twice but a formal ticket against your customer ID is more recognised than anything I paste from a community post as we can track it better, hope that helps.

I don’t know of a way to intercept and see the root call, however I have no reason to doubt it’s the URL we publish in our docs, and I suppose I don’t doubt Rapid7 doesn’t sent it.  I think the parser not parsing this KeyValue is most probably at this point

Andy

Thanks Andy, I appreciate the feedback. I’ll get a ticket logged to track it.

Last question, the raw logs in the events search, those are the complete raw log that came from the source (in this case Rapid7) or are they the raw log after some processing occurs?

I would have thought that if the parser wasn’t parsing the KeyValue, you would see the KeyValue in the raw log for the event, is that not the case?

My second request also hit challenges due to the anonymous nature of Community chat. So yes please log a formal tickets with through your official channels.

Sorry and thanks

Andy