Automating Google searches?

Hey ​@donkos , have you tried the HTTPv2 Action?

Seconding this.

With the HTTPv2 integration and Execute HTTP Query, try something like this, where the principal.hostname placeholder is what you want to Google:

Results:

Google will likely see these are automated in time and request a CAPTCHA, preventing the search from returning. To get around that, I would recommend registering for the Google Custom Search JSON API. From there, change the URL to https://www.googleapis.com/customsearch/v1, and add your key (API Key) and cx (Search Engine ID) to the URL parameters alongside the q query parameter. Example:

{
  "key": "YOUR_API_KEY",
  "cx": "YOUR_SEARCH_ENGINE_ID",
  "q": "[Event.udm.principal.hostname]",
  "num": "10"
}

The Custom Search API will return a JSON object that will be easier to work with vs response_data in subsequent steps as well.

​@cmorris Looks like an interesting solution. I visited the link for the customsearch endpoint and it mentioned that:

Note: The Custom Search JSON API is closed to new customers. Vertex AI Search is a favorable alternative for searching up to 50 domains. Alternatively, if your use case necessitates full web search, contact us to express your interest in and get more information about our full web search solution.

Existing Custom Search JSON API customers have until January 1, 2027 to transition to an alternative solution.


Are there any other options youre aware of?

For Google native - unfortunately not that I am aware of, other than the Vertex AI Search option. A Scraping API (non-Google) could work as well.