Scraping a Website's Search URL with Clay: Tips and Techniques

Dillon T. · 2025-04-30T17:38:32.878655+00:00

Hello - how would I scrape a website using clay to get a website's search url. I tried using the clay scrape tool but the search lives in the script tag typically and I did not see that in the data brought back.

To scrape a website's search URL that's typically in the script tag, you can try these approaches: 1. Use the Clay Chrome Extension for more targeted scraping: • Install the extension from the Chrome Web Store • Navigate to the website you want to scrape • Use the "Select Data" option to manually choose the elements you need, including script tags 2. If using the Scrape Website action: • Set "Keep Non-Text in Body" to true to include script tags in the output • Use the "Extract Custom Regex" field to target specific patterns in the script tags If these methods don't work, you may need to request human assistance for a more tailored solution.

Sources: Scrape Website with Clay Get transcript of any YouTube video in seconds | Clay

3 comments