Scraping a Website's Search URL with Clay: Tips and Techniques

Hello - how would I scrape a website using clay to get a website's search url. I tried using the clay scrape tool but the search lives in the script tag typically and I did not see that in the data brought back.

3 comments

Channeled
APP
·
·
To scrape a website's search URL that's typically in the script tag, you can try these approaches: 1. Use the Clay Chrome Extension for more targeted scraping: • Install the extension from the Chrome Web Store • Navigate to the website you want to scrape • Use the "Select Data" option to manually choose the elements you need, including script tags 2. If using the Scrape Website action: • Set "Keep Non-Text in Body" to true to include script tags in the output • Use the "Extract Custom Regex" field to target specific patterns in the script tags If these methods don't work, you may need to request human assistance for a more tailored solution.
Sources: Scrape Website with Clay Get transcript of any YouTube video in seconds | Clay
Owen C.
·
·
Hey Dillon, thanks for reaching out! Are you referring to the Google Search URL when searching for a companies website? Or what URL are you referring to here specifically? One frame I like to use when thinking about how to do something within Clay is, how would I do this task manually excluding Clay. Once we have the steps to follow there we can work backward and try to figure out how this could be applied within Clay.
Channeled
APP
·
·
This thread was picked up by our in-app web widget and will no longer sync to Slack. If you are the original poster, you can continue this conversation by logging into https://app.clay.com and clicking "Support" in the sidebar. If you're not the original poster and require help from support, please post in 02 Support.