Using Claygent for URL Scraping to Build Company Lists

Can Claygent be used to scrape a specific URL to build an initial list of companies, or do we have to provide the company list first, then use Claygent to add details about each company?

10 comments

Tanvi R.
·
·
Hey Tom, thanks for reaching out. Can you specify which URL you are looking to scrape? If you're only looking to scrape one URL, the Clay Chrome Extension might be a better approach here as it can pull lists from websites. However, if there are multiple different URLs in your Clay table to scrape the Scrape website enrichmentsmight be better. Feel free to share more information so I can assist you better!
Tom E.
·
·
Thanks! I want to scrape a site that requires clicking an element to proceed to “Page 2” etc, and grab some text and URLs from each individual listing on each page. See: https://hrtech2024.smallworldlabs.com/exhibitors
Tom E.
·
·
So I think the answer to your question is “I’m scraping one URL, but it requires paginating”
Andrei B.
·
·
Try this. Worked like a charm 🙂
Tom E.
·
·
I’d prefer to keep it all inside Clay and have a single scraping tool, but thanks Andrei B.
🫡1
Tanvi R.
·
·
Got it Tom, unfortunately the chrome extension cant scrape multiple paginated pages at once. However, since there are only 10 pages here, you could use the Chrome extension to scrape each one of the 10 pages. Feel free to give the chrome extension a try and let me know what you think!
Anas A.
·
·
Adnan M. Can we build this scraper using Clay?
Adnan M.
·
·
Anas A. Tom E. Instant data scraper would be the best way to go about this as Andrei B. suggested because the pagination is being carried out using client side JavaScript, which Clay can't detect. If going from one page to another changes the URL in a logical manner, Clay can handle it.
Tom E.
·
·
Interesting
Channeled
APP
·
·
Hi Tom E.! This thread was recently closed by our Support team. If you have a moment, please share your feedback: