Using Claygent for URL Scraping to Build Company Lists

Tom Elliott · 2024-10-30 19:49:11.892612+00

Can Claygent be used to scrape a specific URL to build an initial list of companies, or do we have to provide the company list first, then use Claygent to add details about each company?

Using Claygent for URL Scraping to Build Company Lists | Clay

10 comments

· Sorted by Oldest

Tanvi (Clay)
·
·
Hey Tom, thanks for reaching out. Can you specify which URL you are looking to scrape? If you're only looking to scrape one URL, the Clay Chrome Extension might be a better approach here as it can pull lists from websites. However, if there are multiple different URLs in your Clay table to scrape the Scrape website enrichmentsmight be better. Feel free to share more information so I can assist you better!
Tom Elliott
·
·
Thanks! I want to scrape a site that requires clicking an element to proceed to “Page 2” etc, and grab some text and URLs from each individual listing on each page. See: https://hrtech2024.smallworldlabs.com/exhibitors
Tom Elliott
·
·
So I think the answer to your question is “I’m scraping one URL, but it requires paginating”
Andrei Baloleanu (Adyntel)
·
·
Try this. Worked like a charm 🙂
Loading attachments
Tom Elliott
·
·
I’d prefer to keep it all inside Clay and have a single scraping tool, but thanks Andrei Baloleanu (Adyntel)
🫡1
Tanvi (Clay)
·
·
Got it Tom, unfortunately the chrome extension cant scrape multiple paginated pages at once. However, since there are only 10 pages here, you could use the Chrome extension to scrape each one of the 10 pages. Feel free to give the chrome extension a try and let me know what you think!
Anas
·
·
Adnan thegtm.company Can we build this scraper using Clay?
Adnan thegtm.company
·
·
Anas Tom Elliott Instant data scraper would be the best way to go about this as Andrei Baloleanu (Adyntel) suggested because the pagination is being carried out using client side JavaScript, which Clay can't detect. If going from one page to another changes the URL in a logical manner, Clay can handle it.
Tom Elliott
·
·
Interesting
Channeled
APP
·
·
Hi Tom Elliott! This thread was recently closed by our Support team. If you have a moment, please share your feedback:

Tanvi (Clay)
·
·
Hey Tom, thanks for reaching out. Can you specify which URL you are looking to scrape? If you're only looking to scrape one URL, the Clay Chrome Extension might be a better approach here as it can pull lists from websites. However, if there are multiple different URLs in your Clay table to scrape the Scrape website enrichmentsmight be better. Feel free to share more information so I can assist you better!
Tom Elliott
·
·
Thanks! I want to scrape a site that requires clicking an element to proceed to “Page 2” etc, and grab some text and URLs from each individual listing on each page. See: https://hrtech2024.smallworldlabs.com/exhibitors
Tom Elliott
·
·
So I think the answer to your question is “I’m scraping one URL, but it requires paginating”
Andrei Baloleanu (Adyntel)
·
·
Try this. Worked like a charm 🙂
Loading attachments
Tom Elliott
·
·
I’d prefer to keep it all inside Clay and have a single scraping tool, but thanks Andrei Baloleanu (Adyntel)
🫡1
Tanvi (Clay)
·
·
Got it Tom, unfortunately the chrome extension cant scrape multiple paginated pages at once. However, since there are only 10 pages here, you could use the Chrome extension to scrape each one of the 10 pages. Feel free to give the chrome extension a try and let me know what you think!
Anas
·
·
Adnan thegtm.company Can we build this scraper using Clay?
Adnan thegtm.company
·
·
Anas Tom Elliott Instant data scraper would be the best way to go about this as Andrei Baloleanu (Adyntel) suggested because the pagination is being carried out using client side JavaScript, which Clay can't detect. If going from one page to another changes the URL in a logical manner, Clay can handle it.
Tom Elliott
·
·
Interesting
Channeled
APP
·
·
Hi Tom Elliott! This thread was recently closed by our Support team. If you have a moment, please share your feedback: