Hello Clay :claymation:! I’m looking to improve my Web Scraping skills...

Hello Clay ! I’m looking to improve my Web Scraping skills. I have a bunch of websites that I need to check for some custom links (all from the same domain). The current Clay Scraper is good but only seems to be working with links that are in the front page and does not go deeper into the site’s structure. Is there a way of searching deeper with Clay? 🤔

✅1

8 comments

Clay Team
APP
·
·
Hey hey! Can you drop the URL for your table here? (if applicable) 🫶 Don't worry, only admins will be able to open your table 😉
Casper S.
·
·
https://app.clay.com/workspaces/48942/tables/t_osJ2X6GbEAG6/views/gv_34w1Shu5iNaw
Clay Team
APP
·
·
Great question, Casper S.! So, we do have an integration that can help in some cases. For example, let's say you scrape a list of companies and the link it collects is a redirect to the company's profile in that site, which you also want to scrape further.You can create a new recipe in that subpage and then use Find Data from Page, which is another variation of a scraper and would detect the recipe for each link. Let me know if that sounds like what you were trying to achieve!
31d82d72-f638-4967-92eb-02bac96f7ef4.png(17 kB)
Clay Team
APP
·
·
Of course, the extension has not been prioritized but will be having updates soon! I know there are other scraping tools/extensions that do have pagination or can scrape subpages as well but not very familiar with them.
Adam E.
·
·
Hey Casper S. - have you tried using the Claygent for this?
Casper S.
·
·
Adam E. yes but these links are sometimes deeper in the filetree and AFAIU Claygent only scrapes root?
Adam E.
·
·
Casper S. - nope Claygent will look for links for you! It would help especially if you tell it what types of links you're looking for, can you try it out and let me know if it works?
Casper S.
·
·
OK, I will circle back with an example of where it didn’t work ✌️