Troubleshooting 403 Error with Clay Scraper on Zillow Profiles

Nir Betan · 2024-10-17 15:31:34.644721+00

Hey Support - I'm currently using the clay scraper to scrape agent data on zillow profiles and it was working for quite a while, now i get a 403 error that says i'm blocked. Any advice on how to overcome this? i tried apify but there are no scrapers for zillow agent profile data there

Troubleshooting 403 Error with Clay Scraper on Zillow Profiles | Clay

14 comments

· Sorted by Oldest

Channeled
APP
·
·
👋 Hey there! Our support team has got your message - we'll be back in touch within 24 hours (often sooner!). If you haven't already, please include the URL of your table in this thread so that we can help you as quickly as possible!
Bruno (Clay)
·
·
Hi Nir, thanks for reaching out. Can you share the table you're working in for this specific setup?
Nir Betan
·
·
sure
Nir Betan
·
·
https://app.clay.com/workspaces/31186/tables/t_UfkgzyuWTT3P/views/gv_YNAqppPAUnjU
Nir Betan
·
·
as you can see - row 1 at the top
Nir Betan
·
·
we ran that earlier and it worked
Nir Betan
·
·
i had several other rows that worked as well in other tables but they've all stopped now
Bo (Clay)
·
·
Hey, thanks for reaching out! 😊 It sounds like the issue you’re experiencing is related to pages that block scraping, which is common for some sites. Zenrows is indeed a great option for handling this, and I see you’ve already got it installed in your table. I re-ran it as well, and it worked except for one instance. If you want to stick with Zenrows, you can use the AI Formula to extract data from the text similar as what you were trying to extract from Find Data From Page. Here’s a helpful guide on how to do that: How to Use AI Formulas. Let me know if you need more help! 😊
Nir Betan
·
·
this is so weird - so it like fails for a few hours and now it's working again. Very strange. Did you change anything on your end?
Nir Betan
·
·
i would use Zenrows BUT i have specific fields i need - is there a way to use zenrows to set up a scrape template
Nir Betan
·
·
similar to how clay scraper worked? it's much easier if i can target specific page elements
Nir Betan
·
·
otherwise i will need to use AI to parse out the entire HTML payload
Bo (Clay)
·
·
No, we didn’t change anything on our end. The issue is likely caused by scraping prevention measures. They may have temporarily blocked our actions, which can cause it to fail for a while and then work again. For the specific fields you need, you can use a formula to extract them (though parsing might take some time). That can be done via prompting. Another option is to use Zenrows Scrape and then feed the HTML payload into ChatGPT to extract the fields you want by specifying the outputs in the enrichment fields. You can also look into using the Zillow API: Zillow API on RapidAPI for similar structured data. Re-runing them after a little while should also get the data back without needing to do anything else Let me know if you need further help! 😊
Channeled
APP
·
·
Hi Nir Betan! This thread was recently closed by our Support team. If you have a moment, please share your feedback:

Channeled
APP
·
·
👋 Hey there! Our support team has got your message - we'll be back in touch within 24 hours (often sooner!). If you haven't already, please include the URL of your table in this thread so that we can help you as quickly as possible!
Bruno (Clay)
·
·
Hi Nir, thanks for reaching out. Can you share the table you're working in for this specific setup?
Nir Betan
·
·
sure
Nir Betan
·
·
https://app.clay.com/workspaces/31186/tables/t_UfkgzyuWTT3P/views/gv_YNAqppPAUnjU
Nir Betan
·
·
as you can see - row 1 at the top
Nir Betan
·
·
we ran that earlier and it worked
Nir Betan
·
·
i had several other rows that worked as well in other tables but they've all stopped now
Bo (Clay)
·
·
Hey, thanks for reaching out! 😊 It sounds like the issue you’re experiencing is related to pages that block scraping, which is common for some sites. Zenrows is indeed a great option for handling this, and I see you’ve already got it installed in your table. I re-ran it as well, and it worked except for one instance. If you want to stick with Zenrows, you can use the AI Formula to extract data from the text similar as what you were trying to extract from Find Data From Page. Here’s a helpful guide on how to do that: How to Use AI Formulas. Let me know if you need more help! 😊
Nir Betan
·
·
this is so weird - so it like fails for a few hours and now it's working again. Very strange. Did you change anything on your end?
Nir Betan
·
·
i would use Zenrows BUT i have specific fields i need - is there a way to use zenrows to set up a scrape template
Nir Betan
·
·
similar to how clay scraper worked? it's much easier if i can target specific page elements
Nir Betan
·
·
otherwise i will need to use AI to parse out the entire HTML payload
Bo (Clay)
·
·
No, we didn’t change anything on our end. The issue is likely caused by scraping prevention measures. They may have temporarily blocked our actions, which can cause it to fail for a while and then work again. For the specific fields you need, you can use a formula to extract them (though parsing might take some time). That can be done via prompting. Another option is to use Zenrows Scrape and then feed the HTML payload into ChatGPT to extract the fields you want by specifying the outputs in the enrichment fields. You can also look into using the Zillow API: Zillow API on RapidAPI for similar structured data. Re-runing them after a little while should also get the data back without needing to do anything else Let me know if you need further help! 😊
Channeled
APP
·
·
Hi Nir Betan! This thread was recently closed by our Support team. If you have a moment, please share your feedback: