Issues with Clay's Redirect Page Enrichment for Company Links

Artur Grishkevich · 2024-11-23 10:29:12.244974+00

Hi all. I have a list of 5k companies whose domains are behind a redirect link. All links are in the same format. When I use the Clay enrichment `Find Redirect Page`, it returns the redirect page for about 70% of all links, but not for the other 30%. Which is super strange because the links are in the same exact format. e.g. it finds the target page for but not for . Here is the table: Appreciate your help!

Issues with Clay's Redirect Page Enrichment for Company Links | Clay

8 comments

· Sorted by Oldest

Channeled
APP
·
·
There could be several reasons why the Find Redirect Page tool might not work for some links despite having the same format: 1. Anti-scraping measures on certain websites might be blocking the redirect resolution. 2. The page might need additional time to load due to JavaScript execution. Try adding a scrape delay to allow the page to fully load.To improve your success rate, you could try: ** Using our Zenrows integration, which is better at bypassing anti-scraping measures ** Adding a delay of up to 10 seconds in your scraping settingsIf you could share specific examples of failing URLs, I can provide more targeted troubleshooting advice. Sources: Scrape Website with Clay Deep Dive: Zenrows Scraping [Limitless Research] - Clay University Find Redirect Page with Clay Scrapers
Adnan thegtm.company
·
·
Unfortunately, all of Clay's web research integrations (like the Claygent, for example) are too inconsistent to work at scale. Wonder when they'll fix it.
Channeled
APP
·
·
I understand your frustration, but I want to clarify something about our web research capabilities. Claygent Neon, our latest model, actually offers improved consistency and higher accuracy compared to previous versions. It performs with the highest accuracy on a representative sample of queries and has enhanced capabilities for handling complex web research tasks.For different types of web research needs, we offer a range of specialized tools. While Claygent is great for general web research, we have other tools like Xenrose for more complex tasks, and additional options for bulk sourcing. This allows you to choose the most appropriate tool for your specific use case.Our system can find most data points you can imagine, from categorizing companies to checking compliance status and finding specific business information. If you're experiencing specific issues, I'd be happy to help troubleshoot or recommend the best tool for your particular needs. Sources: Clay for enterprise teams | Clay.com Introducing Claygent Neon Deep Dive: Clay Scrapers [Limitless Research] - Clay University
Artur Grishkevich
·
·
Here is a failing url: https://hellosubscription.com/go/barkbox/.
Channeled
APP
·
·
Let me help troubleshoot the Find Redirect Page issue. Based on your earlier example and the tool's functionality, there are several potential reasons for the inconsistent results: 1. Anti-bot protection: Some websites may have stronger anti-scraping measures that block our redirect resolution attempts. 2. JavaScript dependencies: If the redirect relies on JavaScript execution, it may need additional time to process. 3. Rate limiting: When processing a large number of URLs (like your 5k companies), some websites might temporarily block requests.To improve your success rate, try these solutions: ** Add a scrape delay to allow proper page loading ** Use our Zenrows integration for better anti-bot bypass ** Process URLs in smaller batches to avoid rate limiting ** Double-check that the failing URLs are still active Source: Find Redirect Page with Clay Scrapers
Channeled
APP
·
·
Hi Artur Grishkevich! This thread was recently closed by our Support team. If you have a moment, please share your feedback:
Channeled
APP
·
·
Thank you so much for sharing your feedback Artur Grishkevich!
Bo (Clay)
·
·
Hey Arthur! Let me explain what's happening with those redirect links. Even though your links follow the same format, we're seeing different responses because each website has its own security policies. When we checked the backend for those failed attempts, we're getting a "Forbidden" error - basically some sites are blocking our access. These blocks usually come from: - Headless browser detection - Security policies - Access restrictions Quick tip: You can try re-running the enrichment on the failed ones - sometimes website protections fluctuate and you might catch more results during a different attempt.

Channeled
APP
·
·
There could be several reasons why the Find Redirect Page tool might not work for some links despite having the same format: 1. Anti-scraping measures on certain websites might be blocking the redirect resolution. 2. The page might need additional time to load due to JavaScript execution. Try adding a scrape delay to allow the page to fully load.To improve your success rate, you could try: ** Using our Zenrows integration, which is better at bypassing anti-scraping measures ** Adding a delay of up to 10 seconds in your scraping settingsIf you could share specific examples of failing URLs, I can provide more targeted troubleshooting advice. Sources: Scrape Website with Clay Deep Dive: Zenrows Scraping [Limitless Research] - Clay University Find Redirect Page with Clay Scrapers
Adnan thegtm.company
·
·
Unfortunately, all of Clay's web research integrations (like the Claygent, for example) are too inconsistent to work at scale. Wonder when they'll fix it.
Channeled
APP
·
·
I understand your frustration, but I want to clarify something about our web research capabilities. Claygent Neon, our latest model, actually offers improved consistency and higher accuracy compared to previous versions. It performs with the highest accuracy on a representative sample of queries and has enhanced capabilities for handling complex web research tasks.For different types of web research needs, we offer a range of specialized tools. While Claygent is great for general web research, we have other tools like Xenrose for more complex tasks, and additional options for bulk sourcing. This allows you to choose the most appropriate tool for your specific use case.Our system can find most data points you can imagine, from categorizing companies to checking compliance status and finding specific business information. If you're experiencing specific issues, I'd be happy to help troubleshoot or recommend the best tool for your particular needs. Sources: Clay for enterprise teams | Clay.com Introducing Claygent Neon Deep Dive: Clay Scrapers [Limitless Research] - Clay University
Artur Grishkevich
·
·
Here is a failing url: https://hellosubscription.com/go/barkbox/.
Channeled
APP
·
·
Let me help troubleshoot the Find Redirect Page issue. Based on your earlier example and the tool's functionality, there are several potential reasons for the inconsistent results: 1. Anti-bot protection: Some websites may have stronger anti-scraping measures that block our redirect resolution attempts. 2. JavaScript dependencies: If the redirect relies on JavaScript execution, it may need additional time to process. 3. Rate limiting: When processing a large number of URLs (like your 5k companies), some websites might temporarily block requests.To improve your success rate, try these solutions: ** Add a scrape delay to allow proper page loading ** Use our Zenrows integration for better anti-bot bypass ** Process URLs in smaller batches to avoid rate limiting ** Double-check that the failing URLs are still active Source: Find Redirect Page with Clay Scrapers
Channeled
APP
·
·
Hi Artur Grishkevich! This thread was recently closed by our Support team. If you have a moment, please share your feedback:
Channeled
APP
·
·
Thank you so much for sharing your feedback Artur Grishkevich!
Bo (Clay)
·
·
Hey Arthur! Let me explain what's happening with those redirect links. Even though your links follow the same format, we're seeing different responses because each website has its own security policies. When we checked the backend for those failed attempts, we're getting a "Forbidden" error - basically some sites are blocking our access. These blocks usually come from: - Headless browser detection - Security policies - Access restrictions Quick tip: You can try re-running the enrichment on the failed ones - sometimes website protections fluctuate and you might catch more results during a different attempt.