Issues with Claygent Analyzing Google Search Listings and Errors

I find that when I attempt to get Claygent to analyze search listings from a Google Search, even going so far as to feed it the URLs directly, it will often claim that "a URL cannot be accessed" or that it "returns a 405 error" when this is not the case. For example, with my table here: https://app.clay.com/workspaces/322200/tables/t_AKtrkMVpmkRh/views/gv_Fhxba7D2Yy4e With my column, "Claygent Ratings", I got the following attached answer from Claygent. However, for the Eating Well URL that I shared, https://www.eatingwell.com/article/8071288/magic-spoon-low-carb-high-protein-cereal-review/, the page is indeed accessible.

James C.
·
·
Note: I have since hid these columns in my table, but feel free to un-hide them in order to investigate
Bo (.
·
·
Hey James - Looks like that's because that's not done with the right approach. There are two effective approaches to resolve this. 1. Process URLs Individually: - Extract each URL using the "Extract Values from Data" feature with custom regex - Use "Write to Table" to send these URLs to a new table - Perform individual enrichments on each URL separately with Claygent 2. Alternative Scraping Approach: - Scrape individual URLs directly - Feed the scraped content to your preferred AI model (like ChatGPT or Claude) This can be good to ensure less hallucinations. Since different AI models have varying capabilities, it's worth testing each one (Claude, Clay's model, ChatGPT) to find which works best for your specific use case before proceeding with bulk analysis. This method helps avoid the "URL cannot be accessed" errors and provides more reliable results than trying to process multiple URLs simultaneously.
James C.
·
·
Very helpful! Thanks Bo (.
Channeled
APP
·
·
Hi James C.! This thread was recently closed by our Support team. If you have a moment, please share your feedback:

Issues with Claygent Analyzing Google Search Listings and Errors | Clay