Troubleshooting Apify Scraping Issues with Clay: Need Help!
Hi Clay support and community - got an issue. For context, I’ve been successfully using Clay and Apify for about a month now! Before I could let it run overnight and it would scrape data for 5000 leads plus without me having to touch it. But overtime it’s gradually gotten worse and worse to the point of now the Apify actor only scrapes data for just one lead before stopping. I’m in contact with Apify’s support team for the last half a week and we’re still struggling to solve this. They suggested I try increasing the memory within Apify. I went ahead and took their advice on adjusting the allocated memory for each run. I changed it from 4mb to 32mb (which is the maximum on my current paid plan with Apify). It was an improvement. Instead of it scraping only one before stopping, it scraped three before stopping. But obviously, that doesn’t come as much help if I have thousands of leads & websites to scrape. Something to note that I found strange, which made me think it could be a Clay issue rather than an Apify issue like I originally thought. I managed to set up a second Apify account with another email address (you get $5 free credits). I thought perhaps if I used a fresh Apify account I wouldn’t run into the same issue because it worked so well at the beginning with my first Apify account. But weirdly I did. Yet again, it was only able to scrape one or two and then couldn’t do anymore (approx $0.50 was spent before it stopped). Any ideas on how I can fix this problem? And whether it’s an Apify problem or Clay problem in your opinion? Thanks in advance 🙂