Implementing Sequential Runs for Apify Tasks to Avoid Errors

Sean McGrath · 2024-11-13 16:52:09.712539+00

I'm trying to run an apify task call from . When I run one row manually, it works. But, when I run the column, i get an error. Is there a way (setting in Apify or edit to my code) to implement sequential or staggered runs if multiple Actors are currently running in parallel? Appreciate any help!

Implementing Sequential Runs for Apify Tasks to Avoid Errors | Clay

owen (Clay)
·
·
Hey Sean, thanks for reaching out! Which Apify Actor are you running? also do you mind sending over the URL to this Clay table so we can take a closer look?
1
Sean McGrath
·
·
I'm running Contact Info Scraper. Sure, here it is: https://app.clay.com/shared-table/share_NJsxVU76eHVJ
owen (Clay)
·
·
Got it, do you minding sharing the URL to the Clay table where this error is occurring? This link appears to be the template used. Regardless, I believe you could use the maxConcurrency setting within Apify to control the maximum number of concurrent requests. By setting this to 1, you can enforce that only one request is made at at time which effectively makes this process sequential. In Apify, this would look like: maxConcurrency: 1,
Sean McGrath
·
·
Ok, here's the url: https://app.clay.com/workspaces/415777/tables/t_J8ZA76MCaBr7/views/gv_Dx6E4BYKz7qy
Sean McGrath
·
·
That makes sense. Would I add "maxConcurrency: 1," to the JSON?
Sean McGrath
·
·
Unfortunately, even w/ maxconcurrency:1 I'm still running into the memory limit error.
owen (Clay)
·
·
Got it, what does your JSON input look like right now? Apologies, it's sometimes challenging to troubleshoot without the full visibility into your side of Apify. I'd be curious if the following would work as JSON input: { "start_url": "https://www.example.com", "depth": 1, "maxConcurrency": 1 }
Sean McGrath
·
·
{ "considerChildFrames": false, "maxDepth": 2, "maxRequests": 1000, "maxRequestsPerStartUrl": 2, "sameDomain": true, "startUrls": [ { "url": "WEBSITE", "method": "GET" } ], "maxConcurrency": 1 }
owen (Clay)
·
·
Got it, as I'm checking this again it appears to be working in a proper manner. Most of these emails appear to be general business emails which I understand may have to be utilized for reaching out to businesses such as this. Which change did you make that allow you to get improved results? https://downloads.intercomcdn.com/i/o/w28k1kwz/1250750739/84a8827db3a8fc22f322e2b56e7f/CleanShot+2024-11-13+at+13_43_10.png?expires=1731525300&signature=3aa30feb359ea174edc59b5966014364ff25807006869f5d95a6fbb3c2f7d877&req=dSIiFs57nYZcUPMW1HO4zebfaKwx1J4XzRYRIUd692BYZ%2Fpab170p579h1ZO%0AirFV%0A
Sean McGrath
·
·
So, those are all rows I ran manually (one at a time). If I try to run multiple rows at once, it only runs 2 rowas, and gives errors for the rest of them...
owen (Clay)
·
·
Ah got it, thank you. Looking into this further
1
owen (Clay)
·
·
Hey Sean, after checking in with our team we've confirmed this is a limitation on Apify's end. Since Clay processes rows independently to Apify, you'll need to upgrade your Apify plan to handle concurrent requests. The current 8192MB memory limit on your Apify account is causing this error when multiple rows try to run simultaneously. You can check out Apify's pricing here: Apify Until you upgrade, you can: 1. Process fewer rows at a time 2. Split your data into smaller batches Hope this clears things up! Let me know if you have any other questions :)
1
Sean McGrath
·
·
Ah, okay, I really appreciate the time you took trying to help me find a solution! I appreciate it.
owen (Clay)
·
·
Of course! Let us know if you have any other questions or concerns :)
1
Channeled
APP
·
·
Hi Sean McGrath! This thread was recently closed by our Support team. If you have a moment, please share your feedback:

16 comments