I'm trying to scrape data via HTTP API in my Clay table so that the scraped data outputs as a JSON. 9/10 times the record can't run because I hit a rate limit. What's the most efficient solution for avoiding rate limit errors without manually running batches of cells?
Use the built-in rate limit controls in your HTTP API integration. In Step 9 of the HTTP API setup, define your Request Limit (maximum requests) and Duration (in ms) (time window). For example, set 10 requests per 1000ms (1 second) to stay within API limits. Clay is also deploying automatic retries and smarter pacing features to better handle rate limits, reducing errors and ensuring consistent performance without manual intervention. Can you copy and paste your current JSON configuration so I can help troubleshoot any specific formatting issues?
Follow redirects: ON Response timeout: 90000 Max retries: 6 Status codes to retry: 429, 500, 502, 503, 504
Your retry configuration looks good for handling rate limits (429 errors). However, you're missing the key rate limit controls. In your HTTP API setup, go to Step 9 and configure the Request Limit and Duration (in ms) fields. For example, set 5 requests per 2000ms to match your API's limits. This prevents hitting rate limits in the first place rather than just retrying after errors. Can you copy and paste your current JSON configuration so I can help troubleshoot any specific formatting issues?
OK, I just did that, but any number I put in the Duration, Clay returns "Rate limit too low"
The "Rate limit too low" error occurs because the rate limit must average at least 5 requests per second to work properly. For example, if you set 10 requests, your duration must be 2000ms or less (10 requests ÷ 2 seconds = 5 requests per second). If you set 5 requests, use 1000ms maximum. Can you copy and paste your current JSON configuration so I can help troubleshoot any specific formatting issues?
I can see you've attached an image, but I can't view the attachment content directly. To help troubleshoot your "Rate limit too low" error, please copy and paste your current JSON configuration from the HTTP API setup into the chat. The error typically occurs when the rate limit doesn't average at least 5 requests per second. For example, if you set 10 requests, your duration must be 2000ms or less (10÷2=5 requests/second).
Can the HTTP API step throttle below 5 RPS? Need guidance for scraper APIs (429s) I attempted to use Step 9 Custom rate limit, but the UI prevents saving values that average under 5 requests/second (I get “Rate limit too low”).
No, our HTTP API step cannot throttle below 5 requests per second. The rate limit must average at least 5 requests per second to work properly. For scraper APIs that require slower rates to avoid 429 errors, you'll need to use alternative approaches like processing data in smaller batches (e.g., 200 rows at a time) to stay within rate limits. Since you're getting JSON configuration issues, please copy and paste your current JSON configuration so I can help troubleshoot any specific formatting problems.
.png){kind=logo}
