Handling UK Companies House API Rate Limits for Large Data Sets

Hi Clay team, I'm running a table using the UK Companies House API to check for registered charges. The API itself works fine, but when I try to run the table with 10,000 rows, I get a timeout error. This is likely due to the Companies House rate limit of 600 requests per 5 minutes (or effectively 2 per 1000ms). I tried using the Custom Rate Limit feature to handle this, but it won’t let me set it to anything lower than 5 requests per 1000ms. A couple of questions:

1.
Is there any way to reduce the rate limit below 5 per 1000ms?
2.
What's the rationale behind not allowing lower limits? Many public APIs have fairly strict rate caps, so being able to run large datasets slowly and reliably seems like the whole point of this feature.
3.
Are there plans to address this feature change in the near future?

🎫1

7 comments

Channeled
APP
·
·
Clay
APP
·
·
Currently, our custom rate limit feature has a minimum setting of 5 requests per 1000ms and cannot be set lower. For APIs with stricter limits like UK Companies House (2 requests per second), you'll need to use workarounds like: ** Breaking datasets into smaller chunks (500-1000 rows) ** Running tables step-by-step with pauses * Using external tools like n8n for preprocessing We understand this limitation affects workflows with strict API rate limits, and your feedback about needing lower rate limit options is valuable for future improvements.
Source: HTTP API with JWT authentication | Documentation | Clay University
🤡1
Hugo D.
·
·
Agreed ! Will Y. I'm looking into solution with n8n and I am trying to implement the same work around but were not able to do it, would you be ok to share how you managed that ? (we can also see that over a 10 min call if you prefer)
Will Y.
·
·
Good responses guys, the data is around 900k rows so anything manual isn't feasible. In the end we just stuck into n8n with a custom flow (inc. a 5 minute wait time between each 600 pull), it worked perfectly, but it's still a shame you can't do this in Clay. Seems like an oversight from the clay team and hopefully they'll update it.
Hugo D.
·
·
Up on this request, I have the exact same problem, is there a way to increase the rate limit, to something like 1 call every 5sec ? Being able to batch-enrich with APIs seems to be the whole point of Clay I don't really understand why you guys would limit on such a key feature, having to go through another tool doesn't make much sense to me
Vinit
·
·
Will Y. Why not create a script in Python and get the data that you need from API and then import into Clay... that's what I do every time because I tried many other options and this is the easiest and accurate one.
Abdul S.
·
·
Hey Will Y. you're correct that the UK Companies House API has a strict rate limit (600 requests per 5 mins), which comes out to 2 requests per second the issue with clay custom Rate Limit setting not allowing anything below 5 per 1000ms is a bit of a blocker in cases like this. what I usually recommend (as someone who's pretty deep into clay workflows) is setting up a batch strategy break your dataset into chunks of 500–1000 rows and run them sequentially with pauses in between using the "run table step-by-step" or via webhook-triggered runs. It's a bit of a manual workaround, but it respects the API's limit and avoids timeouts. totally agree that being able to define stricter custom limits (like 1–2 per second) would make clay even more powerful, especially for public APIs like this hopefully something the team considers soon

Loading comments

Channeled
APP
·
·
Clay
APP
·
·
Currently, our custom rate limit feature has a minimum setting of 5 requests per 1000ms and cannot be set lower. For APIs with stricter limits like UK Companies House (2 requests per second), you'll need to use workarounds like: ** Breaking datasets into smaller chunks (500-1000 rows) ** Running tables step-by-step with pauses * Using external tools like n8n for preprocessing We understand this limitation affects workflows with strict API rate limits, and your feedback about needing lower rate limit options is valuable for future improvements.
Source: HTTP API with JWT authentication | Documentation | Clay University
🤡1
Hugo D.
·
·
Agreed ! Will Y. I'm looking into solution with n8n and I am trying to implement the same work around but were not able to do it, would you be ok to share how you managed that ? (we can also see that over a 10 min call if you prefer)
Will Y.
·
·
Good responses guys, the data is around 900k rows so anything manual isn't feasible. In the end we just stuck into n8n with a custom flow (inc. a 5 minute wait time between each 600 pull), it worked perfectly, but it's still a shame you can't do this in Clay. Seems like an oversight from the clay team and hopefully they'll update it.
Hugo D.
·
·
Up on this request, I have the exact same problem, is there a way to increase the rate limit, to something like 1 call every 5sec ? Being able to batch-enrich with APIs seems to be the whole point of Clay I don't really understand why you guys would limit on such a key feature, having to go through another tool doesn't make much sense to me
Vinit
·
·
Will Y. Why not create a script in Python and get the data that you need from API and then import into Clay... that's what I do every time because I tried many other options and this is the easiest and accurate one.
Abdul S.
·
·
Hey Will Y. you're correct that the UK Companies House API has a strict rate limit (600 requests per 5 mins), which comes out to 2 requests per second the issue with clay custom Rate Limit setting not allowing anything below 5 per 1000ms is a bit of a blocker in cases like this. what I usually recommend (as someone who's pretty deep into clay workflows) is setting up a batch strategy break your dataset into chunks of 500–1000 rows and run them sequentially with pauses in between using the "run table step-by-step" or via webhook-triggered runs. It's a bit of a manual workaround, but it respects the API's limit and avoids timeouts. totally agree that being able to define stricter custom limits (like 1–2 per second) would make clay even more powerful, especially for public APIs like this hopefully something the team considers soon