Automating Batch Processing for API Records in Datasets

Googling and searching on this, but can't find any info: Is there a way to setup a job to run the "next" set of records/rows in a dataset, rather than the whole table? i.e. API limit might be 200 records every 1 hour - is there a way to get it to continue to run the remaining records every 1 hour without having to manually run it?

4 comments

Channeled
APP
·
·
Yes, we've recently introduced a scheduling feature that can help automate this process. You can set up scheduled columns to run your workflows automatically at your preferred frequency (daily, weekly, or monthly). To set this up: 1. Click the "⛭ icon" in your table 2. Under "Run the table," select "On a schedule" 3. Choose your frequency and whether to run all columns or only selected ones This feature allows you to update data and run workflows automatically, which can help manage API limits by processing a set number of records at regular intervals. However, it's worth noting that we don't currently have a feature to specifically process the "next" set of records. If you need more precise control, you might need to explore custom solutions or request additional features.
Sources: Clay Scheduling, Trigify integration, group column templates, Metaprompter updates, and more! Can I schedule data enrichments so they re-run on their own? - FAQ | Clay.com
Justin
·
·
This is limited to minimum once per day, need every hour for example.
Bo (.
·
·
Hey there,
Unfortunately, there's no built-in way to automatically run the "next" set of records based on API limits. Your only scheduling option is through the gear icon next to the action button, where you can set it to run once per day, week, or month.
This doesn't provide the hourly API limit workaround you're looking for though. The system currently processes the entire table when scheduled rather than picking up where it left off.
Let me know if you need any other clarification about scheduling options.
Channeled
APP
·
·

Automating Batch Processing for API Records in Datasets | Clay