.png){kind=logo}
Tips to Speed Up Enrichment Process for Large Datasets
Any tips on how to get an enrichment to run faster? I'm running a simple Lookup Single Row in Other Table on 10k rows and its been running for 20 minutes and still have 4k in queue.
To speed up your enrichment process for 10,000 rows, consider these strategies: 1. Use conditional runs: Apply filters or create a filtered view to enrich only the necessary rows. 2. Leverage existing data: Before running new enrichments, check if the data already exists in your CRM or another Clay table. Use our "write to table" feature to pull existing data, avoiding duplicate enrichments. 3. Process in smaller batches: If you're using external APIs (like Anthropic), try processing data in smaller batches of around 50 rows at a time to manage API limits. 4. Optimize your lookup criteria: Ensure you're using efficient lookup conditions to minimize processing time. Remember, only enrichment actions consume credits, so optimizing your process can save both time and resources.
Dave R.··nope
Bo (.··Hey Dave, Unfortunately, there aren't any user-side tricks to speed up your lookup operation. This kind of data task is resource-intensive, which explains the processing time for 10k rows. The bottleneck is on the architecture side, not something you can adjust. We're actively working on performance improvements that should roll out soon, but until then, we'll need to be patient with these larger operations. Very sorry for the wait time you're experiencing.