Problem: When you run an AI-based function across 10 rows - the output is highly accurate and prompt adherence is strong. When you run this same function across hundreds or thousand of rows - the accuracy of the output is reduced and prompt adherence is week - due the processing of the prompts across a small n (perhaps 1) of LLM instances creating context window overload.
Put another way: spreading 10 requests across 1 LLM instance yields better accuracy than spreading 400 requests across 1 LLM instance.
Sub problem: I need to run this function and achieve the same accuracy when running for 60k rows as I get for 10 rows
Feature request #1: Feature to opt out of shared LLM instance across multiple rows e.g. use a distinct LLM instance for each row
Feature request #2: Feature to batch 10,20,30 rows per run and automatically run batches in sequence (perhaps conditional run logic to only run when previous 10 have finished processing)
Mitch H. - let me know if this is clear
