Aubry S.

Is it possible that the recent product updates on 11/26 modified the system prompt of these LLMs to constrain tool calls to only 1? I never saw these issues on runs prior to this

Row 11 with current filtering

LuisArturo Hi, here's an example of one that is still having this issue

Clay S. Can you please elaborate on the "recent updates to some of our AI integrations"? Would love a list of changes made

Sayli G. Further info supporting my theory that Clay has modified the system prompt or some other parameter to constrain the model

I've been trying to whittle down the number of failed rows. I made another "Use AI" column named "Web Search Error" that analyzed the Reasoning output from the Tax Prep US Status Use AI column and parsed if the web search failed or not. After filtering to the failed ones, I've re-ran the main LLM call and repeated a few times. Have now gone from 2.4k failed rows to only 129. I've ran nearly identical prompts in the past though with nowhere near this high of a failure rate, so I feel like something funky is going on

Sayli G. Sure thing! https://app.clay.com/workspaces/835672/workbooks/wb_0t6pr04zHam5QWGUMWv/tables/t_0t6r09ad7dDqieCQjRr/views/gv_0t6r09aoTbAEW96B9JP

Using Clay's key or your own? If your own, what usage tier are you?

Some examples of this

No. I use my own OpenAI API key and am Tier 4 so should not be hitting any rate limits. I am not hitting budget limits on these prompts. If anything, I'd love for them to use more tokens to respond more consistently/accurately.