Troubleshooting AI Mismatch: Understanding Incorrect Results in Clay's AI Option

Arnaud · 2024-02-06T14:31:32.692Z

Hi there, I used the AI option in Clay with a prompt that runs on all my DB (± 4200 lines with GPT3.5 and my API key). The prompt is simple: from a name, tell me if there is a match with one of the 3 offered options. It seemed to work well and gave me all the results, but I realized that the answer was incorrect for many of them. And when I try again by clicking manually on each one, it gives me the correct answer. How can we explain that? Thanks a lot!

Troubleshooting AI Mismatch: Understanding Incorrect Results in Clay's AI Option | Clay

13 comments

· Sorted by Oldest

Arnaud
·
·
https://app.clay.com/workspaces/161457/tables/t_OvBkZNETqkc4/views/gv_gFknGzvLHtCB
Arnaud
·
·
For a concrete example you can try to run again line 931 (163401 with Ctrl+F)
Arnaud
·
·
Column "Company Match" (you can use my own credits)
Arnaud
·
·
Some other examples (you can use Ctrl+F to find them):
38501
138203
Arnaud
·
·
Here the answer should be 1 And if you run it again, you will have the correct answer
Clay Team
APP
·
·
Hey Arnaud!!
Clay Team
APP
·
·
Took a deeper look at this but thinking that since it's AI, it may have misunderstood the prompt in some of these rows. Perhaps if it's given a few examples it may be a lot better at determining what a match looks like though.
Arnaud
·
·
Hey Arturo O.! It's weird because if it didn't understand it, it would always give the wrong answer, no? But here, it seems that it only makes mistakes when it's done in bulk (it's always correct when I do it individually)
Arnaud
·
·
Don't you think that doing it in bulk has an impact on the AI performance?
Clay Team
APP
·
·
Sorry for the delay, Arnaud!
Clay Team
APP
·
·
Perhaps, I've seen similar reports before where people run calls in bulk so not sure if the AI's server just can't process some of them accordingly and runs into this issue. I do find that providing examples tends to help decrease these instances though, but there are always edge cases.As AI gets a lot better, this may not be an issue in coming months. 🤞
Arnaud
·
·
Ok thanks for the feedback Arturo O.! And if I add some examples, will it be more expensive to run if I use my own API? (does it make the prompt longer?) Thanks!
Clay Team
APP
·
·
It does make the prompt slightly longer but since it's just a few lines of text, the token count increase should be very minimal, which should only be a few cents.-> OpenAI's token calculator: https://platform.openai.com/tokenizer

Arnaud
·
·
https://app.clay.com/workspaces/161457/tables/t_OvBkZNETqkc4/views/gv_gFknGzvLHtCB
Arnaud
·
·
For a concrete example you can try to run again line 931 (163401 with Ctrl+F)
Arnaud
·
·
Column "Company Match" (you can use my own credits)
Arnaud
·
·
Some other examples (you can use Ctrl+F to find them):
38501
138203
Arnaud
·
·
Here the answer should be 1 And if you run it again, you will have the correct answer
Clay Team
APP
·
·
Hey Arnaud!!
Clay Team
APP
·
·
Took a deeper look at this but thinking that since it's AI, it may have misunderstood the prompt in some of these rows. Perhaps if it's given a few examples it may be a lot better at determining what a match looks like though.
Arnaud
·
·
Hey Arturo O.! It's weird because if it didn't understand it, it would always give the wrong answer, no? But here, it seems that it only makes mistakes when it's done in bulk (it's always correct when I do it individually)
Arnaud
·
·
Don't you think that doing it in bulk has an impact on the AI performance?
Clay Team
APP
·
·
Sorry for the delay, Arnaud!
Clay Team
APP
·
·
Perhaps, I've seen similar reports before where people run calls in bulk so not sure if the AI's server just can't process some of them accordingly and runs into this issue. I do find that providing examples tends to help decrease these instances though, but there are always edge cases.As AI gets a lot better, this may not be an issue in coming months. 🤞
Arnaud
·
·
Ok thanks for the feedback Arturo O.! And if I add some examples, will it be more expensive to run if I use my own API? (does it make the prompt longer?) Thanks!
Clay Team
APP
·
·
It does make the prompt slightly longer but since it's just a few lines of text, the token count increase should be very minimal, which should only be a few cents.-> OpenAI's token calculator: https://platform.openai.com/tokenizer