Troubleshooting Runtime Errors During Snowflake Upsert Operations

Hi, we are running into issues where a lot of records will throw Runtime Error without any other details when we try to upsert into Snowflake. When we re-run them, some of them will complete successfully, while others will fail with the same error again. We are having to do re-run manually over and over until all the records are Upserted Successfully, which does eventually happen with a lot of retries. Can you take a look at what may be the issue here? Thanks!

31 comments

Max K.
·
·
we don't see any errors from the Snowflake side
Channeled
APP
·
·
👋 Hey there! Our support team has got your message - we'll be back in touch within 24 hours (often sooner!). If you haven't already, please include the URL of your table in this thread so that we can help you as quickly as possible!
Arturo O.
·
·
Hey Max, thanks for the message and reporting! Happy to check this out. Do you mind sending the link to the table so we can take a look? https://downloads.intercomcdn.com/i/o/1171717996/ac7729f84b426e1d0358af02/image.png?expires=1727396100&signature=5ec796fa02b9157ffac1a6ab2b332ccc86f4f0e4df3c4ad3bfd40d259dc9df43&req=dSEgF85%2FmohWX%2FMW1HO4zYTywCz%2FD5EX2BKsam%2FhVPsvMg%3D%3D%0A
Max K.
·
·
Sure, it has happened to pretty much all of our tables that leverage the snowflake upsert, but this is the latest one that we ran today and ran into the above issues. (They have since been all upserted after close to 10 retries) https://app.clay.com/workspaces/239721/tables/t_TxfcVu4dTJ8R/views/gv_sCFHV6NxQKex
Daniela D.
·
·
Hey Max! Sorry about this. Checking this with the team and will get back as soon as we know more. For context, cells usually returns an error if there's an issue with the format for objects mapped in the integration column. Do you have any table that still has errored cells? Happy to check the error message there to troubleshoot further.
Max K.
·
·
Hey Daniela, yep i'm aware of the different errors that happen when it's due to the format and have addressed those before as those have an error message associated with them. But these ones we are encountering are Runtime error that has no error details and would succeed on a next retry without any changes. I don't think we have any table atm that still has those errored cells as we re-ran them until they were all successful, but we can probably re-run few of them if you need to see the actual error
Arturo O.
·
·
Thank you for the context, Max! We shared this feedback with the team to keep an eye on it. Will try to reproduce this behavior but feel free to follow up if you do come across another example. I've attached a ticket to this conversation in case our engineers find and update us on this issue, so we can share with you. Thanks!
Max K.
·
·
Thanks Arturo! Will post back here if we run into it again. As an additional context, it happens when we try to sync over a lot of rows (thousands of rows)
Max K.
·
·
Hi, is there an update to this by any chance? we are trying to automate some of our workflow and this may block us from doing so. Thank you!
Bo (.
·
·
Hey Max, I’m really sorry for the delay! I did a follow-up on this, and we’re actively looking into it. I understand how important it is for your workflow automation, and we’ll get back to you as soon as we have an update. Thank you for your patience, and if there’s anything else I can assist with in the meantime, feel free to let me know! 😊
Bo (.
·
·
Could you share some examples via Loom? That would be really helpful. I just heard back from our team, and they’re missing some context. 😊
Max K.
·
·
Hi, i didn't create a Loom as it would not be useful for the most part as we wait for the cells to update and run into this issue, but I just re-ran it for the same table and I have left the cells with Runtime Error so that you can take a look. Feel free to re-run them if you need to on your end. Clay table with Runtime error (check the ❄️ Upsert Row): https://app.clay.com/workspaces/239721/tables/t_TxfcVu4dTJ8R/views/gv_sCFHV6NxQKex
Arturo O.
·
·
Thank you for sharing, Max! I made sure to check the logs about this issue and shared + pinged the engineering to continue the investigation. I'll follow up with any insights as soon as they dive into it.
Arturo O.
·
·
Appreciate your patience on this one, Max! The team let us know they've implemented some fixes for this, they're in the final process before releasing but will update you once we get confirmation this ticket was closed.
Max K.
·
·
That's a great news! thanks for following up with me. lmk when it's released
Arturo O.
·
·
For sure! We have that ticket attached to this conversation so we can quickly re-open and share any progress on them. It still says it's in review on our end but will follow up soon to confirm if it will take a bit longer or if it should already be working.
🙏1
Arturo O.
·
·
Great news! It's live. Let us know if you see any issues!
1
Max K.
·
·
Hi Arturo, I have just tried re-running the upsert for all the rows in that table, but are running into the same Runtime Error again unfortunately. I'm using the same table as the one I linked above
Arturo O.
·
·
Thanks for letting us know, Max! I'll check again with the team to confirm what the expected next steps would be after this update was released and share more soon. Appreciate you!
1
Arturo O.
·
·
Hey Max! I re-ran the errored rows since they don't retry on their own. Of course, let me know if any new rows that come in still run into that error.
Max K.
·
·
Got it. For reference, we don't actually need those synced at the moment. I was just using it to test out for this particular issue. We will have others soon that will require that we able to consistently run this automatically without having to retry manually, so would love an update on the issue when you get them!
Arturo O.
·
·
Thank you, Max! Totally understandable. Uncertain if there was a hiccup after the re-run but checked again with the team and they stated the issue should be fixed. If you're still experiencing this in that or different table, let me know. I'll make sure they have as many examples as they can consider to ensure the fix covers them all.
Max K.
·
·
Hi Arturo, I just force re-ran for all the rows in that table, and im still running into the Runtime Error unfortunately 😞. I can try it on a different table if you want me to, but it seems like it hasn't been fully resolved yet from my end
Arturo O.
·
·
Thanks for testing it! Curious if the Snowflake API is the one getting overwhelmed instead of Clay. When you run your projects, will they include several thousand rows running at once, or will it be more of a steady stream of a few coming in at different times? I'll share this feedback with the engineers to consider those scenarios and confirm if that's just a Snowflake limitation or something we can do within Clay.
Max K.
·
·
There are definitely a lot of them coming at the same time, but it seems like there's queueing from Clay side to ensure it's not submitting thousands of queries at once. From Snowflake's side, i dont' see any failures or issues (no queueing either) except for transaction blocking, which is expected since there are numerous queries trying to update the same table
Arturo O.
·
·
Thank you, Max! Already reported that so they can dive deeper. Just waiting on a follow up.
Arturo O.
·
·
Appreciate your patience, Max! I just got a confirmation that our team pushed a final fix, these errors should be mitigated now. If you try re-running, it should be able to handle the volume and performance overall!
Max K.
·
·
thanks for the update Arturo! I have tested it and it looks like it's been resolved and working with those errors. Appreciate all the follow-up and here here 🙏
Arturo O.
·
·
Wooo!! Nice, glad those final changes did the trick. Thank you for testing and confirming! Closing the loop for now, but do feel free to let us know if anything.
1
Channeled
APP
·
·
Hi Max K.! This thread was recently closed by our Support team. If you have a moment, please share your feedback:
Channeled
APP
·
·
Thank you so much for sharing your feedback Max K.!