Best Practices for Handling Async API Requests with Clay

Philip Nordenfelt · 2024-12-04 16:49:59.16872+00

Team - I have an external API that takes some time to return the results. It returns an immediate task_id and provides a status update API as well as an API for fetching the result. What is the best architectural approach with Clay for such scenarios? I don't see an option to set up async HTTP API requests nor do I see any callback URL options either. Curious how people have solved this and what the best way would be. Thanks!

Best Practices for Handling Async API Requests with Clay | Clay

Channeled
APP
·
·
For handling external APIs with asynchronous processing in Clay, here's the recommended approach: 1. Use Clay's webhook functionality to receive initial data and trigger your workflow. 2. Leverage Clay's HTTP API integration to: • Make the initial API call that returns a task ID • Create subsequent calls to check status • Fetch final results when readyYou can set up multiple HTTP API actions in sequence within your Clay workflow to handle the different API endpoints needed for task creation, status checking, and result retrieval. Source: HTTP API with Clay
Channeled
APP
·
·
I'm delighted to hear that! Remember, we're always here to assist you with any future questions or concerns.
Channeled
APP
·
·
Hi Philip Nordenfelt! This thread was recently closed by our Support team. If you have a moment, please share your feedback:
Channeled
APP
·
·
Thanks! We've reopened this thread. You can continue to add more detail directly in this thread.
Philip Nordenfelt
·
·
I understand the part about using the HTTP API action. But how do I "poll" the status with a HTTP action? Do I need to manually trigger the run of the column that is polling? Or can it be done on schedule?
Channeled
APP
·
·
You can use the auto-update feature in the HTTP API column to automatically poll for status updates. When enabled, the column will automatically update whenever you add a new row.To set this up: 1. Configure your HTTP API column with the status check endpoint 2. Enable the auto-update toggle in the column settings 3. Be aware that this will update whenever new rows are addedNote that while this feature is convenient, you should be mindful of potential stale data issues if you choose to disable auto-update.If you need more precise control, you can also set up conditional run criteria to determine specific circumstances under which the status check should run. Sources: HTTP API with JWT Authentication - Clay University HTTP API - Clay University
Philip Nordenfelt
·
·
When you say new rows, do you mean new rows in the sheet? Because I already have all of the rows populated, I'm enriching these rows which is what is being done with the HTTP API action. So no new rows will be added Column 1: HTTP action getting a task_id
Can I output the task_id to Column 2?
Column 2: HTTP action, set to auto-update, which will run whenever Column 1 outputs a new task_id? Column 3: HTTP action, runs api request to /result/{task_id}/ when Column 2 shows "completed" Is the above possible?
Channeled
APP
·
·
Yes, you can set this up using conditional runs in Clay. Here's how you can structure it: 1. Set up Column 1 to get your task_id 2. For Column 2 (status check), use conditional runs that reference the task_id from Column 1 3. For Column 3 (results), create a conditional run that only executes when Column 2 shows 'completed' statusThis type of workflow helps conserve credits and ensures efficient data processing by only running certain operations when specific conditions are met. You can use the AI formula generator to help create the exact conditional logic you need for your specific use case. Sources: Flexibly run any workflows based on conditional logic | Clay AI Formulas + Conditional Runs [CRM Enrichment] - Clay University
Philip Nordenfelt
·
·
That's pretty cool!
Channeled
APP
·
·
I'm thrilled you find it cool! We're always here to assist you with any questions or concerns.
Channeled
APP
·
·
Hi Philip Nordenfelt! This thread was recently closed by our Support team. If you have a moment, please share your feedback:
Channeled
APP
·
·
Thanks! We've reopened this thread. You can continue to add more detail directly in this thread.
Philip Nordenfelt
·
·
How do I set up the conditional run for Column 2? It now gets a task_id when Column 1 has fired. However, it will be processing for a few minutes so I need to poll the status somehow.
Philip Nordenfelt
·
·
Can I update a particular row via a webhook? If so I could send the webhook + id with the request and in turn the other API could use that to insert data on the correct row
Channeled
APP
·
·
Yes, you can set up webhooks in Clay to receive data. Here's how it works: 1. First, you'll need to set up a webhook as a data source in Clay 2. You'll get a unique webhook URL that can serve as the destination for your incoming data 3. You can secure the webhook by including an authentication token in the request header using 'x-clay-webhook-auth'Make sure to copy your authentication token when it's provided, as it can only be accessed once.Note: While webhooks can receive data into Clay, for your specific use case of updating particular rows, you'll want to ensure your incoming webhook payload contains the necessary identifiers to match with the correct rows in your table. Source: Webhooks - Clay University

19 comments