Troubleshooting Apify Data Import: Only 1k Records Retrieved

Marielle C. · 2024-11-28T08:39:37.691Z

Hi, I was trying to import data from Apify, everything is finished running. It should be 20k+ but its only getting 1k data,what am I missing? Thank you

Troubleshooting Apify Data Import: Only 1k Records Retrieved | Clay

6 comments

· Sorted by Oldest

Marielle C.
·
·
https://app.clay.com/workspaces/280547/tables/t_BvRiPwQoujMa/views/gv_YPiWoyt3httx
Channeled
APP
·
·
The Apify integration in Clay has a limit of 5,000 records per single run. If you're seeing only 1,000 records imported when you expected 20,000+, this is because you've hit this limitation. The integration is designed to retrieve up to 5,000 records from a single Apify actor run.To work around this limitation, you may need to: 1. Break down your data collection into smaller batches 2. Adjust your Apify actor's parameters to focus on specific subsets of data 3. Run multiple imports and combine the data in Clay Source: Pull Data from Apify Actor with Apify
Palash W.
·
·
You can use apify api directly and set limit to 20k.
Channeled
APP
·
·
You can use Clay's HTTP API integration and webhooks to handle larger data imports. Here's how: 1. For incoming data to Clay: • Use webhooks to automate data imports • Note that webhooks have a processing rate of 10 records per second, with a burst capacity of 20 records • Each webhook payload must be under 100KB 2. For outgoing data using HTTP API: • You can control the rate of requests using the rate limit settings • Specify the maximum number of allowed requests and the time window in milliseconds • This helps manage larger datasets while staying within API limitsRemember that while Clay doesn't have its own API endpoints, you can use webhooks for incoming data and the HTTP API integration for outgoing data to connect with your external services. Sources: HTTP API with JWT Authentication - Clay University Import Data to Clay Using Webhooks
Channeled
APP
·
·
Hi Marielle C.! This thread was recently closed by our Support team. If you have a moment, please share your feedback:
Channeled
APP
·
·
Thank you so much for sharing your feedback Marielle C.!

Marielle C.
·
·
https://app.clay.com/workspaces/280547/tables/t_BvRiPwQoujMa/views/gv_YPiWoyt3httx
Channeled
APP
·
·
The Apify integration in Clay has a limit of 5,000 records per single run. If you're seeing only 1,000 records imported when you expected 20,000+, this is because you've hit this limitation. The integration is designed to retrieve up to 5,000 records from a single Apify actor run.To work around this limitation, you may need to: 1. Break down your data collection into smaller batches 2. Adjust your Apify actor's parameters to focus on specific subsets of data 3. Run multiple imports and combine the data in Clay Source: Pull Data from Apify Actor with Apify
Palash W.
·
·
You can use apify api directly and set limit to 20k.
Channeled
APP
·
·
You can use Clay's HTTP API integration and webhooks to handle larger data imports. Here's how: 1. For incoming data to Clay: • Use webhooks to automate data imports • Note that webhooks have a processing rate of 10 records per second, with a burst capacity of 20 records • Each webhook payload must be under 100KB 2. For outgoing data using HTTP API: • You can control the rate of requests using the rate limit settings • Specify the maximum number of allowed requests and the time window in milliseconds • This helps manage larger datasets while staying within API limitsRemember that while Clay doesn't have its own API endpoints, you can use webhooks for incoming data and the HTTP API integration for outgoing data to connect with your external services. Sources: HTTP API with JWT Authentication - Clay University Import Data to Clay Using Webhooks
Channeled
APP
·
·
Hi Marielle C.! This thread was recently closed by our Support team. If you have a moment, please share your feedback:
Channeled
APP
·
·
Thank you so much for sharing your feedback Marielle C.!