Hey, I’ve built a few working scrapers, but when the data gets imported into Clay daily, it reimports the entire dataset, not just the new rows. Not sure if the problem is with:
the Apify actor
the scheduled task in Apify
or the integration setup between Apify and Clay.
Would really appreciate some help troubleshooting this!
When using Apify with Clay, to import only new data rather than the entire dataset each time, make sure you don't specify a particular run as an input. Instead: 1. In the Source Modal, select Apify (under "other") 2. Leave the "Get Data from Specific Run" option unchecked 3. Don't specify a particular run ID This setup will automatically update your table with only new data from subsequent future runs of your actor, refreshing every 24 hours.
I thought I did, so I will try again with a new table. or do I have way to update tis setting after creating my table?
Yes, you can update your table's auto-update settings after creation. To modify how your Apify integration imports data: 1. Click on your table name in the top bar 2. Select "Enable auto-update" or "Disable auto-update" from the dropdown menu This controls whether Clay automatically processes new data coming in from your Apify integration.
Can I update the the APify integration? not the auto-update
To update your Apify integration settings for an existing table: 1. Go to Settings > Connections in your Clay dashboard 2. Find and click on the Apify integration 3. Click the "..." next to your Apify account 4. Select "Edit" to modify your account details This allows you to update your API key or rename your account.
just checked, I had toggled off “Get Data from Specific Run.”
Hey, glad to hear that toggling “Get Data from Specific Run” helped. Another option you might consider is using webhooks — that way, the data can be pushed directly into Clay only when new runs are completed. Let me know if you’d like help setting that up.