Troubleshooting Low Capacity: Daily SF Report Only Generating 2k Rows

Jamie M. · 2023-12-19 15:37:29.206883+00

Hey doing a daily sf report - its capping at 2,000, we need around 5-6k rows. Can someone let me know whats happening?

Troubleshooting Low Capacity: Daily SF Report Only Generating 2k Rows | Clay

7 comments

· Sorted by Oldest

Clay Team
APP
·
·
Hey Jamie! Took a quick look; tagging Colin P. to see if he can help us out if there's an issue with the import or if there's another limitation we can't see.
Jamie M.
·
·
thanks Arturo O., Colin P. - any help would be appreciated!
Clay Team
APP
·
·
hey Jamie M. - sorry for the super slow response time here! it's challenging to wrangle salesforce's documentation here (they don't list this out on the records endpoint), but according to their documentation on request headers, the maximum total number of records that can be returned from an API endpoint is 2000. it doesn't matter how you batch those calls; once you hit 2000 records, according to them, that's the limit: https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/headers_queryoptions.htm
Clay Team
APP
·
·
one way around this would be to create a date-created sort, and use that as a sneaky form of "pagination," deduping any results on the day when one sort hits the 2000 record limit and the next begins. if this would still be valuable to you, let me know and i can add this as a ticket to our to-dos!
Jamie M.
·
·
Hey Colin, thank you for this, this was really helpful. Any features would be great. One other question, when we return records from salesforce we’re limited to 5 records. I.e pulling a contact search it only 5 records even if 20 exist on an account. Do you know why it would do that
Clay Team
APP
·
·
1. will add it to the to-dos! 2. yep, lookup record is really only intended to be a 1-to-1 mapping, not to pull in all the results from a search. we've sort of made an exception / tried to be permissive of some 1-to-many use cases by allowing up to 5 to be pulled in, but if we let in more than 5, we'll start to break some customer workflows. if you want to segment out 20 records, i'd use the salesforce source and view, so you can properly load the records into the table
Clay Team
APP
·
·
hey Jamie M.! just merged a big change for this today. there should now no longer be a limit on the number of records you can pull into clay when creating a table from a salesforce view! (besides the maximum rows / clay table limit, of course)one caveat - if you're creating the table from a SOQL-incompatible salesforce view, you'll still be limited to 2000 records. SOQL-incompatible salesforce views are relatively rare; for documentation on them check out here: https://salesforce.stackexchange.com/questions/304733/what-is-issoqlcompatible-field-of-listview-being-used-for

Clay Team
APP
·
·
Hey Jamie! Took a quick look; tagging Colin P. to see if he can help us out if there's an issue with the import or if there's another limitation we can't see.
Jamie M.
·
·
thanks Arturo O., Colin P. - any help would be appreciated!
Clay Team
APP
·
·
hey Jamie M. - sorry for the super slow response time here! it's challenging to wrangle salesforce's documentation here (they don't list this out on the records endpoint), but according to their documentation on request headers, the maximum total number of records that can be returned from an API endpoint is 2000. it doesn't matter how you batch those calls; once you hit 2000 records, according to them, that's the limit: https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/headers_queryoptions.htm
Clay Team
APP
·
·
one way around this would be to create a date-created sort, and use that as a sneaky form of "pagination," deduping any results on the day when one sort hits the 2000 record limit and the next begins. if this would still be valuable to you, let me know and i can add this as a ticket to our to-dos!
Jamie M.
·
·
Hey Colin, thank you for this, this was really helpful. Any features would be great. One other question, when we return records from salesforce we’re limited to 5 records. I.e pulling a contact search it only 5 records even if 20 exist on an account. Do you know why it would do that
Clay Team
APP
·
·
1. will add it to the to-dos! 2. yep, lookup record is really only intended to be a 1-to-1 mapping, not to pull in all the results from a search. we've sort of made an exception / tried to be permissive of some 1-to-many use cases by allowing up to 5 to be pulled in, but if we let in more than 5, we'll start to break some customer workflows. if you want to segment out 20 records, i'd use the salesforce source and view, so you can properly load the records into the table
Clay Team
APP
·
·
hey Jamie M.! just merged a big change for this today. there should now no longer be a limit on the number of records you can pull into clay when creating a table from a salesforce view! (besides the maximum rows / clay table limit, of course)one caveat - if you're creating the table from a SOQL-incompatible salesforce view, you'll still be limited to 2000 records. SOQL-incompatible salesforce views are relatively rare; for documentation on them check out here: https://salesforce.stackexchange.com/questions/304733/what-is-issoqlcompatible-field-of-listview-being-used-for