How to Scrape Programmatic Pages for Mortgage Rates by State

hey all, im looking to understand what the best way is to list out and scrape all of the programmatic pages that exist for a certain website? for example, if i wanted to scrape all of smartasset.com's programmatic pages that state the average 30 year fixed home mortgage rate across the 50 US states (one page updated daily per state, NY example here), what would be the best way to get that output in a clay table? I know how to run claygent neon to get the data, but not sure about the first step which is identifying all of the pages with their corresponding links themselves