Thanks Bo. I tried using Apify for now making changes into Page Function for each directory and will hopefully schedule the runs and dedupe it. I will also try your method out to extract more data. Will update the thread if I face issues. Thanks for helping.