When claygent visits a website, can it crawl subpages ?

Josh Burgin: Yes. Can crawl and retrieve all information (in some instances also from .pdfs uploaded on e.g. career subpage). Exception: paywall / hidden content with necessary registration up-front