When claygent visits a website, can it crawl subpages ?

Josh B.: Yes. Can crawl and retrieve all information (in some instances also from .pdfs uploaded on e.g. career subpage). Exception: paywall / hidden content with necessary registration up-front