How to Scrape Full Text Behind See More Links with Claygent

Lara F. · 2025-07-22T19:46:25.470Z

I am using claygent to scrape a website for bio information. The problem is that it some of the text is behind a "see more" link. How can I get the full block of text?

How to Scrape Full Text Behind See More Links with Claygent | Clay

8 comments

· Sorted by Oldest

Clay
APP
·
·
Claygent can access and retrieve virtually any publicly available data on the internet, but it has limitations with content hidden behind interactive elements like "see more" links. Since Claygent is designed to scrape static content that's publicly available, it may not be able to interact with dynamic elements that require clicking to expand content. For better results with bio information, you could try being more specific in your Claygent prompt about looking for complete biographical information or full descriptions. The effectiveness of Claygent relies heavily on the context and instructions you provide, so mastering the art of prompting is key to unlocking its full potential. What specific website are you trying to scrape bio information from, and what does your current Claygent prompt look like?
Sources: Claygent: AI Web Scraper [CRM Enrichment] - Clay University Claygent: AI Web Scraper [Automated Outbound] - Clay University Claygent: AI Web Scraper [Limitless Research] - Clay University
Lara F.
·
·
https://www.zillow.com/profile/chrisklug
Lara F.
·
·
I would like to get the full text under Get to know Chris Klug
Clay
APP
·
·
For content hidden behind "see more" links, try ScrapeMagic instead of Claygent. ScrapeMagic can extract structured data from web pages and may handle dynamic content better. Alternatively, use the Clay Chrome Extension to scrape the page directly: 1. Open the extension on the target page 2. Click "Select Data to Add from Page" 3. Choose "Select a Single Attribute" 4. Map the bio text field you want to extract The Chrome extension can often capture expanded content that Claygent might miss.
Sources: Parse Data from URL with ScrapeMagic Clay Chrome extension | Documentation | Clay University Clay Chrome extension | Documentation | Clay University
Hari K.
·
·
Hey Lara! Once you click into the Claygent column, you should be able to hit "Add as Column" next to the field to access the full block of text.
Hope this helps!
Lara F.
·
·
It doesn't get the full block of text from the website because the full text is partially behind "see more"
Hari K.
·
·
Hey Lara! In that case, I believe you would have to use Clay Navigator as the model and specify to click the see more button before scraping the text. The other models won't be capable of clicking into actions on the website unless they are operators as well. Hope this helps!
Channeled
APP
·
·
This thread was picked up by our in-app web widget and will no longer sync to Slack. If you are the original poster, you can continue this conversation by logging into https://app.clay.com and clicking "Support" in the sidebar. If you're not the original poster and require help from support, please post in Clay Support.

Clay
APP
·
·
Claygent can access and retrieve virtually any publicly available data on the internet, but it has limitations with content hidden behind interactive elements like "see more" links. Since Claygent is designed to scrape static content that's publicly available, it may not be able to interact with dynamic elements that require clicking to expand content. For better results with bio information, you could try being more specific in your Claygent prompt about looking for complete biographical information or full descriptions. The effectiveness of Claygent relies heavily on the context and instructions you provide, so mastering the art of prompting is key to unlocking its full potential. What specific website are you trying to scrape bio information from, and what does your current Claygent prompt look like?
Sources: Claygent: AI Web Scraper [CRM Enrichment] - Clay University Claygent: AI Web Scraper [Automated Outbound] - Clay University Claygent: AI Web Scraper [Limitless Research] - Clay University
Lara F.
·
·
https://www.zillow.com/profile/chrisklug
Lara F.
·
·
I would like to get the full text under Get to know Chris Klug
Clay
APP
·
·
For content hidden behind "see more" links, try ScrapeMagic instead of Claygent. ScrapeMagic can extract structured data from web pages and may handle dynamic content better. Alternatively, use the Clay Chrome Extension to scrape the page directly: 1. Open the extension on the target page 2. Click "Select Data to Add from Page" 3. Choose "Select a Single Attribute" 4. Map the bio text field you want to extract The Chrome extension can often capture expanded content that Claygent might miss.
Sources: Parse Data from URL with ScrapeMagic Clay Chrome extension | Documentation | Clay University Clay Chrome extension | Documentation | Clay University
Hari K.
·
·
Hey Lara! Once you click into the Claygent column, you should be able to hit "Add as Column" next to the field to access the full block of text.
Hope this helps!
Lara F.
·
·
It doesn't get the full block of text from the website because the full text is partially behind "see more"
Hari K.
·
·
Hey Lara! In that case, I believe you would have to use Clay Navigator as the model and specify to click the see more button before scraping the text. The other models won't be capable of clicking into actions on the website unless they are operators as well. Hope this helps!
Channeled
APP
·
·
This thread was picked up by our in-app web widget and will no longer sync to Slack. If you are the original poster, you can continue this conversation by logging into https://app.clay.com and clicking "Support" in the sidebar. If you're not the original poster and require help from support, please post in Clay Support.