.png){kind=logo}
How to Scrape Full Text Behind See More Links with Claygent
I am using claygent to scrape a website for bio information. The problem is that it some of the text is behind a "see more" link. How can I get the full block of text?
I am using claygent to scrape a website for bio information. The problem is that it some of the text is behind a "see more" link. How can I get the full block of text?
Claygent can access and retrieve virtually any publicly available data on the internet, but it has limitations with content hidden behind interactive elements like "see more" links. Since Claygent is designed to scrape static content that's publicly available, it may not be able to interact with dynamic elements that require clicking to expand content. For better results with bio information, you could try being more specific in your Claygent prompt about looking for complete biographical information or full descriptions. The effectiveness of Claygent relies heavily on the context and instructions you provide, so mastering the art of prompting is key to unlocking its full potential. What specific website are you trying to scrape bio information from, and what does your current Claygent prompt look like?
I would like to get the full text under Get to know Chris Klug
For content hidden behind "see more" links, try ScrapeMagic instead of Claygent. ScrapeMagic can extract structured data from web pages and may handle dynamic content better. Alternatively, use the Clay Chrome Extension to scrape the page directly: 1. Open the extension on the target page 2. Click "Select Data to Add from Page" 3. Choose "Select a Single Attribute" 4. Map the bio text field you want to extract The Chrome extension can often capture expanded content that Claygent might miss.
Hey Lara! Once you click into the Claygent column, you should be able to hit "Add as Column" next to the field to access the full block of text.
Hope this helps!
It doesn't get the full block of text from the website because the full text is partially behind "see more"
Hey Lara! In that case, I believe you would have to use Clay Navigator as the model and specify to click the see more button before scraping the text. The other models won't be capable of clicking into actions on the website unless they are operators as well. Hope this helps!
This thread was picked up by our in-app web widget and will no longer sync to Slack. If you are the original poster, you can continue this conversation by logging into https://app.clay.com and clicking "Support" in the sidebar. If you're not the original poster and require help from support, please post in 02 Support.