.png){kind=logo}
Web Scraper Tool Issues: Unable to Pull Company Name and Website
Hello Team, I am working on web scraper tool in Clay and trying to pull the company name and website from the given portfolio link but It is not working properly.
Hello Team, I am working on web scraper tool in Clay and trying to pull the company name and website from the given portfolio link but It is not working properly.
I see - For this use case, you would need to approach this in a different way. Since each page has sub-pages and within those, they have more information on the company, you would need to do this like this: 1. Use our Chrome Extension to scrape the page and have all the sub-page links, export it to a new clay table. 2. Use our "Find Data from Page" enrichment and the recipe within our Chrome Extension to extract the data from the sub-page. This will allow you to get all their company information from their pages. It seems like they don't have a domain, so I would recommend using a "Domain" waterfall found in the enrichment panel by clicking the "add enrichment" button at the top right of the page as well.
Let me know what you think of doing it like this instead: 1. First table: Scrapethe portfolio pages and use "Write to Other Table" to collect all URLs and their source companies (make sure to send a value that can be matched in the "other data from this table part" 2. Second table: Add "Scrape Website" and set a rule to scrape non-root domains (not domain.com format). That can be specified in the Run Conditions with the AI Formula. E.g don't run if /domain isn't a root domain. Use "Write to Other Table" again to: - Send root domains to table 3 - Include source company data 3. Third table: You'll have your collection of root domains with data - filter and clean as needed 4. Return to table 1 and use "Lookup" to match companies from table 3, bringing in all discovered root URLs
Hey Sachin. Thanks for the reply! I added an example workflow to your table and made a quick video walking through the steps we took: https://www.loom.com/share/fc275a6593424c7f9727f4458fd5d952 Let me know if you have any questions!