Setting up a Recurring Web Scraper to Find RFP Opportunities

Hi, what would be the best way to set up a table to scrape the web for active RFP postings with specific keywords in them and return the links? (Ideally this would be a recurring search to help us get notified of new RFP opportunities for our business when they arise)

7 comments

Whenever W.
·
·
Table not applicable
Clay Team
APP
·
·
Hi, thanks for getting in touch!
Clay Team
APP
·
·
do you have a table with all the websites?
Whenever W.
·
·
Hi Juan, no. I’d likely have Claygent/OpenAI search the web for opportunities with certain keywords
Clay Team
APP
·
·
Check this example please: https://scribehow.com/shared/How_to_Enrich_Data_by_Scraping_a_Website_in_Claycom__lZHA_RoVTOqsqTqraVfUwQ
Whenever W.
·
·
Thanks, would it be best to use claygent for this?
Clay Team
APP
·
·
Hey Whenever W.!! Happy to chip in here. Since Claygent is not necessarily a source you can use to import unique records from a list, the next best option would likely be to find a dedicated site that would continuously update or scrape google searches. We have the integration to start a list with a Google Search...Otherwise, you could find a scraper that runs on the regular and collects any new results and auto-imports into Clay. Apify seems to be a marketplace of these scrapers so perhaps there's an option there.Here's a post about Apify: https://www.linkedin.com/posts/outboundphd_clay-added-1514-data-sources-and-you-didnt-activity-7153428101748727810-qQ1d?utm_source=share&utm_medium=member_desktop