Clay Icon

Building a Career Page Scraper in Clay with Claygent: Steps and Tips

Ā·
Ā·

I want to build a career page scrapper scenario in Clay. It should be with Claygent (AI Web researcher) to scrape career page? Can you share a loom on how to do that. Also, is there a way we can get this done in Clay without using credits.

  • Avatar of Arturo O.
    Arturo O.
    Ā·
    Ā·

    Hey Noor! Thanks for reaching out about this use case. Happy to help. Of course, would like to better understand what data you're looking to scrape from those pages as it really depends on that when you're tasking Claygent to achieve this. We have this video about what Claygent could do and how to potentially structure your prompts to retrieve the information you're looking for. You can do this with or without Clay credits. Claygent allows you to bring your own API keys with most of the AI models offered in it, which allow you to use your own credits with OpenAI or Anthropic, for example. Alternative scraping tools that don't use AI but can still scrape given links are Zenrows, which you can also use your API key with, or Clay's Scrape Website integration too. Hope this helps for now!

  • Avatar of Noor S.
    Noor S.
    Ā·
    Ā·

    Hey Arturo, I am looking to scrape jobs

  • Avatar of Owen C.
    Owen C.
    Ā·
    Ā·

    Hey Noor, got it! Iā€™d love to fully understand what success looks like here. A few questions come to mind: 1. Data to Scrape: Are you looking to scrape job titles, job descriptions, salary, other data points, or a combination of these? 2. Job Source: How do you plan on finding companies to target? 3. Why Clay? Have you considered using other scrapers to perform this action like Apify or Phantom Buster? (we integrate directly with these softwares) If so, what other data points, enrichment, or research are you hoping to get out of Clay? Iā€™d love to help you brainstorm this! Just want to make sure I fully understand your situation, preferred setup, and use case.

  • Avatar of Noor S.
    Noor S.
    Ā·
    Ā·

    1. Data to Scrape: Are you looking to scrape job titles, job descriptions, salary, other data points, or a combination of these? Yes, a combination of all. 2. Job Source: How do you plan on finding companies to target? Scrape websites like indeed, Glassdoor, LinkedIn to get the relevant job posting. 3. Why Clay? Have you considered using other scrapers to perform this action like Apify or Phantom Buster? (we integrate directly with these softwares) If so, what other data points, enrichment, or research are you hoping to get out of Clay? I've used Apify in the past but my leader wants me to use Clay for scrapping and later also document it how to use website scrapper in Clay. Maybe with claygent using Chatgpt option. Or something else that you way advice

  • Avatar of Bo (.
    Bo (.
    Ā·
    Ā·

    Then in that case, you can use our ā€œFind Jobsā€ feature. Hereā€™s how it works: 1. Find Jobs While Creating a New Table ā€¢ Use the Find Jobs Source Enrichment feature. ā€¢ This option uses our sources directly and keeps things simple. 2. Find Jobs Waterfall from the Enrichment Panel ā€¢ Click the ā€œAdd Enrichmentā€ button at the top-right of the page. ā€¢ Select the ā€œFind Jobsā€ waterfall option. 3. Common Ways to Use It: ā€¢ Option 1: ā€¢ Create a new workbook and select ā€œFind Jobs.ā€ ā€¢ Option 2: ā€¢ Create a new workbook and select ā€œFind Companies.ā€ ā€¢ Open a new workbook tab, then choose ā€œFind Jobsā€ and apply it to the companies you found earlier. ā€¢ Option 3: ā€¢ Find companies first. ā€¢ Go to the enrichment panel, click ā€œAdd Enrichment,ā€ type ā€œFind Jobs,ā€ and select the ā€œJobs Waterfallā€ option. 4. Credits and Sources: ā€¢ The ā€œJobs Waterfallā€ option checks multiple sources and uses credits for successful findings. ā€¢ The ā€œFind Jobsā€ option while creating a table checks only LinkedIn and is free. Let me know if youā€™d like more details or help setting this up! šŸ˜Š

  • Avatar of Channeled
    Channeled
    APP
    Ā·
    Ā·

    Hey there - just wanted to check in here to see if you needed anything else! Feel free to reply back here if you do.

  • Avatar of Channeled
    Channeled
    APP
    Ā·
    Ā·

    We haven't heard back from you in a bit, so we're going to go ahead and close things out here - feel free to let us know if you still need something!

  • Avatar of Channeled
    Channeled
    APP
    Ā·
    Ā·

    Hi Noor S.! This thread was recently closed by our Support team. If you have a moment, please share your feedback: