Join the Community

Best Methods for Scraping LinkedIn Activity with Web Agents

·

·

Hey everyone, I wonder how I can best set up a reliable Web Agent to scrape LinkedIn activity. I was testing several approaches (Apify through API, Apify on their interface, Clay's "Find post" enrichment and Claygent) and the only reliable ones seem to be the filter in LinkedIn SalesNav and Phantombuster which both come with their own disadvantages. I would love if you could help me solve this use case but also if you could generally help me understand how Claygent works and when it might not be reliable. I was running several variations of prompt (see in thread) on GPT 4o and Claude Haiku but did not get consistent or always accurate results on the LinkedIn activity of that person. For example, the agents would output that there was no recent activity of that person while there actually was some (example in thread). Is it generally impossible to get this from LinkedIn?

4 comments

Alexander F.
·
·
Prompt 1: Visit the profile LinkedIn and see if there was any recent activity on the profile like posts, comments etc. Output "no" if there was no recent activity and "yes" if there was any. If you see under activity that it shows anything like "commented on a post", please include that as a "yes" If "yes", add ", date_last, date_second_last" where date_last is the latest activity and date_second_last is the date of the second latest activity.
Alexander F.
·
·
Two examples where the Claygent returned that there was no activity on the profile but it was actually wrong (>50% of cases): http://www.linkedin.com/in/jacques-holter-653285b7 http://www.linkedin.com/in/steffenzoernig
Mahdi G.
·
·
You can bypass that by using Phantombuster or Texau in automation platform like n8n than send the data back to a new table with a webhook -then you can leverage simply AI to analyze the text
Alexander F.
·
·
Thanks a lot guys! Good to know that Claygent cannot scrape LinkedIn reliably My main concern with Phantombuster were costs. I will also check out Texau - thanks for the poiner!