It's me again !
I have a worflow where I scan linkedin job offers via AI to detect projects in line with the solution I'm selling.
The problem is, I'm scrapping the full DOM from the LinkedIn job page, which includes:
Duplicates of the same job description
Company marketing, policies, and benefits
it costs me too much tokens.
I was thinking about pre-scrapping the text and target only the main job description Div, but it often fail to scrap it.
What would you recommand ?