David L.

Bo (. thanks for the update. Is there any other way to do this for a given set companies (with URL:s for the domain). Your approach is quite a lot of manual labor and not specifically for a given set of companies. Meaning, trying your approach I receive websites for huge companies such as H&M and Samsung. These would never be a potential prospect, as such the approach is not viable. My main goal is to find prospects in Clay, then for their domain (e.g., testsite.com) retrieve an answer if they have a noindex tag in their robot file (if they do have a robot file).

To clarify, given a set of URL:s (in most scenarios this would be the starting page). Is there a smooth way to find if the HTML code for the given URL has a noindex tag. Meaning, if I could find a reliable way to find if the nooindex is present for a given set of URL:s I would be more than happy. Hence, it does not matter if other pages on the website contains the noindex tag or not 🙂

Sure I could provide a list of URL:s, but the question is more general and not regarding a specific list, it could be whatever websites. I am just asking to see if anyone has found any luck in prompts or other ways to in a scalable way find websites with the noindex tag.

Many websites have what is know as robots tag in their HTML code. The purpose is to give information to crawlers for search engines (Google, Bing etc.) how to handle their website. One robot tag is defined as "noindex", which essentially means that the website should not be found in search results on Search engines.