To clarify, given a set of URL:s (in most scenarios this would be the starting page). Is there a smooth way to find if the HTML code for the given URL has a noindex tag. Meaning, if I could find a reliable way to find if the nooindex is present for a given set of URL:s I would be more than happy.
Hence, it does not matter if other pages on the website contains the noindex tag or not 🙂