.Google's John Mueller addressed a concern about why Google indexes webpages that are actually forbidden from creeping through robots.txt as well as why the it's risk-free to neglect the related Explore Console records about those crawls.Bot Web Traffic To Concern Criterion URLs.The person asking the concern documented that crawlers were making hyperlinks to non-existent concern specification URLs (? q= xyz) to webpages along with noindex meta tags that are additionally shut out in robots.txt. What motivated the question is that Google is creeping the links to those pages, receiving shut out by robots.txt (without seeing a noindex robots meta tag) at that point receiving reported in Google.com Search Console as "Indexed, though obstructed through robots.txt.".The person asked the adhering to inquiry:." Yet listed here's the huge question: why would certainly Google.com index web pages when they can't even observe the information? What's the benefit because?".Google.com's John Mueller affirmed that if they can not crawl the web page they can't see the noindex meta tag. He additionally creates an appealing acknowledgment of the website: search driver, recommending to overlook the outcomes because the "average" users won't observe those end results.He wrote:." Yes, you're proper: if our team can't crawl the page, we can not observe the noindex. That claimed, if our company can not crawl the pages, then there's not a lot for our company to index. So while you might observe a few of those web pages with a targeted web site:- inquiry, the average consumer won't find all of them, so I would not bother it. Noindex is additionally alright (without robots.txt disallow), it merely suggests the URLs will definitely wind up being actually crawled (and end up in the Search Console document for crawled/not catalogued-- neither of these conditions induce problems to the rest of the internet site). The integral part is actually that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the constraints being used the Site: hunt evolved search operator for diagnostic reasons. One of those factors is actually considering that it's certainly not attached to the normal search mark, it's a different thing altogether.Google's John Mueller talked about the website hunt driver in 2021:." The short solution is that a web site: question is actually certainly not suggested to be total, neither utilized for diagnostics objectives.A site query is a specific sort of search that confines the end results to a particular web site. It's essentially simply words website, a colon, and after that the web site's domain.This question restricts the end results to a specific web site. It's certainly not indicated to be a detailed collection of all the pages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is actually great for these type of situations where a crawler is actually linking to non-existent pages that are actually acquiring found out by Googlebot.3. Links along with the noindex tag will produce a "crawled/not catalogued" entry in Browse Console and also those will not possess a damaging impact on the remainder of the web site.Review the question and respond to on LinkedIn:.Why would certainly Google.com index web pages when they can not even view the content?Included Image through Shutterstock/Krakenimages. com.