.Google's John Mueller responded to an inquiry concerning why Google.com marks web pages that are prohibited from creeping by robots.txt as well as why the it is actually risk-free to neglect the associated Explore Console documents concerning those crawls.Bot Web Traffic To Inquiry Guideline URLs.The individual asking the question chronicled that bots were producing links to non-existent query guideline Links (? q= xyz) to web pages with noindex meta tags that are likewise obstructed in robots.txt. What urged the inquiry is that Google is actually crawling the links to those web pages, receiving blocked out by robots.txt (without envisioning a noindex robotics meta tag) then getting shown up in Google Look Console as "Indexed, though blocked out by robots.txt.".The person talked to the observing concern:." But listed here's the major question: why will Google index pages when they can't even find the web content? What is actually the conveniences because?".Google.com's John Mueller verified that if they can not crawl the webpage they can't view the noindex meta tag. He additionally creates an exciting acknowledgment of the web site: search driver, advising to neglect the end results given that the "common" customers will not find those end results.He created:." Yes, you are actually correct: if we can't crawl the webpage, our company can't observe the noindex. That stated, if we can't creep the web pages, after that there is actually not a lot for us to index. Therefore while you could observe a few of those webpages along with a targeted web site:- question, the normal consumer will not observe them, so I would not fuss over it. Noindex is additionally alright (without robots.txt disallow), it just implies the Links will certainly end up being actually crawled (as well as end up in the Search Console report for crawled/not indexed-- neither of these statuses trigger problems to the remainder of the site). The fundamental part is that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's solution confirms the restrictions being used the Website: search advanced search driver for analysis factors. One of those main reasons is actually since it is actually certainly not attached to the normal hunt index, it is actually a separate point entirely.Google's John Mueller talked about the site search driver in 2021:." The short answer is that a website: inquiry is certainly not implied to be comprehensive, neither made use of for diagnostics reasons.A site question is actually a particular type of search that limits the outcomes to a certain web site. It's basically only words web site, a digestive tract, and afterwards the internet site's domain name.This concern limits the outcomes to a details site. It is actually not indicated to become a thorough compilation of all the web pages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is actually great for these kinds of scenarios where a bot is actually connecting to non-existent webpages that are actually obtaining found out through Googlebot.3. URLs with the noindex tag are going to produce a "crawled/not listed" item in Search Console and also those will not have an adverse effect on the rest of the site.Read the question as well as respond to on LinkedIn:.Why would certainly Google mark pages when they can not even see the content?Included Photo through Shutterstock/Krakenimages. com.