.Google.com's John Mueller responded to a concern concerning why Google.com indexes pages that are actually forbidden from crawling through robots.txt as well as why the it's secure to dismiss the relevant Look Console files regarding those crawls.Crawler Web Traffic To Concern Criterion URLs.The individual talking to the concern recorded that robots were actually creating links to non-existent query criterion Links (? q= xyz) to webpages with noindex meta tags that are additionally blocked in robots.txt. What triggered the question is that Google is actually crawling the links to those webpages, acquiring blocked out through robots.txt (without envisioning a noindex robots meta tag) after that obtaining shown up in Google Explore Console as "Indexed, though shut out by robots.txt.".The individual asked the following question:." However listed below is actually the large inquiry: why will Google index web pages when they can not even view the information? What is actually the benefit because?".Google's John Mueller confirmed that if they can not crawl the webpage they can't find the noindex meta tag. He likewise helps make an appealing mention of the site: search driver, recommending to dismiss the end results considering that the "average" customers will not view those end results.He composed:." Yes, you're appropriate: if our experts can not creep the webpage, our company can't view the noindex. That pointed out, if our company can't crawl the web pages, then there is actually certainly not a lot for us to mark. Therefore while you could observe a few of those pages with a targeted website:- concern, the average individual won't find them, so I definitely would not bother it. Noindex is actually likewise alright (without robots.txt disallow), it merely suggests the Links will definitely wind up being crept (and end up in the Search Console record for crawled/not listed-- neither of these conditions create concerns to the remainder of the internet site). The vital part is that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the limits being used the Website: hunt accelerated search operator for analysis factors. Some of those factors is because it is actually not linked to the regular search mark, it is actually a different trait entirely.Google's John Mueller talked about the site hunt operator in 2021:." The short answer is that a website: inquiry is actually not indicated to be comprehensive, nor made use of for diagnostics functions.A site inquiry is actually a details type of search that confines the end results to a particular web site. It's generally just words web site, a bowel, and afterwards the web site's domain.This concern limits the outcomes to a details site. It is actually not meant to be a comprehensive selection of all the pages coming from that web site.".2. Noindex tag without using a robots.txt is fine for these type of situations where a bot is actually connecting to non-existent webpages that are obtaining found out through Googlebot.3. URLs along with the noindex tag will produce a "crawled/not indexed" item in Look Console and also those won't have an unfavorable result on the rest of the web site.Go through the concern as well as address on LinkedIn:.Why would Google mark webpages when they can not even find the material?Featured Picture through Shutterstock/Krakenimages. com.