{"id":7543,"date":"2019-11-23T11:24:26","date_gmt":"2019-11-23T11:24:26","guid":{"rendered":"https:\/\/www.brainlabsdigital.com\/?p=7543"},"modified":"2025-06-20T20:06:28","modified_gmt":"2025-06-20T20:06:28","slug":"web-based-robots-txt-parser-using-googles-open-source-code","status":"publish","type":"post","link":"https:\/\/www.brainlabsdigital.com\/web-based-robots-txt-parser-using-googles-open-source-code\/","title":{"rendered":"Web-based robots.txt parser using Google’s open-source Code"},"content":{"rendered":"

The punchline: I\u2019ve been playing around with a toy project recently and have deployed it as a free web-based tool for checking how Google will parse your robots.txt files, given that their own online tool does not replicate actual Googlebot behaviour. Check it out at realrobotstxt.com<\/a>.<\/p>\n

\"free<\/p>\n

While preparing for my recent presentation at SearchLove London<\/a>, I got mildly obsessed by the way that the deeper I dug into how robots.txt files work, the more surprising things I found, and the more places I found where there was conflicting information from different sources. Google\u2019s open source robots.txt parser should have made everything easy by not only complying with their newly-published draft specification<\/a>, but also by apparently being real production Google code.<\/p>\n

Two challenges led me further down the rabbit hole that ultimately led to me building a web-based tool<\/a>:<\/p>\n

    \n
  1. It\u2019s a C++ project, so needs to be compiled, which requires at least some programming \/ code administration skills, so I didn\u2019t feel like it was especially accessible to the wider search community<\/span><\/li>\n
  2. When I got it compiled and played with it, I discovered that it was <\/span>missing crucial Google-specific functionality<\/a> to enable us to see how Google crawlers like the images and video crawlers will interpret robots.txt files<\/span><\/li>\n<\/ol>\n

    Ways this tool differs from other resources<\/h2>\n

    Apart from the benefit of being a web-based tool rather than requiring compilation to run locally, my realrobotstxt.com tool<\/a> should be 100% compliant with the draft specification<\/a> that Google released, as it is entirely powered by their open source tool except for two specific changes that I made to bring it in line with my understanding of how real Google crawlers work:<\/p>\n

      \n
    1. Googlebot-image, Googlebot-video and Googlebot-news(*) should all fall back on obeying Googlebot<\/strong> directives if there are no rulesets specifically<\/em> targeting their own individual user agents – we have <\/span>verified<\/a> that this is at least how the images bot behaves in the real world<\/span><\/li>\n
    2. Google has a range of bots (AdsBot-Google, AdsBot-Google-Mobile, and the AdSense bot, Mediapartners-Google) which <\/span>apparently<\/a> ignore<\/strong> User-agent: * directives and only<\/em> obey rulesets specifically targeting their own individual user agents<\/span><\/li>\n<\/ol>\n

      [(*) Note: unrelated to the tweaks I\u2019ve made, but relevant because I mentioned Googlebot-news, it is very much not well-known that Googlebot-news is not a crawler<\/a> and hasn\u2019t been since 2011, apparently. If you didn\u2019t know this, don\u2019t worry – you\u2019re not alone. I only learned it recently, and it\u2019s pretty hard to discern from the documentation which regularly refers to it as a crawler. The only real official reference I can find is the blog post announcing its retirement<\/a>. I mean, it makes sense to me, because having different crawlers for web and news search opens up dangerous cloaking opportunities, but why then refer to it as a crawler\u2019s user agent throughout the docs? It seems<\/em>, though I haven\u2019t been able to test this in real life, as though rules directly targeting Googlebot-news function somewhat like a Google News-specific noindex<\/a>. This is very confusing, because regular Googlebot blocking does not<\/strong> keep URLs out of the web index, but there you go.]<\/p>\n

      I expect to see the Search Console robots.txt checker retired soon<\/h3>\n

      We have seen a gradual move to turn off old Search Console features<\/a> and I expect that the robots.txt checker will be retired soon. Googlers have recently been referring recently to it being out of step with how their actual crawlers work – and we can see differences in our own testing:<\/p>\n

      \"incorrect<\/p>\n

      These cases seem to be handled correctly by the open source parser – here\u2019s my web-based tool<\/a> on the exact same scenario:<\/p>\n

      \"robotstxt<\/p>\n

      This felt like all the more reason for me to release my web-based version, as the only official web-based tool we have is out of date and likely going away. Who knows whether Google will release an updated version based on their open source parser – but until they do, my tool<\/a> might prove useful to some people.<\/p>\n

      I\u2019d like to see the documentation updated<\/h3>\n

      Unfortunately, while I can make a pull request<\/a> against the open source code, I can\u2019t do the same with Google documentation. Despite implications out of Google that the old Search Console checker isn\u2019t in sync with real Googlebot, and hence shouldn\u2019t be trusted as the authoritative answer about how Google will parse a robots.txt file, references to it remain widespread in the documentation:<\/p>\n