{"version":"1.0","provider_name":"Brainlabs","provider_url":"https:\/\/www.brainlabsdigital.com","author_name":"claudiu","author_url":"https:\/\/www.brainlabsdigital.com\/author\/claudiu\/","title":"Web-based robots.txt parser using Google's open-source Code - Brainlabs","type":"rich","width":600,"height":338,"html":"<blockquote class=\"wp-embedded-content\" data-secret=\"FTqzbFybH1\"><a href=\"https:\/\/www.brainlabsdigital.com\/web-based-robots-txt-parser-using-googles-open-source-code\/\">Web-based robots.txt parser using Google&#8217;s open-source Code<\/a><\/blockquote><iframe sandbox=\"allow-scripts\" security=\"restricted\" src=\"https:\/\/www.brainlabsdigital.com\/web-based-robots-txt-parser-using-googles-open-source-code\/embed\/#?secret=FTqzbFybH1\" width=\"600\" height=\"338\" title=\"&#8220;Web-based robots.txt parser using Google&#8217;s open-source Code&#8221; &#8212; Brainlabs\" data-secret=\"FTqzbFybH1\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" class=\"wp-embedded-content\"><\/iframe><script>\n\/*! This file is auto-generated *\/\n!function(d,l){\"use strict\";l.querySelector&&d.addEventListener&&\"undefined\"!=typeof URL&&(d.wp=d.wp||{},d.wp.receiveEmbedMessage||(d.wp.receiveEmbedMessage=function(e){var t=e.data;if((t||t.secret||t.message||t.value)&&!\/[^a-zA-Z0-9]\/.test(t.secret)){for(var s,r,n,a=l.querySelectorAll('iframe[data-secret=\"'+t.secret+'\"]'),o=l.querySelectorAll('blockquote[data-secret=\"'+t.secret+'\"]'),c=new RegExp(\"^https?:$\",\"i\"),i=0;i<o.length;i++)o[i].style.display=\"none\";for(i=0;i<a.length;i++)s=a[i],e.source===s.contentWindow&&(s.removeAttribute(\"style\"),\"height\"===t.message?(1e3<(r=parseInt(t.value,10))?r=1e3:~~r<200&&(r=200),s.height=r):\"link\"===t.message&&(r=new URL(s.getAttribute(\"src\")),n=new URL(t.value),c.test(n.protocol))&&n.host===r.host&&l.activeElement===s&&(d.top.location.href=t.value))}},d.addEventListener(\"message\",d.wp.receiveEmbedMessage,!1),l.addEventListener(\"DOMContentLoaded\",function(){for(var e,t,s=l.querySelectorAll(\"iframe.wp-embedded-content\"),r=0;r<s.length;r++)(t=(e=s[r]).getAttribute(\"data-secret\"))||(t=Math.random().toString(36).substring(2,12),e.src+=\"#?secret=\"+t,e.setAttribute(\"data-secret\",t)),e.contentWindow.postMessage({message:\"ready\",secret:t},\"*\")},!1)))}(window,document);\n<\/script>\n","thumbnail_url":"https:\/\/www.brainlabsdigital.com\/wp-content\/uploads\/2020\/11\/BL-Blog-Placeholder-V1.png","thumbnail_width":1136,"thumbnail_height":445,"description":"The punchline: I\u2019ve been playing around with a toy project recently and have deployed it as a free web-based tool for checking how Google will parse your robots.txt files, given that their own online tool does not replicate actual Googlebot behaviour. Check it out at realrobotstxt.com. While preparing for my recent presentation at SearchLove London, [&hellip;]"}