Fetch the robots.txt of any domain and read it the way a crawler would. You get every user-agent group and directive, whether the site is accidentally blocked, and the specific line to change when something is wrong.
The tool requests https://yourdomain/robots.txt and parses the response into user-agent groups and directives. It then reports on the things that actually go wrong in practice: a missing User-agent line, a wildcard group that disallows the whole site, rules that are silently ignored, malformed lines, and sitemap references that are not absolute URLs.
It also shows the file exactly as it was retrieved, so you can compare our reading against the raw text.
/robots.txt is fetched and the response is split into lines.User-agent lines share one group, and a new User-agent after a rule starts a new group.<meta name="robots"> tag or an X-Robots-Tag header, and it cannot tell you whether a page is actually indexed — that information exists only inside Google Search Console.User-agent token to its own name. This tool analyses the group for * because that is what Googlebot uses, but other crawlers may match a different group, and they do not all support the same directives.Your domain name is sent to the retrieval services listed in the Privacy Policy. This static site has no backend and stores no results.
* means “any crawler”. Two or more User-agent lines in a row share the rules that follow them.Disallow: /admin means “do not crawl anything under /admin”. Disallow: with an empty value means the opposite — allow everything. The longest matching rule wins, and Allow wins a tie.* matches any sequence of characters and $ means “end of the URL”. So Disallow: /*.pdf$ blocks every PDF, while Disallow: /*.pdf would also block file.pdf.html.Disallow: / inside the * group. Fix: delete that line, or re-allow the paths you want crawled with a more specific Allow line — for example Allow: / followed by Disallow: /admin/.User-agent:. Fix: add User-agent: * above your rules.Disallow path is more specific than your Allow path, and the longest match wins. Fix: make the Allow path at least as long as the blocking one.Disallow: /admin also covers /administrator, because robots.txt matches prefixes. Fix: write Disallow: /admin/ if you only mean that directory.Disallow rule asks a crawler not to fetch a URL. It does not remove that URL from a search index — a page blocked in robots.txt can still appear in results as a bare link. To keep a page out of results you need a noindex directive on the page itself, via a meta robots tag or an X-Robots-Tag header./robots.txt is completely valid, and is what most small sites have. This tool reports that as a warning rather than an error so you know which case you are in.