What this tool does
It finds and reads the robots.txt file on the domain of the URL you enter and works out whether that URL is blocked for the crawler you select. It also lists every declared sitemap and all rule groups in the file.
Rule selection follows the logic Google documents: the longest (most specific) matching rule wins, and where lengths tie, Allow takes precedence.
What robots.txt does and does not do
- It does: stop a crawler from crawling the page — the content is never downloaded.
- It does not: remove the URL from the index. With external links pointing at it, Google can keep a blocked URL in the index without content.
robots.txt and also putting noindex inside it. Because Google never reads a blocked page, it never sees the noindex, and the URL can stay in the index. The correct order: unblock first, let the noindex be seen, then block again once it has dropped out if you still want to.When to use it
- When launching a new site — to catch the
Disallow: /left over from staging (the most common disaster of all). - When a page is not being indexed — to check whether a robots block is the cause.
- After a CMS update — to confirm the file has not changed by accident.
- When managing crawl budget — to verify filter and search URLs are blocked correctly.
Frequently asked questions
What if there is no robots.txt?
Nothing bad — with no file, every URL is treated as open to crawling. It is not an error.
Is robots.txt enough to remove a page from the index?
No. To remove it, the page needs noindex and must not be blocked, or it must return 404/410. More: 404, 410 and soft 404.
Does the Sitemap line matter?
It is not required, but it helps — robots.txt is the first file every crawler reads, so declaring the sitemap there makes discovery easier.
Does Crawl-delay work?
Googlebot ignores that directive. Improving server response time is a far more effective way to influence crawl rate.
To check the sitemap file itself: Sitemap checker.