Is my robots.txt blocking Google?

Enter your domain and the path in question. The tester reads your live robots.txt, applies Google’s matching rules and names the line that decides. A blanket Disallow: / under User-agent: * is the rule that empties a site out of the index.

Why do Googlebot and other crawlers get different answers?

A group written for a named crawler replaces the * group for that crawler rather than adding to it. A Googlebot group with one rule means the * rules no longer apply to Google at all, which is why this tester answers for three crawlers at once.

Does robots.txt keep a page out of Google?

No. robots.txt controls crawling, not indexing. A blocked URL can still appear in results without a description if other pages link to it. To keep a page out of the index, allow crawling and serve a noindex meta tag or X-Robots-Tag header.

What happens if robots.txt returns a 500?

Google treats a 5xx on robots.txt as a reason to pause crawling the whole site, because it cannot tell what it is allowed to fetch. A 404 is safe and means everything is allowed.

Free tool, no signup

robots.txt tester

Enter a site, and the path you want to check. This reads the live robots.txt, applies the same longest-match rules Google applies, and tells you whether the URL is allowed, which line decided it, and what the file is silently ignoring.

Answered for Googlebot, Bingbot and our own crawler, because they do not always get the same answer.

How the matching actually works

Three rules explain almost every robots.txt surprise.

The most specific group wins outright. A crawler uses exactly one group: the one naming it, if there is one, otherwise the * group. Rules are not merged across groups. Add a Googlebot group to set a crawl-delay and you have just removed every other rule from Google's view.

The longest matching pattern wins, not the first. Order in the file is irrelevant. With Disallow: /admin and Allow: /admin/public, the path /admin/public/page is allowed because the matching Allow is longer.

Anything not disallowed is allowed. There is no need to write Allow rules for the pages you want crawled, and a file with no rules at all permits everything.

User-agent: *
Disallow: /admin
Allow: /admin/public
Disallow: /*?sort=

Sitemap: https://example.com/sitemap.xml

The wildcard * matches any run of characters and $ anchors the end, so Disallow: /*.pdf$ blocks PDFs and nothing else.

Directives that look official and do nothing

noindex. Google stopped obeying it in robots.txt in September 2019. Pages relying on it quietly returned to the index. Use a meta tag or header on the page.

crawl-delay. Bing and Yandex read it. Google ignores it entirely; crawl rate is a Search Console setting.

host. Was a Yandex directive for the preferred mirror. Nothing else has ever used it.

The tester flags each of these with the line number, because a directive that does nothing is worse than no directive: it makes you think the problem is handled.

After the tester

A single path is the question you usually arrive with. The question underneath is normally whether anything important on the site is blocked, and that needs the whole site. A crawl walks your internal links and records what was fetched, what was skipped, and why, including every URL your robots.txt turned away.

Questions people ask

Is my robots.txt blocking Google?: Enter your domain and the path you are worried about. The tester reads your live robots.txt, applies Google's matching rules, and names the line that decides. A blanket Disallow: / under User-agent: * is the one that empties a site out of the index, and it usually arrives with a staging deployment.
Why do Googlebot and other crawlers get different answers?: Because a group written for a named crawler replaces the * group for that crawler rather than adding to it. If you write a Googlebot group with one rule in it, every rule in your * group stops applying to Google. That surprises almost everybody, so this tester answers for three crawlers at once.
Does robots.txt keep a page out of Google?: No, and this is the most expensive misunderstanding in SEO. robots.txt controls crawling. A blocked URL can still be indexed, without a description, if other pages link to it. To keep a page out of the index you have to let it be crawled and serve a noindex robots meta tag or X-Robots-Tag header.
What happens if robots.txt returns a 500?: Google treats a server error on robots.txt as a reason to pause crawling the whole site, on the grounds that it cannot tell what it is allowed to do. A 404 is safe and means everything is allowed. A 500 is not safe, and this tester calls it out.

Other free tools here

XML sitemap validator — Find a site’s sitemap, check it parses, and report the entries search engines will reject.
Redirect chain checker — Follow a URL hop by hop and show every redirect between the address you typed and the page that answers.
Hreflang checker — Read the hreflang annotations on a page and check the language codes, the self-reference and x-default.
All the free tools — one page each, no account on any of them.
Download a whole crawl as SQLite — when one page is not enough and you want the data itself.

One page is a start. Crawl the whole site.

These tools check a single address. The Site Spider crawls up to 5,000 pages and runs fifteen checks over all of them, still free and still without an account.

Scan a site, free