One broad Disallow line can block a whole section from Google or AI search. Robots.txt Tester fetches your robots.txt, or takes one you paste, and checks whether a URL is allowed for 13 crawlers: Googlebot, Bingbot, the AI search bots, the user-triggered AI fetchers and the AI training crawlers. You can add any custom user-agent.
Matching follows RFC 9309, the robots.txt standard, including group selection, wildcards, end anchors and longest-match precedence. For each crawler you see allowed or blocked, plus the rule and line number that decided it. The tool also lists Sitemap lines, the groups in the file and syntax problems. Paste a robots.txt to test changes before you publish them.
If robots.txt answers with a 5xx error, a 429 or a timeout, the tool reports everything as blocked, because RFC 9309 tells crawlers to treat it that way. A 4xx answer means no rules, so everything is allowed. The tool reads the first 500 KiB of the file, the limit RFC 9309 sets for parsers.
Get results in just a few simple steps
Enter a URL or domain, or paste a robots.txt and a path
Add a custom user-agent if you want to test one
The tool fetches robots.txt from that host, unless you pasted one
It parses the file and picks the group that applies to each crawler
It matches the path against Allow and Disallow rules by RFC 9309
You see allowed or blocked for each crawler, with the deciding rule and line
It lists Sitemap lines, groups and syntax problems
Don't make these frequent errors
Blocking important sections with overly broad Disallow rules
Testing only with Googlebot and ignoring AI crawlers
Using the wrong path format in tests
Assuming rule order decides, when the longest matching rule wins
No. Robots.txt blocks crawling. Google can still index a blocked URL without its content if other pages link to it. To keep a page out of the index, let it be crawled and add a noindex tag.
Yes. The test covers GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User, Google-Extended, Applebot-Extended and CCBot. You can also add any other user-agent.
Adjust the Disallow rules or add Allow exceptions for key paths.
Dive deeper into these topics with our comprehensive guides and templates.
Enter your domain for a free AI visibility audit. It asks AI search real buyer questions for your market and shows who it cites instead of you.