Rankwise logoRankwise
Free audit
PricingCompareResources
Sign inContinue with GoogleStart free trial

Footer

Rankwise

Articles that get you found on Google and named by AI tools, checked every week.

One article a day, written for you

Product

  • Pricing
  • Templates
  • Free blog
  • Developers
  • WordPress guide

Solutions

  • For agencies
  • For in-house SEO
  • For founders
  • Use Cases
  • Integrations
  • Compare
  • Alternatives

Resources

  • Resources
  • Articles
  • Guides
  • Learn
  • Glossary
  • Topics
  • Free tools

Company

  • About
  • Contact

Free tools

SEO Tools

  • AI Overview Checker and Rank Tracker
  • AI Overview Content Score
  • AI Overview Opportunities Finder
  • AI Overview Readiness Score
  • AI Visibility Score
  • Claude Rank Tracker
  • Copilot Rank Tracker
  • Duplicate Titles & Metas Finder
  • E-E-A-T Signals Checker
  • Google Preferred Sources Button
  • Internal Link Finder
  • One-Page SEO Audit Tool
  • Perplexity Rank Tracker
  • Rich Results Checker
  • Search Intent Checker
  • Search Visibility Checker
  • SEO Ranking Report
  • SERP Snippet Preview
  • Topic Gap Checker

Content Tools

  • AI Snippet Extractor
  • AI vs SERP Gap Report
  • Answer Density Analyzer
  • Answer-First Formatting Checker
  • Citation Readiness Score
  • Entity Extractor
  • FAQ Builder
  • FAQ Schema Generator
  • Featured Snippet Optimizer
  • Headline Analyzer
  • Listicle Outline Generator
  • Meta Description Generator
  • Prompt Pack Generator
  • Snippet A/B Generator
  • Sourceability Score
  • Thin Content Detector
  • Title Generator
  • Title Tag Generator
  • YouTube Description Generator for AI Answers
  • YouTube Transcript Page Generator

Technical Tools

  • AI Crawler Checker
  • AI Visibility Checker
  • Broken Links Checker
  • Canonical Tag Validator
  • Facet Index Risk Checker
  • GPTBot Allowlist Tester
  • HTTP Header Checker
  • Indexability Checker
  • Internal Redirects Finder
  • LLM Bot Crawl Checker
  • llms.txt Generator
  • Mixed Content Checker
  • Orphan Pages Finder
  • Redirect Chain Checker
  • Robots.txt Tester
  • Robots.txt Tester for Google and AI Crawlers
  • Schema Markup Generator
  • Site Health Audit
  • Sitemap Validator

Business & ROI Calculators

  • Agency Content Margin Calculator
  • Competitor Content Gap Cost Calculator
  • Content Cost Comparator
  • Content Payback Period Calculator
  • Content ROI Calculator
  • Content Scaling Gap Calculator
  • GEO Readiness Score
  • GEO vs SEO Traffic Risk Estimator
  • Headcount vs Automation Calculator
  • Topical Authority Timeline Calculator
TermsPrivacy

© 2026 Rankwise. All rights reserved. Built by Founder Ventures.

Tracking ChatGPT · Perplexity · Claude · Google AI Overviews
  1. Home
  2. Tools
  3. Robots.txt Tester
Free Technical Tool

Robots.txt Tester

One broad Disallow line can block a whole section from Google or AI search. Robots.txt Tester fetches your robots.txt, or takes one you paste, and checks whether a URL is allowed for 13 crawlers: Googlebot, Bingbot, the AI search bots, the user-triggered AI fetchers and the AI training crawlers. You can add any custom user-agent.

Matching follows RFC 9309, the robots.txt standard, including group selection, wildcards, end anchors and longest-match precedence. For each crawler you see allowed or blocked, plus the rule and line number that decided it. The tool also lists Sitemap lines, the groups in the file and syntax problems. Paste a robots.txt to test changes before you publish them.

If robots.txt answers with a 5xx error, a 429 or a timeout, the tool reports everything as blocked, because RFC 9309 tells crawlers to treat it that way. A 4xx answer means no rules, so everything is allowed. The tool reads the first 500 KiB of the file, the limit RFC 9309 sets for parsers.

How It Works

Get results in just a few simple steps

1

Enter a URL or domain, or paste a robots.txt and a path

2

Add a custom user-agent if you want to test one

3

The tool fetches robots.txt from that host, unless you pasted one

4

It parses the file and picks the group that applies to each crawler

5

It matches the path against Allow and Disallow rules by RFC 9309

6

You see allowed or blocked for each crawler, with the deciding rule and line

7

It lists Sitemap lines, groups and syntax problems

Common Mistakes to Avoid

Don't make these frequent errors

Blocking important sections with overly broad Disallow rules

Testing only with Googlebot and ignoring AI crawlers

Using the wrong path format in tests

Assuming rule order decides, when the longest matching rule wins

Frequently Asked Questions

Does robots.txt block indexing?

No. Robots.txt blocks crawling. Google can still index a blocked URL without its content if other pages link to it. To keep a page out of the index, let it be crawled and add a noindex tag.

Can I test AI crawlers here?

Yes. The test covers GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User, Google-Extended, Applebot-Extended and CCBot. You can also add any other user-agent.

How do I fix accidental blocks?

Adjust the Disallow rules or add Allow exceptions for key paths.

See also

AI Visibility CheckerCheck if AI search engines can access your siteSchema Markup GeneratorCreate JSON-LD structured dataRobots.txt TesterTest URL access in your robots.txtSitemap ValidatorValidate an XML sitemap and test its URLsLLM Bot Crawl CheckerCheck AI crawler access to your site

Related Resources

Dive deeper into these topics with our comprehensive guides and templates.

Glossary Terms

robots tester

See how AI search reads your site

Enter your domain for a free AI visibility audit. It asks AI search real buyer questions for your market and shows who it cites instead of you.

Browse all tools