Rankwise logoRankwise
Free audit
PricingCompareResources
Sign inGet started

Footer

Rankwise

The infrastructure layer for AI search: measure what AI says, publish what wins, and pinpoint the fixes that get you cited.

Built for the answer engines

Product

  • Pricing
  • Templates

Solutions

  • For agencies
  • For in-house SEO
  • For founders
  • Use Cases
  • Integrations
  • Compare
  • Alternatives

Resources

  • Resources
  • Articles
  • Guides
  • Case Studies
  • Learn
  • Glossary
  • Topics
  • API

Company

  • About
  • Contact
ImpressumAGBDatenschutz

© 2026 Rankwise. All rights reserved. Built by Founder Ventures.

Tracking ChatGPT · Perplexity · Claude · Google AI Overviews
  1. Rankwise
  2. /AI crawlers
  3. /Claude-User
Anthropic · User-triggered fetch

Claude-User

Visits a website when a person asks Claude a question that needs it.

Source: privacy.claude.com
Check your robots.txt for AI crawlers

Operator

Anthropic

Job

User-triggered fetch

Visits a page because a person asked a question that needs it. This is the visit that can end in a citation.

robots.txt

Follows robots.txt

Anthropic says its bots honour standard robots.txt directives. Blocking Claude-User stops Claude retrieving your content for a user's question.

Source: privacy.claude.com
User agent

The token is all the vendor publishes

Anthropic documents the robots.txt token Claude-User but no full user-agent string. Match on the token.
robots.txt token
Claude-User
Source: privacy.claude.com
robots.txt

Allow or block Claude-User

Add one of these groups to the robots.txt at the root of your domain.

A crawler obeys only the most specific group that names it (RFC 9309), so a named group does not inherit the rules under User-agent: *. Repeat any paths you keep private, as the example does with /admin/.

Allow Claude-User
User-agent: Claude-User
Allow: /
Disallow: /admin/
Block Claude-User
User-agent: Claude-User
Disallow: /
What tryrankwise.com does

We name Claude-User and let it in

Read from the robots.txt we serve. We want AI engines to read and cite this site, so every AI crawler is allowed everywhere except the app, the API and the sign-in pages (/dashboard/, /api/, /login, /signup, /sign-in, /sign-up). The file also declares Content-Signal: ai-train=yes, search=yes, ai-input=yes.

tryrankwise.com/robots.txt
User-agent: Claude-User
Allow: /
Disallow: /dashboard/
Disallow: /api/
Disallow: /login
Disallow: /signup
Disallow: /sign-in
Disallow: /sign-up

See the whole file. The same rules repeat in every group, because a crawler that matches a named group ignores the wildcard one.

Other Anthropic crawlers
Claude-SearchBotSearch index
ClaudeBotModel training
AI engines it matters for
ClaudeHow it picks sources, and what Rankwise measures on it

Sources, checked 2026-09-24

  • privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
Check your own file

See which AI crawlers your robots.txt lets in.

The free AI crawler checker reads your robots.txt and reports which of the major AI crawlers it allows or blocks. No signup.
Check AI crawler accessOr start free →