Rankwise logoRankwise
Free audit
PricingCompareResources
Sign inGet started

Footer

Rankwise

The infrastructure layer for AI search: measure what AI says, publish what wins, and pinpoint the fixes that get you cited.

Built for the answer engines

Product

  • Pricing
  • Templates

Solutions

  • For agencies
  • For in-house SEO
  • For founders
  • Use Cases
  • Integrations
  • Compare
  • Alternatives

Resources

  • Resources
  • Articles
  • Guides
  • Case Studies
  • Learn
  • Glossary
  • Topics
  • API

Company

  • About
  • Contact
ImpressumAGBDatenschutz

© 2026 Rankwise. All rights reserved. Built by Founder Ventures.

Tracking ChatGPT · Perplexity · Claude · Google AI Overviews
  1. Rankwise
  2. /AI crawlers
  3. /OAI-SearchBot
OpenAI · Search index

OAI-SearchBot

OpenAI's search crawler. It is used to surface websites in the search features of ChatGPT.

Source: developers.openai.com
Check your robots.txt for AI crawlers

Operator

OpenAI

Job

Search index

Builds the index an AI answer searches. Block it and you drop out of that product's results.

robots.txt

Follows robots.txt

OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.

Source: developers.openai.com
User agent

How it identifies itself

Copied from the vendor's documentation. Version numbers inside these strings change over time, so match on the token rather than the whole string.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Source: developers.openai.com
robots.txt

Allow or block OAI-SearchBot

Add one of these groups to the robots.txt at the root of your domain.

A crawler obeys only the most specific group that names it (RFC 9309), so a named group does not inherit the rules under User-agent: *. Repeat any paths you keep private, as the example does with /admin/.

Allow OAI-SearchBot
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Block OAI-SearchBot
User-agent: OAI-SearchBot
Disallow: /
What tryrankwise.com does

We name OAI-SearchBot and let it in

Read from the robots.txt we serve. We want AI engines to read and cite this site, so every AI crawler is allowed everywhere except the app, the API and the sign-in pages (/dashboard/, /api/, /login, /signup, /sign-in, /sign-up). The file also declares Content-Signal: ai-train=yes, search=yes, ai-input=yes.

tryrankwise.com/robots.txt
User-agent: OAI-SearchBot
Allow: /
Disallow: /dashboard/
Disallow: /api/
Disallow: /login
Disallow: /signup
Disallow: /sign-in
Disallow: /sign-up

See the whole file. The same rules repeat in every group, because a crawler that matches a named group ignores the wildcard one.

Worth knowing

More from OpenAI

  • OpenAI says each of its robots.txt settings is independent: you can allow OAI-SearchBot to appear in search while disallowing GPTBot. A robots.txt change can take about 24 hours to take effect.

    Source: developers.openai.com
Other OpenAI crawlers
ChatGPT-UserUser-triggered fetch
GPTBotModel training
AI engines it matters for
ChatGPTHow it picks sources, and what Rankwise measures on it

Sources, checked 2026-09-24

  • developers.openai.com/api/docs/bots
Check your own file

See which AI crawlers your robots.txt lets in.

The free AI crawler checker reads your robots.txt and reports which of the major AI crawlers it allows or blocks. No signup.
Check AI crawler accessOr start free →