Google's main crawler. It feeds Google Search, including all Search features, and other Google products.
Source: developers.google.comOperator
Job
Search index
Builds the index an AI answer searches. Block it and you drop out of that product's results.
robots.txt
Follows robots.txt
Google says its common crawlers always obey robots.txt rules when crawling automatically.
Source: developers.google.comMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Add one of these groups to the robots.txt at the root of your domain.
A crawler obeys only the most specific group that names it (RFC 9309), so a named group does not inherit the rules under User-agent: *. Repeat any paths you keep private, as the example does with /admin/.
User-agent: Googlebot Allow: / Disallow: /admin/
User-agent: Googlebot Disallow: /
Read from the robots.txt we serve. We want AI engines to read and cite this site, so every AI crawler is allowed everywhere except the app, the API and the sign-in pages (/dashboard/, /api/, /login, /signup, /sign-in, /sign-up). The file also declares Content-Signal: ai-train=yes, search=yes, ai-input=yes.
User-agent: Googlebot Allow: / Allow: /api/markdown/ Disallow: /dashboard/ Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /sign-in Disallow: /sign-up
See the whole file. The same rules repeat in every group, because a crawler that matches a named group ignores the wildcard one.
Google names robots.txt rules for Googlebot as the control for AI features in Search. To limit what is shown, it points to nosnippet, data-nosnippet, max-snippet and noindex.
W.X.Y.Z in the strings above stands for the Chrome version, which changes.