Lets a site choose whether content Google crawls may be used to train future Gemini models, and for grounding answers in Gemini Apps and in Grounding with Google Search on Vertex AI.
Source: developers.google.comOperator
Job
Model training
Collects pages for training future models. Slower to pay off, and separate from search and citations.
robots.txt
A robots.txt token, not a crawler
Google-Extended has no user-agent string of its own. Crawling is done by Google's existing crawlers, and the token is read from robots.txt as a control.
Source: developers.google.comGoogle-Extended
Disallowing Google-Extended opts your content out of the uses described above. It does not stop any crawler visiting.
A crawler obeys only the most specific group that names it (RFC 9309), so a named group does not inherit the rules under User-agent: *. Repeat any paths you keep private, as the example does with /admin/.
User-agent: Google-Extended Allow: / Disallow: /admin/
User-agent: Google-Extended Disallow: /
Read from the robots.txt we serve. We want AI engines to read and cite this site, so every AI crawler is allowed everywhere except the app, the API and the sign-in pages (/dashboard/, /api/, /login, /signup, /sign-in, /sign-up). The file also declares Content-Signal: ai-train=yes, search=yes, ai-input=yes.
User-agent: Google-Extended Allow: / Allow: /api/markdown/ Disallow: /dashboard/ Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /sign-in Disallow: /sign-up
See the whole file. The same rules repeat in every group, because a crawler that matches a named group ignores the wildcard one.
Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal.
For AI features in Search, such as AI Overviews and AI Mode, Google names robots.txt rules for Googlebot as the control, not Google-Extended.