Google-Extended is a product token, not a separate crawler, that site owners can name in robots.txt. Google introduced it in 2023 so sites could opt out of having content Google crawls used for its generative AI models. Google's documentation says it covers training future Gemini models and grounding in Gemini apps and certain Google Cloud products. It does not affect a site's inclusion or ranking in Google Search, and it does not control AI Overviews or AI Mode.
How it works
Googlebot does the crawling. When Google-Extended is disallowed, Google says it will not use the crawled content for the covered AI purposes. Because there is no separate fetch, you will not see Google-Extended in your server logs.
# Opt out of Gemini training and grounding uses
User-agent: Google-Extended
Disallow: /
# Search crawling continues as normal
User-agent: Googlebot
Allow: /Deciding whether to block it
The trade-off is between control over training use and presence in Gemini app answers that ground on web content. Publishers whose business is the content itself often block it. Many SaaS companies want their product facts in as many assistants as possible and leave it allowed. Either way, make the choice deliberately rather than as a side effect of a blanket block on everything AI-related.
If your goal is to limit how your content appears in AI Overviews or AI Mode, Google points to Search controls such as nosnippet, data-nosnippet and max-snippet instead. Those also affect regular snippets, so use them carefully.
Example
An accidental opt-out
Wickerlane, a fictional AP automation tool, copied a robots.txt template that blocks a long list of AI agents, including Google-Extended. Months later, the team notices the Gemini app describes Wickerlane mostly from third-party sources. Reviewing the file, they decide their product pages should be usable for grounding and remove the Google-Extended block while keeping other rules.
Common mistakes
Expecting it to remove you from AI Overviews
AI Overviews and AI Mode are part of Search. Google-Extended does not control them.
Looking for it in server logs
It is a robots.txt token only. Googlebot does the fetching.
Questions
Google states it is not a ranking signal and does not affect inclusion in Google Search.
Google's documentation describes it as controlling future use. It does not describe retroactive removal from existing models.
