What is Google-Extended and should I block it?
What this token actually controls, what it does not affect, and the mistake that removes you from Google search entirely.
Google-Extended is a robots.txt control token governing whether your content may be used for certain Google AI purposes. It is not a crawler and it does not affect Google Search indexing. Blocking it does not remove you from search — but blocking Googlebot by mistake certainly does.
It is a token, not a crawler
This is the source of most confusion. Google-Extended never makes a request and will never appear in your access logs. It exists only as a name you can address in robots.txt, and Googlebot applies the rule when deciding how content may be used.
So there is nothing to detect and nothing to block at the network level. The only place it means anything is your robots file.
What each token controls
| Token | Controls | Blocking it means |
|---|---|---|
Googlebot | Crawling for Google Search | You are removed from Google Search. Rarely what anyone wants. |
Google-Extended | Use of content for certain Google AI purposes | No effect on search indexing or ranking |
Google-CloudVertexBot | Fetching for Vertex AI agents on a site owner's behalf | Unrelated to search |
Googlebot among them. The site then leaves Google Search — a far larger loss than whatever the AI opt-out was protecting, and it is often noticed weeks later via a traffic collapse.Deciding
Reasons to block it
- Your content is licensed, and unlicensed AI use conflicts with the terms you sell under.
- You publish original research or data whose value depends on people coming to you for it.
- You have a legal or contractual obligation restricting how content may be reused.
- You object on principle, which is a legitimate reason and does not require a business case.
Reasons not to
- Your goal is reach, and appearing in AI answers is a route to it.
- Your content is promotional or informational — you want it repeated.
- You would rather be described accurately from your own words than from third-party summaries of you.
The rules
# Opt out of the AI use, stay fully in Google Search
User-agent: Google-Extended
Disallow: /
User-agent: Googlebot
Allow: /The second group is not strictly required — Googlebot is allowed by default unless a rule says otherwise — but stating it explicitly protects against a later edit to a catch-all group accidentally sweeping it up.
The blanket rule to be careful with:
User-agent: *
Disallow: /This applies to every crawler that has no more specific group of its own, and it is how sites remove themselves from everything at once while believing they addressed one thing.
The asymmetry worth weighing
Blocking Google-Extended is reversible and its effect is gradual — you can change your mind and content becomes eligible again. Being absent from AI answers while competitors are present is a slow, compounding loss that is difficult to detect, because nothing tells you about the citations you did not receive.
For most publishers whose goal is reach, allowing it and focusing on being the best available source is the stronger position. For those selling access to their content, blocking is straightforwardly correct. The decision is genuinely about your business model rather than about best practice.
Check what you actually publish
curl -s https://yoursite.com/robots.txtRead it rather than trusting your memory of it. Hosting platforms and CDNs sometimes generate or override this file, so the version being served may not be the one in your repository — and that difference is invisible until you look.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Whether
Google-Extended,Googlebotand every other named crawler is allowed or blocked, with the exact rule responsible. - Whether a catch-all group is unintentionally blocking crawlers you rely on.
- The distinction between search crawlers and AI-use control tokens, so an opt-out does not silently cost you search visibility.
- Whether the served
robots.txtmatches what you published, which platform-level overrides can silently change.
For agents and scripts, the same measurement is at
/api/v1/ai?url=yoursite.com —
see the API documentation.
Related questions
Does blocking Google-Extended hurt my rankings?
No. It governs certain AI uses of your content and is separate from Search indexing and ranking. The risk is not the block itself — it is accidentally blocking Googlebot in the same edit.
Will I still appear in AI Overviews if I block it?
Google has stated that Google-Extended does not control AI Overviews, which draw on Search indexing. If you want to be excluded from those, that is a Search-level control, and it costs you ordinary search visibility too.
Is there one setting for all AI crawlers?
No, and there is unlikely to be. Each operator uses its own tokens with its own semantics, and the distinction between training use, search indexing and user-triggered fetching differs between them. A blanket rule almost always blocks more than intended.
Can I allow AI search but block AI training?
For the major operators, largely yes — allow the search crawlers and disallow the training tokens. It requires naming each one specifically rather than using a catch-all, and behaviour for user-triggered fetches varies by operator.
Read next
How do I allow AI search but block AI training?
A robots.txt that keeps you visible in AI answers while opting out of training corpora — with the precedence rules that make it work.
ReadShould I block AI crawlers from my site?
An honest look at what you gain and what you give up, and why the answer is usually "block some, allow others".
ReadDoes blocking GPTBot hurt my traffic?
Blocking GPTBot alone costs you almost nothing. The damage comes from the other rules people add at the same time.
ReadWhich AI crawlers have visited my site?
Finding GPTBot, ClaudeBot, PerplexityBot and the rest in your own logs — and what their presence or absence actually tells you.
Read