Should I block AI crawlers from my site?
An honest look at what you gain and what you give up, and why the answer is usually "block some, allow others".
Rarely all of them. Training crawlers and retrieval crawlers are different things: blocking training keeps your content out of future models, while blocking retrieval makes you uncitable and invisible in the products people actually use. Most sites want the second and are ambivalent about the first.
The distinction that decides everything
| Type | Examples | Blocking costs you |
|---|---|---|
| Training | GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended | Nothing visible. Your content is excluded from future training corpora. |
| Retrieval / search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Citations, referral traffic and any presence in AI answers about your subject. |
| Live fetch | ChatGPT-User, Claude-User, Perplexity-User | The experience of anyone who explicitly asks an assistant about your page. |
Nearly every "we blocked AI" configuration blocks all three, because the snippets circulating do. The result is a site that gained nothing and lost its presence in a growing share of how people find things.
Reasonable cases for blocking training
- Your content is the product — a paid archive, licensed photography, subscriber journalism.
- You hold rights on behalf of others and have no permission to hand their work to a training corpus.
- You have a licensing position and want the leverage of an explicit opt-out.
- You object on principle. That is a legitimate reason and does not need a commercial justification.
Reasons that do not survive scrutiny
- "It stops scrapers." It does not. robots.txt is honoured by the operators who publish their bot names and ignored entirely by anyone acting in bad faith. You are constraining exactly the parties who were behaving well.
- "It protects my SEO." There is no mechanism by which allowing an AI crawler harms search ranking.
Google-Extendedcontrols AI training use and is explicitly separate fromGooglebotand from search ranking. - "AI traffic is worthless." Referral volume is lower than search, but arrives at the moment someone is asking about your exact subject. Measure it before deciding.
A configuration that reflects most people's actual preferences
# Retrieval and live fetches: allowed, so we can be found and cited
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Training: opted out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /Know what robots.txt cannot do
It is a published request, not a control. It does not prevent access, it does not protect anything, and it does not stop a determined scraper for a moment. If you need actual protection, that is authentication and rate limiting. Treat robots.txt as a statement of preference to parties who are willing to honour preferences.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Which of twelve named crawlers your site currently allows or blocks, and the exact rule doing it.
- Whether your policy separates training from retrieval, or blocks both indiscriminately.
- Whether a catch-all rule is quietly overriding the specific allowances you wrote.
- Whether the robots.txt served is the one you published, or one substituted by your host.
For agents and scripts, the same measurement is at
/api/v1/ai?url=yoursite.com —
see the API documentation.
Related questions
Does blocking Google-Extended hurt my search ranking?
No. Google documents Google-Extended as controlling AI training and grounding use only. Googlebot governs search crawling and ranking and is a separate token that you would have to block separately.
If I block training crawlers, is my content safe from training?
It is excluded going forward by the operators who honour the file. It does not remove content from models already trained, and it does nothing about parties who ignore robots.txt or who obtained your content from a third-party dataset.
Is there a middle ground?
Yes, and it is the common one: block training, allow retrieval and live fetches. You stay findable and citable while opting out of corpus collection. The configuration above does exactly that.
Read next
Does blocking GPTBot hurt my traffic?
Blocking GPTBot alone costs you almost nothing. The damage comes from the other rules people add at the same time.
ReadHow do I check if AI crawlers can read my site?
Four tests you can run yourself in about ten minutes, in the order that finds the problem fastest.
ReadWhy is my website not showing on ChatGPT?
The four things that decide whether ChatGPT can find, read and cite your site — and how to tell which one is stopping you.
ReadWhat is llms.txt and do I need it?
A proposed convention for pointing language models at your best content. Cheap to add, unproven, and no substitute for the things that do work.
Read