How do I allow AI search but block AI training?
A robots.txt that keeps you visible in AI answers while opting out of training corpora — with the precedence rules that make it work.
Name each crawler explicitly. Allow the search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot — and disallow the training tokens — GPTBot, Google-Extended, Applebot-Extended, CCBot. A catch-all cannot express this distinction.
The precedence rule everything depends on
One rule makes this possible: the most specific matching user-agent group wins, and only that group applies.
A crawler with its own named group reads that group and ignores User-agent: * entirely — including any Allow lines in the catch-all. Get this wrong and you produce a file that says the opposite of what you intended.
A working file
# Search crawlers — allowed, so you can appear in AI answers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Training and corpus collection — disallowed
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# Everything else
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlWhat each group is doing
| Token | Setting | Effect |
|---|---|---|
OAI-SearchBot | Allow | You can appear in ChatGPT search results |
GPTBot | Disallow | Your content is not collected for training |
ChatGPT-User | Not listed | User-triggered fetches — see the note below |
Claude-SearchBot | Allow | Eligible for Claude search |
Google-Extended | Disallow | Opted out of certain Google AI uses, search unaffected |
Googlebot | Allow | Ordinary Google Search indexing continues |
CCBot | Disallow | Excluded from the Common Crawl corpus many models use |
ChatGPT-User and Claude-User — sit in an awkward middle. They fetch a page because a person asked about that specific page, so they are closer to a browser than to a crawler, and operator documentation on whether robots.txt governs them is less determined than for the others. Most publishers want these allowed: someone has explicitly asked about your page.The mistakes that break it
Assuming Allow in the catch-all applies
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /This works, and only because GPTBot has its own group. But the reverse trips people constantly:
User-agent: *
Disallow: /
User-agent: OAI-SearchBot
Allow: /This is also correct — OAI-SearchBot reads its own group. The failure is having a blanket Disallow: / and no group for the crawler you want, then wondering why it never appears.
Blocking a search crawler by accident
Copying a "block AI" snippet from an article and finding it includes OAI-SearchBot or Googlebot alongside the training tokens. Read every line of anything you paste in.
Believing robots.txt is enforcement
It is a request, honoured by the major operators and ignored by everyone else. It removes you from products that respect it and protects nothing. If content must not be accessed, it needs authentication.
Verifying it
- Fetch
yoursite.com/robots.txtin a browser and read the served file, not your local copy. - For each crawler you care about, find the group that applies to it — remembering that a named group wins outright.
- Confirm no catch-all
Disallow: /is stranding a crawler that has no group of its own. - Check your edge configuration too: a WAF blocking these crawlers overrides anything robots.txt permits.
- Re-check after any platform change. Hosting providers sometimes generate or override this file.
Step four catches a case robots.txt analysis alone cannot: a perfectly permissive file in front of bot protection that turns the crawler away regardless.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Per-crawler resolution against your actual rules, naming the group and line that decides each verdict.
- Whether a catch-all rule is stranding crawlers that have no named group of their own.
- The role of each token — search, training or user-triggered — so an opt-out has the effect intended.
- Whether the served
robots.txtdiffers from what you published, and whether the edge is blocking crawlers the file permits.
For agents and scripts, the same measurement is at
/api/v1/ai?url=yoursite.com —
see the API documentation.
Related questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. Appearing in ChatGPT search results is governed by OAI-SearchBot. Blocking the first and allowing the second is exactly the allow-search-block-training position.
Will operators actually honour this?
The major ones publish their tokens and state that they honour robots.txt, and their crawlers are broadly observed to do so. Smaller and undisclosed crawlers may not. Robots.txt is a request, not a control.
Should I block CCBot?
Common Crawl is a public corpus many models train from, so blocking it is consistent with an anti-training position. Note that it is also used for research and archiving, so you are opting out of more than model training.
What about crawlers that do not identify themselves?
Robots.txt cannot address what it cannot name. If content must not be collected, authentication is the only reliable control — everything else depends on the other party choosing to cooperate.
Read next
What is Google-Extended and should I block it?
What this token actually controls, what it does not affect, and the mistake that removes you from Google search entirely.
ReadShould I block AI crawlers from my site?
An honest look at what you gain and what you give up, and why the answer is usually "block some, allow others".
ReadHow do I fix robots.txt blocking Google?
How to read the file correctly, what the precedence rules actually are, and the three mistakes that cause almost every accidental block.
ReadWhat is the difference between crawling, indexing and citing?
Three separate stages, each with its own failure mode — and why fixing the wrong one wastes months.
Read