Which AI crawlers have visited my site?
Finding GPTBot, ClaudeBot, PerplexityBot and the rest in your own logs — and what their presence or absence actually tells you.
Your server access log holds the answer. Grep for the user-agent tokens and you get dates, paths and frequency. Absence usually means one of three things: you are blocking them, your content is not reachable without JavaScript, or nothing has pointed them at you yet.
The tokens to search for
| Token | Operator | Purpose |
|---|---|---|
GPTBot | OpenAI | Training data collection |
OAI-SearchBot | OpenAI | Builds the index ChatGPT search retrieves from |
ChatGPT-User | OpenAI | Fetches a page because a user asked about it |
ClaudeBot | Anthropic | Crawling |
Claude-SearchBot | Anthropic | Search indexing |
Claude-User | Anthropic | User-triggered fetch |
PerplexityBot | Perplexity | Search indexing |
Google-Extended | A robots.txt control token, not a crawler that appears in logs | |
Applebot-Extended | Apple | A control token for AI training use |
CCBot | Common Crawl | Open crawl corpus that many models train from |
Bytespider | ByteDance | Crawling |
Google-Extended and Applebot-Extended are tokens you use in robots.txt to control how content is used — they do not identify separate crawlers and you will not see them making requests. Looking for them in logs and concluding Google is not visiting is a common misreading.Searching your logs
# Count visits by AI crawler
grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|CCBot|Bytespider' \
/var/log/nginx/access.log | sort | uniq -c | sort -rn
# What one of them actually fetched
grep -i 'GPTBot' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
# And what status codes it received
grep -i 'GPTBot' /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -cThat third command is the most useful and the least run. A crawler receiving 403s or 429s is being turned away by your edge, which looks identical to "not interested" if you only count visits.
Reading the result
| What you see | What it means |
|---|---|
| Regular visits, 200 responses | Working as intended |
| Visits returning 403 | Your WAF or bot protection is blocking them, not your robots.txt |
| Visits returning 429 | You are rate-limiting them into uselessness |
Only ChatGPT-User, never OAI-SearchBot | People are pasting your URLs, but you are not in the index |
| Nothing at all | Blocked in robots.txt, or nothing has ever pointed them at you |
| Crawled often, never cited | Mechanics are fine; this is now a content question |
The fourth row is a genuinely useful diagnostic. It means your content is good enough that people share it directly, and the search crawler that decides whether you appear unprompted has not indexed you.
If you cannot reach your logs
Shared hosting often hides them. Two alternatives:
- Most control panels expose an access-log viewer or a raw log download, frequently under a "statistics" or "logs" section.
- If your platform offers neither, you can infer a great deal from configuration: check whether robots.txt permits each crawler, whether your content is in the delivered HTML, and whether your edge is challenging non-browser clients. Those three account for nearly every case of zero visits.
A caution about user agents
A user-agent string is self-reported and trivially spoofed. Traffic claiming to be GPTBot may not be, and operators publish IP ranges precisely so that verification is possible. For deciding whether to serve content this matters; for the diagnostic question "is anything reaching me at all", the string is good enough.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Whether each named AI crawler is allowed or blocked, naming the exact
robots.txtrule responsible. - The distinction between search crawlers, training crawlers and user-triggered fetchers, since blocking each has entirely different consequences.
- Whether your content is present in the delivered HTML, which decides whether a visit is worth anything.
- Which crawler directives are genuinely honoured versus which are control tokens that never appear as traffic.
For agents and scripts, the same measurement is at
/api/v1/ai?url=yoursite.com —
see the API documentation.
Related questions
Why do I see no AI crawler traffic at all?
Check robots.txt first, then your edge configuration — bot protection blocks these crawlers far more often than owners realise, and it produces the same silence as an explicit disallow. Third, check that anything links to you; crawlers have to learn you exist.
Is ChatGPT-User traffic a good sign?
It means people are pasting your URLs into conversations, which is genuine interest. It does not put you in the index — that is OAI-SearchBot. Seeing one without the other is worth investigating.
Can I trust the user-agent string?
It is self-reported and spoofable. Operators publish IP ranges for verification, which matters if you are making serving decisions. For diagnosing whether anything is reaching you, the string is sufficient.
Does more crawling mean more citations?
No. Crawling is necessary and not sufficient. Being crawled frequently while never being cited is a content and structure question, not an access one — and it is a much better problem to have.
Read next
How do I know if AI assistants are sending me traffic?
Identifying referrals from ChatGPT, Perplexity and Claude in your analytics — and why most AI-driven visits are invisible.
ReadHow do I check if AI crawlers can read my site?
Four tests you can run yourself in about ten minutes, in the order that finds the problem fastest.
ReadWhat is Google-Extended and should I block it?
What this token actually controls, what it does not affect, and the mistake that removes you from Google search entirely.
ReadWhy is my website not showing on ChatGPT?
The four things that decide whether ChatGPT can find, read and cite your site — and how to tell which one is stopping you.
Read