What is the difference between crawling, indexing and citing?

The three stages

They are strictly sequential. You cannot be indexed without being crawled, or cited without being indexed. But success at one stage says nothing about the next — being crawled daily and never cited is entirely normal.

Diagnosing which stage you are stuck at

Is it crawling?

# Does robots.txt permit them?
curl -s https://yoursite.com/robots.txt

# Are they actually arriving, and with what status?
grep -iE 'GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot' /var/log/nginx/access.log \
  | awk '{print $9}' | sort | uniq -c

No requests at all means a crawling problem. Requests returning 403 or 429 also mean a crawling problem — one caused by your edge rather than your robots file.

Is it indexing?

# Is your content in the delivered HTML at all?
curl -sL https://yoursite.com | grep -i 'a distinctive sentence from your page'

# Is anything telling engines not to index?
curl -sI https://yoursite.com | grep -i x-robots-tag
curl -sL https://yoursite.com | grep -i 'name="robots"'

Crawled successfully but nothing comes back from the first command means the page is built in the browser. The crawler received an empty shell, so there was nothing to index.

Is it citing?

If you are crawled and indexed and still not cited, the mechanics are done and this is a content question. Ask an assistant something your page answers, see who is cited instead, and read what those pages do differently.

Why citing is different in kind

Crawling and indexing are mechanical — they either work or they do not, and the cause is findable. Citing is competitive and probabilistic. You are not passing a check; you are being selected over alternatives, per question, by systems that weigh several things at once:

• Extractability. Can a passage be lifted whole and still make sense? A direct answer under a clear heading can. A point developed across four paragraphs cannot.
• Specificity. Pages answering one question precisely get cited more than pages covering a topic broadly.
• Primacy. Original data and first-hand detail get cited; summaries of other people's work get skipped in favour of the work.
• Corroboration. Claims that agree with other sources are safer to repeat. A claim appearing nowhere else reads as unreliable, however true.
• Attributability. A named author, a visible organisation, a date. Anonymous content is harder to cite responsibly.

The order to work in

1. Confirm crawlers are permitted, in both robots.txt and your edge configuration.
2. Confirm your content exists in the delivered HTML.
3. Confirm nothing is emitting noindex, in the meta tag or the X-Robots-Tag header.
4. Only then work on structure — headings that ask questions, answers immediately beneath them.
5. Only then work on substance — original detail nobody else has.

Steps one to three take fifteen minutes and account for the large majority of cases. Steps four and five are months of work and are wasted if one to three are broken.