What is llms.txt and do I need it?
A proposed convention for pointing language models at your best content. Cheap to add, unproven, and no substitute for the things that do work.
It is a Markdown file at /llms.txt that lists your most useful pages for a language model. It takes fifteen minutes to write and no major operator has committed to reading it. Add it if you like; do not expect it to fix anything.
What it is
A proposal, by analogy with robots.txt, for a file that tells language models where the substance of your site is. Unlike robots.txt it is not permissions — it is curation. The format is plain Markdown:
# Example Company
> One-paragraph description of what this organisation does
> and who it is for.
## Documentation
- [Getting started](https://example.com/docs/start): installation and first steps
- [API reference](https://example.com/docs/api): every endpoint with examples
## Guides
- [Pricing explained](https://example.com/pricing): how billing works
## Optional
- [Changelog](https://example.com/changelog): release historyA companion llms-full.txt contains the full text of the listed pages, so a model can ingest the substance in one request rather than crawling.
The honest status
- No major AI operator has publicly committed to reading it.
- There is no evidence it improves visibility or citation rates.
- It is a community proposal, not a standard, and adoption is currently among sites that publish about AI.
- It costs almost nothing and breaks nothing.
When it is worth doing
- Documentation sites. The use case it was designed for. Pointing a model at your reference rather than your marketing pages is genuinely useful if anything reads it.
- Large sites with a small core. If ten pages out of two thousand carry the substance, saying so costs nothing.
- Sites whose audience includes developers. People will find it and it signals that you thought about the problem.
For a small brochure site with fifteen pages, there is little to curate and correspondingly little point.
If you write one, write it properly
- Serve it at exactly
/llms.txtwith content typetext/plainortext/markdown. - Open with an H1 naming the site and a blockquote saying what it is in one paragraph.
- Group links under H2 headings by purpose, and give each link a short description of what is in it.
- Link only pages that are actually useful. A list of everything is the same as no list.
- Keep it current. A file pointing at three dead URLs is worse than no file.
What to do instead, if you only do one thing
Spend the same fifteen minutes checking that your robots.txt allows the retrieval crawlers and that your content is in the HTML. Those two things measurably determine whether an assistant can read you. Then, if you want, add llms.txt as well.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Whether
/llms.txtexists, is served with a sensible content type, and is well-formed. - Whether the URLs it lists are reachable rather than dead.
- Whether the higher-impact signals — crawler permissions, server-rendered content, structured data — are in place.
For agents and scripts, the same measurement is at
/api/v1/ai?url=yoursite.com —
see the API documentation.
Related questions
Does ChatGPT read llms.txt?
There is no published commitment from OpenAI that it does. The same applies to Anthropic, Google and Perplexity. Anyone claiming otherwise should be asked for the documentation.
Is llms.txt a replacement for robots.txt?
No, and they do unrelated jobs. robots.txt controls whether a crawler may fetch a page and is honoured today by every major operator. llms.txt suggests which pages are worth reading and is honoured by nobody in particular. Keep robots.txt correct first.
Should I generate llms-full.txt automatically?
If you can, yes — a generated file stays current, and a stale one is the main failure mode. Just make sure it does not include pages you would not want ingested wholesale.
Read next
How do I check if AI crawlers can read my site?
Four tests you can run yourself in about ten minutes, in the order that finds the problem fastest.
ReadShould I block AI crawlers from my site?
An honest look at what you gain and what you give up, and why the answer is usually "block some, allow others".
ReadHow do I get my site cited by AI assistants?
The mechanics that must be right, then the content decisions that actually earn the citation — in the order worth doing them.
ReadWhat is structured data and do I need it?
Machine-readable facts about your page. Worth adding where it genuinely describes the content, worthless — and occasionally harmful — everywhere else.
Read