Why do LLMs hallucinate website audits?
The specific reason a model invents plausible findings about your site, and the two design choices that stop it.
Because it was asked a question about the world and given nothing from the world to answer it with. A language model always produces the most plausible continuation; asked to audit a site it cannot see, the most plausible continuation is a well-formed audit of a typical site. It is not malfunctioning — it is doing exactly what it does, on a question it had no business answering unaided.
Hallucination is not a bug in this case
It helps to be precise about the mechanism. A model predicts likely text. "What is wrong with example.com?" has a shape of answer that is overwhelmingly likely: a few common problems, phrased as findings, at plausible severities. Missing alt text. No CSP. Slow images. These are the right answer for the average website, which is why they read so convincingly.
The model is not fabricating in the sense of choosing to deceive. It is producing the statistically expected audit. The problem is that an expected audit and an actual audit are different objects, and nothing in the output distinguishes them.
Four ways it goes wrong, in order of how often you will meet them
| Failure | What it looks like | Why it happens |
|---|---|---|
| Reference-class answer | Generic findings that fit any site | No site-specific input, so the average is the best available guess |
| Stale memory | Confident claims about a site that were true two years ago | Training data has a cutoff; the web does not |
| Silent partial read | Real page-text findings mixed with invented header findings | A fetch tool returned the body, so the model fills the rest |
| Absence read as pass | "No security issues found" when nothing was checked | Nothing in the input distinguished "clean" from "not examined" |
The third and fourth are the dangerous ones, because they occur when a tool is present. Partial grounding is more persuasive than no grounding, and just as wrong about everything it did not cover.
The specific mistake: absence read as a pass
This deserves its own section because it survives every other fix. Suppose a tool checks ten things and reports two failures. A model reading that output has no way to know whether the other eight passed or were never attempted. The natural reading — the plausible continuation — is that they passed.
So a check that timed out becomes "no issues". A check that requires JavaScript becomes "fine". A whole category the tool does not cover becomes an implied clean result. The audit is now wrong in the most reassuring possible direction, and every sentence in it is defensible individually.
What actually fixes it
- Give it observations. Not a summary of observations — the header value, the certificate field, the robots.txt line. A model handed evidence quotes the evidence; a model handed a verdict paraphrases the verdict and drifts.
- Make the boundary explicit. Every result should name what it did not examine. This is the single highest-value field in an audit response for agent use, and it is the one most tools omit.
- Keep undetermined separate. A dedicated bucket with a reason —
blocked,timeout,requires_browser— so nothing ambiguous is silently counted as fine. - Make it reproducible. If two runs of the same tool on an unchanged site disagree, no amount of grounding helps: the model cannot tell a real change from noise in the instrument.
How to catch it happening to you
- Ask for evidence, not conclusions: "quote the header that told you that." A hallucinated finding has no header to quote.
- Ask what was not checked. A grounded answer can list it. An invented one changes the subject.
- Ask the same question in a fresh conversation and compare. Measured facts repeat exactly; invented ones vary.
- Introduce a known fault — remove a header on a staging copy — and check whether the audit notices. If it does not, it was never looking.
Test four is worth doing once against any tool you intend to rely on. It takes ten minutes and it is the only test that cannot be passed by sounding right.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Every finding is returned with the raw observation behind it, so a model can quote the reason instead of paraphrasing a verdict.
- A
notCoveredarray on every response naming what this service never measures at all. - An
undeterminedbucket for checks that reached no verdict, with the reason, kept separate from the passes. - Version stamps on every response, so a score that moved can be attributed to the site changing rather than the rules changing.
For agents and scripts, the same measurement is at
/api/v1/summary?url=yoursite.com —
see the API documentation.
Related questions
Does a bigger or newer model hallucinate less about my site?
It hallucinates more fluently. Model capability improves reasoning over available facts; it does not create facts that were never supplied. A stronger model with no measurement produces a more convincing wrong answer, not a more accurate one.
Is a browsing tool enough to stop this?
It fixes content questions and leaves configuration questions exactly as they were, while making the whole answer feel grounded. That combination is arguably worse than no tool, because it removes the reader's natural scepticism.
How do I tell a real finding from an invented one?
Real findings name specifics that could be wrong: a header that is absent, a certificate expiring on a date, a robots.txt line at a line number. Invented findings describe categories of problem without ever naming an observation.
Can I just tell the model not to guess?
It reduces confident phrasing somewhat and does not change what the model knows. The fix has to happen at the input, not in the instructions — a model with no observations has nothing to be honest with.
Read next
How do AI agents inspect websites?
What actually happens when you ask an assistant about a website — and why the answer is usually inferred rather than observed.
ReadCan I trust what ChatGPT says about my website?
A practical guide to which parts of an AI answer about your site are reliable, which are guesses, and how to tell them apart in seconds.
ReadWhy is AI-generated SEO advice usually wrong?
Four reasons an assistant gives you outdated, generic or inapplicable SEO advice — and the one change that fixes most of it.
ReadWhat should an AI agent do when it cannot determine something?
Why "not checked" and "passed" must never be the same answer, and how a well-built tool keeps them apart.
Read