What is a deterministic website audit?
Why running the same audit twice should give the same answer, what breaks that property, and why it matters more than accuracy.
An audit is deterministic when the same observations under the same ruleset produce the same result. It matters because without it you cannot tell whether a change in the score means your site changed or the instrument wobbled — which makes before-and-after comparison, regression detection and CI gating all meaningless.
The precise claim
Determinism is often stated loosely as "the same site always scores the same", which is not true and not what is meant. Sites change. Certificates renew, headers get edited, DNS propagates.
The exact guarantee is narrower and more useful: the same observations, evaluated under the same ruleset, produce the same result. If the score moves, either an observation changed — which is a real fact about your site — or the ruleset changed, which is recorded in a version field you can check.
Why it matters more than accuracy
A slightly harsh but consistent measurement is more useful than a perfectly calibrated but noisy one. With a consistent instrument you can answer the questions people actually have:
- Did my change help? — compare before and after.
- Did the deploy break something? — compare against yesterday.
- Are we improving? — compare across months.
- Should this build be blocked? — compare against the last known good state.
Every one of those is a difference. A noisy instrument makes all of them unanswerable, no matter how well calibrated it is on average.
What destroys determinism
| Cause | Effect | Avoidable? |
|---|---|---|
| A language model scoring the output | Same input, different verdict, no explanation | Yes — do not use one in the scoring path |
| Random sampling of pages or resources | Different subset measured each run | Yes — fixed, documented selection |
| Third-party APIs in the path | Their availability becomes your variance | Yes — measure directly |
| Timing-derived checks | Network conditions leak into the score | Partly — keep them out of scoring or weight them low |
| Undetermined treated as pass | A blocked run silently scores higher | Yes — exclude from scoring entirely |
| Genuine site change | Score moves | No — and this is the point |
Why a language model cannot be in the scoring path
This is the most common source of non-determinism in modern audit tools, and it is structural rather than fixable by configuration. Sampling temperature can be set to zero and the output still varies with model version, context, and phrasing. More fundamentally, a model asked to judge produces a plausible judgement, and plausibility is not a stable function of the input.
A model is a fine tool for explaining a finding once the finding exists. It cannot be the thing that decides whether the finding exists, if you want two runs to agree.
How to test any tool for it
- Run the audit twice in a row on a site you control and do not touch in between.
- Compare the two outputs field by field, not just the headline score.
- Repeat an hour later. Content-derived scores sometimes drift with caching behaviour.
- Change one specific thing — add a header — and confirm exactly the related check moved and nothing else did.
Step four is the strongest test. A tool where an unrelated finding also shifted is telling you its numbers are not attributable, and attribution is the whole reason you are measuring.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Every score produced by hand-written rules over direct observations, with no language model anywhere in the scoring path.
- Ruleset, schema and collector versions stamped on every response, so a score change can be attributed correctly.
- Undetermined checks excluded from scoring in both directions rather than folded in with passes.
- The observed value returned with each verdict, so two runs can be diffed at the evidence level rather than at the score level.
For agents and scripts, the same measurement is at
/api/v1/summary?url=yoursite.com —
see the API documentation.
Related questions
So the score never changes?
It changes whenever your site changes — that is the intended behaviour. What does not change is the result for an unchanged site under an unchanged ruleset. If both are the same and the score moved, something is wrong with the instrument.
What if the ruleset is updated?
Every response carries versions.ruleset. When comparing two audits over time, check it: a score can move because the site changed or because the rules did, and only the version distinguishes them.
Are timing checks excluded entirely?
Latency observations are reported because they are useful context, and they are weighted so that ordinary network variation does not move the score meaningfully. Configuration checks, which are stable, carry the weight.
Does determinism mean the audit is correct?
No — those are separate properties. A consistently wrong ruler is still wrong. Determinism is what makes the ruler usable for comparison; the rules being right is a separate question, which is why every check is documented.
Read next
Why do two SEO tools give my site different scores?
Different checks, different weights, different definitions of failure — and why comparing scores across tools is meaningless.
ReadWhat does a website score actually mean?
How a score out of 100 is built, what it can honestly tell you, and the four things it cannot.
ReadWhy did my website score change when I did not change anything?
The four causes of an unexplained score movement, and how to tell which one you are looking at in under a minute.
ReadWhat is evidence-based website analysis?
The difference between a tool that tells you a verdict and one that shows you what it saw — and why the second is worth insisting on.
Read