How do AI agents inspect websites?

The three modes, and how to tell which one you got

When someone asks an assistant "is my site secure?", the answer arrives in the same confident prose regardless of how it was produced. The difference is entirely in what the model had access to.

The failure mode people notice is the first one. The failure mode people miss is the second: a model that fetched your homepage will happily discuss your security posture, having seen none of it. Security lives in headers, not in body text.

What a page fetch does not contain

This is the crux. A tool that returns page text — which is what most browsing tools do — discards the part where the interesting facts live.

• Response headers. Content-Security-Policy, Strict-Transport-Security, X-Frame-Options, cache directives, cookie flags. All of it stripped before the model sees anything.
• The TLS handshake. Which protocol version was negotiated, which cipher, when the certificate expires, who issued it, whether the chain is complete.
• DNS. The MX records that decide whether your email arrives, the SPF and DMARC policies that decide whether someone can forge it, the CAA record that decides who may issue certificates for you.
• Redirect history. Whether http:// upgrades to https://, how many hops it takes, whether the canonical host is the one you think.
• robots.txt resolution. Not the file — the outcome of applying its rules to a specific crawler and a specific path, which is where the subtle mistakes hide.

Why the inferred answer sounds so good

A language model asked about a specific site produces the most plausible description of a site like that one. If your domain is well known, the plausible description is "well configured" — which is a statement about the reference class, not about you. It will be right often enough to feel reliable and wrong exactly when it matters, because the sites with problems are the ones that look normal from the outside.

The tell is unfalsifiability. "Likely has TLS configured correctly and follows common security practices" cannot be checked, because it does not name a single thing that was observed. Compare: "HSTS is present with max-age=31536000; there is no Content-Security-Policy header." Both are one sentence. Only one of them can be wrong.

What a measurement tool returns instead

When an agent has a tool that measures, the result is structured rather than prose, and the structure is what keeps the model honest:

• The finding — what failed, how severe, how much work the fix is.
• The evidence — the header value, certificate field or robots.txt line the verdict came from, so the model can quote the reason rather than the conclusion.
• What was not covered — the areas this tool never examines, so a missing finding is not read as a clean bill of health.
• What reached no verdict — checks that timed out, were blocked, or need a browser to evaluate. Kept separate from the passes, never folded in.

The last two matter more than they look. A tool that returns only findings teaches a model that silence means safety. A tool that states its own boundaries lets the model qualify the answer accurately.

Checking which mode your assistant is in

1. Ask about a detail that cannot be inferred: "when does the TLS certificate for my domain expire?" A model without measurement will hedge, decline, or invent a date.
2. Ask the same question twice in separate conversations. Inferred answers drift; measured ones do not.
3. Ask it to name its evidence: "which header told you that?" If no header can be named, none was read.
4. Change something real — add a security header — and ask again. A measuring tool notices within minutes. Training data will not notice for a year, if ever.

That last test is the definitive one, and it takes five minutes.