How do I audit a website I don't own?
What is legitimate to measure on someone else's site, where the line is, and how to do it without being a nuisance.
Reading publicly served responses — headers, HTML, DNS, certificates, robots.txt — is what every browser and crawler does, and is entirely legitimate. Probing for vulnerabilities, attempting authentication, or generating enough load to matter is a different activity, and needs permission.
The line
| Activity | Status |
|---|---|
| Requesting public pages | Fine. Identical to a browser visit. |
| Reading response headers | Fine. Part of every response. |
| DNS and certificate lookups | Fine. Public by design. |
| Reading robots.txt and sitemaps | Fine. Published for exactly this. |
| Crawling public links, paced | Fine. What search engines do. |
| Ignoring robots.txt | Poor practice; may breach terms of use |
| Guessing at admin paths | Probing. Not measurement. |
| Testing for injection or misconfiguration | Needs permission. This is penetration testing. |
| Attempting authentication | Do not. Likely unlawful in most jurisdictions. |
| High-rate requests | Do not. Indistinguishable from an attack. |
The useful test: would this be reasonable if a person did it manually in a browser? Reading a page, viewing source and checking a certificate all pass. Trying passwords does not.
Legitimate reasons to do it
- Competitive benchmarking. What peers configure that you do not — one of the cheapest sources of good ideas available.
- Vendor due diligence. Assessing the technical hygiene of a supplier before signing.
- Research. Measuring configuration practices across a population of sites.
- Client work. Auditing before a pitch or an engagement.
- Verification. Checking whether a site's public claims about itself hold up.
Doing it considerately
- Identify yourself honestly in the user agent. Do not impersonate a browser to evade rules.
- Respect robots.txt, including when it inconveniences you.
- Pace requests. Seconds between them, not milliseconds.
- Request the smallest thing that answers your question rather than crawling exhaustively.
- Stop immediately on 429 and honour Retry-After rather than retrying through it.
- Do not attempt anything behind authentication, and do not guess at paths.
What you will and will not learn
External measurement gives you a genuinely useful picture and a bounded one. You will see their security headers, transport configuration, DNS and email authentication, crawler policy, third-party footprint, metadata hygiene and accessibility markup — all of it factual and comparable.
You will not see their application security, their internal practices, their dependency versions or their data handling. A site with excellent headers can be trivially exploitable through its code, and nothing observable from outside will suggest it.
Reporting a problem you find
If you discover something genuinely concerning — an exposed key, an open directory, an expired certificate — tell them. Check /.well-known/security.txt for a contact route; a great many sites publish one specifically so that finders have somewhere to go.
Report what you observed, without probing further to see how far it goes. Confirming the extent of a vulnerability is testing, and testing needs permission that a finder does not have.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Only ordinary public requests: no probing, no path guessing, no authentication attempts.
- An honest, identifiable user agent on every request, with no browser impersonation.
- robots.txt respected, and rate limits applied so an audit cannot become a load test.
- A published scope statement naming what is never attempted, so the boundary is verifiable rather than asserted.
For agents and scripts, the same measurement is at
/api/v1/summary?url=yoursite.com —
see the API documentation.
Related questions
Is it legal to audit someone else's site?
Reading publicly served responses is what every browser does and is not in dispute. Probing for vulnerabilities and attempting access are different activities governed by computer-misuse law in most jurisdictions, and need permission. This is general information rather than legal advice.
Will they know I audited them?
It appears in their logs as ordinary requests from an identifiable user agent. Nothing is hidden, which is the point — an audit that had to conceal itself would be doing something it should not.
Can I audit a competitor and publish the results?
Publishing factual, verifiable observations is generally defensible. Take care that the framing matches the evidence: "no CSP header on 17 August" is a fact; "their security is negligent" is a characterisation you would have to stand behind.
What if a site blocks the audit?
Respect it. Checks are reported as undetermined with reason blocked, rather than scored or worked around. A site declining automated inspection has made a decision that is theirs to make.
Read next
How do I compare two websites objectively?
Benchmarking against a competitor without fooling yourself — what is fair to compare, and what is not comparable at all.
ReadHow do I audit hundreds of websites at once?
Running a portfolio, a client list or a research sample through a measurement API without abusing anyone — including the sites you are measuring.
ReadHow do I tell if a website is down or just blocking me?
Distinguishing a real outage from bot protection, geoblocking or a network problem — and what each failure mode looks like.
ReadWhat is a security.txt file and do I need one?
A standard file telling researchers how to report a vulnerability. Five lines, and it is the difference between a private report and a public one.
Read