How do I monitor a website for regressions automatically?

The failures uptime monitoring never sees

A site returning 200 can be badly broken in ways that cost you for months before anyone notices.

• A staging robots.txt deployed to production. Disallow: / everywhere. The site is up, fast, and being progressively removed from every search index. Typically noticed weeks later via a traffic chart.
• A certificate approaching expiry. Fine until the exact moment it is catastrophic, and then every visitor gets an interstitial.
• Security headers dropped by an edge change. A CDN configuration update, a new WAF rule, a rewritten origin config. Your CSP is in your repository and not in the response.
• A canonical tag pointing at staging. Every page telling search engines the real version lives on a host nobody can reach.
• A noindex left in from a launch. The single most expensive one-line mistake in web publishing.
• DMARC or SPF broken by a DNS edit. Silent until deliverability drops, and hard to attribute afterwards.

None of these change the status code. All of them are visible in a full audit.

The shape of the check

Run a complete audit on a schedule, compare it with the previous run, and alert on the difference rather than on the absolute state. Alerting on state means alerting constantly about things you have decided not to fix; alerting on change means you hear about exactly what moved.

curl -s "https://outrings.com/api/v1/changes?url=example.com" | jq '{
  comparable,
  scoreDelta,
  direction,
  newProblems: [.newProblems[]? | "\(.id) (\(.status))"],
  resolved:    [.resolved[]?    | "\(.id) (was \(.was))"]
}'

The endpoint re-audits, compares against the last stored snapshot, and reports what appeared, what cleared and how each category moved. On the first run it returns comparable: false — that is history starting, not an error.

A daily job

#!/usr/bin/env bash
set -euo pipefail

site="example.com"
result=$(curl -sf "https://outrings.com/api/v1/changes?url=$site")

comparable=$(jq -r '.comparable' <<<"$result")
[ "$comparable" = "true" ] || { echo "first snapshot stored for $site"; exit 0; }

new=$(jq -r '[.newProblems[]?.id] | join("; ")' <<<"$result")
delta=$(jq -r '.scoreDelta // 0' <<<"$result")

if [ -n "$new" ]; then
  echo "REGRESSION on $site (score $delta): $new" >&2
  exit 1
fi

echo "$site unchanged (score $delta)"

Run it from cron, a scheduled CI job, or any scheduler you already have. Exit code 1 on regression is what makes it composable with everything else.

What to alert on, and what to ignore

Attributing a change correctly

When a score moves, there are three possible causes and only one is worth waking up for: the site changed, the ruleset changed, or a check that previously succeeded is now undetermined because the auditor was blocked. Every response carries versions.ruleset — if that differs between two runs, compare like for like before concluding anything. And a drop accompanied by new undetermined entries usually means bot protection started intercepting the audit rather than that anything regressed.