How do I audit hundreds of websites at once?

Ask for the smallest thing that answers the question

Scope first. Across hundreds of sites the difference between endpoints is the difference between a script that finishes and one that does not.

Five hundred sites at full is around sixty megabytes and a great deal of work for both ends. The same five hundred at score is under a megabyte.

A workable script

#!/usr/bin/env bash
set -uo pipefail

mkdir -p out

while read -r site; do
  [ -z "$site" ] && continue
  safe="${site//\//_}"

  # Skip anything already fetched, so a rerun resumes instead of starting over.
  [ -s "out/$safe.json" ] && continue

  if curl -sf --max-time 45 \
       "https://outrings.com/api/v1/summary?url=$site" \
       -o "out/$safe.json"; then
    printf '%-30s %s\n' "$site" "$(jq -r '.score' "out/$safe.json")"
  else
    printf '%-30s FAILED\n' "$site" >&2
    rm -f "out/$safe.json"
  fi

  sleep 5
done < sites.txt

The two details that matter most are the resume check and the sleep. Long runs get interrupted, and restarting from zero wastes everyone's bandwidth.

Pacing and limits

• Rate limit: 40 audits per hour, 200 per day per client. Exceeding it returns 429 with Retry-After, which your script should honour rather than retry through.
• Cached results are free. Repeating a URL within ten minutes does not consume quota, so a resumed run costs nothing for what it already fetched.
• A large portfolio takes days, by design. Two hundred a day is the ceiling. Plan for a rolling schedule rather than a single sweep.
• Pace for the targets, not for the API. Each audit issues dozens of requests to the site being measured. A five-second gap is courteous; no gap at all is not.

Store the raw JSON

Write the complete response to disk, one file per site, before extracting anything. Analysis questions change; refetching hundreds of sites because you decided you also wanted the certificate expiry is avoidable and slow.

Once collected, aggregate locally:

# Score distribution across the sample
jq -s '[.[] | .score] | {
  n: length,
  mean: (add / length),
  min: min,
  max: max
}' out/*.json

# The twenty most common failing checks
jq -r '.problems[]?.id' out/*.json | sort | uniq -c | sort -rn | head -20

Interpreting a portfolio

Two habits keep large-sample conclusions honest. First, check how many sites returned undetermined checks — a cohort behind aggressive bot protection will look systematically different for reasons that have nothing to do with quality. Second, confirm the ruleset version is identical across the run; a sweep spanning a ruleset update is comparing two different measurements.