How do I find every page on a website?
Enumerating a site's URLs from sitemaps, crawling and internal links — and why no method gives you a complete list.
Start with sitemap.xml, then crawl internal links from the homepage, then compare the two. The gaps between those lists are the interesting part: pages in the sitemap that nothing links to, and pages linked from the site that the sitemap omits.
Start with the sitemap
curl -s https://example.com/robots.txt | grep -i sitemap
curl -s https://example.com/sitemap.xml | grep -oE '<loc>[^<]+' | sed 's/<loc>//'robots.txt is the right first stop because a sitemap need not live at the conventional path, and large sites often use a sitemap index pointing at several child files. If the first command returns nothing, try /sitemap.xml, /sitemap_index.xml and /sitemap-index.xml before concluding there is none.
Then crawl the links
A sitemap is a claim about what should exist. Crawling shows what is actually reachable by following links, which is what a search engine mostly does.
curl -sL https://example.com \
| grep -oE 'href="[^"]+' \
| sed 's/href="//' \
| grep -E '^(/|https://example\.com)' \
| sort -uThat is one level deep. Real crawling means following each result, deduplicating, respecting robots.txt and not hammering the origin — which is why it is usually worth using a tool rather than a shell loop.
The gaps are the finding
| Situation | What it means |
|---|---|
| In the sitemap, not linked anywhere | Orphaned. Crawlable in principle, but nothing signals it matters. |
| Linked, not in the sitemap | Sitemap is stale or generated from a partial source |
| In neither, but exists | Reachable only by direct URL — invisible to search entirely |
| In the sitemap but returns 404 | Sitemap generated from a database rather than from reality |
| In the sitemap but carries noindex | Directly contradictory instructions |
The last row is worth checking on any site with a generated sitemap. Telling a search engine "here is an important page" and "do not index this page" in the same breath is a common templating accident.
Why no list is ever complete
It is worth being clear that complete enumeration of a website from the outside is not possible in general.
- Pages behind authentication. Not reachable, and correctly so.
- Pages generated from parameters. Search results and filtered views are effectively infinite.
- Pages with no inbound link, absent from the sitemap. Undiscoverable without guessing.
- Content rendered client-side. A crawler that does not execute JavaScript sees no links at all if navigation is built in the browser.
What to do with the result
- Fix contradictions first — sitemap entries returning 404, or carrying noindex.
- Link orphaned pages from somewhere relevant, or accept they are not meant to be found and remove them from the sitemap.
- Check crawl depth: anything more than three clicks from the homepage gets crawled less often and less reliably.
- Confirm the sitemap is regenerated on publish rather than by hand — hand-maintained sitemaps go stale immediately.
- Verify links resolve without a redirect chain. Every hop is a place to lose signal.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- Whether a sitemap exists, is referenced from robots.txt, is valid, and whether its entries actually resolve.
- Internal link structure and crawl depth, identifying pages buried more than three clicks from the homepage.
- Contradictions between sitemap inclusion and indexability directives such as noindex or canonical.
- How much of the navigation exists in the raw HTML versus being constructed by JavaScript.
For agents and scripts, the same measurement is at
/api/v1/discoverability?url=yoursite.com —
see the API documentation.
Related questions
Is crawling someone else's site acceptable?
Following public links at a reasonable rate is what search engines do. Respect robots.txt, identify yourself honestly in the user agent, and pace requests — the difference between a crawl and a nuisance is almost entirely about rate.
Why does my crawl only find one page?
Almost always client-side routing. If navigation is built by JavaScript, the raw HTML contains no links to follow, so a crawler stops at the entry point. This affects AI crawlers far more than Googlebot.
How deep should a site be?
Every page that matters should be within three clicks of the homepage. Beyond that, crawl frequency drops noticeably and pages are re-checked less often, so updates take longer to register.
Do I need a sitemap if my site is small?
For a small, well-linked site it adds little, since crawlers will find everything by following links. It becomes valuable for large sites, for pages with few internal links, and for signalling update frequency.
Read next
Do I need a sitemap.xml?
Not strictly. It helps discovery on larger or poorly-linked sites, and it costs nothing — but it will not fix a site that cannot be crawled.
ReadDoes JavaScript hurt AI crawlers?
Yes, more than it hurts search engines. If your content only exists after JavaScript runs, most AI crawlers see an empty page.
ReadHow do I fix robots.txt blocking Google?
How to read the file correctly, what the precedence rules actually are, and the three mistakes that cause almost every accidental block.
ReadWhat is a web property graph?
Seeing a domain as a connected set of observations — hosts, certificates, records and dependencies — rather than as a single page.
Read