How can I see all the pages on a website?

Crawl it. A crawler starts at the home page, follows every internal link and reads the sitemap, and ends with a list of every address it found and how each one answered. Enter the domain above for a list of up to 5,000 pages, free.

Can I find all the pages on a website I do not own?

Yes, the public ones. A crawl needs no access to the site: it reads what any visitor could. It obeys robots.txt, so closed sections are not fetched. Tick “Keep this scan private” under More options if you would rather there were no public report.

How do I find hidden pages on a website?

Pages nothing links to cannot be found by following links. Look where they leave a trace: the XML sitemap, old addresses that redirect, Search Console for a site you own, and the CMS or server logs. A page in none of those cannot be found from outside.

Why does a site: search show a different number of pages?

Because it counts what the search engine has indexed, not what exists, and the count is an estimate. It leaves out pages that are not indexed and can include addresses that no longer exist.

Can I export the list of URLs?

Yes. Every table in a scan exports as a CSV, and the whole crawl downloads as a SQLite database with one row per URL.

How to

How to find all pages on a website

The complete way to find all the pages on a website is to crawl it: start at the home page, follow every internal link, read the sitemap, and keep the list. The other methods are quicker and each leaves something out.

Enter a domain to crawl it now, up to 5,000 pages, or read on for all five methods and what each one misses.

One email when the scan finishes, with the score and every check. No newsletter.

More options: paths, depth, JavaScript rendering, privacy

What to crawlOnly crawl paths matching (one per line)Leave out paths matching (one per line)Maximum link depth from the start pageDrop these query parameters (comma-separated, * allowed)

Drop tracking parameters utm_*, gclid, fbclid and the like, so tagged links are not crawled twice.Stay inside the start folder Start at /blog/ and only pages under /blog/ are crawled.Crawl subdomains of this host Starting at example.com includes shop.example.com and blog.example.com. Starting at blog.example.com includes only that host and hosts beneath it.Lower-case paths Treat /About and /about as the same page.

Include JavaScript rendering checks Each page is also loaded in a real browser and compared with the HTML the server sent. Slower.Keep this scan private No public report page, no place in the rankings or the sitemap.

Crawls up to 5000 pages plus their images, scripts and stylesheets. No account needed: you get a private link to the results.

1. Read the XML sitemap

Most sites publish a list of their own pages. Try /sitemap.xml after the domain, or look for a Sitemap: line in /robots.txt. The sitemap validator finds it for you and counts the entries.

What it misses: whatever the owner or the CMS left out, which is often a great deal, and it may list pages that no longer exist. A sitemap is what the site says it has.

2. Search with site:

Type site:example.com into Google or Bing to see pages of that domain in the index. Add a word to narrow it: site:example.com pricing.

What it misses: every page that is not indexed, and you can only page through a few hundred results. It answers “what has the search engine kept”, not “what is there”.

3. Ask Search Console (your own site)

The Pages report under Indexing lists the addresses Google knows about, indexed and not, with the reason for each. It is the only source that includes pages Google found through links from other sites.

What it misses: pages Google has never discovered, and the export stops at 1,000 rows per report. It also needs you to have verified the site, so it is no help with someone else's.

4. Export from the CMS

WordPress lists every post and page in its admin and can export them; Shopify exports products and pages as CSV; most platforms have something similar. This is the source of truth for what was created.

What it misses: what is actually served. Archive, tag, search and filter addresses are generated on the fly and appear in no export, and a published page can still answer with an error or a redirect.

5. Crawl the site

A crawler does what a search engine does: fetch a page, collect its links, fetch those, and repeat until there is nothing new. It finds the generated addresses the CMS does not list and the pages the sitemap forgot, and it records how each one answered.

The crawl on this page gives you the list in four forms once it finishes:

Link Score lists every page that answered 200, with how many pages link to it. Export it as a CSV and you have the working pages of the site.
Non-200 responses and Redirects list everything else: the errors, and the addresses that forward somewhere.
Download an XML sitemap gives the indexable pages as a ready-made sitemap. See the sitemap generator.
The SQLite database holds every URL the crawl saw, pages and files alike. One query lists them:

SELECT url, status_code, depth
FROM urls
WHERE kind = 'page'
ORDER BY depth, url;

What it misses: pages nothing links to and no sitemap lists, pages behind a login, and sections robots.txt closes to crawlers. Links that only appear after JavaScript runs are followed when you tick “Include JavaScript rendering checks”.

Putting the methods together

No single source is complete, and the gaps between them are the interesting part. The crawl here already combines two: it follows links and reads the sitemap, and reports the difference in both directions: pages only the sitemap knows, and crawled pages the sitemap leaves out.

For your own site, add the third. Import a Search Console export on the scan page and the pages Google shows that the site no longer links to are listed as well. The orphan page checker covers that in full.

To check a list you already have, such as a CMS export, choose list mode under More options and paste the addresses: the crawl fetches exactly those and tells you how each one answers.

Once you have the list

A list of pages is usually the start of another job: a migration, a content audit, a redirect map. The same crawl has already run 37 checks over those pages, so the titles, word counts, status codes and link counts you would gather next are there. The SQL cookbook has twenty queries to start from.

Questions people ask

How can I see all the pages on a website?: Crawl it. A crawler starts at the home page, follows every internal link and reads the sitemap, and ends with a list of every address it found and how each one answered. Enter the domain above for a list of up to 5,000 pages, free.
Can I find all the pages on a website I do not own?: Yes, the public ones. A crawl needs no access to the site: it reads what any visitor could. It obeys the site's robots.txt, so sections the owner has closed to crawlers are not fetched. If you would rather the result not appear as a public report, tick “Keep this scan private” under More options.
How do I find hidden pages on a website?: Pages nothing links to cannot be found by following links. Look where they leave a trace: the XML sitemap, which the crawl reads for you; old addresses that redirect; Search Console, for a site you own; and the CMS or server logs. A page in none of those cannot be found from outside.
Why does a site: search show a different number of pages?: Because it counts what the search engine has indexed, not what exists, and the count it shows is an estimate. It leaves out pages that are not indexed and can include addresses that no longer exist.
Can I export the list of URLs?: Yes. Every table in a scan exports as a CSV, and the whole crawl downloads as a SQLite database with one row per URL.

List every page of a site

A crawl of up to 5,000 pages, exported as a CSV or a database. Free, no signup.

Scan a site, free