Site Spider
A free site crawler. No signup, no card, no trial.
Type a domain. The crawler follows your internal links, reads your sitemap, obeys robots.txt, and stores every page, image, script and stylesheet it fetches, up to 5,000 pages per scan.
You get a private link to the results while it is still running, 37 SEO checks against the stored crawl, and the whole crawl as a database you can download. There is no account step anywhere in that sentence.
An SEO audit without creating an account
Most crawlers put the interesting part behind a signup, a trial or a seat. The limits here do not move when you register, because there is no tier to sell you: the scan you can run as a stranger is the scan an account holder runs.
While the crawl runs, your results live at an unguessable URL that only you have. Signed out, that link is remembered in this browser so the scan is still in your list when you come back. If you later create an account, the scans from this browser are attached to it automatically, so nothing is lost by starting anonymously.
Every finished scan also gets a public summary page under /domain: the score, the counts and a one-line summary per check. The row-level detail, the stored pages and the downloadable database stay on the private link.
What the crawl gives you
37 checks, each one a list of URLs and what is wrong with them: titles, meta descriptions, headings, broken links, redirects, non-200 responses, indexability, sitemap coverage, canonicals, hreflang, structured data, image alt text, heavy assets, orphan pages, internal links, thin, duplicate and near-duplicate content, soft 404s, spelling and more.
A health score out of 100, computed from the share of pages each check flags, so two scans of the same site are comparable and so is every other site that has been scanned.
A fix plan, written from your own scan results, listing what to change first and why it matters for that site.
A passive security review of what the crawl saw: security headers, cookie flags, mixed content, version banners and known-vulnerable front-end libraries. It reads what was served; it does not probe or attack anything.
An internal link map you can drag around, drawn from the links between your own pages, with orphans and clusters visible at a glance.
The whole crawl as one file: a SQLite database you can download with every fetched page and asset in it, so the data outlives this site and the analysis is not limited to the checks we thought of.
One crawl, and a page for each question
Most people arrive with one question. Each of these is answered by the same crawl, and has a page of its own that explains how:
- Orphan page checker — Pages that answer 200 and that no other page links to, found through your sitemap, your redirects and your Search Console export.
- Internal link checker — Links in and out of every page, click depth, anchor text, where on the page each link sits, and an internal PageRank score.
- Noindex checker — Every page carrying noindex or another robots directive, in a meta tag or an X-Robots-Tag header, across the whole site.
- Soft 404 checker — Pages that answer 200 OK while their title or heading says the page is not there.
- Near-duplicate content checker — Pages on your own site whose text is 95% the same as another page, grouped, with the closest match named.
- Website spell checker — Words an English dictionary does not know, across every page of the site, with how many pages use each one.
- Bulk schema markup validator — JSON-LD, microdata and RDFa on every page: what does not parse, and what a rich result requires that is missing.
- Whole-site accessibility checker — axe-core run in a real browser on every page of the crawl, not only the home page, worst impact first.
- XML sitemap generator — An XML sitemap of up to 5,000 indexable pages built from a crawl, and an image sitemap beside it.
- Find all pages on a website — Five ways to list every URL of a site, what each one misses, and a crawl that gives you the list as a CSV.
- Website source code search — Search the HTML of every page for text, a regular expression or a CSS selector, and export what it pulls out.
- SEO MCP server — Point Claude or another assistant at a finished crawl of your site over MCP: one command, nothing to install.
What it deliberately does not do
It does not run JavaScript unless you ask: turn on “Use browser” under the address box and each page is loaded in a browser too. It does not crawl off your domain: links to other sites are checked once for errors, never crawled. It does not log in, submit forms or touch anything behind authentication. It will not crawl private or internal addresses.
Those limits are why it is safe to point at a live production site, and they are worth knowing before you read a clean result as proof that everything is fine.
Questions people ask
- Can I run an SEO audit without creating an account?
- Yes. Enter a domain and the crawl starts. You get a private, unguessable link to the results, and every check runs there. Nothing asks for an e-mail address, and nothing is held back for signed-in users.
- What does an account add?
- Somewhere to keep scans. Signed in, your scans are listed together, you can put a domain on a repeat schedule, and you get e-mailed when a scheduled scan finds something different. Scans you ran before signing up are attached to your account automatically. The crawl limits are the same either way.
- How many pages will it crawl?
- Up to 5,000 pages and 1,000 assets per scan, staying on the domain you entered. That is the limit for everybody; there is no paid tier that raises it. Files over 10 MB are counted and skipped rather than stored.
- Will crawling hurt my server?
- Each scan starts with one request at a time and a pause between them, and only goes up to four at once while your server answers quickly; a slow answer or an error slows it straight back down. It identifies itself as DIYSEOHubBot, follows your robots.txt and honours a Crawl-delay in it. If your server answers with 429, it backs off and waits, and it gives up on a site that keeps refusing rather than hammering it.
- Does it run JavaScript?
- Only if you ask. By default it reads the HTML your server sends, which is what a crawler sees first. Turn on “Use browser” under the address box and every page is also loaded in a headless browser, and the JavaScript check lists what only appears, or changes, once scripts run.
- Can I crawl a site I do not own?
- The crawler only reads pages that are already public, the way any search engine does, and every scan gets a public summary report. Use it on sites you own or are authorised to check. Site owners can claim their listing and unlist it.
Read next
- The DIY SEO guide — what to do with the findings, in the order that pays off.
- Free broken link checker — the broken-link half of the crawl, on its own.
- Download your crawl as SQLite — the schema, and the queries worth running.
- How this site is built — event sourcing, one SQLite database per scan.