Site Spider
A free site crawler. No signup, no card, no trial.
Type a domain. The crawler follows your internal links, reads your sitemap, obeys robots.txt, and stores every page, image, script and stylesheet it fetches, up to 5,000 pages per scan.
You get a private link to the results while it is still running, fifteen SEO checks against the stored crawl, and the whole crawl as a database you can download. There is no account step anywhere in that sentence.
An SEO audit without creating an account
Most crawlers put the interesting part behind a signup, a trial or a seat. The limits here do not move when you register, because there is no tier to sell you: the scan you can run as a stranger is the scan an account holder runs.
While the crawl runs, your results live at an unguessable URL that only you have. Signed out, that link is remembered in this browser so the scan is still in your list when you come back. If you later create an account, the scans from this browser are attached to it automatically, so nothing is lost by starting anonymously.
Every finished scan also gets a public summary page under /domain: the score, the counts and a one-line summary per check. The row-level detail, the stored pages and the downloadable database stay on the private link.
What the crawl gives you
Fifteen checks, each one a list of URLs and what is wrong with them: titles, meta descriptions, headings, broken links, redirects, non-200 responses, indexability, sitemap coverage, image alt text, heavy assets, orphan pages, internal link counts, thin content, duplicate content and rate-limited URLs.
A health score out of 100, computed from the share of pages each check flags, so two scans of the same site are comparable and so is every other site that has been scanned.
A fix plan, written from your own scan results, listing what to change first and why it matters for that site.
A passive security review of what the crawl saw: security headers, cookie flags, mixed content, version banners and known-vulnerable front-end libraries. It reads what was served; it does not probe or attack anything.
An internal link map you can drag around, drawn from the links between your own pages, with orphans and clusters visible at a glance.
The whole crawl as one file: a SQLite database you can download with every fetched page and asset in it, so the data outlives this site and the analysis is not limited to the checks we thought of.
What it deliberately does not do
It does not run JavaScript. It does not crawl off your domain, so links to other sites are recorded but not fetched or status-checked. It does not log in, submit forms or touch anything behind authentication. It will not crawl private or internal addresses.
Those limits are why it is safe to point at a live production site, and they are worth knowing before you read a clean result as proof that everything is fine.
Questions people ask
- Can I run an SEO audit without creating an account?
- Yes. Enter a domain and the crawl starts. You get a private, unguessable link to the results, and every check runs there. Nothing asks for an e-mail address, and nothing is held back for signed-in users.
- What does an account add?
- Somewhere to keep scans. Signed in, your scans are listed together, you can put a domain on a repeat schedule, and you get e-mailed when a scheduled scan finds something different. Scans you ran before signing up are attached to your account automatically. The crawl limits are the same either way.
- How many pages will it crawl?
- Up to 5,000 pages and 1,000 assets per scan, staying on the domain you entered. That is the limit for everybody; there is no paid tier that raises it. Files over 10 MB are counted and skipped rather than stored.
- Will crawling hurt my server?
- The crawler fetches one URL at a time per scan with a pause between requests, identifies itself as DIYSEOHubBot, and follows your robots.txt. If your server answers with 429, it backs off and waits, and it gives up on a site that keeps refusing rather than hammering it.
- Does it run JavaScript?
- No. It reads the HTML your server sends, which is what a crawler sees first. If your content only appears after JavaScript runs in the browser, this crawl will not see it, and that is itself worth knowing.
- Can I crawl a site I do not own?
- The crawler only reads pages that are already public, the way any search engine does, and every scan gets a public summary report. Use it on sites you own or are authorised to check. Site owners can claim their listing and unlist it.
Crawl a site now
Read next
- The DIY SEO guide — what to do with the findings, in the order that pays off.
- Free broken link checker — the broken-link half of the crawl, on its own.
- Download your crawl as SQLite — the schema, and the queries worth running.
- How this site is built — event sourcing, one SQLite database per scan.