Download your site crawl as a SQLite database

Run a scan and the download button is on your private scan page. Every finished scan also offers the same database from its public report under /domain, so you can take a crawl away without having an account at all.

The file is a normal SQLite database, snapshotted while the scan is quiet so it is consistent. Nothing needs importing.

sqlite3 scan.db
sqlite> .tables
sqlite> SELECT pages_fetched, assets_fetched, total_bytes FROM scan;

Twelve tables. The crawl is stored as an append-only event stream, and everything else is a projection of it, which means the tables you query are derived and can be rebuilt from the events if a question ever needs different shaping.

The fifteen checks on the site are SQL over these tables. Here are the ones people write first.

Duplicate titles, worst first

SELECT title, COUNT(*) AS pages
FROM pages
WHERE title IS NOT NULL AND TRIM(title) <> ''
GROUP BY title
HAVING COUNT(*) > 1
ORDER BY pages DESC;

Every broken internal link, with the page that links to it

SELECT l.from_url, l.to_url, u.status_code, u.error, l.text
FROM links l
JOIN urls u ON u.url = l.to_url
WHERE l.internal = 1
  AND ((u.status_code >= 400 AND u.status_code != 429) OR u.state = 'failed')
ORDER BY l.to_url;

Slowest pages the crawler saw

SELECT url, duration_ms, content_length
FROM urls
WHERE kind = 'page' AND state = 'fetched'
ORDER BY duration_ms DESC
LIMIT 20;

Pages nothing links to

SELECT p.url, p.title, p.word_count
FROM pages p
LEFT JOIN links l ON l.to_url = p.url AND l.internal = 1
WHERE l.to_url IS NULL
ORDER BY p.word_count DESC;

Heaviest assets, and how many pages carry them

SELECT u.url, u.content_length, COUNT(DISTINCT r.page_url) AS used_on
FROM urls u
JOIN resources r ON r.url = u.url
WHERE u.kind = 'asset' AND u.content_length IS NOT NULL
GROUP BY u.url, u.content_length
ORDER BY u.content_length DESC
LIMIT 20;

In your sitemap but never crawled, and the other way round

SELECT s.url AS in_sitemap_not_crawled
FROM sitemap_entries s
LEFT JOIN urls u ON u.url = s.url AND u.state = 'fetched'
WHERE u.url IS NULL;

The stored HTML of one page, exactly as it was served

SELECT CAST(b.body AS TEXT)
FROM urls u
JOIN blobs b ON b.hash = u.body_hash
WHERE u.url = 'https://example.com/';

That last one is the point of the whole design. The crawl is not a report about your site; it is your site, as a search engine received it, on a date, in a file you hold.

Storing every fetched byte is not free, which is why most tools keep a summary and throw the rest away. Keeping it turns a crawl from a moment into a record. Two scans a month apart can be diffed properly, not by comparing two dashboards but by joining two tables. A question that occurs to you in March can be asked of January's crawl.

Content addressing makes it affordable: files are keyed by hash, so a logo on every page is stored once. A scan of up to 5,000 pages stays a file you can e-mail to a colleague.

It also means there is no lock-in to argue about. The export is not a feature bolted on beside the product; it is the product, handed over.