Near-duplicate content
Near-duplicate content checker for your own site
Exact copies are easy to find: two files are identical or they are not. The duplicates that cause trouble are the near ones, pages that differ by a place name, a date or a product option and read as the same page to a search engine.
Enter a domain. The crawl fingerprints the readable text of every page and groups the pages that are about 95% the same as another, with the closest match beside each one.
What you get
One row per page, grouped so a cluster of twelve similar pages reads as one problem:
- The group it belongs to, and how many pages are in it.
- The page it is most like, and how alike they are as a percentage.
- Its word count, because near-duplicates of thirty words and of three thousand are different problems.
The table exports as a CSV. Byte-identical pages are reported separately by the Duplicate content check in the same scan, since those have a different cause: usually one page reachable at two addresses.
How the comparison works
Only the content area is compared. Navigation, headers, footers, sidebars, scripts and styles are left out before anything is measured. Otherwise every page on a site with a large menu would look like every other.
Each page gets a SimHash. The text is cut into overlapping three-word sequences and reduced to a 64-bit fingerprint with the property that similar texts get similar fingerprints. Pages with fewer than fifty words of content are not fingerprinted: there is too little text to compare.
Fingerprints within three bits are a match. That is roughly 95% the same text. The pages are found without comparing every pair, so a crawl of thousands of pages costs no more than a small one.
Pages already dealt with are left out. A page that is noindex, or whose canonical points at another page, has told search engines what to do. Only pages competing for the index are compared.
Where near-duplicates come from
Location and service pages built from one template with the town swapped. The commonest case, and the one most worth rewriting.
Product variants, one address per size or colour with the same description.
Addresses with parameters: sorting, filtering, tracking and session parameters that produce the same page with a line changed.
Archives: tag, category, author and date pages that list the same few posts.
Print and mobile versions, and staging copies left reachable.
Fixing a group
One page should win. Add rel="canonical" on the others pointing at it. The copies stay usable and the search engine is told which one counts. The canonical checker confirms a single page afterwards.
Only one page is needed. Merge them and 301 the others to the survivor. This is the strongest fix, because links to the copies follow the redirect.
Each page has a reason to exist. Then make them different: real detail about that location, that product, that audience. If there is nothing different to say, there was one page.
The copies are for visitors, not for search. Filtered and sorted listings can carry noindex. To see what the site looks like without them first, the scan form can drop chosen query parameters before crawling.
Where it stops
The threshold is fixed at about 95%. Pages that share half their text are not reported, though they may still compete with each other; duplicate and near-identical titles are a better signal for those, and the title check lists them. The comparison reads the HTML the server sent, so text that only appears after JavaScript runs is not part of the fingerprint.
The crawl runs 37 checks in all. Thin pages, soft 404s and exact duplicates are reported beside this one.
Questions people ask
- What is near-duplicate content?
- Two pages at different addresses that say almost the same thing: the same paragraphs with a city name swapped, the same product in another colour, the same article on a print-friendly address. Not byte-for-byte copies, which are exact duplicates, but close enough that a search engine sees one page twice.
- Does this check my pages against other websites?
- No. It compares the pages of one site with each other. That is the duplicate content a site owner causes and can fix. Finding copies of your text elsewhere on the web is plagiarism detection, a different job needing an index of the whole web.
- Is there a duplicate content penalty?
- Not in the sense of a punishment. What happens is quieter: the search engine picks one of the near-identical pages to show and filters the rest, its choice may not be yours, links pointing at the copies are split between them, and crawling is spent on pages that add nothing.
- How similar do two pages have to be?
- About 95% alike. Each page's readable text is reduced to a 64-bit fingerprint, and two pages are reported when their fingerprints differ in three bits or fewer. Pages that share a template but have different content are well outside that.
- How many pages can it compare?
- Up to 5,000 pages per crawl, free and without an account. Every page is compared with every other in effect, without the crawl slowing down as the site grows.
Find the pages that say the same thing twice
Read next
- Website spell checker — words an English dictionary does not know, across every page of the site, with how many pages use each one.
- Bulk schema markup validator — JSON-LD, microdata and RDFa on every page: what does not parse, and what a rich result requires that is missing.
- Whole-site accessibility checker — axe-core run in a real browser on every page of the crawl, not only the home page, worst impact first.
- The free site crawler — the crawl all of these checks read from, and what else it records.