How do I find all noindex pages on a website?

Crawl it and read two places on every page: the robots meta tag in the HTML and the X-Robots-Tag response header. Enter the domain above and the Robots directives check lists each page that carries an instruction, which one, and where it was declared.

What is the difference between a meta robots noindex and an X-Robots-Tag?

They mean the same thing and live in different places. The meta tag is in the page’s HTML. The X-Robots-Tag is an HTTP header set by the server or a CDN, invisible in the page, and able to cover whole folders, PDFs and images. Header noindex is the one that gets missed.

Can a page blocked in robots.txt be noindex?

Not usefully. A crawler that may not fetch a page never sees its noindex, so the address can still be indexed from links alone. To keep a page out of search, allow crawling and give it noindex. This crawler obeys robots.txt too, so blocked pages are counted as blocked and their tags are not read.

Does it catch a noindex added by JavaScript?

When you ask for it. Tick “Include JavaScript rendering checks” under More options and each page is also loaded in a browser; the JavaScript rendering check lists pages where scripts added or changed a noindex, a canonical or a title.

How many URLs can it check?

Up to 5,000 pages in one crawl, free and without an account. To check a specific list instead of a whole site, choose list mode under More options and paste the URLs.

Noindex and robots directives

Bulk noindex checker: find every noindex page on a site

A single noindex outranks every other SEO problem a page can have, because a page that may not be indexed earns nothing. It is also the fault most often shipped by accident: a staging setting that survived launch, a plugin default, a header added to one folder too many.

Enter a domain. The crawl reads the robots meta tag and the X-Robots-Tag header of every page and lists the ones that tell search engines to stay away.

One email when the scan finishes, with the score and every check. No newsletter.

More options: paths, depth, JavaScript rendering, privacy

What to crawlOnly crawl paths matching (one per line)Leave out paths matching (one per line)Maximum link depth from the start pageDrop these query parameters (comma-separated, * allowed)

Drop tracking parameters utm_*, gclid, fbclid and the like, so tagged links are not crawled twice.Stay inside the start folder Start at /blog/ and only pages under /blog/ are crawled.Crawl subdomains of this host Starting at example.com includes shop.example.com and blog.example.com. Starting at blog.example.com includes only that host and hosts beneath it.Lower-case paths Treat /About and /about as the same page.

Include JavaScript rendering checks Each page is also loaded in a real browser and compared with the HTML the server sent. Slower.Keep this scan private No public report page, no place in the rankings or the sitemap.

Crawls up to 5000 pages plus their images, scripts and stylesheets. No account needed: you get a private link to the results.

What the crawl reports

Robots directives. Every page that carries an instruction, with the instruction and where it was declared: meta tag, HTTP header, or both. It reads noindex, nofollow, none, noarchive, nosnippet, noimageindex, unavailable_after and the max-snippet, max-image-preview and max-video-preview limits. The summary says how many pages carry one and how many carry none.

Indexability. The contradictions. A noindex page that is still listed in your XML sitemap asks to be indexed and refuses in the same breath; the check lists each one.

Internal link counts. Noindex pages the site still links to, which is how you tell a deliberate noindex (a thank-you page nothing links to) from an accidental one (a category in the main menu).

Canonicals. Pages whose canonical points at a noindex page: two instructions that cannot both be followed.

Telling an accident from a decision

Most noindex pages are meant to be. Work down the list with three questions:

Is it in the sitemap? A sitemap is a list of pages you want indexed. A noindex page in it is a mistake in one place or the other.
Is it linked from the navigation or from page content? Pages you link to prominently are pages you expect people to find.
Is it a whole pattern? Every URL under /blog/ or every page of one template carrying the same header is a server or CMS setting, and one change undoes it.

Search results pages, cart and account pages, filtered listings and thank-you pages are the usual deliberate ones. Service pages, product pages and articles are not.

Where a noindex comes from

WordPress: “Discourage search engines from indexing this site” under Settings, Reading, and the per-page and per-post-type switches in an SEO plugin.

Site builders: a “hide this page from search results” switch in the page settings, sometimes on by default for new pages.

Servers and CDNs: an X-Robots-Tag header added in the web server configuration or a CDN rule. Check one address from the command line:

curl -sI https://yourwebsite.com/page | grep -i x-robots-tag

Frameworks: a metadata default in a layout that every page under it inherits, or a preview-environment setting that reached production.

After the fix, a page comes back into search only when it is crawled again. Request indexing for the important ones in Search Console and re-run the scan to confirm the directive is gone.

Where it stops

The crawl reads <meta name="robots"> and the X-Robots-Tag header. A tag addressed to one crawler only, such as <meta name="googlebot">, is not read. For that case, search the stored HTML of every page with the source search. It reports what the pages say, not whether Google has indexed them: that answer is in Search Console's Pages report.

For one address, the canonical checker and the robots.txt tester answer without a crawl. The crawl runs 37 checks in all.

Questions people ask

How do I find all noindex pages on a website?: Crawl it and read two places on every page: the robots meta tag in the HTML and the X-Robots-Tag response header. Enter the domain above and the Robots directives check lists each page that carries an instruction, which instruction, and where it was declared.
What is the difference between a meta robots noindex and an X-Robots-Tag?: They mean the same thing and live in different places. The meta tag is in the page's HTML, where anyone viewing the source can see it. The X-Robots-Tag is an HTTP header set by the server or a CDN, invisible in the page, and able to cover whole folders, PDFs and images at once. Header noindex is the one that gets missed.
Can a page blocked in robots.txt be noindex?: Not usefully. A crawler that is not allowed to fetch a page never sees its noindex, so the address can still be indexed from links alone, without its content. To keep a page out of search, allow it to be crawled and give it noindex. This crawler obeys robots.txt the same way, so blocked pages are counted as blocked and their tags are not read.
Does it catch a noindex added by JavaScript?: When you ask for it. Tick “Include JavaScript rendering checks” under More options and each page is also loaded in a browser; the JavaScript rendering check then lists pages where scripts added or changed a noindex, a canonical or a title.
How many URLs can it check?: Up to 5,000 pages in one crawl, free and without an account. To check a specific list instead of a whole site, choose list mode under More options and paste the URLs.

Check a whole site for noindex

Every page, both places a directive can hide. Up to 5,000 pages, free, no signup.

Scan a site, free