noindex tags and robots.txt blocks

The most common reason a page is crawled but never indexed is that the page itself tells search engines not to index it. These settings override any indexing signal, ours included.

The four usual blockers

1. A noindex meta tag

<meta name="robots" content="noindex">

Google can crawl the page, sees this tag, and drops it from the index. SEO plugins often add it to tag pages, archives, staging sites, or whole sites when "Discourage search engines" is ticked in WordPress.

2. An X-Robots-Tag header

The same instruction can be sent as an HTTP header (X-Robots-Tag: noindex). It's invisible in the page source, so it's easy to miss. Check it with the HTTP status & header checker.

3. A robots.txt Disallow rule

User-agent: *
Disallow: /blog/

A disallowed URL can't be crawled at all. Test any URL against your live robots.txt with the robots.txt tester. Need a clean file? Use the robots.txt generator.

Don't block a page in robots.txt to remove it from Google. If Googlebot can't fetch the page, it can't see a noindex either.

4. A canonical pointing elsewhere

If page A has <link rel="canonical" href="page-B">, Google will usually index B instead of A. Check with the Canonical & hreflang checker.

Find all of them at once

Run the page through the free Indexability checker. It checks the status code, robots.txt, meta robots, X-Robots-Tag, canonical, redirects and more, and grades the page A to F with plain-English fixes.

Fixing and resubmitting

1. Remove the noindex (in your SEO plugin, theme or server config), fix the robots.txt rule or correct the canonical.
2. Clear any page cache or CDN cache so Googlebot sees the new version.
3. Re-run the Indexability checker to confirm.
4. In History, click Resubmit (1 credit).

Before you submit in bulk

If you're sending a sitemap or a big list, spot-check a few URLs from each section first. One leftover noindex in a template can affect hundreds of pages. See also Crawled, not indexed.