Indexing Checklist for a New Website

A new website starts with nothing: no crawl history, no external links and no track record with search engines. Crawlers are cautious with unknown domains, so every technical mistake that blocks or confuses them costs you more than it would on an established site.

The good news is that most indexing problems on new sites come from a short list of avoidable mistakes. Work through this checklist before launch, then again a week after.

Before launch

1. Remove staging blocks from robots.txt

Staging sites are often protected with a blanket disallow:

User-agent: *
Disallow: /

If that file goes live with the site, nothing gets crawled. Check the production robots.txt and make sure it only blocks what you actually want blocked, such as admin areas, internal search results and cart pages. Add a line pointing to your sitemap. Test important URLs with the robots.txt tester.

2. Remove leftover noindex tags

Many CMS platforms have a "discourage search engines" setting that adds noindex to every page. Developers also add X-Robots-Tag: noindex headers to staging servers. Both regularly survive launch. Check the HTML source and the response headers of your key templates (homepage, category, article, product).

Remember that robots.txt and noindex interact. If a page is blocked in robots.txt, the crawler can't see its noindex tag. Use one mechanism per URL, not both.

3. Check canonical tags

Every indexable page should have a self-referencing canonical with the final production URL. Look out for:

• Canonicals still pointing at the staging domain
• Every page canonicalising to the homepage, a common template bug
• http vs https, or www vs non-www mismatches
• Paginated pages all canonicalising to page one, which can stop deeper content from being discovered

4. Choose one URL version and redirect the rest

Pick one version (for example https://www.example.com) and 301-redirect the others to it. Keep trailing-slash behaviour consistent. Avoid redirect chains: each old URL should go to its final destination in one hop.

5. Check status codes

Every page you want indexed should return 200. Missing pages should return a real 404 or 410, not a 200 with a "not found" message (a soft 404). Server errors (5xx) during crawler visits slow crawling down for the whole site.

You can check status, robots directives and canonicals for a list of URLs in one pass with the indexability checker.

6. Build a clean XML sitemap

Your sitemap should only contain canonical, indexable URLs that return 200. Include accurate lastmod dates. Leave out redirected URLs, noindex pages, parameter URLs and paginated archives you don't want indexed. For larger sites, split into several sitemaps under an index file (up to 50,000 URLs or 50 MB uncompressed per file). Check it with the sitemap validator.

7. Make sure content is visible without JavaScript problems

If your site relies on client-side rendering, check that the main content and navigation links appear in the rendered HTML. Server-side rendering or static generation for key pages avoids most problems. Links should be normal <a href> elements, not click handlers.

At launch

8. Verify in webmaster tools

• Google Search Console: verify (a domain property is ideal), then submit your sitemap
• Bing Webmaster Tools: import from Search Console or verify separately, then submit your sitemap. This also covers Yahoo, and much of DuckDuckGo's web results come from Bing.
• Yandex Webmaster: if you target markets where Yandex is used

9. Set up IndexNow

Host an IndexNow key and configure your CMS or CDN to ping on publish and update. It reaches Bing, Yandex, Seznam, Naver and Yep. It doesn't reach Google, which relies on sitemaps, links and Search Console.

10. Build internal links deliberately

Every page you want indexed should be reachable through normal links from the homepage, ideally within a few clicks. Use clear navigation, category pages, breadcrumbs and contextual links in your content. Orphan pages that only appear in the sitemap are the usual cause of "Discovered – currently not indexed".

After launch

11. Publish pages that deserve indexing

Search engines are selective, especially with new domains. Pages most likely to be left out:

• Thin pages with a few sentences and lots of template
• Near-duplicate pages, such as location pages with only the city name changed
• Empty category or tag pages
• Pages copied from manufacturers or other sites

Start with fewer, stronger pages. It's better to launch with 30 substantial pages than 3,000 thin ones. Add tag archives and filtered pages only once they have real content, or keep them noindex.

12. Earn some first links, then monitor

A few genuine links from relevant, crawled sites (partners, directories that people actually use, local organisations, press coverage) give crawlers a route in and a reason to come back. Then keep watching:

• The Page indexing report in Search Console, weekly
• Crawl stats, for errors and response times
• Specific URLs with URL Inspection or the index checker

If pages show up as "Crawled" or "Discovered – currently not indexed", our guide on why Google is not indexing your page explains what each status means and what to do.

The short version

1. Production robots.txt allows crawling and lists the sitemap
2. No stray noindex tags or headers
3. Self-referencing canonicals on the production domain
4. One URL version, with single-hop 301s
5. 200 for real pages, 404 or 410 for missing ones
6. A clean sitemap with accurate lastmod
7. Content and links present in the rendered HTML
8. Verified in Google, Bing (and Yandex if relevant)
9. IndexNow configured
10. No orphan pages
11. Substantial, original content
12. A few genuine first links, and regular monitoring

Launching with QuickIndexx

Once your checklist is clean, QuickIndexx can help new URLs get their first crawl sooner. Submit your launch pages and see each crawler visit as it happens. If no crawl is recorded within 24 hours, the credit is returned. Search engines still decide what to index. We help make sure they look. Plans are on the pricing page.