Find out whether Googlebot, Bingbot and other crawlers may crawl a URL — and which line of robots.txt decides it. Uses Google’s rules: the most specific user-agent group, the longest matching path, Allow wins ties, and * / $ wildcards.
robots.txt controls crawling, not indexing. A blocked page can still appear in Google (without a snippet) if other sites link to it — use noindex to keep a page out.
A robots.txt that returns a 5xx error makes Google pause crawling the whole site.
Pages fixed and ready? Send Googlebot to them in minutes — credit back if it doesn't visit.
Submit URLs free