Robots.txt, noindex, and canonical tags: which one actually keeps a page out of search

"Can you just noindex it in robots.txt" is a request we get regularly, and it doesn't quite make sense - because robots.txt, the noindex directive, and the canonical tag are three separate mechanisms that solve three different problems. Picking the wrong one is how a page you wanted removed ends up permanently stuck in the results with a blank description.

It's worth being precise about what each one does, because the failure modes are specific and they compound.

Robots.txt controls crawling, not indexing

A `Disallow` rule in robots.txt tells Googlebot not to fetch the page. It does not tell Google to keep the URL out of the index. If other pages link to it, Google can still list the URL - usually with no snippet and a "No information is available for this page" line. And because Google was told not to crawl it, it never sees any noindex tag you put on the page. Blocking a page in robots.txt and adding a noindex tag to it at the same time means the noindex is never read.

Noindex is the only reliable way to remove a page

A `<meta name="robots" content="noindex">` tag in the page head, or an `X-Robots-Tag: noindex` HTTP header, is a directive Google honours. The page has to stay crawlable for the tag to be seen. Once Google processes it, the page drops out of the index and stays out as long as the tag remains - remove it later and the page becomes eligible again. If you need the removal to happen quickly, Search Console's removals tool hides the URL for around six months while the noindex propagates on its own.

Canonical is a hint, not a command

A `rel="canonical"` tag tells Google which URL, among a set of duplicates or near-duplicates, you consider the primary one. Google usually respects it, but it can and does override it when other signals disagree - internal links, the sitemap, or the page content itself pointing somewhere else. It does not remove anything: both URLs stay crawlable, and either can still appear in results. Use it to consolidate duplicates onto one URL, never to suppress a page you want gone.

The combinations that quietly break

• noindex plus a robots.txt Disallow on the same URL - the crawler never reaches the page, so the noindex is never applied.
• A canonical pointing at a URL that is itself noindexed - a contradiction, and Google may end up dropping both.
• A canonical that points at a redirect, or a chain of pages each canonicalising to the next.
• Using canonical to hide staging, thank-you, or filter pages instead of noindexing them - they stay indexable.
• Blocking a page in robots.txt and then filing a bug because it still shows up in search.

Which one for which job

1. Page should never appear and never be crawled (admin screens, internal search results): add noindex first, wait for it to leave the index, then optionally add a Disallow to save crawl budget - in that order, not both from day one.
2. Parameter or duplicate versions of a real page: canonical to the primary URL, and keep every version crawlable.
3. Thin, thank-you, or checkout pages: noindex, leave them crawlable.
4. An entire environment like staging: put it behind HTTP authentication, not robots.txt.

To check what a live page is actually telling Google, view its HTML source or use Search Console's URL Inspection. Our SEO Analyzer pulls the robots meta tag and the canonical URL out of a page's markup so you can see both in one place, and the Sitemap Validator flags URLs you're submitting in a sitemap that are actually set to noindex - one of the more common contradictions we find.

Almost every "why is this page still in Google" case we look at comes down to the same thing: it was blocked in robots.txt, so Google could never crawl it to see the noindex that would have removed it.

Does robots.txt remove a page from Google?

No. Robots.txt only stops Google crawling the page; the URL can still be indexed and shown if other sites link to it, just without a description. Worse, a page it can't crawl is a page where it never sees a noindex tag. To remove a page, allow crawling and add a noindex directive.