Keeping a URL out of Google's index sounds like it should be a one-line fix. In practice it's one of the most commonly misconfigured pieces of technical SEO, because there are three different mechanisms — robots meta tag, X-Robots-Tag header, and robots.txt — that look related but do genuinely different things, and combining them the wrong way silently cancels the one you actually need.
Auditing whether a noindex actually took effect on a live page starts with seeing what Google renders for that URL. Check a localized Google SERP for the page's target query to confirm it's actually gone from results, not just assumed to be.
Why You'd Noindex a URL at All
Noindex tells Google (and any other compliant crawler) "you may crawl this page, but don't put it in your index." It's the right tool for pages that need to exist and be reachable, but shouldn't compete for rankings or show up in search: internal search results pages, thin filtered/faceted URLs, staging or duplicate content, thank-you and checkout-confirmation pages, paginated archive pages beyond the first, and low-value tag or author archives on a blog.
It's the wrong tool for pages you don't want crawled at all — for that, robots.txt is closer to what you want, though even that only stops crawling, not necessarily indexing (more on that below).
The Robots Meta Tag: The Standard Method
For any HTML page, the standard, most reliable method is a meta tag in the <head>:
<meta name="robots" content="noindex">A few things worth knowing about this tag specifically:
- It has to be crawlable to work. Google has to fetch the page and see the tag to honor it. If the page is blocked in robots.txt, Google may never see the noindex directive at all — this is the single most common way noindex silently fails, covered in detail below.
- Combine directives with commas.
content="noindex, follow"keeps the page out of the index but still passes link equity through its outbound links — useful for thin pages you still want crawled for discovery.content="noindex, nofollow"does neither. - Target specific bots if needed.
<meta name="googlebot" content="noindex">only speaks to Google;name="robots"is the universal directive most sites should use. - Placement matters. It has to be in the
<head>, not the<body>. A noindex tag rendered into the body via JavaScript after the head has already been parsed is a real, documented way this breaks on JS-heavy sites.
The X-Robots-Tag Header: for Non-HTML Files
PDFs, images, and other non-HTML file types can't carry a <meta> tag — there's no <head> to put it in. For those, the equivalent directive is sent as an HTTP response header instead:
X-Robots-Tag: noindexThis is typically set at the server or CDN level (an Nginx/Apache rule, or a Cloudflare Worker/Transform Rule) rather than in application code, since it has to be attached to the raw HTTP response for that file type. It accepts the exact same directive values as the meta tag — noindex, nofollow, noarchive, and so on — and is honored identically by Google. It also works on HTML responses, and is sometimes preferred there specifically because it doesn't require touching template code — a CDN rule can noindex an entire URL path pattern in one place.
The Mistake That Breaks Noindex: Blocking in robots.txt
This is the failure mode that catches nearly everyone at least once. The instinct is: "I want this page gone from Google, so I'll noindex it and block it in robots.txt for good measure." That combination usually backfires.
robots.txt controls crawling, not indexing. If a URL is disallowed in robots.txt, Google's crawler never fetches it — which means it never sees the noindex meta tag sitting in the page's head, because it's not allowed to look. Google can still index a robots.txt-blocked URL anyway, using only the anchor text and context from links pointing to it, and it will often show up in results as a bare URL with the note "No information is available for this page." That's the opposite of what a noindex was supposed to achieve.
The correct sequencing: if a page currently has real inbound links and needs to be de-indexed, leave it crawlable (don't disallow it in robots.txt) with a noindex tag in place, wait for Google to recrawl and drop it, and only add a robots.txt disallow afterward if you want to stop future crawl budget being spent on it once it's already out of the index.
Noindex vs. Canonical vs. Disallow
These three get conflated constantly. They solve different problems:
- Noindex — "crawl this, but don't show it in search results." Use for pages that must exist but shouldn't rank.
- Canonical tag — "index the other URL as the authoritative version of this content, not this one." Use for genuine duplicate or near-duplicate content where you want equity consolidated onto one URL, not removed entirely. A page can rank under its canonical target even if it isn't the URL that was crawled.
- Robots.txt disallow — "don't crawl this at all." Use to preserve crawl budget on large sites, or to keep genuinely sensitive paths (internal tools, staging environments) out of the crawl queue entirely. Not a reliable de-indexing method on its own, as covered above.
A common, correct combination for faceted navigation with dozens of filter-parameter URLs: canonical each filtered URL back to its clean parent category page, rather than noindexing every permutation individually. That consolidates ranking signals instead of just discarding them.
How Long Noindex Actually Takes to Work
There's no fixed SLA — it depends entirely on how often Google recrawls the URL. For a page with reasonable existing crawl frequency, expect the directive to be picked up within days to a couple of weeks. For a low-authority page Google rarely revisits, it can take considerably longer, because the whole mechanism depends on a fresh crawl seeing the new tag. Requesting indexing for the specific URL in Google Search Console's URL Inspection tool after adding the tag is the standard way to accelerate this rather than waiting for organic recrawl.
Verifying a Noindex Actually Worked
Three checks, in order of reliability:
- Search Console's URL Inspection tool. It reports the live-tested robots directive Google sees for that exact URL, which is the ground truth.
- A
site:search for the exact URL, some time after the directive was added. If it's genuinely gone from the index, it won't return a result. This is a blunt instrument — Google'ssite:operator isn't perfectly exhaustive — but it's a fast sanity check. - A live, geolocated search for a query the page used to rank for, to confirm it's not quietly still showing up in results for users in a specific market. This is exactly where a UULE-based local SERP checker earns its keep — it shows the actual page Google serves for a real searcher, not an assumption based on your own account and location.
The Bottom Line
Noindex is simple in concept and easy to misconfigure in practice, almost always because of the interaction between robots.txt and the noindex directive itself. The rule that prevents nearly every real-world failure: never disallow a URL in robots.txt while you're relying on its noindex tag to take effect — Google can't read a directive on a page it isn't allowed to crawl. Get the sequencing right, use X-Robots-Tag for anything that isn't HTML, and verify the result with Search Console rather than assuming the tag did its job.