Home / Blog / SEO
SEOHow to Exclude Pages from Search Engines: A Step-by-Step Guide
When and how to keep pages out of Google — robots.txt, noindex, canonical and access control — without hurting the rankings you want to keep.
TL;DR
To keep a page out of search results, use a noindex directive — not robots.txt. Robots.txt blocks crawling, which can leave a URL indexed with no snippet. Reserve exclusion for thin, private, duplicate or staging pages, and never noindex a page you still want to rank.
Why you would exclude a page at all
Not every page belongs in search results. Internal search results, faceted filter URLs, thank-you and confirmation pages, staging environments, gated PDFs, tag archives, and near-duplicate variants add clutter without earning traffic. Left indexed, they dilute crawl budget, split link equity, and can drag down the site-wide quality signals that decide how the pages you do care about rank.
In our technical SEO audits, roughly a third of a typical B2B site turns out to be low-value or duplicate content that would be better off out of the index. Exclusion is a maintenance discipline, not a one-time task.
The three exclusion tools — and when each applies
1. The noindex meta tag. Placed in the <head> of a page, <meta name="robots" content="noindex"> tells search engines to drop the page from results while still allowing them to crawl it. This is the default choice for HTML pages you want removed. Add follow (the default) so link equity still flows through the page while it’s being de-indexed.
2. The X-Robots-Tag HTTP header. Same effect as the meta tag, but delivered in the server response. Use it for non-HTML files — PDFs, images, spreadsheets — where you can’t insert a meta tag. A single header rule can noindex an entire directory of documents.
3. Access control. If a page is genuinely private, put it behind authentication. A password wall or login requirement is the only method that guarantees the content never appears in results, because engines can’t reach it at all.
The robots.txt trap
Robots.txt looks like the obvious tool — it literally lists what crawlers may not fetch. But a Disallow rule blocks crawling, not indexing. If other sites link to a blocked URL, Google can still index the address and show it with a placeholder snippet reading “No information is available for this page.”
Worse, if you disallow a page in robots.txt and add a noindex tag, Google never crawls the page, never sees the noindex, and the exclusion silently fails. The correct sequence is: allow the crawl, serve the noindex, wait for recrawl, then — if you like — add a robots.txt rule to save crawl budget once the page is fully de-indexed.
Step-by-step: excluding a page correctly
- Confirm the page is currently crawlable. Check that no robots.txt rule blocks it and that it returns a 200 status.
- Add the noindex directive. Insert the meta tag for HTML, or the X-Robots-Tag header for files and bulk directories.
- Keep internal links pointing at it — for now. Removing links too early slows the recrawl. Once the page is de-indexed, you can prune the links.
- Request removal in Search Console. The Removals tool hides the URL within about 24 hours, buying time while the noindex propagates over the next one to three weeks.
- Verify. Use the URL Inspection tool to confirm Google sees the noindex and has dropped the page.
Excluding at scale
For large sites, exclusion is a pattern problem, not a page problem. Faceted navigation can spawn thousands of filter permutations; a CMS can generate a tag archive for every keyword. Handle these with rules, not one-off edits: a parameter-based X-Robots-Tag policy, template-level noindex logic, and canonical tags that consolidate near-duplicates onto a single authoritative URL.
Canonical tags deserve a caution, though — they’re a suggestion, not a directive. Google honors them only about half the time, and never for genuinely different content. Use canonicals to consolidate duplicates; use noindex to remove.
Common mistakes to avoid
- Noindexing a page you still want to rank. A stray template rule can wipe a money page off Google overnight. Audit noindex directives after every deploy.
- Blocking crawl before de-indexing. As above — this freezes the page in the index forever.
- Forgetting staging sites. A public staging environment with no exclusion is one of the most common accidental-indexing sources we find. Protect it with authentication, not robots.txt.
- Excluding pages that pass link equity. Before removing a page with strong backlinks, 301-redirect it so the authority isn’t lost.
Exclusion as part of a healthier site
Done well, exclusion is one of the highest-leverage, lowest-cost levers in technical SEO — it concentrates crawl budget and authority where they earn revenue. If you’re not sure which pages are quietly dragging your site down, a free technical audit will map your indexable footprint and flag the pages that should come out. And if search visibility is only part of the picture, our SEO program ties index hygiene to the rankings and pipeline that actually move the business.
Want this done for you?
Get a free audit →FAQ
Should I use robots.txt or noindex to remove a page?
Noindex, in almost every case. Robots.txt only blocks crawling — a blocked URL can still be indexed from external links, just without a snippet. To truly remove a page from results, allow the crawl and serve a noindex directive.
How long until an excluded page disappears from Google?
Usually one to three weeks after Google recrawls the page. To speed it up, request removal in Google Search Console's Removals tool, which hides the URL within a day while the noindex propagates.
Does excluding pages help my SEO?
Indirectly, yes. Removing thin, duplicate, or low-value pages concentrates crawl budget and link equity on the pages you want to rank, and can lift overall site quality signals.
