Because nothing told Google to stay away. Every page is indexable by default. If Googlebot can reach a URL, it can put that URL in search results. The fix is a noindex rule, added either as a meta tag in the page head or as an X-Robots-Tag response header, on every page you want kept out.
We see this on almost every site we take over. A client shows us their Google results and there is a staging domain sitting in the list, next to a thank you page and three landing pages from a campaign that ended two years ago. None of it was meant to be public. It got indexed because nobody ever set a rule.
The noindex rule is one of the smallest pieces of technical SEO you can learn. It is also one of the easiest to get wrong, because the most common way people try to hide a page is the one thing that stops the rule from working at all.
A noindex tag is an instruction that tells search engines not to show a page in search results. Google Search Central documents it as a robots meta tag carrying the value noindex. The crawler still visits the page and still reads it. It simply leaves that page out of the index, which means the page cannot rank.
The word tag is a little loose here. Google supports the same rule in two places. One is the HTML meta tag in the head of the page. The other is an HTTP response header called X-Robots-Tag, which matters for files that have no HTML head at all, like PDFs, spreadsheets, and images.
Noindex belongs to a wider family of robots rules. Google's robots meta tag documentation lists nofollow, nosnippet, noarchive, notranslate, noimageindex, indexifembedded, max-snippet, max-image-preview, max-video-preview, and unavailable_after alongside it. There is also a value called none, which Google defines as the equivalent of noindex plus nofollow in one word.
One detail matters a lot when you combine these. Google's documentation states that when rules conflict, the more restrictive rule applies. So a page carrying both an index rule and a noindex rule is treated as noindex. You cannot argue a page back into the index by adding a second, friendlier tag on top.
Add a single meta element to the page head with the name set to robots and the content set to noindex. That covers every search engine that honours the rule. If you only want to block Google, set the name to googlebot instead. For files that are not HTML, send the X-Robots-Tag response header with the value noindex.
In Webflow, this lives in the SEO section of page settings for a static page, or in the collection page settings for a CMS template. Webflow also has a site-wide toggle that discourages search indexing on the webflow.io staging domain. That toggle is the one most teams forget to check before they connect a real domain.
On WordPress, plugins such as Yoast SEO expose the same rule as a per-page switch. On Shopify, Squarespace, and Framer the wording changes, but the output is the same robots meta tag underneath. Whatever the platform, the thing worth verifying is the rendered HTML, not the checkbox in the editor.
Verifying matters more than it sounds, because a builder can write the tag into a template that a crawler never actually receives. If your page depends on JavaScript to render, the rule has to be present in the HTML that arrives before any script runs. We have written separately about that failure mode, because JavaScript rendering is where a lot of otherwise correct SEO work quietly disappears.
Because a crawler has to read a page before it can see a rule inside that page. Google Search Central is explicit that for the noindex rule to be effective, the page must not be blocked by a robots.txt file. If robots.txt disallows the URL, Googlebot never fetches the HTML, never sees the noindex, and can still list the URL.
This is the single most common mistake we find in technical audits. Someone wants a page gone, so they do both things at once for good measure. They add the noindex rule and they block the path in robots.txt on the same afternoon. The two instructions fight, the block wins, and the noindex is never read.
It helps to remember what robots.txt actually is. The Robots Exclusion Protocol was written by Martijn Koster in 1994 and was only formally standardised by the IETF as RFC 9309 in September 2022. Read the abstract and you will see it governs how crawlers access content. It says nothing about whether a URL appears in search results.
So the order is simple. Allow the crawl first, then serve the noindex. Once the page has actually dropped out of the index, you can block it in robots.txt if you also want to stop wasting crawler requests on it. We go deeper on that trade-off in our guide to controlling AI crawlers with robots.txt.
Use noindex on pages that need to exist for people but add nothing for searchers. That covers staging sites, internal search result pages, thank you pages, gated download pages, tag archives holding a single post, filtered category URLs, and print versions of articles. The test is simple. If a stranger arriving from Google would be confused, keep the page out.
Thank you pages are the clearest example. They exist to confirm a form submission. If someone lands on one directly from a search result, they see a message about a form they never filled in, and your conversion tracking records an event that never happened. Nothing good comes from that page ranking.
Internal search results are the second clearest. A large ecommerce or content site can generate thousands of these URLs, each one thin and near-identical to the last. Left open, they bloat the crawlable surface of your site without ever earning a click. Whether that bloat is a real problem depends on the size of your site, and on most small sites it is not.
Here is where we push back on common advice. Plenty of guides tell you to noindex every low-traffic page. We disagree. A page with low traffic and a clear purpose is not the same as a page with no purpose. Noindex the ones that should never have been public. Improve or delete the rest.
Use a canonical tag when two URLs hold near-identical content and you want their ranking signals to combine on one preferred version. Use noindex when the page should not rank at all, under any URL. A canonical is a hint about which copy to prefer. Noindex is an instruction to leave the page out entirely.
Mixing the two up is expensive. If you noindex a duplicate that was collecting links, you throw those signals away instead of passing them to the version you care about. If you canonical a page that should be private, you have not hidden anything, because a canonical is advisory and Google can ignore it.
Our rule of thumb is to ask what would happen if the page ranked. If the answer is that the wrong version of a good page would rank, that is a canonical problem. If the answer is that a page nobody should see would rank, that is a noindex problem. We unpack the first case in our guide to canonical URLs and duplicate content.
It takes as long as Google needs to recrawl that page. Google Search Central states that depending on the importance of the page on the internet, it may take months for Googlebot to revisit. Until that recrawl happens the page can keep appearing in results, and there is no setting that speeds it up.
Google's own guidance addresses the impatience directly. If a page is still appearing in results after you add the rule, the documentation says it is probably because the page has not been crawled since the change. That is a crawl scheduling issue, not a broken tag, and the cure is patience rather than a second tag.
You can nudge it along. Requesting indexing on the specific URL in Google Search Console asks for a fresh crawl of that page. Making sure the URL still appears in your XML sitemap during the removal window sounds backwards, but it keeps the page discoverable long enough for the crawler to come back and read the new rule.
The Removals tool in Google Search Console hides a URL from results quickly, but only for a limited period. Google's documentation says requests made in the Removals tool last for about 6 months. Noindex is the permanent fix. On an urgent leak the right move is both, with the tool buying time while the rule takes hold.
We treat this as an emergency protocol rather than routine maintenance. A pricing page that went live before the announcement, a client draft that got shared, an internal document that was never meant to be public. In those cases six months of breathing room is exactly what you need while the recrawl catches up.
Google's removal guidance also names the alternatives. Beyond noindex, the documentation points to removing or updating the content itself and to password protecting the page. Password protection is the only one of the three that stops a determined human as well as a crawler, which matters when the content is genuinely sensitive.
Not reliably. Noindex is a search indexing instruction aimed at search engines. Whether an AI crawler respects it depends entirely on that crawler's published policy, not on the tag itself. If you want to control how ChatGPT, Perplexity, or any other answer engine treats your pages, you need separate user-agent rules and you need to read each vendor's own documentation.
This trips people up because the two goals feel identical. Keeping a page out of Google results and keeping it out of an AI answer are different requests handled by different systems. A page can be absent from Google and still be reachable, readable, and quotable by anything else that fetches it.
Our honest position is that noindex is the wrong tool for this job. It was designed for search results in an era before answer engines, and stretching it to cover AI access leaves gaps. Treat AI crawler control as its own project, with its own rules and its own review schedule, rather than a side effect of an SEO tag.
Start in Google Search Console. The Pages report groups every URL by the reason it is not indexed, and one of those groups covers pages excluded by a noindex tag. Then run a full site crawl using a tool such as Screaming Frog, Semrush, or Ahrefs so you can see every noindex rule on the site in one list.
The two views answer different questions. Search Console tells you what Google has already seen and acted on, which lags behind reality. A crawler tells you what your site serves right now. When the lists disagree, the gap is usually a recent change that has not been recrawled.
What we look for is the surprise. Every audit turns up at least one noindex nobody remembers adding, often on a page the client considers important. It usually traces back to a template, a bulk edit, or a launch checklist that was applied a little too broadly and never reviewed afterwards.
If a page you want ranking is sitting in the noindex bucket, that is a different problem from a page that ranks poorly. We separated those two situations in our article on why a page can be indexed but not ranking, because the fixes have nothing in common.
Yes, if you have launched, migrated, or redesigned anything in the past year. Noindex mistakes are silent. A wrongly tagged page does not throw an error or look broken to a visitor. It just quietly stops earning traffic, and nobody notices until somebody thinks to check the index coverage report.
The audit itself is short. Pull the noindex list from a crawl, compare it against the pages you actually want in search, and check that nothing on the keep list is blocked in robots.txt. Most sites we look at need one or two corrections. A few need a dozen.
If you are staring at a Pages report you cannot make sense of, or you found a staging site in Google this morning and want a second pair of eyes, let's talk. We are happy to walk through what is indexed, what should not be, and what it is costing you. You can reach us at phoenix.studio.
Tell us where you want to go. We'll tell you how we'd get you there.