Noindex is an instruction to search engines to leave a page out of their index. You set it with a robots meta tag in the page's HTML or an X-Robots-Tag HTTP header. The crawler has to be able to fetch the page to read it, which is why noindex and robots.txt blocking should not be combined on the same URL.
Two ways to set it
The meta tag works for HTML pages. The header works for any file type, including PDFs, images and API responses, and is handy when you cannot edit the HTML. You can target a specific crawler by replacing robots with its name, such as googlebot.
<!-- In the page <head> -->
<meta name="robots" content="noindex" />
# As an HTTP response header
X-Robots-Tag: noindexWhen to use it
Use noindex for pages that are useful to visitors who reach them directly but add nothing to search results.
- Internal search results and thin filter combinations
- Thank-you, confirmation and post-signup pages
- Staging and preview environments (alongside password protection)
- Login, account and app pages that are publicly reachable
- Thin tag or author archives that duplicate other listings
- Programmatic pages that lack enough unique content yet
Noindex and robots.txt
If robots.txt blocks a URL, Google never fetches it and never sees the noindex. The URL can stay indexed, usually as a bare link with no description, if other pages link to it. To remove a page from the index, allow crawling until Google has seen the noindex and dropped the page. Only then, if you also want to stop crawling, add the robots.txt block.
Google ended support for a noindex rule written inside robots.txt in 2019, so that method no longer works for Google.
Side effects worth knowing
Google has said that a page left on noindex for a long time is eventually crawled less often, and links on it may be followed less. So noindex is not a neutral way to hide a hub page that links to important content. For temporary hiding, a short-lived noindex is fine; for long-term structure, think about whether the page should exist.
For urgent removals, Search Console's Removals tool hides a URL from Google results for about six months while you put a permanent fix in place.
Example
The staging tag that went live
A fictional security SaaS, Vaultline, launches a redesigned marketing site. The new templates were built on a preview environment with a sitewide noindex meta tag, and the tag shipped to production. Over the next weeks, pages start dropping from Google.
A deploy check that fails the build if production HTML contains a noindex tag on templates expected to be indexed would have caught it before release. After the fix, pages return as Google recrawls them, on Google's schedule rather than instantly.
Common mistakes
Blocking noindexed pages in robots.txt
Google cannot read a noindex on a page it is not allowed to fetch. Let it crawl until the page is gone from the index.
Noindex plus a canonical to another page
These send conflicting signals. Pick one: canonical to consolidate, noindex to exclude.
Environment variables that flip noindex
Many frameworks toggle noindex by environment. Test the production output, not the config.
Questions
No. The page drops out after Google recrawls it and processes the tag, which can take days or weeks. Use the Removals tool for urgent cases.
Usually not. "noindex, follow" is the default behavior anyway, and blocking links can stop discovery of other pages. Add nofollow only if you have a specific reason.
Major search engines, including Bing, support the robots meta tag and X-Robots-Tag. Support among AI crawlers is less well documented; check each vendor's documentation.
