The robots meta tag sits in the head of an HTML page and gives crawlers page-level instructions. The most used directive is noindex, which keeps a page out of search results, but the tag also controls link following, snippets, image previews and expiry dates. It applies only to the page it is on and only after the page is crawled, which is the key difference from robots.txt, which controls crawling, not indexing. The X-Robots-Tag HTTP header carries the same directives for files that have no HTML.
Need help with SEO?
Comppound runs SEO and AI search for SaaS companies. We plan what to publish, produce it with AI, and every page passes automated checks, and a person reviews samples and anything high-risk.
- Published to your site and tracked in Search Console
- One person answers for your account
- Plans from $400 a month, terms from 3 months
The directives Google supports
With no tag, the default is index, follow. Writing index or follow explicitly does nothing.
- noindex: do not show the page in search results
- nofollow: do not follow the links on the page
- none: the same as noindex, nofollow
- nosnippet and max-snippet:[number]: limit text snippets, including in AI Overviews
- max-image-preview and max-video-preview: limit image and video previews
- noimageindex: do not index images on the page
- notranslate: do not offer a translation in results
- indexifembedded: allow indexing of content embedded via iframes despite a noindex
- unavailable_after:[date]: drop the page from results after a date
<meta name="robots" content="noindex, follow">
<meta name="googlebot" content="max-snippet:200">Things to know
Google must be able to crawl a page to read its meta tag, so do not block a noindexed page in robots.txt. Use name="robots" for all crawlers or a specific name such as googlebot for one. If several directives conflict, Google applies the most restrictive. On JavaScript sites, adding or changing the tag with script is unreliable: if the served HTML says noindex, Google may not render the page to see a change.
Example
Thank-you pages in search
A fictional scheduling tool, Slotwise, finds its post-signup thank-you page ranking for its brand name. Adding a robots meta tag with noindex to that template keeps the page out of results while its links still work.
Common mistakes
Shipping a staging noindex to production
A sitewide noindex from staging can remove a whole site from search. Check the tag on key templates after every release.
Blocking the page in robots.txt as well
If Google cannot crawl the page, it never sees the noindex, and the URL can stay indexed from links.
Frequently Asked Questions
robots.txt controls whether a URL is crawled. The robots meta tag controls whether a crawled page is indexed and how it appears.
No. Pages without one are indexed and followed by default. Add it only where you want to restrict something.
Keep reading
- Noindex: Noindex is a directive, set in a meta tag or HTTP header, that tells search engines not to include a page in their results.
- X-Robots-Tag: X-Robots-Tag is an HTTP response header that sends robots directives like noindex. It works for PDFs and other non-HTML files a meta tag cannot reach.
- robots.txt: robots.txt is a plain-text file at a site's root that tells crawlers which paths they may or may not request.
- Snippet Controls: Snippet controls are robots directives such as nosnippet, max-snippet and data-nosnippet that limit how much of a page Google may show as a text or image preview, including in its AI features.
- Bing Webmaster Tools: Bing Webmaster Tools is Microsoft's free console for site owners. It shows how Bing crawls and indexes a site and offers SEO reports and URL submission.
- Content delivery network: A CDN is a network of servers worldwide that caches and serves a site's files close to each visitor, cutting load times and shielding the origin server.
- Crawl Errors: Crawl errors are failures that happen when a search engine tries to fetch a URL, such as server errors, redirect problems, blocked pages or DNS failures.
- Crawlability: Crawlability is how easily search engine and AI crawlers can discover and fetch the pages on a site.
