The meta robots tag sits in a page's head and gives crawlers page-level instructions, such as noindex to keep it out of results or nofollow to ask them not to follow its links. It only works if the page itself can be fetched.
Need help with SEO?
Comppound runs SEO and AI search for SaaS companies. We plan what to publish, produce it with AI, and every page passes automated checks, and a person reviews samples and anything high-risk.
- Published to your site and tracked in Search Console
- One person answers for your account
- Plans from $400 a month, terms from 3 months
Common values
- noindex: do not show this page in results
- nofollow: do not follow the links on this page
- nosnippet: do not show a text snippet
- max-snippet and max-image-preview: limit snippet size
<meta name="robots" content="noindex, follow">Pairing with robots.txt
If robots.txt blocks a page, crawlers never see its meta tag. To remove a page from results, let it be crawled and use noindex.
Common mistakes
Blocking and noindexing the same page
The block hides the noindex, so the URL may stay listed.
Frequently Asked Questions
Search engines combine directives, and the most restrictive one generally applies. Keep them consistent.
Keep reading
- Noindex: Noindex is a directive, set in a meta tag or HTTP header, that tells search engines not to include a page in their results.
- X-Robots-Tag: X-Robots-Tag is an HTTP response header that sends robots directives like noindex. It works for PDFs and other non-HTML files a meta tag cannot reach.
- robots.txt: robots.txt is a plain-text file at a site's root that tells crawlers which paths they may or may not request.
- Indexability: Indexability is whether a page is allowed and able to be stored in a search engine's index.
- Bing Webmaster Tools: Bing Webmaster Tools is Microsoft's free console for site owners. It shows how Bing crawls and indexes a site and offers SEO reports and URL submission.
- Content delivery network: A CDN is a network of servers worldwide that caches and serves a site's files close to each visitor, cutting load times and shielding the origin server.
- Crawl Errors: Crawl errors are failures that happen when a search engine tries to fetch a URL, such as server errors, redirect problems, blocked pages or DNS failures.
- Crawlability: Crawlability is how easily search engine and AI crawlers can discover and fetch the pages on a site.
