Meta Robots, X-Robots-Tag and Noindex on SaaS Sites

The meta robots tag is a line of HTML telling search engines how to treat a page, such as whether to index it or show a snippet. The X-Robots-Tag does the same job in the HTTP response header, which means it also works for files such as PDFs. The value that matters most is noindex, which keeps a page out of search results.

On a SaaS site, these controls decide what the public can find among your marketing pages, docs, app routes and customer-created content. They're simple to use, and one wrong setting on a shared template can take a whole section out of Google.

The syntax in one place

The meta tag belongs in the page's <head>:

<meta name="robots" content="noindex, nofollow">

The header version is set by your server or framework:

X-Robots-Tag: noindex

Google's documentation on robots meta tags lists the supported rules. These are the ones you are most likely to use:

Rule	What it does
noindex	Keeps the page out of search results
nofollow	Tells search engines not to follow the links on the page
nosnippet	Shows no text snippet or video preview, and also keeps the content from being used as direct input for AI Overviews and AI Mode
max-snippet:[number]	Limits the snippet to a given number of characters
max-image-preview:[setting]	Sets the largest image preview allowed
unavailable_after:[date]	Stops showing the page after a set date

If no rule is present, the default is index, follow, so you don't need to add a tag saying that. Also, <meta name="googlebot"> targets Google specifically, while robots applies to all crawlers that honor it. If rules conflict, the more restrictive one generally applies.

Noindex and robots.txt do different jobs

This is the most costly misunderstanding in the topic. A page blocked in robots.txt can't be crawled, so Google never sees a noindex tag on it. If other sites link to that URL, it can still appear in search results, usually with no description.

So the rule is:

• To keep a page out of search results, use noindex and leave it crawlable.
• To stop crawlers spending time on a section, use robots.txt.
• To protect anything private, use a login. Neither of the other two is a security control. Robots.txt is public, in fact, and listing a private path there tells everyone it exists.

Our crawl budget guide explains why noindex is also a poor way to save crawling.

What a SaaS site should and shouldn't index

Say you run a 45-person project management SaaS. Here's a reasonable starting policy.

Section	Treatment	Why
Marketing pages, blog, comparison pages	Index	These bring buyers
Public docs and help center	Index, usually	Often valuable for long-tail searches. See SEO for docs and help centers
Integration and template pages	Index if each page is useful	Thin auto-generated versions may need noindex
Logged-in app routes	Login, with noindex on anything publicly reachable	Nobody searches for a dashboard
Customer-shared boards and forms	Noindex by default	Thin, and potentially sensitive
Internal search results	Noindex or block	Nearly endless, low value
Thank-you and confirmation pages	Noindex	Useless as a search entry point, and it can muddy your conversion tracking
Paid-campaign landing pages	Noindex if they duplicate organic pages	Avoids competing with your main page
Staging and previews	Password protection	Noindex can be forgotten or copied to production
Gated PDFs	Keep behind the form, or X-Robots-Tag if public	You can't put a meta tag inside a PDF

Customer content deserves a moment's thought. If users can publish to your domain, a default-index policy puts what they write into Google under your brand, including spam. Google's guidance on user-generated spam makes the site owner responsible for managing it. Default to noindex and allow opt-in only where the content is useful.

When to use the X-Robots-Tag header

Reach for the header when a meta tag won't work, or when applying it at the server is simply cleaner.

Non-HTML files. PDFs, spreadsheets and images have no <head>. An old whitepaper PDF ranking above your current landing page is a common example.

Whole paths at once. A rule at the edge or in your server configuration can cover /app/* in one place, instead of relying on every page template to set a tag.

Environments. Sending noindex on every response from non-production hosts is a helpful backstop, as long as you also confirm it isn't sent from production.

Here's a Next.js example in next.config, using a path pattern:

async headers() {
  return [{
    source: "/shared/:path*",
    headers: [{ key: "X-Robots-Tag", value: "noindex" }],
  }];
}

The mistakes that take pages out of Google

These tend to come from deployment and templates rather than from an SEO decision.

Staging noindex shipped to production. A tag or header added to protect staging goes live with the launch. It's among the first things to check when traffic drops sharply after a release. A check in your deployment pipeline that fails if / or /pricing returns noindex costs little and can save a lot.

Noindex removed by JavaScript. Google may not render a page it has already found to be noindex, as Google's JavaScript guidance explains. So if the original HTML says noindex, a script removing it may not help.

Noindex added through a CMS checkbox. A "hide from search" option left on when a draft goes live. Spot-check new pages in the first days after publishing.

Noindex on pages that hold your internal links. If you noindex a hub page for a long time, Google may eventually crawl it less, along with the links on it. Don't assume that noindex is cost-free for a page that holds up other pages.

Blocking then noindexing. Adding noindex and a robots.txt block at the same time, so that the noindex is never read. To remove a page already indexed, allow crawling until it has dropped out, and only then add a block if you want one.

Removing a canonical's target. A page set to noindex shouldn't also be the target of canonicals from other pages. It sends mixed messages about which should appear.

How to check your setup

For a single page, use URL Inspection in Search Console. It shows whether indexing is allowed, and it reports whether the noindex came from a meta tag or a header. To see the header yourself, run curl -sI https://example.com/page.

For the site, a crawler can export the robots directives of every URL, and you can filter by template. Then compare the list against your policy. In Search Console, the page indexing report has a "Excluded by 'noindex' tag" row: you should recognize everything listed there. If any page from your pricing, feature or comparison sections appears, treat it as urgent.

Going back to the project management tool, the check worth automating first is the simplest: a test that fetches your most important pages after each deployment and fails if it finds noindex. Then add a review of the excluded list to your technical SEO checklist.

If you'd rather have someone watching for this as a matter of routine, see the system we run.