# Meta Robots, X-Robots-Tag and Noindex on SaaS Sites

> How the meta robots tag, X-Robots-Tag header and noindex work on a SaaS site, what to keep out of search, and the mistakes that deindex pages that sell.

Source: https://comppound.com/blog/meta-robots-x-robots-tag-and-noindex-on-saas-sites (published 2026-10-30)

The meta robots tag is a line of HTML telling search engines how to treat a page, such as whether to index it or show a snippet. The X-Robots-Tag does the same job in the HTTP response header, which means it also works for files such as PDFs. The value that matters most is `noindex`, which keeps a page out of search results.

On a SaaS site, these controls decide what the public can find among your marketing pages, docs, app routes and customer-created content. They're simple to use, and one wrong setting on a shared template can take a whole section out of Google.

## The syntax in one place

The meta tag belongs in the page's `<head>`:

```html
<meta name="robots" content="noindex, nofollow">
```

The header version is set by your server or framework:

```
X-Robots-Tag: noindex
```

[Google's documentation on robots meta tags](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) lists the supported rules. These are the ones you are most likely to use:

| Rule | What it does |
| --- | --- |
| `noindex` | Keeps the page out of search results |
| `nofollow` | Tells search engines not to follow the links on the page |
| `nosnippet` | Shows no text snippet or video preview, and also keeps the content from being used as direct input for AI Overviews and AI Mode |
| `max-snippet:[number]` | Limits the snippet to a given number of characters |
| `max-image-preview:[setting]` | Sets the largest image preview allowed |
| `unavailable_after:[date]` | Stops showing the page after a set date |

If no rule is present, the default is `index, follow`, so you don't need to add a tag saying that. Also, `<meta name="googlebot">` targets Google specifically, while `robots` applies to all crawlers that honor it. If rules conflict, the more restrictive one generally applies.

## Noindex and robots.txt do different jobs

This is the most costly misunderstanding in the topic. A page blocked in robots.txt can't be crawled, so Google never sees a noindex tag on it. If other sites link to that URL, it can still appear in search results, usually with no description.

So the rule is:

- To keep a page **out of search results**, use noindex and leave it crawlable.
- To stop crawlers **spending time** on a section, use robots.txt.
- To protect anything **private**, use a login. Neither of the other two is a security control. Robots.txt is public, in fact, and listing a private path there tells everyone it exists.

Our [crawl budget guide](/blog/crawl-budget-does-a-saas-site-need-to-care) explains why noindex is also a poor way to save crawling.

## What a SaaS site should and shouldn't index

Say you run a 45-person project management SaaS. Here's a reasonable starting policy.

| Section | Treatment | Why |
| --- | --- | --- |
| Marketing pages, blog, comparison pages | Index | These bring buyers |
| Public docs and help center | Index, usually | Often valuable for long-tail searches. See [SEO for docs and help centers](/blog/seo-for-saas-docs-and-help-centers) |
| Integration and template pages | Index if each page is useful | Thin auto-generated versions may need noindex |
| Logged-in app routes | Login, with noindex on anything publicly reachable | Nobody searches for a dashboard |
| Customer-shared boards and forms | Noindex by default | Thin, and potentially sensitive |
| Internal search results | Noindex or block | Nearly endless, low value |
| Thank-you and confirmation pages | Noindex | Useless as a search entry point, and it can muddy your conversion tracking |
| Paid-campaign landing pages | Noindex if they duplicate organic pages | Avoids competing with your main page |
| Staging and previews | Password protection | Noindex can be forgotten or copied to production |
| Gated PDFs | Keep behind the form, or X-Robots-Tag if public | You can't put a meta tag inside a PDF |

Customer content deserves a moment's thought. If users can publish to your domain, a default-index policy puts what they write into Google under your brand, including spam. Google's guidance on [user-generated spam](https://developers.google.com/search/docs/essentials/spam-policies) makes the site owner responsible for managing it. Default to noindex and allow opt-in only where the content is useful.

## When to use the X-Robots-Tag header

Reach for the header when a meta tag won't work, or when applying it at the server is simply cleaner.

**Non-HTML files.** PDFs, spreadsheets and images have no `<head>`. An old whitepaper PDF ranking above your current landing page is a common example.

**Whole paths at once.** A rule at the edge or in your server configuration can cover `/app/*` in one place, instead of relying on every page template to set a tag.

**Environments.** Sending `noindex` on every response from non-production hosts is a helpful backstop, as long as you also confirm it isn't sent from production.

Here's a Next.js example in `next.config`, using a path pattern:

```js
async headers() {
  return [{
    source: "/shared/:path*",
    headers: [{ key: "X-Robots-Tag", value: "noindex" }],
  }];
}
```

## The mistakes that take pages out of Google

These tend to come from deployment and templates rather than from an SEO decision.

**Staging noindex shipped to production.** A tag or header added to protect staging goes live with the launch. It's among the first things to check when traffic drops sharply after a release. A check in your deployment pipeline that fails if `/` or `/pricing` returns noindex costs little and can save a lot.

**Noindex removed by JavaScript.** Google may not render a page it has already found to be noindex, as [Google's JavaScript guidance](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) explains. So if the original HTML says noindex, a script removing it may not help.

**Noindex added through a CMS checkbox.** A "hide from search" option left on when a draft goes live. Spot-check new pages in the first days after publishing.

**Noindex on pages that hold your internal links.** If you noindex a hub page for a long time, Google may eventually crawl it less, along with the links on it. Don't assume that noindex is cost-free for a page that holds up other pages.

**Blocking then noindexing.** Adding noindex and a robots.txt block at the same time, so that the noindex is never read. To remove a page already indexed, allow crawling until it has dropped out, and only then add a block if you want one.

**Removing a canonical's target.** A page set to noindex shouldn't also be the target of canonicals from other pages. It sends mixed messages about which should appear.

## How to check your setup

For a single page, use URL Inspection in Search Console. It shows whether indexing is allowed, and it reports whether the noindex came from a meta tag or a header. To see the header yourself, run `curl -sI https://example.com/page`.

For the site, a crawler can export the robots directives of every URL, and you can filter by template. Then compare the list against your policy. In Search Console, the page indexing report has a "Excluded by 'noindex' tag" row: you should recognize everything listed there. If any page from your pricing, feature or comparison sections appears, treat it as urgent.

Going back to the project management tool, the check worth automating first is the simplest: a test that fetches your most important pages after each deployment and fails if it finds noindex. Then add a review of the excluded list to your [technical SEO checklist](/blog/technical-seo-checklist-for-saas-companies-a-step-by-step-guide).

If you'd rather have someone watching for this as a matter of routine, [see the system we run](/system).

## Frequently asked questions

### What is the difference between meta robots and X-Robots-Tag?

They support the same rules. The meta tag lives in a page's HTML, and the X-Robots-Tag is an HTTP header, which also works for non-HTML files such as PDFs and can be applied to whole paths.

### Should I use noindex or robots.txt to hide a page?

Use noindex to keep a page out of search results, and leave the page crawlable so the tag can be read. Use robots.txt to control crawling. Use a login to protect anything private.

### How long does noindex take to work?

The page drops out after Google next crawls it, which can take anywhere from days to much longer. You can use URL Inspection to request a recrawl of an important page.

### Can nosnippet keep my content out of AI Overviews?

Google lists nosnippet and related controls as ways to limit what's shown from your pages in Search, including its AI features. Weigh this carefully, since it also removes your text snippets from regular results.

## About Comppound

Comppound runs SEO and AI search for B2B SaaS companies: a person sets the plan and reviews samples and every high-risk page, and AI speeds up production.

- [Check a page free](https://comppound.com/tools/seo-checker): Check one of your pages before you change anything else
- [See pricing](https://comppound.com/pricing): Want the content written, reviewed and published for you?