X-Robots-Tag is an HTTP header that carries the same indexing directives as the robots meta tag, such as noindex, nofollow and nosnippet. Because it is part of the response rather than the HTML, it works for any file type, including PDFs, images and spreadsheets, and it can be applied across many URLs with a single server rule.
How it works
The server adds the header to a response. Directives can be combined with commas, and you can target a specific crawler by prefixing its user agent token. Google supports the same directives in the header as in the meta tag.
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: googlebot: nosnippet
# nginx: noindex every PDF
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex";
}When to use it
Use X-Robots-Tag for non-HTML files, for whole sections where editing templates is awkward, or for environments such as staging where a server rule is easier than changing every page. On HTML pages it is equivalent to the meta tag; pick one method per page to avoid confusion.
- Gated PDFs and whitepapers you do not want in search
- Image or file directories
- Preview and staging deployments
- API responses or feeds that should not be indexed
Things to know
Crawlers must be allowed to fetch a URL to see the header. If robots.txt blocks it, the noindex is never read and the URL may still be indexed from links. Headers are also easy to forget because they do not show in the page source; check them with curl -I or the browser's network panel.
Example
Keeping lead magnet PDFs out of search
Copperloom gates its contract templates behind a form, but the PDF URLs were being indexed and found directly in search. Adding X-Robots-Tag: noindex to the PDF directory removes them from the index over time, while the landing pages that describe the templates stay indexed.
Common mistakes
Combining it with a robots.txt block
A blocked URL is never fetched, so the header is never seen. Allow crawling if you want noindex to work.
Leaving a staging noindex header in production
A header copied from staging config can deindex the live site. Check headers after every infrastructure change.
Questions
No. It keeps them out of search results only. Anyone with the URL can still download the file. Use authentication for real privacy.
