The sitemap lists the canonical pages you want indexed. Robots.txt keeps crawlers away from areas of no value. They do different jobs.
For anyone who has a sitemap and a robots.txt but is not sure they are right. You will finish with both reviewed against their actual jobs.
Sitemaps and robots.txt are two small files that are often mixed up, and mixing them up is how sites end up with pages that will not leave the index or will not enter it. They do different jobs, and getting each right takes about an hour.
1Know what each file is for
| Sitemap | robots.txt | |
|---|---|---|
| Job | Lists the pages you want search engines to find and prioritise | Tells crawlers which paths to stay out of |
| Contains | Canonical, indexable URLs | Paths to disallow, and the sitemap location |
| Does not do | Force a page into the index | Remove a page from search results |
2Put only the right URLs in the sitemap
A sitemap is a list of the pages you want found. Everything in it should be canonical, indexable and return a 200. Anything else sends mixed signals and clutters your reports.
- Include: canonical pages that should rank
- Leave out: redirected URLs and 404s
- Leave out: noindexed pages
- Leave out: parameter and filter variants
- Leave out: pages blocked by robots.txt
curl -s https://example.com/sitemap.xml \
| grep -o '<loc>[^<]*' | sed 's/<loc>//' \
| xargs -I{} curl -s -o /dev/null -w "%{http_code} {}\n" {} \
| grep -v '^200'3Split big sitemaps by type
Each sitemap file has size and URL limits, and on larger sites it helps to group pages by type so Search Console's per-sitemap reports show where indexing is strong or weak.
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://example.com/sitemap-blog.xml</loc></sitemap>
<sitemap><loc>https://example.com/sitemap-docs.xml</loc></sitemap>
<sitemap><loc>https://example.com/sitemap-product.xml</loc></sitemap>
</sitemapindex>4Use robots.txt to save crawling, not to hide pages
Robots.txt is best used to keep crawlers away from areas that add nothing: internal search results, carts, endless filter combinations, admin paths. Do not use it for content you want out of results, and do not rely on it for anything private.
User-agent: *
Disallow: /search
Disallow: /cart
Disallow: /*?sort=
Sitemap: https://example.com/sitemap.xml5Submit it and read the report
- Add the sitemap line to robots.txt
- Submit the sitemap URL in Search Console
- Check the status shows Success and the discovered URL count looks right
- Open the Pages report and compare indexed against submitted
- Investigate any large group that is submitted but not indexed
Worked example
A sitemap that listed everything (illustrative)
A made-up SaaS site generates a sitemap from its database, so it lists redirected old posts, noindexed tag pages and parameter URLs. Search Console shows thousands of submitted but excluded pages. The team filters the generator to canonical, indexable, 200 pages only, splits it into blog, docs and product files, and disallows internal search. Coverage becomes readable, and the real gaps, such as new guides not yet indexed, stand out.
Say it in 30 seconds
The 30-second version
The sitemap lists the canonical, indexable pages you want found, split by type on bigger sites. Robots.txt keeps crawlers off wasteful areas. Never block a page you want removed from results; let it be crawled and use noindex.
Worksheet
Sitemap and robots.txt review
The full guide and a worksheet to fill in, as a PDF you can print or share with your team. The worksheet covers:
- Export all sitemap URLs and crawl them
- Confirm every URL returns 200
- Remove redirected, errored and noindexed URLs
- Confirm each URL is the canonical version
- and 6 more
One email with the PDF. No list, no follow-ups.
