robots.txt lives at the root of a host, such as https://www.example.com/robots.txt, and lists rules for crawlers by user agent. It controls crawling, meaning which URLs a bot may fetch. It does not control indexing: a blocked URL can still appear in search results if other pages link to it. The format is standardized in RFC 9309, and it is now also the main way site owners set rules for AI crawlers.
How the file works
Rules are grouped under one or more User-agent lines. Each group contains Allow and Disallow lines with path prefixes. A crawler uses the most specific group that matches its name, and within that group, Google applies the most specific (longest) matching rule. The wildcard * matches any sequence of characters and $ marks the end of a URL.
Each host and protocol needs its own file. The robots.txt on www.example.com does not cover docs.example.com or app.example.com.
User-agent: *
Disallow: /app/
Disallow: /search
Disallow: /*?sort=
Allow: /app/signup
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlWhat robots.txt cannot do
It is a request, not access control. Reputable crawlers follow it, but anything sensitive should sit behind authentication. It also does not remove pages from search. Google stopped supporting noindex rules inside robots.txt in 2019; to keep a page out of the index, allow crawling and use a noindex meta tag or header, because Google has to fetch the page to see the noindex.
Google ignores the crawl-delay directive. Some other search engines have historically honored it. Google also enforces a size limit of 500 KiB and ignores content past it.
Rules for AI crawlers
AI companies publish user agent tokens for their crawlers, and many separate training crawlers from search or retrieval crawlers, so you can allow one and block another. Google-Extended is a token that controls whether content Google crawls is used for some of its AI model training and grounding; it does not affect Google Search crawling or ranking. Vendor tokens and their exact scope change, so check each vendor's current documentation instead of copying an old list.
Blocking an AI crawler is a business decision. For a SaaS company that wants to be recommended in AI answers, blocking retrieval crawlers can reduce the chance of being cited.
How errors are handled
Google's documented behavior: a 4xx response for robots.txt (other than 429) is treated as if no file exists, meaning everything may be crawled. A 5xx response is treated as a temporary full block, and if the error persists Google falls back to its last cached copy for a period before changing behavior. A robots.txt that times out or errors during an outage can therefore pause crawling of the whole host.
Example
Keeping crawlers out of the app
Say a scheduling SaaS, Slotwise, serves both marketing pages and the logged-in app from www.slotwise.example, with app routes under /app/ and public booking pages under /book/. Crawlers waste time on app routes that redirect to login.
A rule disallowing /app/ while leaving /book/ open fixes the waste. The team also checks that no CSS or JavaScript needed to render public pages lives under /app/, since blocking those files can stop Google rendering the pages correctly.
Common mistakes
Shipping the staging file to production
A Disallow: / meant for staging is one of the most damaging and most common deploy mistakes. Add a check to your release process.
Blocking pages you want deindexed
If robots.txt blocks a page, Google cannot see its noindex tag. The URL may stay indexed with no snippet.
Blocking rendering resources
Disallowing JavaScript, CSS or API paths that pages need to render can leave Google with a blank or broken view.
Questions
No. It stops crawling, not indexing. A blocked URL can still be indexed from links, usually shown without a description. Use noindex or authentication to keep pages out.
No. Without one, crawlers assume everything is allowed. Most sites still use one to block low-value paths and list the sitemap.
Google generally caches robots.txt for up to 24 hours, so changes are usually picked up within a day. Other crawlers set their own caching.
