A user agent is the software making a web request, and the User-Agent header is the text it sends to say what it is. Browsers send long strings naming the engine and operating system; crawlers send strings that include a product token such as Googlebot or Bingbot. SEOs use user agents to target robots.txt rules, filter server logs and test how pages respond to different crawlers.
What a user agent string looks like
The string is defined loosely by the HTTP specification, so formats vary. Most browser strings start with Mozilla/5.0 for historical compatibility reasons and then list the platform and engine. Crawler strings usually add a token and often a URL explaining the bot.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
(compatible; Googlebot/2.1; +http://www.google.com/bot.html)User agents in robots.txt
Robots.txt groups rules under a User-agent line. The value there is a product token, not the full string: you write Googlebot, not the whole header. Crawlers obey the most specific group that matches them and ignore the rest, so a group for Googlebot replaces, rather than adds to, the rules in the wildcard group.
The same applies to AI crawlers. Tokens such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended each have their own documented behavior, published by their operators. Check each vendor's documentation, since tokens and policies change.
User-agent: *
Disallow: /app/
User-agent: GPTBot
Disallow: /Testing with different user agents
Crawlers and browser dev tools can switch user agent to show how a server responds to a mobile browser or a bot. This is useful for catching redirects, blocks or content differences that only some clients see. Remember that changing the string does not change the IP, so firewall rules that verify real crawler IPs will still treat you as a normal visitor.
Example
A bot rule that blocked more than intended
A developer at Copperloom, a contract management SaaS, adds a WAF rule that blocks any user agent containing the word bot, to stop scrapers. A week later Search Console shows a rise in server errors and fewer crawled pages. The rule caught Googlebot and Bingbot too. Replacing it with IP-verified allowlists for known crawlers and rate limits for everything else fixes the problem.
Common mistakes
Trusting the string as proof
User agents are self-reported and trivial to fake. Use reverse DNS or published IP ranges to confirm a crawler's identity.
Assuming wildcard rules stack
If you add a specific group for a crawler, it ignores the * group entirely. Repeat any shared Disallow lines inside the specific group.
Questions
Chromium-based browsers have reduced the detail in their user agent strings and offer User-Agent Client Hints instead. Crawlers still send identifying tokens, and robots.txt matching is unaffected.
