A user-triggered fetcher retrieves a web page in response to a specific user action: pasting a link into a chat, asking an assistant to check a page, or an agent visiting a site as part of a task. Examples include ChatGPT-User, Claude-User and Perplexity-User, and Google documents its own list of user-triggered fetchers. Vendors treat these differently from crawlers, and some state that robots.txt rules may not apply to them, because the fetch acts on behalf of a person.
How they differ from crawlers
Crawlers decide what to fetch and when, and they generally follow robots.txt. User-triggered fetchers fetch a specific URL because someone asked. Their requests are fewer, tied to live conversations, and closer to a browser visit. Several vendors document that robots.txt may not govern these fetches, though policies differ and have changed over time.
- Purpose: answer one user's request, not build an index
- Volume: usually low and bursty
- Rules: robots.txt treatment varies by vendor
- Signals: may reveal which pages people are discussing with assistants
Why they are useful to watch
In server logs, user-triggered fetches are among the few direct signals that people are asking assistants about specific pages on your site. A spike on a pricing or comparison page can show interest that never appears as a click in analytics. They also tell you whether those fetches succeed: a fetcher blocked by a firewall or served a challenge page means the assistant answers without your current content.
Example
Logs showing assistant interest
Sablecrest, a fictional expense tool, filters its logs for user-triggered agents and finds repeated fetches of its security page and its comparison with a larger rival. Analytics show little referral traffic from assistants for those pages. The logs suggest buyers are checking those pages through assistants, so Sablecrest makes sure both are current and state key facts in plain text.
Common mistakes
Blocking them as if they were crawlers
A blocked user-triggered fetch means a buyer's assistant cannot read your page at the moment they asked about it.
Relying on robots.txt to control them
Vendors document that robots.txt may not apply. Use server or CDN rules if you truly need to block them.
Questions
They overlap. Some agents run in full browsers and do not identify themselves; user-triggered fetchers typically fetch pages directly with a named user agent.
For most SaaS marketing and docs pages, yes, since they serve people researching your product. Protect private or rate-sensitive areas with normal access controls.
