AI crawlers and robots.txt: who is visiting and what to allow
A field guide to the AI user agents hitting your site, the difference between training crawlers and retrieval crawlers, and how to decide what to allow.
· 3 min read · Index-now
Your logs now contain a set of user agents that did not exist a few years ago. Deciding what to do about them requires separating two very different activities that often share a company name.
The distinction that matters
- Training crawlers collect content to train future models. Your content contributes to a model's general knowledge. You get no attribution and no traffic.
- Retrieval crawlers fetch pages in real time to answer a specific user question, usually with a citation and a link. This can send you traffic.
Blocking one does not block the other, and they frequently use different user agents from the same organisation. Blocking indiscriminately is how sites accidentally remove themselves from AI answers while still being trained on.
The main user agents
| User agent | Operator | Purpose |
|---|---|---|
GPTBot | OpenAI | Training |
OAI-SearchBot | OpenAI | Search retrieval |
ChatGPT-User | OpenAI | Live fetch for a user's request |
ClaudeBot | Anthropic | Training |
Claude-User | Anthropic | Live fetch for a user's request |
PerplexityBot | Perplexity | Search index |
Google-Extended | Gemini training opt-out control | |
Applebot-Extended | Apple | Apple Intelligence training opt-out |
CCBot | Common Crawl | Open dataset, widely used for training |
Amazonbot | Amazon | Search and assistant |
Meta-ExternalAgent | Meta | Training |
Note that Google-Extended and Applebot-Extended are not crawlers at all — they are control tokens. Blocking Google-Extended opts you out of Gemini training without affecting Google Search, which is a genuinely useful separation.
Deciding what to allow
If you sell something, or want to be found
Allow retrieval crawlers. When someone asks an assistant "who does X in Leeds", being fetchable is the entire prerequisite for being the answer. Blocking retrieval is functionally removing yourself from a growing discovery channel.
Training is a separate judgement call. Allowing it is a bet that being well-represented in model weights helps you get mentioned; blocking it protects content you consider proprietary. Both positions are defensible.
If your content is the product
Publishers, course creators, paid research, documentation you sell — block training, allow retrieval with citation. You want to be cited and linked, not absorbed.
If you host user content or sensitive material
Be more restrictive, and remember robots.txt is a request, not an access control. Anything genuinely sensitive belongs behind authentication.
A permissive configuration
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xmlBlocking training while allowing retrieval
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /Practical notes
- Crawler names change. Review this once or twice a year; new agents appear and old ones get renamed.
- Not everyone honours robots.txt. Well-known operators generally do. Scrapers do not. Rate limiting and WAF rules are the enforcement mechanism, not robots.txt.
- Server-side rendering matters more here. Most AI crawlers execute JavaScript far less reliably than Googlebot. Content that only appears after client-side rendering may simply not be seen.
- Being crawlable is necessary but not sufficient. Clear, well-structured pages that directly answer a question get cited; marketing prose that circles the point does not.
Frequently asked
Will blocking GPTBot remove me from ChatGPT search results?
No. GPTBot is the training crawler. Search retrieval uses OAI-SearchBot and live fetching uses ChatGPT-User. Blocking GPTBot opts you out of training while leaving retrieval intact.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended controls use of your content for Gemini training only. It has no effect on Google Search crawling, indexing or ranking.
Do AI crawlers respect robots.txt?
The major operators state that they do, and generally behave accordingly. Unidentified scrapers do not. Treat robots.txt as a stated preference and use rate limiting for actual enforcement.