Website guide
Which AI crawlers your robots.txt lets in
Several AI companies run more than one bot, and each bot has a different job. Blocking one name does not block the others, so it helps to know which is which before you edit robots.txt.
Sources read on 30 October 2026; vendors change these pages, so check them before you rely on this guide. A bot or setting that a vendor page did not describe is left out.
What does each bot do?
| Bot | Purpose | If you disallow it |
|---|---|---|
OAI-SearchBot (OpenAI) | Surfaces sites in ChatGPT search | Not shown in ChatGPT search answers; may still appear as a link |
GPTBot (OpenAI) | Collects content that may train OpenAI models | Signals your content should not be used for training |
ChatGPT-User (OpenAI) | Visits a page when a user asks ChatGPT or a custom GPT | Robots rules may not apply, because a user triggers the visit |
Googlebot (Google) | Crawls for Google Search | Affects Search, so most sites leave it allowed |
Google-Extended (Google) | A robots.txt token, not a separate crawler | Controls Gemini training and grounding use; no effect on Search inclusion or ranking |
ClaudeBot (Anthropic) | Collects public content that may train Claude models | Signals your future content should be excluded from training |
Claude-SearchBot (Anthropic) | Improves the quality of Claude search results | Your content is not indexed for search, which may reduce visibility in answers |
Claude-User (Anthropic) | Fetches pages when a Claude user asks a question that needs them | Claude cannot retrieve your content for that user's query |
PerplexityBot (Perplexity) | Surfaces and links sites in Perplexity search | Your pages may not appear in its results |
Perplexity-User (Perplexity) | Fetches a page to answer a user's question | Generally ignores robots.txt |
Sources: OpenAI crawler documentation, Google common crawlers and crawler overview, Anthropic crawler article and Perplexity bots guide, all accessed 30 October 2026.
Training, search and user-triggered: why the difference matters
Training bots (GPTBot, ClaudeBot, the Google-Extended token) relate to building models. Disallowing them is a request about future use of your content.
Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) feed answer and search features that can link to you. OpenAI says a robots.txt change can take about 24 hours to apply to search.
User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) run when a person asks a question. OpenAI says robots rules may not apply to its fetcher, and Perplexity says its fetcher generally ignores them. Do not treat robots.txt as the control for these.
None of this promises rankings or citations. Allowing a search bot lets it read your pages; it does not mean you will be shown.
A robots.txt that allows search and answer bots
This example allows the search and user-triggered bots and, optionally, asks the training bots to stay out. Delete the last three groups if you are happy to be used for training.
User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /
A bot follows the group that names it, so a named group replaces your User-agent: * rules for that bot (robots standard). Add your own Disallow lines to a group if some folders must stay private. Robots rules are requests, not security: keep private data behind a login. Anthropic notes that blocking its IP addresses may not reliably opt you out, and that each subdomain needs its own robots.txt.
What does the Onbiom free check report?
The free website scan reads your robots.txt as a fixed public file (the robots check) and records how many explicit allow and disallow lines it holds for five names. These appear as findings called Googlebot robots scope, OAI-SearchBot robots scope, GPTBot robots scope, ChatGPT-User robots scope and Google-Extended robots scope. Each shows the number of directives and how many are a root Disallow: /.
The scan also reads any Content-Signal values you declare (the content_signals check), where search, ai-input and ai-train are separate.
Limits: Onbiom counts explicit rules per name. It does not infer wildcard inheritance, does not test whether a vendor obeys you, and does not currently report on ClaudeBot, Claude-SearchBot, PerplexityBot or the user-triggered fetchers beyond ChatGPT-User. The full rules are in the crawler page.
Check yours
Run the free website scan on your domain and read the robots.txt findings next to what you intended. If a search bot shows a root disallow you did not mean, that is the first thing to fix.
Sources and further reading
Last reviewed:
More guides
- What AI assistants read on your website
- AI website readability: 10 questions answered
- Onbiom, DIY, an agency or AI search monitoring?
- Do you need llms.txt? What the evidence shows
- Serving Markdown to AI agents: how it works and how to check it
- Help AI agents find your MCP server: server cards and /.well-known discovery
- All guides