Onbiom

Website guide

Which AI crawlers your robots.txt lets in

Several AI companies run more than one bot, and each bot has a different job. Blocking one name does not block the others, so it helps to know which is which before you edit robots.txt.

Sources read on 30 October 2026; vendors change these pages, so check them before you rely on this guide. A bot or setting that a vendor page did not describe is left out.

What does each bot do?

BotPurposeIf you disallow it
OAI-SearchBot (OpenAI)Surfaces sites in ChatGPT searchNot shown in ChatGPT search answers; may still appear as a link
GPTBot (OpenAI)Collects content that may train OpenAI modelsSignals your content should not be used for training
ChatGPT-User (OpenAI)Visits a page when a user asks ChatGPT or a custom GPTRobots rules may not apply, because a user triggers the visit
Googlebot (Google)Crawls for Google SearchAffects Search, so most sites leave it allowed
Google-Extended (Google)A robots.txt token, not a separate crawlerControls Gemini training and grounding use; no effect on Search inclusion or ranking
ClaudeBot (Anthropic)Collects public content that may train Claude modelsSignals your future content should be excluded from training
Claude-SearchBot (Anthropic)Improves the quality of Claude search resultsYour content is not indexed for search, which may reduce visibility in answers
Claude-User (Anthropic)Fetches pages when a Claude user asks a question that needs themClaude cannot retrieve your content for that user's query
PerplexityBot (Perplexity)Surfaces and links sites in Perplexity searchYour pages may not appear in its results
Perplexity-User (Perplexity)Fetches a page to answer a user's questionGenerally ignores robots.txt

Sources: OpenAI crawler documentation, Google common crawlers and crawler overview, Anthropic crawler article and Perplexity bots guide, all accessed 30 October 2026.

Training, search and user-triggered: why the difference matters

Training bots (GPTBot, ClaudeBot, the Google-Extended token) relate to building models. Disallowing them is a request about future use of your content.

Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) feed answer and search features that can link to you. OpenAI says a robots.txt change can take about 24 hours to apply to search.

User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) run when a person asks a question. OpenAI says robots rules may not apply to its fetcher, and Perplexity says its fetcher generally ignores them. Do not treat robots.txt as the control for these.

None of this promises rankings or citations. Allowing a search bot lets it read your pages; it does not mean you will be shown.

A robots.txt that allows search and answer bots

This example allows the search and user-triggered bots and, optionally, asks the training bots to stay out. Delete the last three groups if you are happy to be used for training.

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

A bot follows the group that names it, so a named group replaces your User-agent: * rules for that bot (robots standard). Add your own Disallow lines to a group if some folders must stay private. Robots rules are requests, not security: keep private data behind a login. Anthropic notes that blocking its IP addresses may not reliably opt you out, and that each subdomain needs its own robots.txt.

What does the Onbiom free check report?

The free website scan reads your robots.txt as a fixed public file (the robots check) and records how many explicit allow and disallow lines it holds for five names. These appear as findings called Googlebot robots scope, OAI-SearchBot robots scope, GPTBot robots scope, ChatGPT-User robots scope and Google-Extended robots scope. Each shows the number of directives and how many are a root Disallow: /.

The scan also reads any Content-Signal values you declare (the content_signals check), where search, ai-input and ai-train are separate.

Limits: Onbiom counts explicit rules per name. It does not infer wildcard inheritance, does not test whether a vendor obeys you, and does not currently report on ClaudeBot, Claude-SearchBot, PerplexityBot or the user-triggered fetchers beyond ChatGPT-User. The full rules are in the crawler page.

Check yours

Run the free website scan on your domain and read the robots.txt findings next to what you intended. If a search bot shows a root disallow you did not mean, that is the first thing to fix.

Sources and further reading

Last reviewed:

Run the free website scan Current plans