What is AI crawlers?
Also called: GPTBot, ClaudeBot, AI bots
AI crawlers are automated visitors that read websites for AI search, answers or training. Your robots.txt file states which are welcome, and a firewall can turn them away even when robots.txt allows them.
Why it matters
Blocking the bots that power search and answers can keep you out of those results. Blocking training bots is a separate choice and does not hide you from answers.
How Onbiom checks it
"Your rules let search and AI bots in" reads robots.txt and the X-Robots-Tag header for 8 bots. Search and answer bots: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot. Training bots: GPTBot, ClaudeBot and Google-Extended. It fails if any search or answer bot is denied the homepage. Blocking a training bot is reported as a choice, not a fault.
"AI bots can open your homepage" requests the homepage as a normal browser and as GPTBot, ClaudeBot and PerplexityBot, sent from Onbiom's servers. It fails when a bot is refused, shown a challenge page or rate limited while the browser loads cleanly. It is one observation from one place and time, and is only run when a check of your site is started.
In your report this appears as: Search bots allowed by robots rules; AI bots can load the homepage.
How to check it yourself
- Open yoursite.com/robots.txt and look for Disallow lines under the bots above.
- Ask your host whether bot protection or a firewall blocks AI bots.
curl -s https://example.com/robots.txtNot sure about your own site? Paste your address and see in a minute which of these checks your site passes. Free, no account needed.
How to fix it
- Cloudflare
- In the dashboard open Security, then Bots and check the settings for AI bots and verified crawlers.
- WordPress
- Check your SEO or security plugin for rules that block bots.
- Ask your web person
- Remove or narrow the Disallow rule for the search bots you want, and allow them in the firewall.
Related terms
Sources and further reading
Last reviewed: