Glossary

What is robots.txt?

Also called: Robots exclusion, Crawler rules file

robots.txt sits at the top of your site, at /robots.txt. It lists which crawlers may visit which paths, and can point to your sitemap.

Why it matters

It controls what well-behaved search and AI bots crawl. It is a request, not a lock, so it does not hide private pages.

How Onbiom checks it

robots.txt check: Onbiom requests /robots.txt. It passes if the response is plain text with content. A missing file (404) is fine, as without one all bots may read your site. A file returned in another format fails.

Whether it blocks search or AI bots is checked separately by the bot access checks.

In your report this appears as: Optional crawler policy document.

How to check it yourself

  1. Open yoursite.com/robots.txt in a browser.
  2. Check it shows plain text lines such as User-agent and Disallow.
  3. Confirm nothing blocks pages you want found.
curl -s https://example.com/robots.txt | head -30

Not sure about your own site? Paste your address and see in a minute which of these checks your site passes. Free, no account needed.

Check your site

How to fix it

WordPress
Under Settings, then Reading, check search engine visibility is on. SEO plugins edit robots.txt.
Shopify
Shopify creates robots.txt. You can customise it through the robots.txt.liquid theme template.
Cloudflare
If Cloudflare adds managed rules to robots.txt, review them in the dashboard.
Any website
Ask your web person or hosting company to make the change, and send them the wording from your report.

Related terms

Sources and further reading

Last reviewed: