What is robots.txt?
Also called: Robots exclusion, Crawler rules file
robots.txt sits at the top of your site, at /robots.txt. It lists which crawlers may visit which paths, and can point to your sitemap.
Why it matters
It controls what well-behaved search and AI bots crawl. It is a request, not a lock, so it does not hide private pages.
How Onbiom checks it
robots.txt check: Onbiom requests /robots.txt. It passes if the response is plain text with content. A missing file (404) is fine, as without one all bots may read your site. A file returned in another format fails.
Whether it blocks search or AI bots is checked separately by the bot access checks.
In your report this appears as: Optional crawler policy document.
How to check it yourself
- Open yoursite.com/robots.txt in a browser.
- Check it shows plain text lines such as User-agent and Disallow.
- Confirm nothing blocks pages you want found.
curl -s https://example.com/robots.txt | head -30Not sure about your own site? Paste your address and see in a minute which of these checks your site passes. Free, no account needed.
How to fix it
- WordPress
- Under Settings, then Reading, check search engine visibility is on. SEO plugins edit robots.txt.
- Shopify
- Shopify creates robots.txt. You can customise it through the robots.txt.liquid theme template.
- Cloudflare
- If Cloudflare adds managed rules to robots.txt, review them in the dashboard.
- Any website
- Ask your web person or hosting company to make the change, and send them the wording from your report.
Related terms
Sources and further reading
Last reviewed: