The Onbiom website scanner
Onbiom checks what a website serves before scripts run, so its owner can see what AI assistants and search engines can read. This page says exactly what the scanner sends, when it visits and how to stop it.
Found a report page about your site? You can remove it at once, with no account needed.
How it identifies itself
Every request from the scanner carries this User-Agent header:
OnbiomSiteAudit/1.0 (+https://onbiom.com/crawler)
The scanner itself does not run scripts, send cookies or log in, and it does not follow redirects to other sites.
What it fetches
- Your homepage, plus one request to its
wwwor non-wwwtwin to see how the two relate. /robots.txt,/sitemap.xmland/llms.txt.- One more request to the homepage that asks for a Markdown version.
- Where enabled, a few public DNS lookups for your domain through one public resolver: whether its name is signed (DNSSEC), its name servers, and its certificate rules (CAA) for your address and its parent names. They are not requests to your server.
- Where enabled,
/.well-known/security.txt, the file that tells people how to reach you about a security problem. One request, to check that it is there and not out of date. - The site icon your homepage declares, or
/favicon.icoif it declares none. - The canonical page your homepage declares, if it is a different address on the same host.
- Four extra requests to your homepage as a test: one with an ordinary browser User-Agent and three with the published User-Agent of GPTBot, ClaudeBot and PerplexityBot, to see whether your firewall treats AI bots differently. They are tests sent by Onbiom, not visits by those services, and they never follow redirects. Not sent when Onbiom checks a site from a public list.
- Up to three more pages that your homepage links to on the same host.
- Up to ten other pages on the same host that your homepage links to, one request each, as a small sample to spot broken links. Links with a query string, or that look like logout, login, delete or checkout actions, are not requested, and robots.txt rules for
OnbiomSiteAuditand*are respected. - Up to five pages listed in your sitemap, one request each, plus up to two sitemap files (the one robots.txt names or
/sitemap.xml, and one sitemap it lists), to check that those pages load, do not redirect and are not marked noindex. Same host only, and robots.txt rules are respected. - Where enabled, one request to a page-rendering service to open your homepage in a browser and compare it with the page before scripts run. Only the homepage address is sent; no cookies, logins or other data.
- Where the security checks are enabled, one plain
http://request to the address as you typed it (port 80, no HTTPS), and at most one more to itswwwor non-wwwtwin, to see whether visitors who skip https reach the secure page. It does not follow the redirect, reads no page content and never leaves your site's address pair. - Optionally, the MCP endpoint addresses your homepage points to, to see whether they connect. Not sent when Onbiom checks a site from a public list.
Each response is size-limited and each request times out after a few seconds. Pages are read only to report technical signals such as titles, headings and structured data; raw page bodies are not kept.
When it visits
- Once, when someone starts a check of your address on Onbiom.
- At most once a day, for a site where a person asked Onbiom to email them when it changes.
- At most once every 30 days, if your site is on a public list of business websites: Onbiom checks it to publish its report page. Onbiom checks at most 40 sites an hour this way.
There is no other crawl of the web and no other schedule.
What it does with robots.txt
When someone starts a check, the homepage, robots.txt, sitemap.xml and llms.txt are fetched once whatever robots.txt says, because the check reports on them. When Onbiom checks a site from a public list, it reads robots.txt first and requests nothing else if robots.txt cannot be read or disallows your homepage for OnbiomSiteAudit, OnbiomReportBot or *. The extra linked pages are fetched only if robots.txt could be read, and are skipped when it disallows them for * or for OnbiomSiteAudit by a plain path prefix. If a rule uses wildcards or other patterns that the scanner does not interpret, it skips those pages rather than guess.
A public report page for your domain is published only when robots.txt could be read and does not disallow your homepage for OnbiomReportBot, OnbiomSiteAudit or *. A missing robots.txt is not a Disallow. If it cannot be read, no new report is published. If you add a Disallow, an existing report is deleted the next time the site is scanned; to remove it at once, use the removal form. A report page is shown only for a website checked in the last 30 days; after that the page disappears and its stored results are deleted.
Ask us to stop
The fastest way is the removal form: no account, no payment, no explanation needed. Your public report disappears at once, any recent shared result we hold for your site is deleted at the same moment, it is never published again, and Onbiom stops both its daily re-checks of the address and its checks from public lists. A one-off check that someone starts on our homepage can still run afterwards. Its result describes only what your site shows publicly, and for 3 hours it is shown to anyone who checks the same address or opens the link to that result. You can also email support@onbiom.com with your domain.
To keep your site out of public report pages, add this rule to robots.txt. Onbiom then also stops checking it from public lists:
User-agent: OnbiomReportBot Disallow: /
To also keep the scanner off your other pages, add this rule instead. A check someone starts still reads your homepage and the standard files listed above:
User-agent: OnbiomSiteAudit Disallow: /
To block the scanner entirely, deny requests whose User-Agent contains OnbiomSiteAudit.
Contact: support@onbiom.com.