For site owners

The Onbiom website scanner

Onbiom checks what a website serves before scripts run, so its owner can see what AI assistants and search engines can read. This page says exactly what the scanner sends, when it visits and how to stop it.

Found a report page about your site? You can remove it at once, with no account needed.

How it identifies itself

Every request from the scanner carries this User-Agent header:

OnbiomSiteAudit/1.0 (+https://onbiom.com/crawler)

The scanner itself does not run scripts, send cookies or log in, and it does not follow redirects to other sites.

What it fetches

Each response is size-limited and each request times out after a few seconds. Pages are read only to report technical signals such as titles, headings and structured data; raw page bodies are not kept.

When it visits

There is no other crawl of the web and no other schedule.

What it does with robots.txt

When someone starts a check, the homepage, robots.txt, sitemap.xml and llms.txt are fetched once whatever robots.txt says, because the check reports on them. When Onbiom checks a site from a public list, it reads robots.txt first and requests nothing else if robots.txt cannot be read or disallows your homepage for OnbiomSiteAudit, OnbiomReportBot or *. The extra linked pages are fetched only if robots.txt could be read, and are skipped when it disallows them for * or for OnbiomSiteAudit by a plain path prefix. If a rule uses wildcards or other patterns that the scanner does not interpret, it skips those pages rather than guess.

A public report page for your domain is published only when robots.txt could be read and does not disallow your homepage for OnbiomReportBot, OnbiomSiteAudit or *. A missing robots.txt is not a Disallow. If it cannot be read, no new report is published. If you add a Disallow, an existing report is deleted the next time the site is scanned; to remove it at once, use the removal form. A report page is shown only for a website checked in the last 30 days; after that the page disappears and its stored results are deleted.

Ask us to stop

The fastest way is the removal form: no account, no payment, no explanation needed. Your public report disappears at once, any recent shared result we hold for your site is deleted at the same moment, it is never published again, and Onbiom stops both its daily re-checks of the address and its checks from public lists. A one-off check that someone starts on our homepage can still run afterwards. Its result describes only what your site shows publicly, and for 3 hours it is shown to anyone who checks the same address or opens the link to that result. You can also email support@onbiom.com with your domain.

To keep your site out of public report pages, add this rule to robots.txt. Onbiom then also stops checking it from public lists:

User-agent: OnbiomReportBot
Disallow: /

To also keep the scanner off your other pages, add this rule instead. A check someone starts still reads your homepage and the standard files listed above:

User-agent: OnbiomSiteAudit
Disallow: /

To block the scanner entirely, deny requests whose User-Agent contains OnbiomSiteAudit.

Contact: support@onbiom.com.

Scan your own website Privacy