Website guide
Serving Markdown to AI agents: how it works and how to check it
Some sites return a Markdown version of a page to software that asks for it, and HTML to everyone else. It is optional, and the public evidence that major AI services ask for it is thin. This guide explains the mechanism, one way to set it up and a one-line test.
Sources read on 30 October 2026; these pages change, so check them before you rely on this guide. Where a vendor page was silent, we say so instead of guessing.
How does it work?
The mechanism is HTTP content negotiation. The client sends an Accept header listing the formats it can process, and the server uses it as a hint to choose a representation of the same URL. MDN's guide to content negotiation (last modified 21 December 2025) says the selection algorithm "is server-specific and not defined in the standard", and that a server which varies its answer should send a Vary header so caches can reproduce the decision.
For Markdown, the request is Accept: text/markdown and the answer is a response with Content-Type: text/markdown. That media type is registered in RFC 7763 (March 2016), which is an informational document, not an Internet Standards Track specification.
What about .md URLs?
Some sites also publish a separate address for the Markdown copy, for example the page URL with .md appended. OpenAI's developer documentation site says Markdown versions of its documentation pages are available this way. That describes how one site serves its own docs; we found no vendor page saying a crawler looks for .md addresses on other sites.
Who says they ask for Markdown?
- Google: its AI optimization guide says you do not need to create new machine readable files, AI text files, markup or Markdown to appear in Google Search. It is silent on
Accept: text/markdownrequests and on separate.mdpages. - OpenAI, Anthropic and Perplexity: the OpenAI crawler documentation, the Anthropic crawler article and the Perplexity bots guide say nothing about their bots sending
Accept: text/markdown. - Cloudflare documents the server side, not any AI vendor's behavior (next section).
So we cannot tell you which assistants request Markdown. Tools you or your customers run yourselves may. Do not expect citations, rankings or traffic from it.
A minimal server setup
Cloudflare's Markdown for Agents (last updated 9 October 2026) converts HTML to Markdown at its edge when a request sends Accept: text/markdown. The page says it is available on Pro, Business and Enterprise plans and for SSL for SaaS customers at no cost. You turn it on under AI Crawl Control for the zone, or per path with a Configuration Rule. It documents these behaviors:
- The response is
text/markdown; charset=utf-8and includesvary: accept. - Only HTML is converted, up to 6 MiB of decompressed HTML.
Content-Length,ETagandLast-Modifiedare removed from the converted response.
If you do not use Cloudflare, your web agency can do the same in the web server or application: check the Accept header, return Markdown with the right Content-Type and send Vary: Accept. We did not find a vendor-neutral configuration to cite, so we do not give one here.
Test it with one curl command
curl -sI -H "Accept: text/markdown" https://example.com/
Replace the address with your own. Look at the content-type line of the response. text/markdown means the server answered with Markdown; text/html means it ignored the header, which is the normal result for most sites. Drop the -I to read the body instead.
What does the Onbiom free check report?
The free website scan makes one extra request to the site root (/) with Accept: text/markdown and shows a finding called Markdown response, listed as an optional Markdown representation. It is pass when the server answers HTTP 200 with a non-empty text/markdown body. An HTML response, or a 404, 406 or 410, is reported as not applicable. Any other result, such as a timeout, is reported as unknown. The wording says HTML remains a supported format and a normal site can pass the technical assessment without Markdown.
Limits: only the root address is tested, never other pages and never .md addresses. The body is capped at 256 KB, the request has a four-second deadline and follows at most two redirects on the same host. The check does not compare the Markdown with the page or check its quality. Nothing here shows whether any AI service requests Markdown from your site. See the how Onbiom checks your site.
Check yours
Run the free website scan on your domain and read the Markdown response finding next to the other results. Treat it as optional, and fix the customer-facing basics first.
Sources and further reading
- MDN's guide to content negotiation (developer.mozilla.org)
- RFC 7763 (www.rfc-editor.org)
- AI optimization guide (developers.google.com)
- OpenAI crawler documentation (developers.openai.com)
- Anthropic crawler article (support.claude.com)
- Perplexity bots guide (docs.perplexity.ai)
- Cloudflare's Markdown for Agents (developers.cloudflare.com)
Last reviewed:
More guides
- What AI assistants read on your website
- AI website readability: 10 questions answered
- Onbiom, DIY, an agency or AI search monitoring?
- Which AI crawlers your robots.txt lets in
- Do you need llms.txt? What the evidence shows
- Help AI agents find your MCP server: server cards and /.well-known discovery
- All guides