The way content and businesses are discovered online is evolving rapidly. Previously, traffic came mainly from traditional search engines, with SEO determining visibility. Now, increasing traffic is driven by AI crawlers and agents that require structured data from the inherently unstructured, human-focused Web. To stay ahead, businesses must reconsider not only human visitors or traditional SEO but also treat AI agents as first-class users. Feeding raw HTML to AI is inefficient—HTML is verbose and token-heavy compared to markdown. For example, a simple "About Us" section costs about 3 tokens in markdown, but 12–15 tokens in HTML, excluding non-semantic elements like div wrappers and scripts that inflate token use without adding meaning. This blog post, when measured in tokens, uses 16, 180 tokens in HTML, but only 3, 150 tokens in markdown—a reduction of 80%. Markdown has become the preferred format for AI systems due to its explicit structure, which supports better AI processing and reduces token waste. However, since the Web is primarily composed of HTML and page sizes have grown, AI agents must filter out extraneous parts to access relevant content.

Converting HTML to markdown in AI workflows is common but imperfect: it adds computational overhead, cost, complexity, and may not reflect the creator’s intent. Cloudflare now offers an innovative solution by performing real-time HTML-to-markdown conversion at the source for sites on its network with Markdown for Agents enabled. AI systems can request markdown versions of pages via the Accept content negotiation header, specifying "text/markdown. " Cloudflare fetches the original HTML and automatically converts it to markdown on the fly, serving a streamlined, AI-optimized response. For example, a curl request with the header “Accept: text/markdown” fetches markdown content from Cloudflare’s developer documentation. AI developers using Cloudflare Workers can similarly specify this header in TypeScript fetch calls. Popular coding agents like Claude Code and OpenCode already utilize this approach, receiving markdown responses that simplify parsing and improve token efficiency. Responses also include an x-markdown-tokens header, estimating token count, aiding AI systems in optimizing context windows and chunking strategies. Markdown for Agents integrates with Cloudflare’s Content Signals framework, introduced during Birthday Week, enabling content creators to control AI usage permissions. Responses include headers like Content-Signal: ai-train=yes, search=yes, ai-input=yes, signaling consent for AI training, search indexing, and agentic use, with options for future custom policies. Cloudflare has enabled this feature in its Developer Documentation and Blog, encouraging AI crawlers to consume markdown rather than HTML, improving efficiency and semantics. For AI systems needing document conversion outside Cloudflare or when Markdown for Agents is unavailable, Cloudflare offers alternative solutions: - Workers AI’s AI. toMarkdown() supports multiple document types and summarization. - Browser Rendering with the /markdown REST API converts dynamic pages after rendering. Cloudflare Radar now tracks AI bot and crawler content-type requests globally and per agent, including markdown retrieval statistics, such as those from OpenAI’s OAI-Searchbot powering ChatGPT’s search. This data helps monitor trends in AI content consumption and is freely available via public APIs and Data Explorer. To enable Markdown for Agents, users can toggle the feature in the Cloudflare dashboard for their zone. The feature is currently in Beta at no cost for Pro, Business, Enterprise plans, and SSL for SaaS customers. In summary, as AI-driven discovery increases, adopting Markdown for Agents allows websites to efficiently serve structured, concise content directly to AI, reducing processing overhead, honoring creator intent, and preparing for the future of web interaction.