Your Website Is Feeding Robots a Five-Course Meal (When All They Want Is a Sandwich)

AI bots are eating your website alive.

Not in the dramatic, sci-fi sense. In the very boring, very expensive sense of automated systems hammering your server thousands of times a day, downloading your entire website—every navigation menu, every stylesheet, every script tag, every cookie banner—just to extract the text on the page. It’s like sending a moving truck to pick up a letter.

An Expensive Problem

Dozens of AI systems now crawl websites constantly, looking for content to answer questions and power the tools that more people are using every day. This is fundamentally different from how Google’s crawler has worked for the past twenty years. Traditional search crawlers were relatively polite, fetching the HTML, indexing the text, and moving on. Modern AI bots are more aggressive. They come back more often, they follow deeper links, and some of them execute JavaScript and simulate full browser sessions just to read one page.

According to WP Engine’s traffic trends report from December 2025, AI-driven bot traffic now consumes up to 70% of the most costly dynamic hosting resources. AI crawlers quadrupled their traffic share in just eight months during 2025, with OpenAI’s GPTBot alone growing by 305%. Kinsta actually changed their entire billing model in November specifically because bot traffic was making customers’ bills unpredictable.

This hits WordPress sites especially hard. Every time a bot visits a page, your server has to wake up PHP, query the database, assemble the page from your theme templates, load all your plugins, and deliver a complete HTML document — often 500KB or more — just so the bot can throw away 98% of it and keep the plain text.

The Industry Response

Over the past few weeks, a wave of solutions has appeared from Cloudflare, Vercel, Laravel Cloud, and the WordPress ecosystem, all converging on the same idea: what if we just gave the bots what they actually want?

The answer is markdown, a simple plain-text formatting language. Just content: headings, paragraphs, links, and lists. No navigation bars, no JavaScript, no CSS, no cookie consent popups. A bot sends a request indicating it’s fine with markdown, and the server responds with a lightweight text version of the same content. Human visitors still see your normal website. Nothing changes for people.

Cloudflare’s implementation, launched February 12th, showed an 80% reduction in data transferred—a blog post that consumed 16,180 tokens as HTML dropped to 3,150 as markdown. Vercel reported even more dramatic numbers: a typical blog post weighing 500KB shrinks to just 2KB as markdown, a 99.6% reduction in payload.

What This Means for WordPress

WordPress sites bear the brunt of the problem because every page request triggers server-side processing. A static site doesn’t feel the pain as much, but a WordPress site with WooCommerce, a handful of plugins, and dynamic content is doing real work every time a bot comes knocking. Multiply that by hundreds or thousands of bot requests per day, and your server is spending most of its time serving robots.

We can already help with this. If your site is using Cloudflare to help serve content, offering markdown is one click away. If not, Joost de Valk's Markdown Alternate plugin is a WordPress-native way to offer this. We can help configure it for unique types of content, but it works out of the box for posts and pages.

And it’s not just about the money. When AI systems can efficiently read your content, your site has a better chance of being cited in AI-generated answers. There’s emerging evidence that markdown content produces better results in AI retrieval—one analysis found 35% better accuracy when AI systems work with markdown versus raw HTML. Making your content easy for these systems to consume is becoming as important as traditional SEO.

We Can Help

While this is all very new—much of what I’m citing here is about a week old—it’s already well worth considering. If you have questions about any of this, or if you’ve noticed your hosting performance degrading and suspect bot traffic might be the culprit, get in touch. This is exactly the kind of thing we’re here for.