AI Crawlers Are Hammering Your WordPress Site

If your WordPress site has gotten mysteriously slow in the last year, or your host has emailed you about resource limits, or the whole thing fell over one afternoon for no reason anybody could explain, there’s a decent chance nobody attacked you. Something just read your website very enthusiastically.

This is different from the security problem, and in some ways it’s a more immediate one, because it doesn’t require anyone to have bad intentions. AI crawlers are not trying to break your site. They just want everything on it, right now, and they are not being careful about how they ask.

Why WordPress Takes This So Badly

A crawler pulling a thousand pages from a static site is a non-event. The server hands over a thousand files. That’s what a server is for.

WordPress doesn’t have a thousand files. It has a database and a program that builds each page on demand. Every request means PHP starting up, a series of database queries, a page assembled from the results, and then the whole thing thrown away. Caching helps enormously, but only for pages that get requested more than once.

Here’s the problem: WordPress generates a nearly unlimited number of URLs that nobody has ever visited.

Every category has an archive page. Every tag has one. Every author, every year, every month, every day. All of those paginate, so there’s page two and page three and page nineteen. There are feed URLs for most of them. Search results are URLs. If you run WooCommerce, every combination of filters is a URL. The site you think of as forty pages is, from a crawler’s point of view, tens of thousands of distinct addresses.

A crawler doesn’t know those are the same forty pages sliced up. It sees links, so it follows them. And because almost none of those URLs have ever been requested before, almost none of them are cached. Every single one is a full database render.

Your forty-page website is, to a crawler, a maze with tens of thousands of rooms.

That’s the fan-out. It’s why the symptom is usually not “my site is a bit slower” but “my site was fine and then it went down at 2pm on a Tuesday.” Several crawlers arrive at once, each opening many connections in parallel, and the server runs out of PHP workers or hits a CPU limit. Legitimate visitors get a timeout. Your host either throttles you or sends you a bill.

robots.txt Is Not Going to Save You

The standard advice is to add crawler rules to robots.txt. Do it, because the well-behaved ones honor it and it costs you nothing.

Understand what it is, though. robots.txt is a note on the door asking politely. It has no enforcement behind it whatsoever. As of 2026, roughly half of AI crawler traffic ignores it outright, and that’s before counting the crawlers that don’t identify themselves honestly in the first place. Nothing stops a bot from claiming to be Chrome.

This is why enforcement has moved down a layer, to the firewall, where a request can actually be refused instead of asked nicely to leave.

The September 15 Change, and the Trap In It

On September 15, 2026, Cloudflare begins blocking AI training and agent crawlers by default. Search crawlers are unaffected, so this doesn’t touch your Google rankings. Alongside it they’ve launched Pay Per Crawl, where a crawler asking for a page gets back a 402 Payment Required and a price, and can either pay or leave.

A lot of coverage has summarized this as “Cloudflare now blocks AI crawlers by default,” and site owners are reading that and concluding they’re covered.

Read the scope carefully. The default applies to ad-supported pages, for new domains, new sites from existing customers, and existing free-tier customers.

If you run a law firm site, a clinic site, a nonprofit, a local services business, or basically any small business site that doesn’t display advertising, the default probably does not apply to you. You have to go turn it on.

What To Do

The Decision Nobody Frames Honestly

There’s a real argument for letting AI crawlers through. If people increasingly find businesses by asking an assistant instead of searching, being absent from that has a cost, and it’s not obviously smaller than the server bill.

That’s a genuine strategic question and I don’t think it has one right answer. But notice that on WordPress you can’t evaluate it cleanly, because the crawling itself is expensive enough to be a problem on its own terms. You end up blocking for infrastructure reasons before you ever get to decide the strategy question on its merits.

On a static site, that pressure disappears. A crawler pulling 10,000 pages from a CDN costs approximately nothing and slows down approximately nobody. The files already exist. Serving them again is the cheapest thing a computer does.

So you get to make the interesting decision, about visibility and content and who gets to use your work, without the boring decision about server capacity forcing your hand first. That’s a smaller argument than the security one, and I find it more persuasive. Disaster is easy to discount. What your infrastructure quietly decides on your behalf is harder to.

If your site is slow for the more ordinary reasons as well, I went through those in why your WordPress site is slow. The security side of unwanted traffic is in your new website is already being attacked, and the full cost picture is in the true cost of running WordPress.