On 15 September 2026, Cloudflare changes its default handling of AI crawlers. If you have seen this mentioned at all, you have probably seen a version of the sentence “Cloudflare will block AI crawlers by default across up to 30% of the world’s top sites.”
That framing is doing a lot of work it cannot support. The change is narrower than that, it applies to fewer sites than that, and the genuinely risky part of it is something almost nobody is mentioning: a setting that can remove Googlebot while you believe you are only declining to be training data.
Here is what Cloudflare’s own announcement actually says.
Three categories, not one
Cloudflare no longer treats “AI bot” as a single thing. It classifies crawlers by what they do with what they take:
- Search — crawls to build an index that points users back to your site.
- Agent — fetches a page on behalf of a user acting through an assistant.
- Training — collects content used to train models.
The framing Cloudflare uses to justify the split is a reasonable one: “What are they doing on my site? What are they storing? And how will they reshare my content?”
This mirrors the distinction we set out in which AI crawlers can actually reach your site. The vendors themselves already separate these functions — OpenAI runs different crawlers for training, search and user-triggered fetches, and so do Anthropic and Perplexity. Cloudflare is now exposing that same distinction at the network edge, which is a genuine improvement over a single on/off switch.
What the new default actually is
Cloudflare states: “On September 15, 2026, we’ll be setting new defaults for each of these three classifications.”
Those defaults are:
- Training and Agent bots: blocked by default on ad-monetized pages.
- Search bots: allowed by default.
Two qualifications matter enormously and both are routinely dropped in summary.
First, it is scoped to ad-monetized pages, not your whole site. Cloudflare’s reasoning is that “an ad is a signal that a website owner meant for a person to land there and see it.” If you do not run advertising on your pages, this default does not describe your situation.
Second, it applies to domains newly onboarding to Cloudflare. Existing customers are not silently reconfigured. Cloudflare is explicit that “if a website owner wants to opt out of these new default configurations, they can easily mark this in their Security settings any time leading up to September 15.”
So where does “30% of the world’s top sites” come from? That figure describes roughly how much of the top web sits behind Cloudflare in the first place — the size of the population the policy could touch. It is not a forecast of how many sites will be blocking anything on 16 September. Those are very different statements, and the second one has been quietly substituted for the first as the claim travelled.
The part worth your attention
Buried in the same announcement is this:
“multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training.”
Read that slowly.
Googlebot is not only a training crawler. It is the crawler — the one that builds the index behind ordinary Google Search results, the one your organic traffic depends on entirely. The same is true of BingBot and Applebot in their respective ecosystems. They serve multiple purposes, and Cloudflare’s Training classification captures them because Training is one of those purposes.
Which means a marketing director who takes the entirely defensible position “we would rather our content wasn’t used to train someone else’s model” and selects the control that appears to express exactly that can, as a direct consequence, remove their site from Google.
Nothing about the interface announces this. The setting is named for the thing you want. The consequence extends well past it.
Why this keeps happening
If this feels familiar, it should. We wrote in August about the robots.txt specificity rule — where adding a friendly-looking named group for a single crawler causes that crawler to stop reading your wildcard rules entirely, silently discarding every restriction you thought applied to it.
Different layer, identical failure mode:
- A control is named after an intention — “block training”, “allow GPTBot”.
- The actual behaviour covers more surface than the name implies.
- Nothing errors. No warning, no failed deploy, no validator complaint.
- The gap between the two is only visible in outcomes — in logs, in impressions, in traffic that quietly stops arriving.
This is the defining characteristic of AI-era visibility problems, and it is why we keep insisting on the same unglamorous discipline: the configuration is a statement of intent; the logs are the outcome. They are not the same thing and they disagree more often than anyone expects.
What to check before 15 September
Four steps. None of them takes long.
1. Establish whether you are behind Cloudflare at all. A surprising number of businesses do not know, because a developer or host set it up years ago. If you are not, this change does not affect you — though the underlying lesson still does.
2. Open your zone Security settings and read the bot classifications. Look specifically at whether Training is set to block, and understand that this selection reaches Googlebot, BingBot and Applebot.
3. Decide the two questions separately. Do you want your content used for model training? Do you want to remain indexed and visible in search? These are genuinely different commercial decisions. Reasonable businesses land in different places on the first. Almost nobody intends to answer no to the second.
4. Verify afterwards in your logs. Configure, then confirm. Every major crawler identifies itself by user agent and most publish verifiable IP ranges. The question “is Googlebot still reaching us” has a factual answer in your access logs, and it is a different question from “does our configuration allow Googlebot”.
If you also maintain content signals in robots.txt — Cloudflare supports declaring content-use preferences of immediate, reference or full — check that those declarations and your edge settings are telling the same story. Two layers making different decisions about the same crawler is the most common way this goes wrong.
What we cannot tell you
We can tell you what Cloudflare has documented, because they published it. We cannot tell you:
- How many sites will actually end up blocking Googlebot as a result. Nobody knows that yet, and anyone quoting a number is estimating.
- Whether Cloudflare will refine these classifications after launch. They describe themselves as still building operator management tools, which suggests the boundaries may move.
- What the downstream effect on AI citations or search traffic will be for any individual site. That depends on your configuration, your traffic mix and your content, and it is measurable afterwards rather than predictable beforehand.
What we can say with confidence is narrower and more useful: a control exists that can remove you from Google while appearing to do something else, the default changes on a known date, and both are checkable this week.
The short version
Cloudflare’s 15 September change is smaller than reported — new domains, ad-monetized pages, Search bots still allowed. It is not the sweeping shutdown the summaries suggest.
The real exposure is not the default at all. It is that “block Training” also blocks Googlebot, Applebot and BingBot, because those crawlers do more than one job and the category does not distinguish.
If you run anything on Cloudflare, that is fifteen minutes of checking, and the deadline is 15 September.