Learn · How-To

How to Choose the Right Cloudflare AI Crawler Setting

On 15 September 2026 Cloudflare replaced a control that conflated two different decisions with one that separates them. You can now refuse to be model training data while staying indexed in search — but only if you pick the right option, and two of the four still take Googlebot, Bingbot and Applebot down with them. This guide walks you to the correct setting and then verifies it in your logs, because a configuration screen states an intention and only logs record an outcome.

1. 1

   Confirm you are actually behind Cloudflare

   Run a DNS lookup on your domain, or check whether your nameservers point at Cloudflare. A surprising number of businesses do not know, because a developer or host set it up years ago and nobody has looked since. If you are not behind Cloudflare, none of these settings exist for you and the equivalent controls are your robots.txt and your WAF rules.

2. 2

   Separate the two decisions before you touch anything

   Write down two answers. Do you want your content used to train AI models? Do you want to remain indexed and visible in search? These are genuinely different commercial questions. Reasonable businesses land in different places on the first. Almost nobody intends to answer no to the second, which is precisely why the old combined control caused damage.

3. 3

   Open the AI crawler controls and read all four options

   In your zone, go to Security and find the AI bot and training controls. The four settings are Allow, Disallow AI Training, Block on pages with ads, and Block. Read them as a set rather than picking the one whose name sounds closest to your intention — the names describe intent, the behaviour covers more ground.

4. 4

   Pick Disallow AI Training if you want both

   This is the option most businesses actually want. Cloudflare publishes the no-training preference through Bot Preference Sync into your robots.txt, and accountable mixed-use crawlers stay allowed for search. You decline to be training data and you keep your Google indexing. Before 15 September 2026 this combination was not available.

5. 5

   Understand what Block and Block on pages with ads still do

   Both of these reach mixed-use crawlers. Googlebot, Bingbot and Applebot do more than one job, and these settings stop them doing any of them on the affected pages. Choose Block only if you genuinely intend those crawlers gone entirely, which for most businesses means leaving Google's index. Cloudflare's own figures suggest almost nobody wants this: 17% of its sites block training in some form, while under 1% block Search bots.

6. 6

   Check what your robots.txt now says

   Bot Preference Sync writes a preference into your robots.txt. Fetch yourdomain.com/robots.txt and read it. If you also maintain your own Content-Signal line or per-agent groups, make sure the two layers are telling the same story — two layers making different decisions about the same crawler is the most common way this goes wrong.

7. 7

   Verify in your logs a week later

   Every major crawler identifies itself by user agent and most publish verifiable IP ranges. Filter your access or edge logs for Googlebot, GPTBot, ClaudeBot and PerplexityBot over the seven days after your change and confirm they are still arriving with 200 responses rather than 403s. The question "is Googlebot still reaching us" has a factual answer in your logs, and it is a different question from "does our configuration allow Googlebot".