Cloudflare's Block setting now stops Google, Bing and Apple search crawlers too. Check yours.

If your website runs through Cloudflare, one setting can now stop Google, Bing and Apple from crawling it for search. On September 15, Cloudflare changed what its Block option does for AI crawlers. It also added a new option for owners who want to stay in search.

What changed

Some large companies use one crawler for both search and AI training. Cloudflare calls these mixed-use crawlers, and it names Applebot, Bingbot and Googlebot.

Until now, Cloudflare’s Block and “Block on pages with ads” settings did not apply to those crawlers, because blocking them could also affect search. Now they do. Cloudflare says Block will stop Applebot, Bingbot and Googlebot from reaching your site, and that this includes search.

For owners who want search without training, Cloudflare added a setting called Disallow AI Training. Cloudflare says it “lets you easily stay indexed for search while refusing to let that same crawler train on your content.” It works by publishing a no-training preference in your robots.txt file.

Three details matter:

1. Your current settings carry over. Cloudflare says most sites need to do “Nothing, in almost every case.” Earlier Training choices of Block or Block on pages with ads move to Disallow AI Training.
2. Bing is not covered yet. Cloudflare says Microsoft is building support for a no-training preference in robots.txt, targeted for early 2027. Until then, Disallow AI Training does not pass a no-training preference to Bing through robots.txt.
3. Every plan has it. The controls are available to all customers on all plans, in the domain’s Security Settings.

Cloudflare also reports that less than 1% of its sites block search bots, and that 17% use some way to block training. Those are Cloudflare’s own figures.

Why this matters if you rank but lose the AI answer

You worked hard to rank. Many owners also want to keep their pages out of AI training.

The risk is the word Block. From now on, choosing Block on Cloudflare keeps Google’s, Bing’s and Apple’s crawlers off your site, search included. A web person who picks Block to stop AI training could keep the site out of the search crawl without meaning to.

A five-minute check

1. Sign in to Cloudflare and open the domain’s Security Settings.
2. Find the AI crawler settings. Cloudflare lists separate settings for Search, Training and Agent.
3. Look at Training. If it says Block and you want to stay in search, switch it to Disallow AI Training.
4. Look at Search. Make sure search crawlers are allowed.
5. Open your site’s robots.txt file and look for the Disallow line Cloudflare publishes for training.
6. Tell whoever manages your site that Block now covers search crawlers too.

If you also want to tell Google directly, Cloudflare’s post says Google lets site owners opt out of training with a Disallow rule for Google-Extended in robots.txt.

What to do this week

• Check the Training and Search settings on every domain you run through Cloudflare.
• Write down what each one says today, with the date.
• Tell your web person or agency what Block means now.
• If Bing training matters to you, put early 2027 on your list to check again.