09/10/2026
If your website runs on Cloudflare, one choice in its new AI crawler settings can block Googlebot too.
On Sept. 15, 2026, Cloudflare replaced its older "Block AI Bots" approach with three separate controls:
• Search: bots crawling to build a search index
• Training: bots crawling to train AI models
• Agent: AI assistants fetching a page for a person
The part to watch is the Training setting. Cloudflare says "Block" now applies to all training crawlers, including mixed-use crawlers like Googlebot, Bingbot, and Applebot. Choose Block there, and you could cut off search crawling along with training.
The newer "Disallow AI Training" option works differently. It publishes a no-training preference while keeping supported mixed-use crawlers available for search.
Existing sites were migrated automatically based on their old settings. So if you (or your host, agency, or developer) set this up months ago, don't assume you know what's active now.
Take 20 minutes:
1. Find the AI crawler controls in your Cloudflare dashboard
2. Write down your Search, Training, and Agent settings
3. Check that robots.txt says what you expect
4. Look for unexpected blocked pages or 4xx errors in Google Search Console
Our write-up covers how old settings were migrated, plus a simple policy framework:
→ https://techhelp.ca/cloudflare-ai-crawler-defaults-changed/