Cloudflare sits in front of roughly a fifth of the web, and on September 15 it changes what that fifth does by default. Since July 1, every customer, free plans included, can sort AI traffic into three buckets: Search, Agent, Training. New domains that block Training will, come September, have that block follow the crawler wherever it also trains, and that sweeps in Googlebot unless the customer opts out. Five days after the announcement, Google's John Mueller went on Reddit and dismissed Cloudflare's companion move, the Content-Signal line in robots.txt, as having "no effects whatsoever for any crawler or LLM." Two facts to hold at once, and a deadline.
Credibility: HIGH. Both claims trace to first-party artifacts: an official Cloudflare blog post, and a verbatim-quoted Reddit post by a named Google employee reported by Search Engine Roundtable. Bias runs in both directions and is worth naming: Cloudflare sells bot management and benefits from framing AI crawling as a threat; Mueller speaks for Google, which benefits when publishers keep crawl access open.
- July 1, 2026: Search / Agent / Training controls made available to all Cloudflare plans including Free (official Cloudflare blog post)
- September 15, 2026: for new domains onboarding to Cloudflare, Training and Agent bots are blocked by default on pages that display ads; Search stays allowed by default
- September 15, 2026: multi-purpose crawlers are judged by all their behaviors; Cloudflare states Googlebot, Applebot, and BingBot "will be blocked by customers who have selected to block Training" unless the customer opts out in Security settings
- July 1, 2026: Cloudflare extended Content Signals with a fourth robots.txt field, use= (immediate / reference / full); its managed robots.txt now appends use=reference
- July 6, 2026: Mueller on Reddit: the content-signal directives were "made up by a CDN," have "no effects whatsoever for any crawler or llm," and using them "just adds bloat & future maintenance to your robots.txt file"; he also stated Google does not use llms.txt
- Cloudflare fronts approximately 21.3% of all websites as of January 2026 (figure cited by Search Engine Roundtable)
Stay ahead of AI search
The changes that move your rankings and AI citations — and the exact move to make — before they cost you. Free, three times a week.
The two announcements are arguing about the same thing from opposite sides: enforcement versus etiquette. Cloudflare blocks at the network, so it needs nobody's agreement to work. The Content-Signal and llms.txt lines in robots.txt are polite notes that, by Google's own account, no crawler reads, and treating a polite note as a lock is how a site quietly ends up half-blocked. September 15 is the date that bites. If a client has Training blocked and Google has not split its crawler into separate search and training agents by then, Googlebot gets caught in the net, and organic plus AI Overviews visibility goes with it. None of that argues for letting every bot in. It argues for making the call on purpose, per client, in writing, before the default makes it for you.
Do this before September 15:
- Have whoever runs your web infrastructure open the Cloudflare Security tab on every client domain and decide explicitly whether a Training block should extend to the multi-purpose crawlers (Googlebot, Bingbot, Applebot). Where search visibility is the priority, opt out. Record the rationale per client.
- Stop treating Content-Signal or llms.txt lines as controls. Where a client needs real blocking, put it in Cloudflare's bot rules or in standard robots.txt disallow directives that crawlers actually honor.
- Put a re-check in the calendar for the week of September 8, 2026 to confirm each domain's live settings match the decision you recorded, before the new defaults land.
Cloudflare: Your site, your rules: new AI traffic options for all customers, Cloudflare blog, July 1, 2026 · https://blog.cloudflare.com/content-independence-day-ai-options/
Search Engine Roundtable: Google: Cloudflare Content Signals Robots.txt Directive Has No Effects, RustyBrick, July 6, 2026 · https://www.seroundtable.com/google-cloudflare-content-signals-41631.html