Cloudflare's September 15 Default Block on AI Training Crawlers Takes Effect, Applying to New Customers and Free-Tier Sites on Ad-Supported Pages

Cloudflare's policy blocking mixed-use AI crawlers on ad-supported pages by default took effect on September 15, 2026, two and a half months after the company announced the change on July 1. The new defaults block Training and Agent crawlers while allowing Search crawlers on pages that display ads. The rules apply to all new domains onboarding to Cloudflare, new customers, and existing customers on the Free tier, with existing paid customers keeping their current settings unless they opt in. Cloudflare defines Training as crawlers collecting content to train or fine-tune models, Agent as automated behavior acting in real time on a person's behalf, and Search as behavior that indexes content to answer questions later. The policy creates immediate operational implications for RAG pipelines and agent systems that rely on generic HTTP crawls of Cloudflare-fronted publishers. Mixed-use crawlers including Googlebot, Bingbot and Applebot are handled under the most restrictive rule that applies to them, meaning publishers who block Training will also lose visibility from those crawlers unless the companies separate their functions. Cloudflare CEO Matthew Prince framed the change as a response to bot traffic having overtaken human traffic on the internet for the first time, arguing a payment layer is required for a sustainable content ecosystem. The company is evolving its Pay Per Crawl experiment into Pay Per Use, where publishers get paid when their content shows up in AI answers rather than per fetch.