What happened
Cloudflare will block "mixed-use" crawlers from ad-supported web pages by default starting September 15, 2026, affecting new and existing free customers. This policy separates traditional search indexing from AI training and agent use, requiring AI companies to use distinct crawlers for different purposes. Cloudflare also evolves its "Pay Per Crawl" initiative into "Pay Per Use," enabling publishers to charge AI companies when their content generates value, not just when fetched, initially partnering with Ceramic.ai and You.com.
Why it matters
AI model providers face increased operational costs and complexity for data sourcing as web-scale training data now requires explicit opt-in or payment. Procurement teams must budget for content licensing, while platform engineers need to manage distinct crawler identities to avoid default blocking. This rebalances power for publishers, offering new monetisation avenues via "Pay Per Use" and reducing bandwidth consumption from redundant AI crawls. This follows Google's recent policy linking AI access to publisher content, intensifying the shift towards paid content for AI training.
