Cloudflare has announced a major policy change that could reshape how artificial intelligence companies access content across the web.
Beginning September 15, 2026, the company will automatically block “mixed-use” web crawlers from ad-supported websites unless site owners choose to allow them.
The new default setting targets crawlers that combine multiple purposes, such as traditional search indexing, AI agent access, and AI model training. According to Cloudflare, these bots will no longer be able to crawl pages displaying advertisements by default. The policy will apply to all new Cloudflare customers, newly created websites, and existing users on the free plan.
The move is designed to give publishers greater control over how their online content is used. While many website owners want their pages to appear in search results and AI-powered services, Cloudflare says they also want stronger protection against having their content freely collected for AI training without compensation.
Cloudflare also criticized what it described as the world’s largest search engine, an apparent reference to Google, saying it has access to significantly more web content than other AI companies because publishers often cannot remain visible in search without also contributing to AI-related features.
Google has previously rejected that criticism, pointing to Google Extended, a dedicated bot that allows website owners to opt out of AI training and products such as Gemini Apps and Vertex AI while remaining indexed in Google Search. However, Google’s primary crawler, Googlebot, is still responsible for powering Search as well as AI-powered experiences like AI Overviews and AI Mode.
Cloudflare CEO and co-founder Matthew Prince said the internet has reached a turning point, with automated bots now generating more traffic than humans for the first time. He said the company wants to encourage AI providers to separate search crawlers from bots used for AI agents and model training, creating a more transparent and sustainable ecosystem for both publishers and AI companies.
Alongside the crawler policy, Cloudflare is expanding its publisher monetization efforts. Its earlier Pay Per Crawl system, which lets websites charge AI companies for scraping content, is evolving into Pay Per Use. Instead of charging only when content is crawled, publishers will now be able to earn revenue whenever their content creates value for AI services.
The company also noted that more than half of AI crawler traffic is spent repeatedly fetching pages that have not changed, creating unnecessary bandwidth and computing costs for publishers. The new approach aims to reduce that overhead while encouraging more efficient access to web content.
To launch the new payment model, Cloudflare is partnering with Ceramic.ai and You.com. Publishers who opt in can receive compensation when their content appears in Ceramic.ai’s AI search results or when You.com accesses premium content. Cloudflare says additional AI companies will be able to adopt and customize the system to match their own services in the future.
If this article helped you, please consider supporting our work. Every small contribution keeps Abijita.com independent and running.
