Cloudflare, which powers around 20% of the internet, has launched a bold new experiment aimed at reshaping how publishers interact with AI companies.
Announced Tuesday, the company’s new “Pay per Crawl” marketplace lets website owners charge AI companies a micropayment every time their content is crawled for training or search purposes.
The initiative arrives amid growing concerns from publishers about unauthorized data scraping by AI bots and the declining value of traditional web traffic. Cloudflare’s new platform gives website owners more transparency and control, allowing them to block, allow, or monetize AI crawlers on an individual basis.
“The proceeds will support xAI’s continued development of cutting-edge AI solutions, including one of the world’s largest data centers and its flagship Grok platform,” Morgan Stanley wrote.
Cloudflare has spent the past year developing tools to manage AI crawler activity, including a one-click AI bot blocker and a dashboard for monitoring bot behavior. The new marketplace builds on that foundation, offering a permission-based system where AI companies and publishers can negotiate crawl rates directly within their Cloudflare accounts. Cloudflare acts as the intermediary, handling transactions and payouts, though cryptocurrency is not involved at this stage.
By default, all new Cloudflare-hosted websites will now block AI crawlers unless explicitly granted access. Prominent media companies such as TIME, The Atlantic, Condé Nast, and The Associated Press have signed on to support this approach.
The stakes are high. Cloudflare reports that AI crawlers are taking far more than they’re giving: in June, OpenAI’s crawler scraped websites 17,000 times per referral, while Anthropic’s crawler did so 73,000 times. In comparison, Google crawled 14 times for each visit it sent — still a poor ratio in today’s web economy.
Cloudflare envisions the “Pay per Crawl” system eventually powering AI agents that fetch information programmatically and pay for access on the fly. This would provide publishers with a real-time, automated business model tailored for the AI age.
If this article helped you, please consider supporting our work. Every small contribution keeps Abijita.com independent and running.
Still, the success of the marketplace hinges on participation. While publishers are eager for solutions, many AI companies currently scrape data for free and may resist a paywall model. Nonetheless, Cloudflare appears uniquely positioned to drive adoption, given its massive web infrastructure footprint and trusted relationships with thousands of publishers.
