Command Palette

Search for a command to run...

Discover

Cloudflare’s AI crawler rules push web toward paid access

Cloudflare will block mixed-use AI crawlers by default on many ad-supported sites starting September 15, pushing AI companies to separate search, agent and training bots or pay publishers through new usage-based models.

Published about 2 hours ago
techcrunch

Cloudflare's new policy pushes AI companies to pay for publishers' content | TechCrunch

techtimes

Cloudflare Separates AI Crawlers by Purpose and Opens Door to Charging Them Directly

martech

The latest AI-powered martech news and releases

tech critter

Cloudflare expands AI content controls, analytics, and monetization ahead of new crawler policy changes

letsdatascience

Cloudflare Gives Publishers Control Over AI Crawlers

Cloudflare’s AI crawler rules push web toward paid access
Click to expand

A harder line at the edge of the web

Cloudflare is turning its network into a checkpoint for AI crawlers, saying that starting September 15, 2026, many ad-supported sites will block “mixed-use” bots by default unless publishers choose otherwisetechcrunch +1. The policy is aimed at crawlers that blend traditional search indexing with AI training or agent activity, a combination Cloudflare says leaves publishers choosing between visibility and uncompensated use of their worktechcrunch +1.

The company is also giving site owners more detailed controls, separating AI traffic into search, agent and training categories rather than treating all AI bots as one classtechtimes +1. That distinction matters because search crawlers can still send referral traffic, while training crawlers collect data to improve models and agent crawlers may fetch pages in real time for a user requesttechtimes.

Payment moves from crawling to usage

Cloudflare’s earlier Pay Per Crawl experiment is being reworked into Pay Per Use, a model meant to compensate publishers when content creates value inside AI products rather than simply when a bot fetches a pagetechcrunch +1. The company says more than half of AI crawler traffic is spent re-fetching pages that have not changed, wasting bandwidth for publishers and compute for AI companiestechcrunch +1.

Two early partners show how Cloudflare wants the system to work. Ceramic.ai plans to pay participating publishers when their content appears in its AI search results, while You.com will let an agent pay on demand for access to specific premium contenttechcrunch +1. The broader technical idea builds on Cloudflare’s 2025 Pay Per Crawl beta, which used HTTP 402 responses to tell authenticated crawlers when payment was required for accesscloudflare.

Google is the pressure point

The sharpest tension is with large search providers whose crawlers serve more than one purpose. Cloudflare says the largest search engine has access to roughly twice as much information as other AI companies because publishers often cannot stay discoverable in search while opting out of AI usestechcrunch +1. Google has argued that Google Extended lets sites restrict use for Gemini Apps and Vertex AI without affecting Google Search, but TechCrunch noted that Googlebot still supports Search features including AI Overviews and AI Modetechcrunch.

Cloudflare framed the change as a push for transparent intent rather than a blanket anti-AI stance. CEO Matthew Prince said the company must “go further and act faster” now that a majority of internet traffic is non-human, while partners including Patreon backed the split between discovery and training accessitbrief. If the defaults stick, AI companies may need cleaner crawler labels, paid-content arrangements, or both.