Cloudflare Lets Sites Charge AI Bots With ‘Pay Per Crawl’

Cloudflare is giving publishers a new option: charge AI bots for crawling their sites using a revived HTTP 402 payment system.

Through its new “Pay Per Crawl”  system website owners can charge AI bots to access their content. The system provides publishers with an alternative approach from their current two-option policy which either allows AI data scraping, or complete blocking of AI access.

“Many publishers, content creators and website owners currently feel like they have a binary choice — either leave the front door wide open for AI to consume everything they create, or create their own walled garden,” Cloudflare said. “But what if there was another way?”

With Pay Per Crawl, content creators can now decide who gets in and at what price. They can let some AI crawlers in for free, block others entirely, or charge for access. “We wanted content creators to have control over who accesses their work,” Cloudflare said. “Creators should be in the driver’s seat.”

The move comes amid growing backlash over AI companies using web content without consent. For example,  YouTube has been criticised for allowing Google to video scrape without notifying creators. Additionally, Google’s AI Overviews feature has reduced traffic to news sites like the  HuffPost and The Washington Post by more than 50%.

“Google just takes content by force and uses it with no return — the definition of theft,” said Danielle Coffey, president of the News/Media Alliance.

The tool works using an old web feature: HTTP response code 402, which stands for “Payment Required.” If an AI bot tries to access a page, the server can now reply with a 402 and include a price tag. If the bot agrees to pay, the server delivers the content.

Cloudflare provides the technical infrastructure and handles payments. Publishers can set a fixed price per crawl request and even apply different rules for different bots. Even if a bot isn’t registered with Cloudflare, it can still be “charged” — essentially blocking access, but leaving room for future deals.

This system could lead to more flexible licensing and dynamic pricing in the future. As Cloudflare puts it: “By providing creators with a robust, programmatic mechanism for valuing and controlling their digital assets, we empower them to continue creating the rich, diverse content that makes the Internet invaluable.”