Cloudflare is rolling out its system that lets website owners charge artificial intelligence companies for training their models on site content to all its customers. The goal is to give online creators an alternative to either blocking all crawlers or allowing them all in for only a handful of referrals.
For years, the internet’s business model depended on Google, which offered websites visibility in its search results in exchange for indexing and monetising their content through ads. Now, AI companies are proposing a similar arrangement: referrals from their models in return for crawl access.
The problem, Cloudflare notes, is scale: Google’s crawl-to-referral ratio stands at 9.4 to 1, while OpenAI’s is closer to 1,600 to 1, and Anthropic’s a staggering 70,900 to 1.
In July, Cloudflare introduced a pay-per-crawl system that allowed website owners to charge tech companies a fee in exchange for providing their crawlers access to their site content to train AI models. This presents another stream of revenue aside from the small trickle of referrals. When a crawler visits a page, they are notified in a dashboard and can choose to allow, block, or charge for access.
If they select the latter, the crawler will be presented with a custom HTTP 402 “Payment Required” response code, signalling the fee in its header, and requiring the crawler to accept the charge and authenticate through Web Bot Auth before gaining access. The custom page may provide contact details or API endpoints for completing payment.
This service is now being extended to all Cloudflare customers, not just those who signed up for the beta version. It is being rebranded from AI Audit Tool to AI Crawl Control to reflect “the tool’s evolution from simple monitoring to detailed insights and control over how AI systems can access your content,” Cloudflare says.
Cloudflare reports that customers are already sending over a billion 402 response codes per day. It plans to add new parameter options for the custom page, such as those indicating content quality and update frequency.
Matthew Prince, Cloudflare’s CEO, strongly believes in changing the internet’s business model so that rich AI companies must pay for content access, and users, who are currently burdened by ads and paywalls, can view it for free. This will put the power back in the hands of creators.
“For these new AI systems, the value of, ‘I’m going to take your data and then in exchange I’m going to send traffic back to your site’—that’s just going to break,” Prince told Fortune in February. “And so we have to invent some other model.”
In the future, Prince wants to create a marketplace for original content, but he hasn’t fully figured out that step yet.
In March, Cloudflare introduced a free tool that creates fake web pages to confuse AI web crawlers and keep them from cluttering sites with bot traffic or scraping information.