The free ride AI bots have enjoyed across the web just got bumpier.
Cloudflare is tightening the rules around how AI companies collect content from websites, introducing new controls that could reshape the relationship among publishers, search engines, and AI companies.
The company announced that beginning Sept. 15, 2026, new default settings on its platform will block AI Training and Agent crawlers from pages that display ads, while traditional Search crawlers will remain allowed. The move also affects mixed-purpose bots, or crawlers that combine search indexing with AI training or agent functions.
Cloudflare said those mixed-use systems will be judged by all of their behaviors, not just one. In practice, that means a crawler used for both search and AI training could be blocked even if search access is still permitted. The changes apply to new customers, new sites created by existing customers, and existing free-tier users who have not adjusted their settings.
A shift away from the old web bargain
Cloudflare argues that the internet’s long-standing exchange — websites allow crawling and receive visitors in return — no longer works as cleanly in the AI era.
Traditional search engines historically indexed websites and redirected users back to publishers. AI systems increasingly absorb information, summarize it, and deliver answers directly to users, reducing the need to visit sources.
The company described this challenge as a difficult trade-off for content owners who want online visibility without freely handing over their work to AI systems.
“Content owners still want to be able to protect their content, and they should be compensated for the original content that they work hard to create, curate, and share. We also know that locking down content isn’t a one-size-fits-all solution; website owners want more options than resorting to ‘block all automation, every time,’” wrote Cloudflare Product Manager Jin-He Lee and Director of Product Bryan Becker.
Cloudflare also renewed its criticism of large search platforms, suggesting that major players gain an advantage when the same crawling systems handle both search indexing and AI-related tasks.
New tools beyond simple blocking
Instead of treating all AI traffic the same, Cloudflare is splitting automation into three categories: Search, Agent, and Training.
Website owners can now create separate rules for each category, giving them more control over how content is accessed. Cloudflare also introduced BotBase, a searchable database for Enterprise customers that identifies bot behavior and classification.
The company is also expanding its content controls to include usage categories such as “Immediate,” “Reference,” and “Full,” allowing publishers to signal how their material should be stored or reused after being crawled.
Why this matters
The announcement reflects a larger fight over who benefits from AI-generated answers.
Publishers depend on traffic, advertising, subscriptions, and referrals. AI companies depend on access to large amounts of content. As AI systems increasingly answer questions directly, fewer users may visit the original websites that produced the information.
Cloudflare is also evolving its previous Pay Per Crawl system into “Pay Per Use,” an approach designed to compensate publishers when their content contributes value to AI-generated responses.
Pressure shifting toward AI companies
The biggest impact of Cloudflare’s move may not be blocking bots; it may be forcing transparency.
Companies running one crawler for search, AI agents, and model training now face pressure to separate those functions if they want uninterrupted access. That could make it easier for publishers to understand who is collecting content and why.
The challenge is that Cloudflare cannot control the entire internet. Large technology firms already have their own opt-out mechanisms, and competitors could pursue alternate methods of gathering data. Still, because Cloudflare sits in front of a large share of internet traffic, its policy changes could influence how AI companies structure their systems going forward.
Also read: Portugal launched Amália, an open-source AI model built for European Portuguese as Europe pushes for more control over AI infrastructure.