What Cloudflare’s AI crawler controls actually do
Cloudflare’s new AI crawler controls and BotBase let website owners manage AI web scraping prevention across Search, Agent, and Training bots, block AI crawlers that misuse content, and set clear content access policies that balance protection with the visibility modern sites still need.
If you run a content site, you are in the middle of a quiet shift: more of your traffic now comes from bots than humans, and a portion of those bots exists to feed AI products rather than your business. Cloudflare’s updated bot control moves past a crude on/off switch and lets you treat different AI crawlers differently, depending on whether they power classic search, live agents, or model training. The end goal is not to vanish from the web, but to stop feeling like free fuel for tools you do not control.
These tools are available to all Cloudflare customers, including free-plan sites, so you do not need a huge infrastructure budget to regain some leverage over how your pages are copied and reused.
How Cloudflare now classifies and filters AI crawlers
The most important change is that Cloudflare no longer treats bots as either “AI” or “not AI.” Instead, each crawler is labeled by what it does with your content: Search, Agent, or Training. That means a bot that fetches snippets to answer user questions is not lumped in with a bot that indexes pages for organic search, and a pure Training crawler is in its own bucket again.
This matters because you can now apply different policies to each category, rather than block all automation outright. Starting September 15, 2026, new domains will default to blocking Training and Agent crawlers on pages that display ads, while keeping Search crawlers allowed; mixed-use crawlers that act as both Search and Training will be blocked if you block Training, even when Search is allowed. Cloudflare also plans to automatically block mixed-use crawlers that index sites for search engines and act as AI agents and trainers at the same time, especially where they do not give you a clear opt-out for AI use.
The gotcha is that if you rely heavily on certain mixed-use bots, tightening Training rules may unexpectedly cut into your search visibility, because those crawlers are evaluated under both policies. You can opt out of these new defaults in your security settings before they kick in if you want to preserve current behavior.
Step-by-step: setting crawler and content use policies
Here is a straightforward way to use Cloudflare bot control to block AI crawlers you do not want, while keeping the automated traffic that still helps you. The idea is to start with visibility, then tune your rules instead of guessing blindly.
- Review your current AI traffic categories (Search, Agent, Training) and note which bots are hitting ad-supported pages most often.
- Decide your default stance for each category: allow Search, and choose whether to block Training and Agent crawlers on ad pages to reduce free AI training.
- Apply category-based rules so Training and Agent crawlers are blocked where you depend on ad revenue, while Search crawlers remain allowed by default.
- For Enterprise Bot Management, open BotBase, search for specific Verified Bots or AI agents, and check how they are classified under the new taxonomy.
- Use BotBase detection IDs to create or adjust security rules that filter traffic by individual bots, tightening or relaxing access as needed.
The sequence here matters: if you start by blocking categories without examining which crawlers are classified as mixed-use, you risk wiping out both AI web scraping and helpful indexing in one move. According to Cloudflare, “the largest search engine has access to about 2X more information than leading AI companies because they make it difficult for customers to remain discoverable without also being used for AI,” so expect some tension when you restrict mixed-use crawlers.
Using content access policies to control how bots reuse your pages
Blocking AI crawlers is one part of the story; the other is telling allowed bots what they may do with the content they collect. Cloudflare is adding content use controls that let certain customers decide how crawlers can store or reproduce material after visiting your site. This is not a legal contract, but it is a clear technical signal that Verified Bots must respond to if they want to keep their status.
There are three levels: Immediate, where bots cannot store or reuse your content; Reference, where indexing, short excerpts, and links back are fine; and Full, where summaries or near-full reproduction are allowed. These preferences are expressed through an extension to robots.txt called a use parameter, building on Cloudflare’s Content Signals format. While robots.txt itself cannot enforce anything, Cloudflare will monitor whether Verified Bots follow your declared use policy, and those that ignore it or reproduce content in full may lose verification.
This approach supports a more balanced relationship between websites and AI companies: you can still appear in tools that send you traffic or revenue opportunities, while restricting bots that mainly strip value away.
Is Cloudflare’s AI crawler control worth using?
If your livelihood depends on content or ads, it is hard to justify letting AI companies train models on your work by default. Cloudflare’s new controls give you a way to block AI crawlers that treat your site as a training dataset, while keeping the search traffic that still matters. You gain both category-level rules and, if you use Enterprise Bot Management, detailed visibility and content use policies through BotBase.
The catch is that mixed-use crawlers blur the line between search and AI training, and stricter Training policies can unintentionally affect discoverability. That is why it helps to start by understanding which bots hit your site, deciding what you are comfortable with, and then applying rules in stages. Overall, these tools are a step toward a fairer deal, where automated traffic is something you choose and shape, instead of a tax you quietly pay in scraped content.
