AI crawler blocking becomes the new default
Cloudflare’s new AI crawler controls are a set of bot and content-use policies that allow website owners to decide, in detail, which AI crawlers may access their pages, how they may use that content, and whether mixed-use bots that serve search and training purposes can touch ad-supported pages at all. This is not a minor configuration tweak; it is a quiet power grab on behalf of publishers after years of unbalanced web scraping. Cloudflare now lets customers manage AI traffic across three categories—Search, Agent, and Training—expanding bot controls beyond a crude allow-or-block model. From September 15, Training and Agent crawlers will be blocked by default on pages that display ads, while Search crawlers remain allowed. In practice, that means AI crawler blocking becomes the baseline for new domains, including free accounts, unless owners actively opt out ahead of the deadline. This change signals a clear judgment: if your business depends on ads or subscriptions, it is no longer acceptable that AI bots quietly repurpose your content without your say.
From blunt robots.txt to Cloudflare bot control
For years, robots.txt has been the only polite fiction standing between bots and a site’s content. It signaled intent but rarely enforced it, and AI companies learned to treat the file as a suggestion rather than a rule. Cloudflare’s bot control system is the first mainstream attempt to attach enforcement and economic consequences to those signals. Instead of classifying bots as “AI” or “non-AI,” Cloudflare now categorizes them by function—Search, Agent, Training—and applies separate policies to each. Multi-purpose crawlers that perform both Search and Training, such as Googlebot, Applebot, and BingBot, will be evaluated under both policies and can be blocked entirely if a site disables Training. According to Cloudflare’s leadership, content owners want protection and compensation but also more nuanced options than “block all automation, every time.” This redesign breaks the long-standing assumption that if you want to be discoverable in search, you must accept your content being reused to train AI models. That trade-off is now negotiable.
BotBase and content-use controls: web scraping policies with teeth
The most consequential shift is not the default blocking alone, but the introduction of BotBase and explicit content-use tiers. Together they turn vague web scraping policies into a structured contract between sites and AI companies. BotBase is a searchable database of known bots—including Verified Bots and AI agents—that gives Enterprise Bot Management customers a centralized view of their classifications and behaviors under Cloudflare’s updated bot taxonomy. On top of that, content use controls let these customers define how bots may use content after crawling it, across three levels: Immediate (no storage or reuse), Reference (indexing, excerpts, and links back), and Full (summaries or reproduction). Cloudflare is extending its Content Signals format in robots.txt with a new use parameter, and it will report whether Verified Bots comply with those declared preferences via BotBase. Verified Bots that ignore these preferences or reproduce content in full may lose their Verified status. This is subtle but important: verification now confirms only a bot’s identity, while access depends on its classification and the site’s policies on Search, Agent, and Training crawlers. Identity without obedience no longer earns a free pass.
AI content protection, ad economics, and a quiet shot at mixed-use crawlers
Behind the technical knobs lies a blunt economic reality. Web traffic used to mean humans viewing ads or paying for subscriptions, but AI models that visit sites on a user’s behalf have upended that system. Non-human traffic now dominates, and much of it extracts value without returning any. Content owners still want to protect their work and believe they should be compensated for the original content they create, curate, and share. Cloudflare’s new defaults—allow search but block training and agent use on pages with ads—explicitly aim to rebalance that relationship. Mixed-use crawlers that index for search and also act as AI trainers or agents will be automatically blocked on those ad-supported pages unless they separate out their roles or respect site choices. The company has already experimented with commercial models, evolving its Pay Per Crawl concept into a Pay Per Use approach where site owners are paid when their content appears in AI chatbot answers, through early partnerships with named AI platforms. The message is clear: indexing is free, training and answer generation should pay or at least respect site-level consent.
What this power shift means for the AI data ecosystem
Cloudflare’s move is less about bot detection and more about who gets to set the terms of AI’s data diet. When the majority of internet traffic is non-human, as its CEO points out, a sustainable ecosystem requires someone to stand up for the humans who run sites and write content. These controls give individual websites agency in an AI data collection landscape that has been dictated by large model providers. Starting September 15, the new defaults will apply to new customers and new domains, with the option for existing users—free included—to opt out through security settings if they want to preserve current behavior. Cloudflare plans more controls later this year so customers can manage automated traffic from the same interface. AI companies face a choice: offer clear, separate crawlers for search, agents, and training that honor content-use settings, or risk systematic blocking on a sizable portion of the web. Site owners, meanwhile, must decide what value exchange they want—visibility, reference traffic, direct payments, or hard AI content protection. The conclusion is straightforward: AI will not stop crawling the web, but with these tools, websites are no longer passive data sources. They become negotiating partners, and the era of one-sided scraping is starting to end.
