AI crawler control is becoming a core publishing decision
AI crawler control is the practice of allowing, limiting, or blocking automated bots from accessing and reusing website content based on their function, such as search indexing, AI assistants, or training data collection, and it is rapidly becoming a core strategic decision for every site owner who depends on ads, subscriptions, or reputation online. Cloudflare’s new approach does not just tweak a few firewall rules—it signals a power shift. By introducing controls that let website owners manage AI traffic across three categories—Search, Agent, and Training—Cloudflare is saying out loud that bot intent matters more than bot identity. For anyone publishing on the web, this is a chance to stop treating AI crawlers as an inevitable background process and start treating them like negotiable business relationships.
From blunt blocking to functional web scraping control
For years, bot management has been a binary choice: allow everything and hope for the best, or slam the door on automation altogether. That era is over. Cloudflare expanded its bot controls beyond a simple allow-or-block model, and that change matters because it maps to how bots use content after crawling it. AI crawler blocking is no longer about punishing AI companies; it is about separating legitimate search indexing from opaque agent and training activity. Instead of classifying bots as AI or non-AI, Cloudflare now categorizes them by function—Search, Agent, and Training—so website owners can apply separate policies to each. That nuance is what real web scraping control looks like. Site administrators can selectively allow or block specific AI crawlers based on their content policies and business goals, rather than lumping every bot into the same bucket. This turns bot filtering into a form of editorial judgment.
Default AI crawler blocking: a line in the sand
The most opinionated part of Cloudflare’s move is not the dashboard; it is the defaults. Starting September 15, 2026, the default settings for new domains will change. Training and Agent crawlers will be blocked by default on pages that display ads, while Search crawlers will remain allowed by default. That is a clear statement that ad-funded pages should not quietly subsidize AI models without consent or compensation. Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. According to Cloudflare’s CEO, “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge”. Free-plan users are not exempt: the feature is available to all Cloudflare customers, including those on the Free plan, and users with free accounts will also switch to these defaults unless they opt-out ahead of the September 15 deadline. Website owners who do not want the new default settings can opt out through the Security settings before September 15. In other words, inaction now means a harder stance against AI training later.
BotBase and AI content policies: transparency with teeth
Visibility has always been the missing piece in Cloudflare bot filtering. BotBase, a searchable database of known bots—including Verified Bots and AI agents—gives Enterprise Bot Management customers a centralized view of their classifications and behaviors based on Cloudflare’s updated bot taxonomy. With BotBase, administrators can browse Cloudflare’s catalog of verified bots, search for specific bots, view their classifications, filter traffic by individual bots, and copy detection IDs for use in security rules. That turns murky bot traffic into something closer to an addressable audience list. More important, Cloudflare is adding content use controls that let Enterprise Bot Management customers define how bots may use content after crawling it. The three levels are Immediate (no storage or reuse), Reference (indexing, excerpts, and links back), and Full (summaries or reproduction). The company is extending its Content Signals format in robots.txt with a new use parameter to express these preferences, and Verified Bots that ignore these preferences or reproduce content in full may lose their Verified status. This is not legal enforcement, but it is reputational pressure backed by technical tracking—a first draft of enforceable AI content policies.
What this shift means for publishers and AI companies
At its core, this move is a response to content owners who “want to be able to protect their content, and they should be compensated for the original content that they work hard to create, curate, and share”. Web traffic used to indicate humans seeing ads or paying for subscriptions, but the popularity of AI models that visit sites on a user’s behalf has upended that system. Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. Previously, all Verified Bots were allowed by default; under the new model, verification confirms only a bot’s identity, while access depends on its classification and the website owner’s policies. The company plans to add more controls later this year, allowing customers to manage automated traffic from the same interface, and it will notify customers before the default changes so they have time to review and update their settings. The message to AI companies is blunt: respect declared AI content policies, separate search from training, and be explicit about content use—or expect to be filtered out. For site owners, the question is no longer whether to engage with AI crawlers, but on what terms.
