What AI crawler blocking means for your site
AI crawler blocking is the practice of using technical controls and platform settings to decide which automated systems can access your website content and whether that content can be stored, indexed, or reused for AI training, search, or agent-like services, giving site owners clearer control over how their work contributes to external AI models and tools. This matters if your site is a real content asset: a blog, SaaS documentation, news archive, or anything that attracts search traffic and ad revenue. The caveat is that you rarely want to block everything—search crawlers still help people find you, and some AI agents might be part of services you value. Think of this as data governance for your website. You are not trying to fight automation; you are trying to stop your articles, product pages, or internal screenshots from being turned into AI training data without consent. New tools from major infrastructure and search platforms now make that possible, if you take the time to configure them.
How Cloudflare’s new bot controls protect your content
If your site runs behind Cloudflare, you now get direct control over different types of AI traffic, even on the Free plan. Instead of treating bots as a single blob, Cloudflare splits AI crawlers into three categories: Search, Agent, and Training. Search covers indexing for search engines; Agent covers bots acting on behalf of users to fetch live answers; Training covers crawlers that collect data to build or refine AI models. Content owners still want to be able to protect their content and be compensated for the original work they create, curate, and share. The key change is that you can apply separate policies to each category, and Cloudflare encourages providers to use separate crawlers for different functions to make access rules clearer. On pages that display ads, new domains will default to blocking Training and Agent crawlers while allowing Search crawlers starting September 15, 2026. Mixed‑use crawlers that do both search and training will be evaluated under both policies and automatically blocked if you block Training, even if Search is allowed.
For Enterprise Bot Management customers, Cloudflare’s BotBase adds another layer of visibility. BotBase is a searchable database of known bots, including Verified Bots and AI agents, and shows how they are classified under the updated bot taxonomy. Administrators can browse this catalog, search for individual bots, filter traffic, and copy detection IDs into security rules. Previously, being a Verified Bot meant automatic access; now verification only confirms identity, while access is controlled by your policies for Search, Agent, and Training. This is where the real website content protection happens: you can see who is hitting your site and tune rules instead of relying on a blunt, global allow-or-block toggle. The gotcha: multi-purpose crawlers like widely used search bots may be blocked if they do not separate their training function. That might slightly affect search visibility if those crawlers cannot adapt, so you will want to monitor analytics after you change your policies.
Step-by-step: Setting Cloudflare policies and content use rules
Here is how you would walk through Cloudflare’s options as a site owner or developer. The goal is to keep search benefits while limiting AI training data use and agent scraping. These controls are especially useful if your pages carry ads or subscription content, where non-human traffic has started to distort the value exchange between visitors and publishers. Around this process, you should remember that blocking Training crawlers can also block mixed-use bots on ad pages, so the sequence below balances control with discoverability.
- Audit your current bot traffic using Cloudflare analytics or, if available, BotBase to see which crawlers are hitting key sections of your site and how they are classified under Search, Agent, or Training.
- Decide policy goals for each content type: for public, search-dependent pages you may allow Search but restrict Training and Agent; for ad-supported or premium content, you may block Training and Agent while keeping Search allowed by default starting September 15, 2026.
- In Cloudflare’s Security or Bot Management settings, set category-based rules so Search crawlers remain allowed where you want visibility, Agent crawlers are limited to parts of the site you are comfortable exposing to live AI agents, and Training crawlers are blocked from pages you do not want used for AI model training.
- For Enterprise Bot Management, use BotBase to fine-tune rules: filter traffic by individual bots, confirm their function, and apply detection IDs in custom security rules to block or allow specific AI crawlers in line with your content policies.
- Define content use controls so that even allowed bots know how they may use your data: choose Immediate for no storage or reuse, Reference for indexing and short excerpts with links back, or Full for broader summaries or reproduction, and express these preferences through the updated Content Signals format in robots.txt.
- Before and after September 15, 2026, review Cloudflare’s default changes for new domains and decide whether you want to opt out to preserve current behavior for Training crawlers that also perform search functions, adjusting settings in the Security area if needed.
- Monitor traffic, ad performance, and search rankings over the following weeks to catch any unintended side effects, especially from mixed-use crawlers like common search bots that might be blocked if they do not separate training from search.
The main warning here is timing: starting September 15, 2026, new customers and new websites will default to allowing search while blocking training and agent use on ad-supported pages. Existing free accounts will also move to these defaults unless they opt out before the deadline. If you rely heavily on a specific crawler that mixes search and training, you should confirm how it behaves under the new policies and adjust ahead of the switch. Still, the upside is clear—site owners can now enforce content use policies and prevent unauthorized AI training on their data using a mix of category-based rules, BotBase visibility, and explicit content use controls. Verified Bots that ignore these preferences or reproduce content in full risk losing their Verified status, which is a strong incentive for AI providers to respect your rules.
Controlling how Google Search uses uploads for AI training
Cloudflare protects the content people and bots see on your site, but you also need to think about what your own team uploads into consumer search tools. Google has updated its Search services settings so that saved media such as images, files, audio, and video from user interactions may be used to improve its AI models and technologies unless users opt out. That saved media includes things like Google Lens images, Search Live recordings, Translate speaking practice, uploaded content, and voice searches. For a business, this means a helpful screenshot or translated document can quietly become AI training material if staff use those features in everyday work. The change lives under two new privacy settings: Search Services History and Personalized Recommendations. Media saved under Search services can be used to develop and improve both the AI models and the services that use them, with the stated aim of delivering safer and more accurate results. The catch is that this setting does not affect media managed by other Google products like Gemini Apps, Google Voice, NotebookLM, or YouTube, so turning it off is only part of a wider data governance picture.
To keep company-sensitive material out of this training pool, you should walk employees through the opt-out process. Users who wish to opt out of having their media saved can turn off the Saved Media setting by going to My Google Activity, selecting Search Services History, and unchecking the Save Media sub-setting. That stops new media interactions with Search services from being saved to Search Services History. However, turning off Save Media does not delete previously saved media, which may continue to be used to improve Google technologies unless users delete it from their accounts. The gotcha here is historical data: simply flipping the toggle does not clean up past uploads. You will need internal guidance about what types of content should be removed, and which search tools are acceptable for work use. Different platforms offer varying levels of control—Cloudflare offers granular bot management by function, while Google gives per-user opt-outs for saved media—so your governance plan must match each platform’s strengths.
Is locking down AI access worth the effort?
If you run a site with meaningful content or revenue, tightening AI crawler access is no longer optional; it is part of protecting your business. Cloudflare introduced new controls that let website owners manage AI traffic across Search, Agent, and Training, available even to Free plan customers. With BotBase and content use rules, you can express clear preferences about how bots may use your content—Immediate for no storage, Reference for indexing and short excerpts, or Full for wider reuse. Meanwhile, you can stop your own uploads into search tools from becoming AI training data by opting out of Google’s Saved Media setting and cleaning up past media where needed. This is worth it, but it is not a one-time toggle. You will need to watch how defaults change, especially around the September 15, 2026 shift for new Cloudflare domains, and keep an eye on how mixed-use crawlers respond. The payoff is straightforward: more website content protection, more AI training data control, and a clearer say in how your work fuels other people’s models.
