By Mehdi HassanIs Cloudflare blocking AI crawlers on your site?
Possibly, and you won't see it in your robots.txt. Cloudflare can refuse AI crawlers at its edge, before a request reaches your server, so a site whose robots.txt says "allow everything" can still be invisible to ChatGPT search or Perplexity. Since mid-2025 Cloudflare has moved its defaults toward blocking AI crawlers, and its controls have been renamed and reorganised more than once.
This is the hands-on check for sites on Cloudflare. Which bots you should let in is covered in our guide to blocking or allowing AI crawlers, and crawler access is one layer of an AI-search-ready website. Here we cover each Cloudflare setting and the order to check them in.
Which Cloudflare settings can block AI crawlers?
Three Cloudflare features decide what AI crawlers can fetch, and they work independently. Any one of them can block a bot that the other two allow, which is why sites end up blocked without anyone deciding it.
Setting | Where it lives | What it does | Plans |
|---|---|---|---|
AI bot policies | Security Settings, Configure AI bot policies | Blocks or allows the Search, Agent and Training categories as a group | Check Cloudflare's docs for your plan |
AI Crawl Control | AI Crawl Control, Security tab | Per-crawler Allow, Block or Charge actions, enforced by a WAF custom rule | All plans; better detection on paid plans |
Managed robots.txt | Generated from your AI bot policies | Adds Disallow rules and a Content Signals Policy above your own robots.txt | All plans |
Block AI bots (legacy) | Security Settings, Block AI bots | The older training-bot toggle, deprecated on September 15, 2026 | Being retired |
Other bot and WAF rules | Security rules | Bot challenges and custom rules can also stop crawlers that don't run JavaScript | Varies |
The first two block at the edge: the crawler gets an error or a challenge and never sees your page. Managed robots.txt only asks, since robots.txt compliance is voluntary, which is why Cloudflare points to AI Crawl Control for enforcement.
What do Cloudflare's Search, Agent and Training categories mean?
Cloudflare's AI bot policies sort AI crawlers into three categories, and each can be set to Block (on all pages), Block on pages with ads, or Allow (do not block). The category, not the company, decides what you're turning off, so it's worth reading them carefully.
- Search: crawlers that collect or index content to answer questions later, which by their vendors' descriptions would include bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot. Block these and you drop out of those engines' answers.
- Agent: real-time activity on a person's behalf, such as chat fetchers like ChatGPT-User and Claude-User, and browser-use agents. Block these and an assistant can't open your page when a buyer asks about it.
- Training: crawlers that train or fine-tune models, such as GPTBot and ClaudeBot. Cloudflare also puts mixed-purpose crawlers here, the ones used for both training and search.
That last point is the trap. Under Cloudflare's documented behaviour, mixed-purpose crawlers are blocked under every AI-training block configuration, so refusing training can also cut off a crawler that feeds someone's search product. Open AI Crawl Control and check the category shown for each crawler before you decide.
What changed on September 15, 2026
Cloudflare's docs say the legacy Block AI bots setting is deprecated from September 15, 2026, and that new domains get updated defaults from that date: Training and Agent bots blocked on pages that display ads, Search left allowed, and mixed-purpose crawlers blocked under every AI-training configuration, including the legacy option.
If your zone predates that change, your settings may still reflect older choices someone made, or the defaults at the time. That's the reason to check rather than assume. If your site shows no ads, confirm in Cloudflare's docs how the ads condition is detected before relying on it.
How do you check whether Cloudflare is blocking AI crawlers?
Check the three gates in order, then confirm with real request data. It tells you whether the problem is a setting, a rule or your own robots.txt.
Open your AI bot policies
In the dashboard, go to the domain's Security Settings and find Configure AI bot policies. Note the setting for Search, Agent and Training. If you still see the legacy Block AI bots toggle, note that too.
Read the AI Crawl Control crawler table
Go to AI Crawl Control and open the Security tab. Each row shows a crawler, its category, its request count, robots.txt violations and the current action. Look for Block on any search or agent crawler you want.
Fetch your live robots.txt
Open yourdomain.com/robots.txt in a browser. If managed robots.txt is on, Cloudflare's directives and Content Signals Policy appear above your own rules. Read the whole served file, not the copy in your codebase.
Look for other security rules
Review custom WAF rules, rate limits and bot challenges. A JavaScript challenge stops most AI crawlers, because they don't run scripts. Blocks created by AI Crawl Control appear here as a WAF custom rule.
Confirm with request data
Check the security events for AI user agents, and your origin logs for the status codes they received. Successful responses on key pages mean the crawler got in; challenges and 403s mean it didn't.
How do you allow AI crawlers through Cloudflare?
Allow AI crawlers by setting the Search and Agent categories to Allow, making sure no AI Crawl Control action blocks the individual bots you want, and keeping managed robots.txt consistent with those choices. For a typical startup marketing site, that means search and agent bots in, and training as a separate decision.
- Search category: Allow. This keeps OAI-SearchBot, Claude-SearchBot and PerplexityBot able to index and cite you.
- Agent category: Allow. This lets assistants open your pricing or product page when someone asks about you mid-conversation.
- Training category: your choice. Most marketing sites can allow it; publishers whose pages are the product may block it. Check which mixed-purpose crawlers you'd also catch.
- AI Crawl Control: set any wrongly blocked crawler back to Allow. You can keep Enforce robots.txt on, so allowed bots still have to respect your file.
- Managed robots.txt: if you block training, the managed file adds Disallow lines for training bots. Make sure it doesn't contradict rules in your own robots.txt.
Cloudflare's managed robots.txt documentation shows the exact directives it adds. Its example output disallows training user agents such as GPTBot, ClaudeBot, CCBot and Google-Extended, and leaves search crawlers alone. That matches the pattern we recommend in our robots.txt guide for AI crawlers.
How does Cloudflare's managed robots.txt interact with your own file?
Cloudflare prepends its managed directives to your robots.txt. If your server returns a robots.txt, Cloudflare serves its managed rules and Content Signals Policy first, followed by your file, as one response. If you don't have one, Cloudflare creates a file with Disallow rules for known AI crawlers.
Two things follow. First, a crawler follows the most specific group that names it, so a Cloudflare group for GPTBot overrides your default group for that bot. Repeat any shared Disallow lines if you need them applied. Second, the Content Signals Policy adds lines that state how your content may be used. Cloudflare documents the managed default as search allowed and AI training not allowed, and these signals are preferences, not enforcement.
What content signals are
The Content Signals Policy is Cloudflare's way of stating preferences inside robots.txt, using three categories: search, ai-input and ai-train. Its docs describe the managed default as search=yes and ai-train=no, plus a content-use field Cloudflare is testing.
Nothing forces a crawler to read these lines. Treat them as a statement of your terms, not as a way to stop a bot that ignores robots.txt.
Does pay per crawl affect AI search visibility?
It can. Pay per crawl, shown as the Charge action in AI Crawl Control, asks a crawler owner to pay for each successful request. Cloudflare lists it as a private beta, so most sites won't have it. A crawler that doesn't pay doesn't get the page, and a search crawler that can't fetch you can't cite you.
For a business whose website is marketing, charging the search bots that send you buyers works against you. It makes more sense for publishers and data businesses whose pages are the product. On paid plans, AI Crawl Control can also return a custom block response with a 403 or 402 status.
What should you check after changing the settings?
Check that the crawlers you allowed now get successful responses. AI Crawl Control's request counts and your origin logs should show search and agent bots receiving your pages once they next visit.
- Run a quick external check. Our free AEO checker looks at robots.txt and other basics from the outside.
- Confirm the content is there. Allowed crawlers still need real HTML. If your pages are client-rendered, read about server-side rendering for AI crawlers.
- Watch referrals. Once access is fixed, track AI referral traffic in GA4 to see whether visits arrive.
- Recheck after Cloudflare changes. New categories, crawlers and defaults appear every few months. A quarterly look at AI Crawl Control is enough for most sites.
If you'd rather have the whole stack checked at once, our AI visibility audit runs through rendering, access and structure in order, and our AEO website service covers building sites with these decisions made on purpose.
