AI crawlers: block or allow?
Block AI crawlers that collect training data if you do not want your content in AI models, but think twice before blocking AI search and assistant bots: they decide whether ChatGPT, Claude or Perplexity can show and link your pages. Check your Cloudflare AI settings too, because new defaults from September 15, 2026 can block more than you expect.
Three kinds of AI crawlers
"AI bot" covers very different visitors. The big AI companies now run separate bots for separate jobs, and the decision to block each one has different consequences:
| Job | Examples | If you block it |
|---|---|---|
| Training: collects content that may be used to train models | GPTBot (OpenAI), ClaudeBot (Anthropic) | Your content is not used for future training. No direct effect on traffic. |
| Search: indexes pages for AI search answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your pages stop appearing and getting links in those AI search answers. |
| User-triggered: fetches a page when a person asks the assistant | ChatGPT-User, Claude-User, Perplexity-User | The assistant cannot read your page when someone asks about it. |
Two more names matter even though they never show up in your logs. Google-Extended is a robots.txt token, not a crawler: Googlebot does the crawling, and the token tells Google whether that content may be used to train Gemini models. Applebot-Extended works the same way for Apple. Blocking these tokens does not affect Google Search or Apple's search features. The full list of user agents, with links to each operator's documentation, is on our bot user agents page.
Which AI crawlers to block, and which to let in
A practical default for a business site that wants traffic:
- Training bots: your call. Blocking them costs you nothing in visits today. Some publishers block on principle, others allow them to stay present in future models.
- AI search bots: allow, unless you have a reason. They work like a search engine: they bring a link and a visitor.
- User-triggered agents: allow. A person asked about your page, and blocking means the answer is written without it.
- Unknown bots that claim to be AI crawlers: verify before you trust them. OpenAI, Anthropic and Perplexity all publish the IP addresses their bots use.
The exception is load. If an AI crawler hits your server so hard that real customers wait, rate limit it or block it for a while, whatever its purpose. A bot traffic report shows how much each crawler actually costs you.
How to block AI crawlers with robots.txt
Robots.txt is the polite way. The training bots of the major companies respect it, and OpenAI documents that disallowing GPTBot signals your content should not be used for training. To opt out of training but stay in AI search:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /Two limits. First, robots.txt is a request, not a lock: bots that ignore it are not stopped. Second, user-triggered agents may not apply robots.txt at all, because a person asked for the page. OpenAI and Perplexity both say so in their documentation.
Blocking AI crawlers in Cloudflare
Cloudflare can enforce what robots.txt only asks for. It classifies verified bots by category, including AI Crawler, AI Search and AI Assistant, and you can use those categories in a custom rule. To block training crawlers and leave AI search and assistants alone:
(cf.verified_bot_category eq "AI Crawler")Action: Block. Because the rule matches only verified bots in that one category, search engines and AI search bots are not touched.
Cloudflare also offers AI crawler policies in the dashboard, with three choices for each type of AI bot: block on all pages, block only on pages that show ads, or allow. The policies also cover unverified bots that behave like the verified ones in that category.
The September 15, 2026 change
On September 15, 2026, Cloudflare changed how it handles AI crawlers:
- New domains get new defaults: training and agent bots are blocked on pages with ads, search bots stay allowed.
- The old one-click "Block AI bots" setting is deprecated.
- Crawlers that collect data both for search and for AI training are now blocked by every configuration that blocks AI training, including the old setting. Until then, the old setting left them out.
The last point is the one to check. Cloudflare's documentation does not list which crawlers it treats as both search and training. If one of them is a search engine that sends you traffic, a setting you switched on to stop AI training may now block it too.
How to check your site:
- Open your AI crawler settings and note which categories are blocked, and whether on all pages or only on pages with ads.
- In Security Analytics, filter on verified bots in the Search Engine Crawler category and look for blocks or challenges since mid-September.
- Test a few pages with the URL Inspection tool in Search Console and check its crawl stats report.
Verify before you trust or block
AI user agents are easy to fake, just like Googlebot's. Scrapers sometimes call themselves GPTBot or ClaudeBot to look legitimate. Check the IP against the operator's published list. OpenAI's lists for GPTBot are on our IP ranges page, together with Google's and Bing's. Cloudflare's verified bot categories already do this check for you: a request only gets the AI Crawler category after Cloudflare has verified the bot.
FAQ
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and page visits on a user's request come from ChatGPT-User. Block those only if you want to stay out of ChatGPT answers.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended is only a robots.txt token that controls use of your content for Gemini training. Googlebot keeps crawling for Search either way.
Should I block AI crawlers on an online store?
Usually not the search and assistant bots: shoppers increasingly ask assistants for product recommendations. Blocking training bots is a separate choice with no direct effect on sales.
Can AI crawlers overload my server?
Yes, some crawl aggressively. If one does, rate limit it rather than blocking a whole category, and check which bot it really is first.