How to let AI agents through Cloudflare and other bot protection
If ChatGPT or Claude can't load your site, bot protection is the usual cause. Here is how to tell the three kinds of AI traffic apart, allow the ones you want, and test it.
When an AI assistant can't load your website, the most common cause is bot protection that treats it like an attacker. Allow the assistants and search crawlers you want, decide separately about training crawlers, and test with the same user agent the assistant sends.
This matters because an assistant that can't read your site still answers the question. It uses other people's pages instead. In a study of 37,927 agent journeys across 1,056 businesses, answers built without the business's own site were 3.7 times as likely to contain none of the facts the buyer asked for (Finder et al., 2026).
Three kinds of AI traffic
AI companies send different bots for different jobs. Blocking one has a different effect from blocking another, so treat them separately.
| Kind | What it does | Examples |
|---|---|---|
| Training crawlers | Collect pages that may be used to train models | GPTBot (OpenAI), ClaudeBot (Anthropic) |
| Search crawlers | Index pages so they can appear in AI search results | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Assistants acting for a person | Fetch a page because someone asked a question or gave a task | ChatGPT-User, Claude-User, Perplexity-User |
Cloudflare's bot reference uses the same split, with the categories AI Crawler, AI Search and AI Assistant. Blocking a training crawler keeps your content out of future training data, while blocking a search crawler or an assistant keeps you out of the answers your customers read.
1. See what an assistant sees
Request a page with an assistant's user agent and look at the status code:
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
https://yourcompany.com/pricing
A 200 means the page loaded. A 403, a 503, or a challenge page means something is blocking it. Repeat with Claude-User and Perplexity-User.
The test only catches rules that match on the user agent. Bot managers also check IP addresses and signatures, so a page that loads for curl can still block the real assistant. The firewall log in step 2 shows what actually happened.
2. Cloudflare
In July 2026 Cloudflare replaced its single "block AI bots" switch with separate Search, Agent and Training controls, available on every plan including Free (Cloudflare).
- In the Cloudflare dashboard, open AI Crawl Control for your domain.
- Allow Search and Agent traffic if you want to appear in AI answers and let assistants act for your customers.
- Choose Training separately. Blocking it does not affect search or assistants.
- Open your security events and filter for
ChatGPT-User,Claude-UserandPerplexity-User. Any blocked or challenged requests point to the rule that caught them, often a custom WAF rule or an older bot setting.
If you write custom rules on a plan without Bot Management, Cloudflare's docs say to match on user agent in WAF rules or robots.txt. User agents can be faked, so prefer Cloudflare's verified-bot signal where you have it. Some assistants now sign their requests with Web Bot Auth, which Cloudflare can verify (Cloudflare).
3. Vercel
Vercel's firewall has two managed rulesets that affect agents, both off by default (Vercel):
The AI Bots ruleset identifies known AI crawlers and can log or deny them. Leave it off, or set it to Log, unless you want to turn away every AI bot, assistants included.
The Bot Protection ruleset challenges clients that don't behave like browsers, and many agents don't. Verified bots such as Google's crawler are excluded. Run it in log mode first to see what it would block, then add a custom rule with a bypass action for automated traffic you trust.
4. robots.txt
Write one rule for everyone, and block only the paths no one needs:
User-agent: *
Disallow: /cart
Disallow: /account
Sitemap: https://yourcompany.com/sitemap.xml
If you want to opt out of training only, add a group for the training crawler:
User-agent: GPTBot
Disallow: /
OpenAI notes that ChatGPT-User visits happen because a person asked, so robots.txt rules may not apply to them (OpenAI). Anthropic says its bots honor robots.txt (Anthropic). In practice the firewall decides whether an assistant gets in, and robots.txt decides what crawlers index.
5. Slow agents down with a 429
If you need to limit traffic, answer with a 429 Too Many Requests status and a Retry-After header, which tells a well-behaved client when to try again (RFC 6585). A challenge page or a 403 gives it no such signal.
Check the result
Run your site through Dedica after the change. The "Not blocked by bot detection" check in the report confirms that agents get through.
Sources
- Overview of OpenAI crawlers, OpenAI.
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic.
- AI Crawl Control bot reference and The Internet has a second audience, Cloudflare, Sep 2026.
- WAF managed rulesets, Vercel.
- Finder et al., AX is the New AEO, arXiv, Sep 2026.