---
title: "How to let AI agents through Cloudflare and other bot protection"
description: "If ChatGPT or Claude can't load your site, bot protection is the usual cause. Here is how to tell the three kinds of AI traffic apart, allow the ones you want, and test it."
canonical: "https://getdedica.com/blog/let-ai-agents-through-bot-protection"
last-updated: "2026-10-23"
---

# How to let AI agents through Cloudflare and other bot protection

When an AI assistant can't load your website, the most common cause is bot protection that treats it like an attacker. Allow the assistants and search crawlers you want, decide separately about training crawlers, and test with the same user agent the assistant sends.

This matters because an assistant that can't read your site still answers the question. It uses other people's pages instead. In a study of 37,927 agent journeys across 1,056 businesses, answers built without the business's own site were 3.7 times as likely to contain none of the facts the buyer asked for ([Finder et al., 2026](https://arxiv.org/abs/2609.34951)).

## Three kinds of AI traffic

AI companies send different bots for different jobs. Blocking one has a different effect from blocking another, so treat them separately.

| Kind | What it does | Examples |
| --- | --- | --- |
| Training crawlers | Collect pages that may be used to train models | `GPTBot` (OpenAI), `ClaudeBot` (Anthropic) |
| Search crawlers | Index pages so they can appear in AI search results | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` |
| Assistants acting for a person | Fetch a page because someone asked a question or gave a task | `ChatGPT-User`, `Claude-User`, `Perplexity-User` |

Cloudflare's bot reference uses the same split, with the categories AI Crawler, AI Search and AI Assistant. Blocking a training crawler keeps your content out of future training data, while blocking a search crawler or an assistant keeps you out of the answers your customers read.

## 1. See what an assistant sees

Request a page with an assistant's user agent and look at the status code:

```bash
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
  https://yourcompany.com/pricing
```

A `200` means the page loaded. A `403`, a `503`, or a challenge page means something is blocking it. Repeat with `Claude-User` and `Perplexity-User`.

The test only catches rules that match on the user agent. Bot managers also check IP addresses and signatures, so a page that loads for `curl` can still block the real assistant. The firewall log in step 2 shows what actually happened.

## 2. Cloudflare

In July 2026 Cloudflare replaced its single "block AI bots" switch with separate Search, Agent and Training controls, available on every plan including Free ([Cloudflare](https://blog.cloudflare.com/agentic-web/)).

1. In the Cloudflare dashboard, open **AI Crawl Control** for your domain.
2. Allow **Search** and **Agent** traffic if you want to appear in AI answers and let assistants act for your customers.
3. Choose **Training** separately. Blocking it does not affect search or assistants.
4. Open your **security events** and filter for `ChatGPT-User`, `Claude-User` and `Perplexity-User`. Any blocked or challenged requests point to the rule that caught them, often a custom WAF rule or an older bot setting.

If you write custom rules on a plan without Bot Management, Cloudflare's docs say to match on user agent in WAF rules or robots.txt. User agents can be faked, so prefer Cloudflare's verified-bot signal where you have it. Some assistants now sign their requests with Web Bot Auth, which Cloudflare can verify ([Cloudflare](https://blog.cloudflare.com/signed-agents/)).

## 3. Vercel

Vercel's firewall has two managed rulesets that affect agents, both off by default ([Vercel](https://vercel.com/docs/vercel-firewall/vercel-waf/managed-rulesets)):

The AI Bots ruleset identifies known AI crawlers and can log or deny them. Leave it off, or set it to Log, unless you want to turn away every AI bot, assistants included.

The Bot Protection ruleset challenges clients that don't behave like browsers, and many agents don't. Verified bots such as Google's crawler are excluded. Run it in log mode first to see what it would block, then add a custom rule with a bypass action for automated traffic you trust.

## 4. robots.txt

Write one rule for everyone, and block only the paths no one needs:

```txt
User-agent: *
Disallow: /cart
Disallow: /account

Sitemap: https://yourcompany.com/sitemap.xml
```

If you want to opt out of training only, add a group for the training crawler:

```txt
User-agent: GPTBot
Disallow: /
```

OpenAI notes that `ChatGPT-User` visits happen because a person asked, so robots.txt rules may not apply to them ([OpenAI](https://developers.openai.com/api/docs/bots)). Anthropic says its bots honor robots.txt ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)). In practice the firewall decides whether an assistant gets in, and robots.txt decides what crawlers index.

## 5. Slow agents down with a 429

If you need to limit traffic, answer with a `429 Too Many Requests` status and a `Retry-After` header, which tells a well-behaved client when to try again ([RFC 6585](https://www.rfc-editor.org/rfc/rfc6585)). A challenge page or a `403` gives it no such signal.

## Check the result

Run your site through [Dedica](/) after the change. The "Not blocked by bot detection" check in the report confirms that agents get through.

## Sources

- [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots), OpenAI.
- [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), Anthropic.
- [AI Crawl Control bot reference](https://developers.cloudflare.com/ai-crawl-control/reference/bots/) and [The Internet has a second audience](https://blog.cloudflare.com/agentic-web/), Cloudflare, Sep 2026.
- [WAF managed rulesets](https://vercel.com/docs/vercel-firewall/vercel-waf/managed-rulesets), Vercel.
- Finder et al., [AX is the New AEO](https://arxiv.org/abs/2609.34951), arXiv, Sep 2026.