Is your site blocking ChatGPT? How to check

Your site can block AI engines without anyone on your team deciding to. Since July 1, 2025, Cloudflare asks every new domain at signup whether to allow AI crawlers, and the starting setting is block. Cloudflare says it handles traffic for about 20 percent of the web. On top of that, the bot most people block on purpose, GPTBot, is not the one that decides whether ChatGPT search can show your business. That one is OAI-SearchBot. You can check all of this yourself in about five minutes, and the steps are below.

The three kinds of AI bot

OpenAI and Anthropic each run more than one crawler, and each crawler has its own name in robots.txt. Blocking one does nothing to the others.

• Training crawlers: GPTBot, ClaudeBot. These collect pages that may be used to train future models. OpenAI says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models." Blocking them is a reasonable choice, and it does not take you out of search answers.
• Search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot. These build the index the engines search when someone asks a question. OpenAI is blunt about this one: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."
• Live fetch: ChatGPT-User, Claude-User. These visit a page because a person in a chat asked about it. Anthropic says Claude-User follows robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, because a person started the request.

So a site that added User-agent: GPTBot and Disallow: / in 2023 made a training decision. A site that blocks every bot it does not recognize made a search decision as well, usually without knowing it.

Where the blocks come from

1. A robots.txt line. Someone added it years ago, or a plugin did. A line reading User-agent: * followed by Disallow: / blocks every bot that obeys the file, AI or not.
2. Cloudflare settings. New domains get the allow or block question at signup. Existing sites can block AI crawlers with one click, and Cloudflare can also write Disallow lines into your robots.txt for you through a setting it calls managed robots.txt. Your robots.txt can show lines nobody on your team typed.
3. Firewall and bot rules. A security rule that challenges or blocks unknown bots will stop an AI crawler before it ever reads robots.txt. Robots.txt can say allow while the firewall says no.

The five-minute check

1. Open your robots.txt. Go to yoursite.com/robots.txt in a browser. Look for any User-agent line that names one of the bots above, or a *, followed by Disallow: /. If you see Cloudflare's name in comments at the top, Cloudflare is managing part of the file.
2. If you use Cloudflare, open the dashboard. Pick your domain, then AI Crawl Control. Each AI crawler is listed with its action, allow or block. Then check Security settings, filtered to bot traffic, for the managed robots.txt setting.
3. Ask whoever runs your hosting about firewall rules. One question covers it: "Do we block or challenge bots we have not verified, and are OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed?"

Here is a robots.txt that keeps you in AI search answers while opting out of training. Adjust the training lines to your own preference:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Why an outside test cannot settle it

There are tools that fetch your site while pretending to be GPTBot and report whether they got through. Treat the result with care. Cloudflare and similar services check where a request comes from, and a fake GPTBot comes from the wrong address. Blocking it is correct behavior and says nothing about what the real GPTBot gets.

We ran into this in our own audits. From outside, a blocked fake crawler and a blocked real one look the same. So when our check hits a block at the Cloudflare level, the report says it cannot be confirmed from outside and asks the owner to look at the dashboard. Robots.txt is the only part anyone can read from outside with certainty. For the rest, your own dashboard or your server logs are the only real answer.

What we recommend

If you want AI engines to name your business, allow the search crawlers and the live-fetch bots, and make the training decision separately, on purpose. Then check again after any change to your hosting, your security plugin, or your Cloudflare plan, because that is when these settings move. Crawler access is the first of the five things that move AI visibility, and the only one you can fix in an afternoon.