Is your site blocking ChatGPT? How to check

Your site can block AI engines without anyone on your team deciding to. On July 1, 2025, Cloudflare started asking every new domain at signup whether to allow AI crawlers, with block as the starting setting. On September 15, 2026, it changed that. A new domain is now offered one of two presets. The one for a site without ads allows AI crawlers. The one for a site that runs ads blocks AI training crawlers and keeps search crawlers allowed. Cloudflare also moved every domain that had the old block switched on to that same training setting. Cloudflare says it handles traffic for about 20 percent of the web. On top of that, the bot most people block on purpose, GPTBot, is not the one that decides whether ChatGPT search can show your business. That one is OAI-SearchBot. You can check all of this yourself in about five minutes, and the steps are below.

The three kinds of AI bot

OpenAI and Anthropic each run more than one crawler, and each crawler has its own name in robots.txt. Blocking one does nothing to the others.

• Training crawlers: GPTBot, ClaudeBot. These collect pages that may be used to train future models. OpenAI says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models." Blocking them is a reasonable choice, and it does not take you out of search answers.
• Search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot. These build the index the engines search when someone asks a question. OpenAI is blunt about this one: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."
• Live fetch: ChatGPT-User, Claude-User. These visit a page because a person in a chat asked about it. Anthropic says Claude-User follows robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, because a person started the request.

So a site that added User-agent: GPTBot and Disallow: / in 2023 made a training decision. A site that blocks every bot it does not recognize made a search decision as well, usually without knowing it.

Where the blocks come from

1. A robots.txt line. Someone added it years ago, or a plugin did. A line reading User-agent: * followed by Disallow: / blocks every bot that obeys the file, AI or not.
2. Cloudflare settings. The single Block AI bots switch is gone. On September 15, 2026, Cloudflare replaced it with separate Search, Training and Agent settings. A new domain is offered a preset at signup, and Search starts on Allow in both presets. Training starts on Allow for a site without ads, and on an option called Disallow AI Training for a site that runs ads. Disallow AI Training blocks the training-only crawlers run by OpenAI, Anthropic, Meta and Amazon, and it keeps Googlebot, Applebot and Bingbot crawling for search. The preset for sites with ads also sets Agent, the group Cloudflare uses for chat fetch bots, to block on pages that serve an ad. Cloudflare can also write Disallow lines into your robots.txt for you. That setting was called managed robots.txt and is now Bot Preference Sync, and it is switched on in both presets. Your robots.txt can show lines nobody on your team typed.
3. Firewall and bot rules. A security rule that challenges or blocks unknown bots will stop an AI crawler before it ever reads robots.txt. Robots.txt can say allow while the firewall says no.

If your robots.txt has a Disallow line under Google-Extended or Applebot-Extended, Cloudflare may have put it there, because Disallow AI Training works by publishing those lines. Google says Google-Extended covers Gemini training and the grounding Gemini does at prompt time, and that it has no effect on a site's inclusion in Google Search. That line leaves your Google rankings alone and tells Google to keep your pages out of what the Gemini app looks up when someone asks it a question.

The five-minute check

1. Open your robots.txt. Go to yoursite.com/robots.txt in a browser. Look for any User-agent line that names one of the bots above, or a *, followed by Disallow: /. If you see Cloudflare's name in comments at the top, Cloudflare is managing part of the file.
2. If you use Cloudflare, open the dashboard. Pick your domain, then Security settings. Search, Training and Agent each have their own setting. If your site was set to Block under the old switch, Cloudflare moved Training to Disallow AI Training for you, so you will see that option selected even if nobody on your team picked it. AI Crawl Control lists each AI crawler with its action, allow or block. Then check whether Preference Sync is on, because that is the setting that writes to your robots.txt.
3. Ask whoever runs your hosting about firewall rules. One question covers it: "Do we block or challenge bots we have not verified, and are OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed?"

Here is a robots.txt that keeps you in AI search answers while opting out of training. Adjust the training lines to your own preference:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Why an outside test cannot settle it

There are tools that fetch your site while pretending to be GPTBot and report whether they got through. Treat the result with care. Cloudflare and similar services check where a request comes from, and a fake GPTBot comes from the wrong address. Blocking it is correct behavior and says nothing about what the real GPTBot gets.

We ran into this in our own audits. From outside, a blocked fake crawler and a blocked real one look the same. So when our check hits a block at the Cloudflare level, the report says it cannot be confirmed from outside and asks the owner to look at the dashboard. Robots.txt is the only part anyone can read from outside with certainty. For the rest, your own dashboard or your server logs are the only real answer.

What we recommend

If you want AI engines to name your business, allow the search crawlers and the live-fetch bots, and make the training decision separately, on purpose. Then check again after any change to your hosting, your security plugin, or your Cloudflare plan, because that is when these settings move. Crawler access is the first of the five things that move AI visibility, and the only one you can fix in an afternoon.