Here is a scenario that confuses thousands of website owners in 2026. Your robots.txt file explicitly allows GPTBot, ClaudeBot and every other AI crawler. You never installed any blocking plugin. Yet AI assistants never cite your content, and when you test your site with an AI visibility scanner, the crawlers receive a 403 Forbidden response. The block is real, but it is not happening on your server. It is happening one layer up, at your CDN.

What Changed in 2025

In July 2025, Cloudflare, the network that sits in front of roughly a fifth of the web, made a landmark decision: AI training crawlers would be blocked by default for newly onboarded domains. The company framed it as giving content owners control and even introduced a marketplace model where AI companies could pay for crawl access. Other CDN and hosting providers followed with similar toggles and defaults through late 2025.

The intention was reasonable. The side effect was not: enormous numbers of small business sites, blogs and web tools are now invisible to AI systems without their owners ever making that choice, or even knowing a choice was made for them.

Why You Do Not Notice It

Edge level blocking is invisible in day to day use for three reasons.

Humans see nothing wrong. The CDN only interferes with requests whose user agent or behavior identifies them as an AI crawler. Every normal visitor, and every normal test you run from a browser, gets a perfect 200 response.

Your origin files are untouched. You can inspect your server all day and find nothing. The 403 is generated at the network edge, before the request ever reaches your hosting.

Your robots.txt may be replaced in transit. This is the part that surprises even experienced developers. Cloudflare offers a managed robots.txt feature that injects or rewrites directives at the edge, including a Content Signals policy block. The robots.txt file sitting on your server and the robots.txt file the internet actually receives can be two different documents. Auditing the file on disk tells you nothing about what crawlers really see.

We Learned This the Embarrassing Way

CheckMy.site is a scanner that measures AI readiness. While upgrading our own detection engine, we ran a scan against our own domain and watched it fail the exact checks we had just built. Our robots.txt allowed every AI bot. Our live fetch test still showed HTTP 403 for both GPTBot and ClaudeBot, and our CDN detection flagged that the robots.txt being served did not match the file on our server. A website built to measure AI visibility was itself blocked at the edge, by a default setting nobody in the team had ever seen. If it happened to us, it can happen to anyone.

How to Check Your Site in Two Minutes

Test 1: Request your homepage as an AI bot. From any terminal, send a request with a real AI crawler user agent and compare it to a normal request:

curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" https://yoursite.com/

curl -I -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" https://yoursite.com/

If the first request returns 403 or 503 while the second returns 200, you have edge level blocking.

Test 2: Compare the two versions of robots.txt. Open yoursite.com/robots.txt in a browser and compare it line by line with the file on your server. Look for a Content-Signal line or comment blocks you never wrote. If they differ, a CDN is managing the file for you.

Test 3: Run an automated scan. The CheckMy.site scanner performs both tests automatically. Its live fetch check requests your page as GPTBot and as ClaudeBot in parallel and compares the responses against a normal visit, and its CDN detection flags managed robots.txt substitution. This catches blocking that pure robots.txt analysis can never see.

How to Fix It in Cloudflare

If your site is behind Cloudflare, the settings live in the dashboard under AI Crawl Control, which older accounts may still see under Security and then Bots.

Step 1: Open the AI Crawl Control section for your zone. You will see a list of known AI crawlers with per bot Allow and Block switches.

Step 2: Set the bots you want to Allow. At minimum, most sites benefit from allowing the answer engine and user triggered bots: OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Perplexity-User. These are the ones that produce citations and referral traffic. Our complete list of AI crawlers explains what each one does.

Step 3: Check whether the managed robots.txt feature is enabled. If it is, either disable it so your own file is served unmodified, or edit the managed policy so it reflects your real preferences instead of the defaults.

Step 4: Retest with the curl commands above. Edge configuration changes typically propagate within a few minutes.

Sites behind other CDN or security providers should search their dashboards for bot management, AI crawlers or bot fight features. The pattern is the same everywhere: a default deny list that you can override per bot.

Should You Allow Everything?

Not necessarily, and this is worth an honest paragraph. Edge blocking exists because many publishers have legitimate objections to their content training commercial models without compensation. A sensible 2026 policy for most businesses looks like this: allow the search index bots and user triggered fetchers, because they cite you and send visitors; then make a deliberate decision about pure training crawlers like GPTBot, ClaudeBot in training mode and CCBot based on your business. A local service company usually wants maximum AI visibility everywhere. A publisher selling original reporting may prefer to keep training bots blocked or negotiate paid access. The point of this article is not that blocking is wrong. The point is that the decision should be yours, made consciously, not a default you never saw.

The Takeaway

In 2026, checking robots.txt is no longer enough. The real question is what an AI crawler experiences when it knocks on your door, and only a live request can answer that. Run the two curl tests above, or get the complete picture with a free 158 point scan that covers edge blocking, robots.txt policy, structured data and everything else that determines whether AI systems can see, understand and cite your website.