Is Your Site Blocking AI Crawlers? A Ten-Point Check After Cloudflare's September Change

What Cloudflare Changed on 15 September

Cloudflare replaced its "Block AI Bots" switch with three independent controls, one each for search, agent and training, defined in its own words as indexing your content to answer questions later, acting in real time on a person's behalf, and taking your content to train a model. New domains are now offered one of two presets, depending on whether the site earns money from advertising.

Three things worth taking from that, because very little of the coverage said them. Search stays allowed in both columns, so this was never a change about being findable. Existing sites were left alone: "your current settings carry over on their own", and a domain that never touched these controls ends up allowing all three. And the two restrictions in the right-hand column work in opposite ways. "Block on pages with ads" is an edge rule that refuses the request. "Disallow AI Training" is not a block at all: it is "named for the Disallow: directive it publishes in your robots.txt".

The part that can actually cost you is the reversal underneath. Cloudflare's harder settings used to spare crawlers that do search and training in one visit; now Block and Block on pages with ads apply to all training crawlers, mixed-use ones included, and selecting Block "will stop Applebot, Bingbot, and Googlebot from reaching your site, search included". Last month that setting left Googlebot alone. This month it does not. If you want training refused without touching search, Disallow AI Training is now the setting that does it.

One caveat on reading any of this second hand, including here. Cloudflare's July announcement was broader than what shipped, and described changes for existing free-plan zones that the September implementation post does not mention. Your dashboard is the authority on your own site.

A Crawler Has a Job, and the Job Is in Its Name

One AI company sends several bots, and they are not interchangeable. Each carries its own user-agent token, its own purpose, and its own consequence if you refuse it. This table is the part of the post worth keeping.

Every row comes from the operator's own documentation: OpenAI describes four, the fourth being OAI-AdsBot, which validates the safety of pages submitted as ads; Anthropic three, Perplexity two, of which it says PerplexityBot "is not used to crawl content for AI foundation models". The last row is a category of its own. Google-Extended and Applebot-Extended never fetch anything, they are opt-out tokens the vendor applies on its own side, and Apple says so: "Applebot-Extended does not crawl webpages". Neither can appear in your logs or return a status code, so a tool that reports one for them is reporting something it made up.

Blocking Training Does Not Cost You Citations

This is the correction worth the whole post. Most published advice treats "block AI crawlers" as one decision with one outcome. It is two decisions with opposite outcomes, and the vendors say so. OpenAI is explicit: "Each setting is independent of the others", with the example of allowing OAI-SearchBot to stay in search results while disallowing GPTBot. The cost sits on the other token: "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers". Anthropic never states the independence outright, but documents three bots with three tokens and pins the search cost on Claude-SearchBot. Our ChatGPT playbook goes further into how each engine uses what it collects.

Google catches the most people. "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal", so blocking it to stay out of AI Overviews does nothing. What governs AI features is Googlebot itself: "robots.txt directives for Googlebot is the control". There is no crawler you can refuse to leave AI Overviews while staying in Search. The lever that exists is the Search generative AI control in Search Console, rolled out to all sites on 31 August 2026, which Google says "isn't used as a ranking or inclusion signal affecting other parts of Search".

That split is the whole industry saying it at once. Publishers are not trying to disappear from AI answers, they are trying not to be free training data. The failure mode is doing the first by accident while attempting the second. One honest caveat on the training half: this is a claim about retrieval, the answers engines fetch and cite. What a model knows about you unprompted comes from its training corpus, which is the argument of our Claude playbook. Refusing a training crawler is a licensing decision with a long-run cost to that, and no cost to being cited today.

Where a Block Can Live

In robots.txt

A request rather than a fence. It works because vendors choose to honour it, and nothing enforces it. Agent bots are carved out of it on two of the three platforms: OpenAI says that because a person initiated the fetch, "robots.txt rules may not apply", and Perplexity says its user fetcher "generally ignores robots.txt rules". Anthropic is the exception and says all three of its bots honour it. So a ChatGPT-User fetch that succeeds against a disallow is documented behaviour, not a broken test, and it is what agentic browsing runs on.

At the edge

Where a refusal happens before your origin is asked, and the layer your robots.txt cannot tell you about, because a dashboard setting is not a line in a file. What the crawler receives is itself a choice: Cloudflare lets you return 403 Forbidden or 402 Payment Required, the second meaning the crawler must pay.

And increasingly in both

Your robots.txt may no longer be written by you. Bot Preference Sync updates the file to reflect what you set in the Cloudflare dashboard, on every plan from Free upward. Read your robots.txt as a report of what is published on your behalf, not a record of what you typed.

How to Read the Result

Running the requests is easy. Interpreting them is where most checks fall down, because a status code on its own says less than it appears to.

A refusal only means something next to a control

Fetch the same URL as an ordinary browser first. Browser served and bot refused is a rule about bots. Everything refused, browser included, is a rule about that page, or a site that turns away anything it does not recognise. Without the control you cannot tell those apart.

A user agent is a claim, not an identity

Your test sets a header that says GPTBot, and so can anyone, which is why Google states that "the HTTP user agent string can be spoofed" and identifies its own crawlers by published IP ranges and a reverse DNS procedure instead. OpenAI, Anthropic and Perplexity all publish IP ranges too. This cuts both ways: a site that refuses every unverified user agent may still admit the genuine crawlers arriving from those ranges, so a blanket refusal is ambiguous rather than a verdict.

A 429 is usually your own test

When we built a fetch-as-each-bot check into Ranqo, the most common false positive was self-inflicted: a dozen requests from one address in three seconds is the shape of a scraper, and a rate limit is what comes back. Space the requests and re-ask anything that was not a 200 before recording it.

A 200 is not proof anyone can read you

Challenge pages arrive with a 200 and a body that says nothing. So do JavaScript shells, and most AI crawlers do not run JavaScript, which is a different failure with the same symptom and one we have walked through before. Compare the words a bot receives with the words the browser got.

The Ten-Point Check

Ten questions in the order that makes each answer interpretable, and the commands to do the fetching part. Replace the domain with your own. These are the identifying tokens rather than the full user-agent strings a real crawler sends, which are longer and contain the token.

# The control. Ask as an ordinary browser first.
curl -s -o /dev/null -w "browser  %{http_code}  %{size_download} bytes\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" \
  https://example.com/

# Then one bot at a time, with a pause between them.
for UA in "OAI-SearchBot/1.0" "ChatGPT-User/1.0" "GPTBot/1.4" \
          "Claude-SearchBot/1.0" "ClaudeBot/1.0" \
          "PerplexityBot/1.0" "Googlebot/2.1" "Bingbot/2.0"
do
  CODE=$(curl -s -o /dev/null -w "%{http_code}" -A "$UA" https://example.com/)
  printf "%-22s %s\n" "$UA" "$CODE"
  sleep 2
done

# Re-ask anything that was not 200, once, after a pause.
# Then read the file the bots are reading.
curl -s https://example.com/robots.txt

AI CRAWLER ACCESS CHECK

  1. Which AI controls does your CDN have set, and did anyone set them on purpose?
  2. Is your robots.txt written by you, or generated for you by your CDN?
  3. Any firewall or origin rule that matches on user agent?
  4. Does robots.txt allow each SEARCH bot by token: OAI-SearchBot,
     Claude-SearchBot, PerplexityBot, Googlebot?
  5. Are the training tokens (GPTBot, ClaudeBot, Google-Extended,
     Applebot-Extended) set the way you actually intend?
  6. Fetch your page as an ordinary browser. That is your control.
  7. Fetch it as each bot, one at a time, a couple of seconds apart.
  8. Re-ask anything that was not 200 after a pause, before you believe it.
  9. Check the body, not just the status: a 200 can be a challenge page or an
     empty JavaScript shell.
 10. Is your sitemap reachable, and does the page a bot gets carry the same
     words the browser got?

READING THE RESULT
  Refused you, served the browser       -> a rule about bots. Real.
  Refused everything, browser included  -> ambiguous. Verify by IP range.
  429                                   -> rate limit, possibly your own test.
  403 or 402 from a CDN                 -> a configured response, not a verdict.
  200 with no readable text             -> a rendering problem, not a block.

VERIFYING A CRAWLER BY IP RANGE
  Every vendor publishes the ranges its bots crawl from, so an ambiguous
  refusal can be checked against the real thing rather than a User-Agent:
    openai.com/searchbot.json · openai.com/gptbot.json
    claude.com/crawling/bots.json
    perplexity.com/perplexitybot.json
    developers.google.com/static/search/apis/ipranges/common-crawlers.json

After You Change Something

Re-testing immediately tells you nothing. OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust", so re-run the check the next day. Then keep the two questions apart. Whether a crawler can reach you is answered by a request and a status code. Whether AI answers name you is a different measurement on a slower clock: restoring access removes a reason you cannot be cited, it does not give an engine a reason to cite you.

Four things to keep. The bot that matters for citations is the search bot, not the training bot. A robots.txt line is a request and an edge rule is enforcement, and only one of them is visible in the file. A refusal means nothing without a browser control beside it. And the failure this post exists for is a setting nobody chose, applied by a platform update, on a site nobody was watching, so put the check on a schedule. Once you know where you stand, our guide to the control files covers what to write.