Every AI Crawler User Agent in 2026, Ranked by How Often Top Sites Block It
We asked the 10,000 most visited website addresses in Chrome for their robots.txt files. 25% of the files we could read block GPTBot, OpenAI's training crawler, but only 10% block OAI-SearchBot, which finds pages for ChatGPT search. One Cloudflare setting writes many of the training blocks, and a fifth of files still name crawlers no company documents. Here are 59 crawler names, ranked.
TL;DR
- In top sites' robots.txt files, a company's AI training crawler is blocked about two to five times as often as its search crawler, and GPTBot is blocked most, at 25%.
- If you use Cloudflare's managed robots.txt, open your file and read what it added: not every name on its block list trains a model.
- Decide by job, not by company: a block on a training crawler asks to keep your pages out of model training, and a block on a search crawler takes you out of that engine's search answers.
- Write each name exactly as its operator documents it, replace old names with current ones, and block at your server the crawlers that fetch pages for a person, which may ignore robots.txt.
The AI crawlers top sites block most are the ones that gather data for model training. Of the robots.txt files we could read from the 10,000 most visited website addresses in Chrome, 25.0% block GPTBot, OpenAI's training crawler, from the whole site, and 23.2% block Anthropic's ClaudeBot. The crawlers that find pages for AI search are blocked far less often: OAI-SearchBot, which feeds ChatGPT search, by 10.0%, and Googlebot, which feeds Google's AI Overviews, by 4.1%.
Much of the gap comes from one setting. Cloudflare's managed robots.txt adds a block list of AI crawlers to the top of a site's file, and files carrying it hold 28.5% of all the GPTBot blocks we found. Outside them, 19.3% of files still block GPTBot.
A fifth of the files also name crawlers that no company documents today, such as anthropic-ai and claude-web. Below are 59 crawler names, with what each operator says the crawler does, how often top sites block it and name it, and what each kind of block costs you.
Which AI Crawlers Top Sites Block Most
Ranked by how often the whole site is blocked, crawlers that gather data for AI training fill most of the top of the list. GPTBot leads at 25.0%, then Common Crawl's CCBot, ByteDance's Bytespider, which ByteDance documents only as Toutiao Search's crawler, and ClaudeBot. Google-Extended, Amazonbot, Meta-ExternalAgent and Applebot-Extended follow at about 20% each. The first crawler in the chart that works for an AI search engine or for a person is PerplexityBot at 13.1%. The search engines sit at the bottom: Bingbot at 5.5% and Googlebot at 4.1%.
Most blocks of the crawlers at the top name them. Of the 1,744 files that block GPTBot, 1,388 name it, and for each of the next seven crawlers between 74% and 78% of the blocks do. Only 5.9% of files tell every unnamed crawler to stay out, and 3.6% name no crawler at all and block everything. In all, 30.7% of the files write out the name of at least one of the 19 AI crawlers in the chart.
Share of top sites' robots.txt files that block each crawler from the whole site
6,969 readable robots.txt files from the top 10,000 origins on the Chrome UX Report list, read 10 November 2026. The teal part of each bar is files carrying Cloudflare's managed list.
- GPTBot
- All files: 25.0%
- Files without Cloudflare's list: 19.3%
- Share of these blocks in files with Cloudflare's list: 28.5%
- CCBot
- All files: 24.8%
- Files without Cloudflare's list: 19.0%
- Share of these blocks in files with Cloudflare's list: 28.7%
- Bytespider
- All files: 24.2%
- Files without Cloudflare's list: 18.4%
- Share of these blocks in files with Cloudflare's list: 29.5%
- ClaudeBot
- All files: 23.2%
- Files without Cloudflare's list: 17.3%
- Share of these blocks in files with Cloudflare's list: 30.7%
- Google-Extended
- All files: 21.2%
- Files without Cloudflare's list: 15.2%
- Share of these blocks in files with Cloudflare's list: 33.5%
- Amazonbot
- All files: 21.1%
- Files without Cloudflare's list: 15.0%
- Share of these blocks in files with Cloudflare's list: 34.0%
- Meta-ExternalAgent
- All files: 20.7%
- Files without Cloudflare's list: 14.7%
- Share of these blocks in files with Cloudflare's list: 34.4%
- Applebot-Extended
- All files: 19.9%
- Files without Cloudflare's list: 13.7%
- Share of these blocks in files with Cloudflare's list: 36.0%
- MistralAI-Training
- All files: 13.6%
- Files without Cloudflare's list: 6.9%
- Share of these blocks in files with Cloudflare's list: 52.5%
- PerplexityBot
- All files: 13.1%
- Files without Cloudflare's list: 13.8%
- Share of these blocks in files with Cloudflare's list: 2.2%
- ChatGPT-User
- All files: 12.5%
- Files without Cloudflare's list: 12.7%
- Share of these blocks in files with Cloudflare's list: 5.2%
- OAI-SearchBot
- All files: 10.0%
- Files without Cloudflare's list: 10.7%
- Share of these blocks in files with Cloudflare's list: 1.3%
- Perplexity-User
- All files: 9.8%
- Files without Cloudflare's list: 9.9%
- Share of these blocks in files with Cloudflare's list: 6.0%
- Claude-User
- All files: 9.5%
- Files without Cloudflare's list: 9.6%
- Share of these blocks in files with Cloudflare's list: 6.2%
- Claude-SearchBot
- All files: 9.1%
- Files without Cloudflare's list: 9.6%
- Share of these blocks in files with Cloudflare's list: 2.0%
- DuckAssistBot
- All files: 8.8%
- Files without Cloudflare's list: 9.5%
- Share of these blocks in files with Cloudflare's list: 0.2%
- MistralAI-User
- All files: 8.7%
- Files without Cloudflare's list: 8.7%
- Share of these blocks in files with Cloudflare's list: 6.6%
- Meta-WebIndexer
- All files: 8.0%
- Files without Cloudflare's list: 8.6%
- Share of these blocks in files with Cloudflare's list: 0.4%
- MistralAI-Index
- All files: 6.5%
- Files without Cloudflare's list: 6.9%
- Share of these blocks in files with Cloudflare's list: 1.8%
- Applebot
- All files: 5.7%
- Files without Cloudflare's list: 6.0%
- Share of these blocks in files with Cloudflare's list: 2.8%
- Bingbot
- All files: 5.5%
- Files without Cloudflare's list: 5.9%
- Share of these blocks in files with Cloudflare's list: 0.3%
- Googlebot
- All files: 4.1%
- Files without Cloudflare's list: 4.4%
- Share of these blocks in files with Cloudflare's list: 0.7%
The last column is the share among files that don't carry Cloudflare's list. Googlebot, Bingbot and Applebot are search engines, shown for comparison. Since 5.9% of files block every crawler they don't name, a crawler falls below that only when sites let it in by name, or, for Applebot, through Googlebot's rules.
Googlebot's low figure is nearly all sites that block everything. Of the 631 files that name Googlebot, 20 block it; the rest name it to let it in. Bingbot is blocked by name more often, by 78 of the 571 files that name it. On the whole, sites treat search engines as crawlers to keep and training crawlers as ones to refuse.
Training Crawlers Get Blocked, Search Crawlers Mostly Don't
The eight companies below each run a training crawler or token beside a search crawler, and every one of them sees the training one blocked more. GPTBot is blocked by 25.0% of files and OAI-SearchBot by 10.0%; ClaudeBot by 23.2% and Claude-SearchBot by 9.1%; Google-Extended by 21.2% and Googlebot by 4.1%. Across the eight, the training crawler is blocked about two to five times as often.
Each company's training crawler is blocked about two to five times as often as its search crawler
Share of the 6,969 readable files that block each crawler from the whole site.
- OpenAI
- GPTBot, training crawler or token: 25.0%
- OAI-SearchBot, search crawler: 10.0%
- Anthropic
- ClaudeBot, training crawler or token: 23.2%
- Claude-SearchBot, search crawler: 9.1%
- Google-Extended, training crawler or token: 21.2%
- Googlebot, search crawler: 4.1%
- Amazon
- Amazonbot, training crawler or token: 21.1%
- Amzn-SearchBot, search crawler: 6.6%
- Meta
- Meta-ExternalAgent, training crawler or token: 20.7%
- Meta-WebIndexer, search crawler: 8.0%
- Apple
- Applebot-Extended, training crawler or token: 19.9%
- Applebot, search crawler: 5.7%
- Mistral AI
- MistralAI-Training, training crawler or token: 13.6%
- MistralAI-Index, search crawler: 6.5%
- Moonshot AI
- KimiBot, training crawler or token: 13.4%
- Kimi-SearchBot, search crawler: 6.3%
Google-Extended and Applebot-Extended are robots.txt names, not crawlers: Googlebot and Applebot fetch the pages, and the token decides the use. Amazon says Amzn-SearchBot follows other search bots' rules when no group names it; we read those as Googlebot's, then Bingbot's.
That doesn't mean sites chose to keep search crawlers in. Of the 1,744 files that block GPTBot, 1,046 don't block OAI-SearchBot from the whole site. In 945 of those, OAI-SearchBot is never named, so it falls under the general rules for every crawler, which in those files close some paths at most. Only 101 name OAI-SearchBot, and 40 of those let it fetch everything. The split is mostly a list that names training crawlers and stops there.
Cloudflare's list adds to the split, because its common 32-name version names none of these companies' search crawlers. Without it the split shrinks but holds for six of the eight companies: in files that don't carry the list, their training crawlers are blocked 1.7 to 3.4 times as often as their search crawlers, and 44.7% of the files blocking GPTBot leave OAI-SearchBot unblocked. Mistral AI and Moonshot AI are the exceptions: outside the list, each one's training and search crawlers are blocked about equally.
Perplexity runs no training crawler. Its search crawler, PerplexityBot, is blocked by 13.1% of files, more than OAI-SearchBot or Claude-SearchBot, and almost all of it in sites' own rules: Cloudflare's list accounts for 2.2% of its blocks.
One Cloudflare Setting Writes Many of the Blocks
Cloudflare's managed robots.txt puts a block of rules at the top of a site's file, and when a site has no file, Cloudflare "creates a new file with managed Disallow rules for known AI crawlers". Customers had turned it on for more than 3.8 million domains by September 2025. Cloudflare now builds the block with Bot Preference Sync, announced in August 2026, which writes it from a site's Search, Agent and Training settings. A new publisher who says its pages carry ads starts with Training set to Disallow.
We can't see anyone's Cloudflare settings, but the block leaves a mark: a Content-Signal line holding use=reference, which Cloudflare says it adds to the managed content. 501 of the readable files carry it (7.2%), and 397 of those list exactly the same 32 crawler names. The 501 files hold 28.5% of all the GPTBot blocks we found, 33.5% of Google-Extended's and 52.5% of MistralAI-Training's. Inside them GPTBot is blocked in 99.2% of files and PerplexityBot in 4.0%.
28.5%
of all GPTBot blocks come from the 501 files carrying Cloudflare's managed list
The 32-name list is aimed at training crawlers, so it leaves out PerplexityBot, OAI-SearchBot and Claude-SearchBot. Not every name on it trains a model, though. Its 32 names include two search engines' crawlers, Baiduspider and PetalBot; an uptime monitor, WARDBot; three crawlers whose makers describe other jobs, Diffbot, GoogleOther and Cotoyogi; four crawlers from brand-monitoring and SEO tools; and six names no company documents today, among them anthropic-ai, claude-web and cohere-ai.
Four more only visit sites whose owners signed up for them: Google-CloudVertexBot, atlassian-bot, QualifiedBot and NavuBot. For most sites those lines do nothing. For a site that uses one of those products, they ask it to stay out. If you use Cloudflare's setting, open your robots.txt and read what it added. Cloudflare's own docs say "robots.txt compliance is voluntary".
Content signals have spread mostly through that setting. Outside Cloudflare's files, 89 files carry a Content-Signal line of their own, 1.4% of them.
Old and Undocumented Names Still Fill Block Lists
21.2% of the readable files name at least one of the 18 names in the table below, none of which any company documents as a crawler today. Cloudflare's list accounts for 498 of those 1,480 files. Outside it, 15.2% of files still carry one.
The most common, anthropic-ai, is one of two old Anthropic names. In July 2024 an Anthropic spokesperson told 404 Media that anthropic-ai and claude-web were "no longer in use" and that ClaudeBot had been configured to respect rules written for them. Anthropic's current crawler page names neither. It lists ClaudeBot, Claude-SearchBot and Claude-User, and says Anthropic's bots honour robots.txt. 81 files block an old name while leaving ClaudeBot unblocked, and 78 of them never name ClaudeBot. Whether those files keep ClaudeBot out rests on a statement from 2024. One ClaudeBot line would settle it.
omgili and omgilibot belong to Webz.io, which in July 2024 wrote that Omgilibot "served us well" and introduced two bots in its place: Webzio, which crawls, and Webzio-Extended, which tags the data as usable or not for AI training. FacebookBot, which Meta once said improved language models for its speech recognition, is gone from Meta's crawler page. cohere-ai, GrokBot and DeepSeekBot appear on many lists, but we found no page where Cohere, xAI or DeepSeek documents a crawler.
Crawler names that no company documents today
18 of the most-named, with the share of the 6,969 readable files that name each one.
| Name in robots.txt | Named in | What we found | Write instead |
|---|---|---|---|
 | 17.1% | Anthropic said in 2024 it was no longer in use | ClaudeBot |
 | 16.0% | We found no crawler page from Cohere | No documented name |
 | 14.4% | Webz.io's old crawler, replaced in 2024 | Webzio, or Webzio-Extended for AI use |
| 14.3% | Anthropic said in 2024 it was no longer in use | ClaudeBot |
 | 7.3% | Meta's crawler page no longer lists it | Meta-ExternalAgent |
| 7.2% | Webz.io's old crawler, replaced in 2024 | Webzio, or Webzio-Extended for AI use |
| FishBot | 7.1% | No operator page found; every file naming it carries Cloudflare's list | No documented name |
| CitibotSiteCrawler | 7.1% | No operator page found; every file naming it carries Cloudflare's list | No documented name |
 | 5.2% | No operator page found | No documented name |
| 3.5% | We found no crawler page from Cohere | No documented name |
| FriendlyCrawler | 3.4% | No operator page found | No documented name |
 | 3.0% | No operator page found | No documented name |
 | 2.5% | We found no crawler page from DeepSeek | No documented name |
 | 2.2% | Ai2's crawler page names only AI2Bot | AI2Bot |
 | 1.9% | Google's former name, supported until August 2026 | Google-GeminiNotebook |
| 1.8% | Not among Google's documented crawlers | No documented name |
 | 1.1% | We found no crawler page from xAI | No documented name |
 | 1.0% | Not one of Perplexity's crawler names | PerplexityBot |
Some lines name a real crawler in a form the standard doesn't allow. RFC 9309 says a crawler's name "MUST contain only" letters, underscores and hyphens, yet files write chatgpt-user/2.0 (25 files) and meta-externalagent/1.1 (10), and two spell Perplexity-User with a non-breaking hyphen, which looks identical and is a different character. Whether a crawler reads any of these as its own name depends on its parser. Write the bare name.
What Blocking Each Kind of Crawler Costs
Training crawlers and tokens
Blocking a training crawler asks its operator to keep your pages out of model training. OpenAI says each of its settings "is independent of the others", so a site can block GPTBot and still appear in ChatGPT search through OAI-SearchBot. Apple says Applebot-Extended "is only used to determine how to use the data crawled by the Applebot user agent". Answers in Siri and Search draw on Applebot, and Apple's opt-out for those answers is the nosnippet tag.
Google-Extended reaches further than the other training tokens. Google says it governs the training of Gemini models and also grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. It "does not impact a site's inclusion in Google Search". AI Overviews and AI Mode are part of Search, and Google says robots.txt rules for Googlebot are the control there.
Anthropic adds one more way in. Its crawler page says that since 30 October 2026 it has begun recognising a general machine-readable "no AI training" preference, based on IETF work, and that a robots.txt rule naming ClaudeBot takes priority over it.
Search crawlers
Blocking a search crawler takes you out of that engine's search answers. OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links". Moonshot AI says blocking Kimi-SearchBot "will prevent your site from appearing in Kimi search results". PerplexityBot, Claude-SearchBot, Meta-WebIndexer, MistralAI-Index, Amzn-SearchBot and DuckAssistBot do the same job for their engines.
Fetchers acting for a person
ChatGPT-User, Perplexity-User, Google-Agent, Google-GeminiNotebook, Meta-ExternalFetcher, Amzn-User and Kimi-User fetch a page because someone asked for it. Their operators say robots.txt may not apply: Google says such fetchers "generally ignore robots.txt rules", and Perplexity says the same of Perplexity-User. A robots.txt line for these is a request at most. To refuse them, block them at your server or CDN, using each operator's published address list where it has one. Claude-User and MistralAI-User are documented as following robots.txt.
Whatever you change, give it time. OpenAI, Perplexity, Meta and Amazon say a robots.txt change takes about a day to reach their systems, and DuckDuckGo says 72 hours.
Writing the rules
- Decide by job, not by company. List the training crawlers you want out and leave the search crawlers and fetchers you want in, using the names in the full list below.
- Write each name exactly as its operator documents it, with nothing after it. One group can hold several User-agent lines.
- If you name a crawler to let it in, copy your general rules into its group. A crawler that finds a group naming it ignores the * group.
- Replace old names with current ones: ClaudeBot for anthropic-ai and claude-web, Webzio and Webzio-Extended for omgili, Meta-ExternalAgent for FacebookBot.
# Keep training crawlers out of the whole site User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Applebot-Extended User-agent: Meta-ExternalAgent Disallow: / # Everyone else, AI search crawlers included User-agent: * Disallow: /admin/
Here the search crawlers aren't named, so they follow the * group like any other crawler and can fetch everything except /admin/. Google-Extended is left out on purpose: it also covers grounding in Gemini Apps, so add it only if you want out of that too. For what llms.txt and ai.txt add beside robots.txt, see our crawler control guide.
The Full List of AI Crawler User Agents
The table lists 59 user agents. They are the names operators document for AI products that we checked, with the search engines' crawlers Googlebot, Bingbot, Applebot and PetalBot for comparison. Google, Apple and Huawei say their indexes feed AI answers too. Each name links to its operator's page, which we read on 11 November 2026. About 5.9% of files block every crawler they don't name, which is why the rarely named crawlers sit near that figure.
Every AI crawler user agent, ranked by how often top sites block it
59 names, each from its operator's own page. Blocked: share of the 6,969 readable files that block it from the whole site. Named: share whose robots.txt names it.
| # | User agent | What it does | Blocked | Named |
|---|---|---|---|---|
| 1 |  GPTBotOpenAI | Crawls content that may be used to train OpenAI's models | 25.0% | 24.1% |
| 2 |  CCBotCommon Crawl | Builds Common Crawl's open web archive, free for anyone to use | 24.8% | 21.6% |
| 3 |  BytespiderByteDance | ByteDance documents it as the crawler for Toutiao Search | 24.2% | 20.1% |
| 4 | ClaudeBotAnthropic | Collects content that could contribute to training Anthropic's models | 23.2% | 21.7% |
| 5 | Google-ExtendedGoogle | A token, not a crawler: Gemini training and grounding in Gemini Apps | 21.2% | 19.8% |
| 6 |  AmazonbotAmazon | Improves Amazon's products; the data may train its AI models | 21.1% | 17.7% |
| 7 | Uses such as training foundation AI models | 20.7% | 17.4% | |
| 8 |  Applebot-ExtendedApple | A token, not a crawler: training Apple's foundation models | 19.9% | 16.4% |
| 9 | PetalBotHuawei | Petal search engine; its index also feeds Huawei's AI Search | 18.9% | 14.2% |
| 10 |  ICC-CrawlerNICT | Research in translation, information analysis and AI | 15.7% | 9.9% |
| 11 |  QualifiedBotQualified | Crawls customers' sites for its AI products | 13.9% | 8.5% |
| 12 |  atlassian-botAtlassian | Indexes sites its customers own, for Rovo search and chat | 13.6% | 7.7% |
| 12 |  MistralAI-TrainingMistral AI | Builds datasets for training Mistral's models | 13.6% | 7.6% |
| 14 | Google-CloudVertexBotGoogle | Crawls a site when its owner builds a Vertex AI agent | 13.5% | 10.1% |
| 15 |  KimiBotMoonshot AI | Crawls content that may train Kimi's models | 13.4% | 7.5% |
| 16 | PerplexityBotPerplexity | Perplexity's search results; not used for foundation models | 13.1% | 12.5% |
| 17 |  NavuBotNavu | Indexes customers' sites for Navu's AI website chat | 13.0% | 7.1% |
| 18 | ChatGPT-UserOpenAI | Fetches pages for ChatGPT users; robots.txt may not apply | 12.5% | 12.0% |
| 19 |  YouBotYou.com | Indexes pages for the You.com search engine | 11.7% | 7.1% |
| 20 | OAI-SearchBotOpenAI | Finds pages for ChatGPT search | 10.0% | 9.5% |
| 21 | AI2BotAi2 | Content used to train Ai2's open language models | 9.8% | 4.4% |
| 21 | Perplexity-UserPerplexity | Fetches pages for Perplexity users; generally ignores robots.txt | 9.8% | 6.3% |
| 23 | Fetches links at a user's request; may bypass robots.txt | 9.7% | 5.0% | |
| 24 | Webzio-ExtendedWebz.io | Tags Webz.io's data as usable or not for AI training | 9.5% | 3.8% |
| 24 | Claude-UserAnthropic | Fetches pages when Claude users ask | 9.5% | 6.2% |
| 26 | Claude-SearchBotAnthropic | Indexes pages for Claude's search results | 9.1% | 6.0% |
| 27 | img2datasetOpen-source tool | Builds image datasets; its opt-out is the X-Robots-Tag header | 9.0% | 3.4% |
| 28 |  DuckAssistBotDuckDuckGo | Fetches pages for DuckDuckGo's AI answers; not used for training | 8.8% | 4.6% |
| 29 | MistralAI-UserMistral AI | Visits pages to answer questions in Mistral's Vibe | 8.7% | 3.3% |
| 30 | Meta-WebIndexerMeta | Improves Meta AI search results | 8.0% | 3.0% |
| 31 | Amzn-UserAmazon | Live answers such as Alexa's; may not follow all robots.txt rules | 7.5% | 1.9% |
| 31 |  FirecrawlAgentFirecrawl | The robots.txt name of a scraping API for AI apps | 7.5% | 1.8% |
| 33 |  SBIntuitionsBotSB Intuitions | AI development and information analysis | 7.2% | 1.3% |
| 34 | Kimi-UserMoonshot AI | Fetches pages for Kimi users; robots.txt may not apply | 6.7% | 0.9% |
| 34 |  LinkupBotLinkup | A web search index that grounds AI applications | 6.7% | 0.8% |
| 34 |  AddSearchBotAddSearch | Site search and AI answers for AddSearch's customers | 6.7% | 0.8% |
| 34 | Google-AgentGoogle | Agents acting for users; generally ignores robots.txt | 6.7% | 1.2% |
| 34 |  AnomuraDireqt | Direqt's search features; not used to train models | 6.7% | 0.8% |
| 34 |  LinerBotLiner | Liner's search and reference index | 6.7% | 0.8% |
| 40 |  KlaviyoAIBotKlaviyo | Klaviyo's AI features, on sites its customers connect | 6.6% | 0.8% |
| 40 | NovaActAmazon | Browser agents built with Amazon Nova Act | 6.6% | 0.8% |
| 40 |  bedrockbotAmazon Web Services | Crawls the URLs a customer sets up for a Bedrock knowledge base | 6.6% | 0.7% |
| 40 |  ZanistaBotZanista.ai | Financial news, summarised with language models | 6.6% | 0.7% |
| 40 |  Channel3BotChannel3 | A product catalog for AI shopping agents | 6.6% | 0.7% |
| 40 | Amzn-SearchBotAmazon | Search in Amazon products such as Alexa; not for AI training | 6.6% | 1.7% |
| 46 | MistralAI-IndexMistral AI | Indexes pages for Mistral search in Vibe | 6.5% | 0.6% |
| 46 | Google-GeminiNotebookGoogle | Fetches sources users add in Gemini Notebook; generally ignores robots.txt | 6.5% | 0.7% |
| 48 | Kimi-SearchBotMoonshot AI | Builds Kimi's search index | 6.3% | 0.4% |
| 49 | Claude-AgentAnthropic | A browser Anthropic hosts to do tasks for Claude users | 6.2% | <0.1% |
| 50 | OAI-AdsBotOpenAI | Checks pages submitted as ChatGPT ads; not used for training | 6.0% | 0.6% |
| 50 |  ExaUserExa | Pages Exa fetches on a user's request | 6.0% | 0.1% |
| 50 | SBIntuitions-SearchBotSB Intuitions | Browses for Sarashina users; not used in AI development | 6.0% | 0.1% |
| 53 | Liner-UserLiner | Fetches pages in real time for Liner users | 5.9% | 0.0% |
| 54 | ApplebotApple | Spotlight, Siri and Safari; reads Googlebot's rules if not named | 5.7% | 4.4% |
| 54 | ExaSearchBotExa | Exa's search engine; reads search engines' rules if not named | 5.7% | 0.7% |
| 54 |  Cloudflare-AI-SearchCloudflare | Crawls only domains in the customer's own Cloudflare account | 5.7% | 0.3% |
| 57 |  BingbotMicrosoft | Bing's standard crawler; Bing lists no separate one for Copilot | 5.5% | 8.2% |
| 58 | GooglebotGoogle | Google Search, including AI Overviews and AI Mode | 4.1% | 9.1% |
| 59 |  YandexAdditionalYandex | Keeps pages out of Yandex AI answers; reads only groups that name it | 1.3% | 1.3% |
A few rows need a note. The fetchers above may ignore robots.txt whatever it says. img2dataset is a tool anyone can run, and its documented opt-out is an X-Robots-Tag header, not robots.txt. Anthropic describes allowing or refusing Claude-Agent's traffic and says nothing about robots.txt for it. And several only fetch the sites their customers point them at, often the customer's own: Google-CloudVertexBot, atlassian-bot, bedrockbot, KlaviyoAIBot and Cloudflare-AI-Search.
Block lists also carry names whose makers describe jobs other than AI. Six of the most common are below. Blocking them is your call, but it isn't an AI decision.
Often on AI block lists, but their makers describe other jobs
Share of the 6,969 readable files that name each one, and what its maker says it does.
| User agent | Named in | What its maker says |
|---|---|---|
 | 15.0% | General crawling for its knowledge graph and web search; "not used for AI training" |
 | 12.2% | Baidu's search crawler |
| 9.5% | Google's generic crawler, for example for internal research and development |
 | 8.8% | Collects Japanese-language data resources |
 | 7.6% | An uptime monitor that checks URLs its users add |
 | 5.9% | Collects public images for Hive's web intelligence products |
How We Measured
We took the top 10,000 origins from the Chrome UX Report top-sites list for September 2026, which ranks origins by page loads in Chrome. On 10 November 2026 RanqoBot asked each one for /robots.txt once, following up to five redirects, with no retries, from one network over about 17 minutes. We kept measurements about each file, never its text, and we name no site.
What the top 10,000 origins sent back
One request for /robots.txt per origin, 10 November 2026. Only the first row is in the shares above.
- A robots.txt file we could read
- 6,969
- No file served (403: 925, 404: 740, other 4xx: 62)
- 1,727
- Connection failed (timeouts: 349, TLS: 348, DNS: 109, other: 32)
- 838
- A web page served in place of the file
- 355
- Rate limited or a server error
- 83
- More than five redirects
- 28
Each file was read with Ranqo's robots.txt parser, which follows RFC 9309: a crawler follows the group that names it, else the * group, else no rule applies. Our parser also lets a crawler follow a group whose user-agent line is the start of its name; following exact names only moves no share in the main chart by more than half a point. Where an operator documents its own fallback, we used it. Applebot, Google-CloudVertexBot and ImagesiftBot read Googlebot's rules. Amazon and Exa say their search crawlers follow other search engines' rules, which we read as Googlebot's, then Bingbot's. And YandexAdditional reads only groups that name it, since Yandex says such robots may ignore even its own User-agent: Yandex groups. Blocked means both the home page and a deeper path are disallowed. A file "carries Cloudflare's list" when its Content-Signal line holds use=reference.
The shares count the files we could read. GPTBot's 25.0% is 20.1% if origins with no file count as having no rules, as the standard says a crawler should treat them, and 17.4% of all 10,000 origins. Counting each registrable domain once, by its best-ranked origin (5,960 of them), moves no share in the main chart by more than 1.4 points. By rank, GPTBot is blocked in 26.1% of the top 1,000, 25.1% of the next 4,000 and 24.7% of the last 5,000, too close to call a trend.
robots.txt can't show blocks made at the server. 925 origins answered our request with a 403, which often means a bot filter turned it away, and those sites may refuse AI crawlers the same way. A 2025 study found that 14% of the top 10,000 sites in October 2024 actively blocked two of Anthropic's crawlers, and that only 2% of those also restricted them in robots.txt.
Earlier counts line up where the groups match. The Web Almanac found GPTBot named in 20.9% of the top 1,000 sites in its July 2025 crawl, which also uses the Chrome UX Report; we find it named in 21.1% of the top 1,000 origins. For the top 10,000 the Almanac had 12.1% and we have 16.8%, or 11.8% without the files carrying Cloudflare's list, which Cloudflare launched the month the Almanac crawled. Cloudflare's own count from June 2025 found GPTBot fully disallowed by 250 of 3,816 top domains with a robots.txt, a much lower rate, but its top 10,000 is ranked by DNS lookups, not visits, and only about 37% of it had a robots.txt file. Among news publishers blocking runs far higher: the Reuters Institute found 48% of leading news sites in ten countries blocking OpenAI's crawlers by the end of 2023.
How Ranqo Does It
A count of top sites tells you what is common. Your own file and your own server decide what happens to you. In Ranqo, Site Access fetches your site as each AI crawler it tracks beside a regular browser, reads your robots.txt the way each one does, and keeps the crawlers that fetch pages for answers apart from training crawlers, so a training block is never reported as a lost citation. It flags a robots.txt that names Anthropic's crawler only by its old names, and a disallow of Google-Extended, since Google says that one also reaches Gemini's answers.
Without an account, the free AI Crawler Inspector reads any page's robots.txt for each crawler Ranqo tracks, fetches the page as each one the file lets in and shows what each one got. The robots.txt generator writes the groups for the crawlers you choose.
See how AI crawlers read your site
Ranqo reads your robots.txt and fetches your site as each AI crawler it tracks, then measures how often ChatGPT, Perplexity and Google's AI answers name you. To check your setup by hand first, work through our ten-point check.
Start a 7-day trialWritten by
Nisha Kumari
Nisha Kumari is Co-Founder at Ranqo, where she leads growth strategy and client acquisition. With a background in digital marketing and financial management, she specializes in SEO, Generative Engine Optimization, and helping brands build visibility across AI platforms.
Share this article
Related articles
Is Your Site Blocking AI Crawlers? A Ten-Point Check After Cloudflare's September Change
Most sites are not blocking AI crawlers, and most that are did not decide to. Refusing the crawler that trains a model costs nothing in AI answers; refusing the one that fetches pages at answer time is what removes you. What Cloudflare changed in September, which bot is which, and a ten-point check to run on your own site.
AI.txt vs Llms.txt vs Robots.txt: The Complete AI Crawler Control Guide for 2026
Most articles about AI crawlers ask the wrong question -- whether to block them. The strategic question is which crawlers should train your model, which should retrieve from you for citation, and where licensing replaces both. This guide covers the three control files (robots.txt, llms.txt, AI.txt), the AI crawler taxonomy, the crawl-to-referral economics that should drive your decisions (ClaudeBot 20,583:1 vs PerplexityBot 194.8:1), the Perplexity stealth-crawling case study, and industry-specific decision frameworks. Every claim is verified against published sources.
The Page Checks Most Sites Fail for AI Search, Ranked
We ran 20 common AI search checks on 3,691 pages. More than 80% of pages on company sites had no FAQ markup, and neither did 72.5% of pages AI already cites, so it can't be a requirement. Failures that can stop AI reading a page are rare, under 4%, and they're the ones to fix first. Here's the full ranking, with fixes.