AI.txt vs Llms.txt vs Robots.txt: The Complete AI Crawler Control Guide for 2026

Three files compete for the role of “tell AI what to do with my content.” They are not interchangeable. Each has a different purpose, a different adoption profile, and a wildly different track record on whether AI platforms actually read it.

Before we touch any control file, you need to know what you're trying to control. AI crawlers split into two functional categories, with one important dual-purpose outlier:

• Training crawlers fetch web content to train future AI models. They give you nothing back -- your content shapes a model that competes with you. GPTBot, ClaudeBot, CCBot, Bytespider, and Google-Extended fall here.
• Retrieval crawlers fetch web content so an AI platform can cite you in real-time answers. They potentially drive traffic and brand visibility. ChatGPT-User, Claude-User, Claude-SearchBot, OAI-SearchBot are pure retrieval bots.
• Dual-purpose bots do both. PerplexityBot is the canonical example -- it indexes for the answer engine but the same crawl feeds training as well. Google-Extended is technically training- only but its data path overlaps with the broader Google ecosystem.

For a complete glossary of crawler-related terms, see our 100-term AI citation dictionary.

The most important data point in this entire post is the ratio of pages each AI bot crawls per user it actually sends back to your site. Cloudflare Radar tracks these ratios across hundreds of thousands of sites and the spread between bots is enormous.

robots.txt is the oldest of the three control files (1994, formalized as RFC 9309 in 2022) and the only one with anything resembling universal adoption. Every major AI crawler -- GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider -- claims to read and respect it.

The syntax is simple. To block GPTBot from training on your entire site:

That configuration says: don't use my content to train future models, but do cite me when a user asks a real-time question. It's a coherent strategic position, not a contradiction.

The catch: robots.txt is voluntary. There is no protocol-level enforcement. A bot reads it because the vendor chooses to honor it. When a vendor decides not to -- or builds a parallel non-declared crawler -- there is no automatic recourse. That's section 8's case study. For now, treat robots.txt as the necessary baseline that works against cooperative bots and means nothing against uncooperative ones.

llms.txt was proposed by Jeremy Howard at Answer.AI in September 2024. The pitch: instead of blocking, give LLMs a curated map of your most important content. Help them find what you'd want them to summarize.

The syntax is markdown. A typical /llms.txt looks like:

That's the optimistic story. The honest story is harder. Adoption is climbing -- our analysis in the llms.txt complete guide put adoption at roughly 10% of surveyed domains as of early 2026, up from near-zero a year earlier. But the same analysis found no measurable correlation between llms.txt presence and AI-citation frequency. None of the major AI platforms have publicly committed to reading the file when generating answers. Google said explicitly that it relies on standard SEO signals, not on llms.txt.

What to do with that reality:

• Ship one if it's cheap. Generating an llms.txt from existing site structure is a 30-minute task. The cost is low and the option value if AI platforms ever read the file is real.
• Don't treat it as a visibility lever. No 2026 evidence supports the claim that llms.txt drives AI citations. Skeptical vendors who pitch “llms.txt SEO” are selling hype, not measurement.
• Watch the consumption side. If a major platform starts reading llms.txt, the calculus flips. Until then, treat it as forward-investment, not current-strategy.

AI.txt is the Spawning-led proposal that lets creators express preferences about whether their work can be used for AI training. It's separate from robots.txt (which controls crawl access) and llms.txt (which curates content for AI consumption). Where robots.txt says “you may or may not crawl,” AI.txt says “you may or may not train.”

The Spawning project also operates a “haveibeentrained.com” database that artists and creators can opt out of. Major image generators (Stability AI, Adobe Firefly) have integrated some of these preferences into their training pipelines. Major LLM vendors (OpenAI, Anthropic, Google) have not formally committed to AI.txt compliance.

Other emerging standards in the same space include C2PA (Coalition for Content Provenance and Authenticity, focused on cryptographic content signing), the IETF's evolving opt-out drafts, and various creator-focused registries. None has reached the universal-recognition status of robots.txt; most are creative-industry specific.

On August 4, 2025, Cloudflare published a detailed investigation showing that Perplexity was running undeclared crawlers alongside its declared PerplexityBot user agent. The stealth crawlers used generic Chrome / macOS user-agent strings (indistinguishable from real users at first glance), rotated across multiple IP addresses and ASNs, and ignored robots.txt directives that explicitly disallowed Perplexity.

The numbers Cloudflare reported: 3-6 million daily stealth requests vs 20-25 million from declared bots -- meaning roughly 13-30% of Perplexity's effective crawl was happening outside the standard control mechanism. Cloudflare responded by de-listing PerplexityBot from its verified bot directory and rolling out network-level blocks for customers that opted in.

What the case taught the industry:

• Cloudflare-class WAF enforcement matters. A robots.txt rule plus network-level bot detection plus rate limiting is the minimum 2026 stack for sites that genuinely want compliance.
• Vendors caught violating face commercial consequences. Loss of Cloudflare verification, public credibility damage, and potential regulatory exposure are now real costs of stealth crawling.
• The detection bar is rising. Tools that previously trusted user-agent strings now fingerprint TLS handshakes, behavioral patterns, and IP-vs-claimed-vendor mismatches. Stealth crawling gets harder year over year.

For Perplexity-specific optimization beyond crawler controls, see how to get cited by Perplexity.

Here is the framing nobody else elevates as the central decision: a single AI vendor (OpenAI, Anthropic) operates multiple distinct crawlers, and the right move is to evaluate each one against what you get back, not against the vendor brand collectively.

OpenAI's three-bot architecture, for example:

• GPTBot -- training only. Trains future GPT models on your content. You get nothing back.
• ChatGPT-User -- retrieval only. Fetches your page when a ChatGPT user actively asks for live content. You get visibility, citations, and traffic.
• OAI-SearchBot -- retrieval index only. Builds the search index ChatGPT Search uses to cite sources. You get visibility in ChatGPT Search results.

Treating those as one vendor decision is a category error. Blocking GPTBot while allowing ChatGPT-User and OAI-SearchBot is a strictly dominant strategy for most brands: you keep your content out of training data and you stay visible in real-time AI answers. Anthropic has the same three-bot structure (ClaudeBot training; Claude-User and Claude-SearchBot retrieval) and the same logic applies. The same is true of Perplexity -- though there the stealth-crawling history complicates trust.

For a small set of sites -- premium publishers, large content platforms, anyone whose content is structurally valuable as training data -- the right answer to “block or allow?” is “neither: license it.”

The deals shipped to date:

• Reddit-OpenAI (May 16, 2024) -- financial terms not publicly disclosed. First major AI training-data licensing deal. Established the model that every subsequent publisher deal followed.
• Financial Times-OpenAI (April 2024) -- exact terms not publicly disclosed; reported by industry press as a multi-year deal that integrates FT content into ChatGPT and licenses training data. Premium positioning for paywalled business news.
• News Corp-OpenAI (May 2024) -- reported at $250M+ over 5 years. Covers WSJ, Barron's, NY Post, the Telegraph. The largest publicly reported AI training-data deal.
• AP, Axel Springer, Vox, Le Monde -- multiple lower-disclosure deals following the same pattern.

The legal landscape that drives these deals: the New York Times v. OpenAI lawsuit is still active as of May 2026, with copyright infringement claims allowed forward and OpenAI compelled to produce ~20M ChatGPT user logs in discovery. The case has not been settled, which means the legal precedent for unlicensed training data is still being established. Publishers that sign licensing deals are buying certainty in a regulatory vacuum -- and AI vendors that sign them are buying insulation against future judgments.

The optimal AI-crawler control strategy varies sharply by industry. What works for a B2B SaaS company is wrong for a news publisher; what works for an e-commerce site is wrong for a research firm. Three industry patterns:

Three reference configurations you can adapt. None are universally right -- pick the one that matches your industry framework above and modify from there.

We see these failure modes repeatedly across customer audits:

1. Blocking GPTBot but forgetting OAI-SearchBot. You think you've removed yourself from ChatGPT, but ChatGPT Search continues to cite you because it uses a different crawler. The unintended outcome: you block training (good if intended) but stay visible in retrieval (probably good but should be a deliberate choice).
2. Blocking Google-Extended and worrying about Google Search rankings. Google-Extended is for AI training only. Blocking it does not affect Googlebot, regular Google Search, or AI Overview retrieval (which uses Googlebot infrastructure). The risk is over-blocking out of caution.
3. Treating llms.txt as a substitute for robots.txt. They solve different problems. llms.txt is a curation map for AI consumers (currently aspirational); robots.txt is access control (currently real). You may need both, but skipping robots.txt is the costlier omission.
4. Relying on robots.txt alone for enforcement. Per the Perplexity stealth-crawling case study, an honor-system file is insufficient against vendors who choose not to honor it. Pair robots.txt with WAF/CDN-level bot detection if you genuinely need to enforce blocking.
5. Setting allow/disallow for non-existent user agents. Bots that don't exist (typos, deprecated names, vendor-specific strings like “ChatGPT” that aren't real bots) clutter the file without doing anything. Use the verified taxonomy from section 3.

Audit your current setup against these 12 questions. Each should have a defensible answer; if any are “don't know” you have a gap to close.

AI crawler control in 2026 is a business decision dressed up as a technical one. The technical part -- the syntax of robots.txt, the structure of llms.txt, the choice of which bots to allow -- is the easy part. The hard part is being honest about what each crawler is actually costing you and what it's actually returning.

The framework that works:

• Treat training crawlers and retrieval crawlers as different decisions, not one vendor-level decision. Block training where the economics don't work; allow retrieval where the citations matter.
• Use the crawl-to-referral ratio as the primary numerical input. ClaudeBot at 130,330:1 is a different argument than PerplexityBot at 194.8:1.
• Don't rely on robots.txt alone for enforcement. Pair it with WAF-level detection if you genuinely need compliance.
• Treat llms.txt as forward-investment, not visibility lever. Ship one if cheap; don't expect citation lift in 2026.
• License where it's available and the math works. The Reddit, FT, and News Corp deals are the model -- if your content is valuable enough to interest AI vendors, extract value from that interest rather than just blocking it.
• Review quarterly. The crawler ecosystem moves faster than annual SEO reviews. Your robots.txt should evolve with it.