Skip to main content
Ranqo
Features
Search VisibilityMonitor how AI sees your brand — visibility, position, and sentimentPrompt IntelligenceThe exact questions buyers ask AI — and whether you're in the answerCompetitor BenchmarkingSee who AI recommends instead of you — share of voice, position, and gapsSource AnalyticsThe domains AI trusts when it answers — and how to become one
Visitor AnalyticsSee every AI bot and human visitor — the traffic GA4 missesAction CenterKnow what to do next — every action ranked by impact, tracked to donePage OptimizationSix-dimension audits that show why AI cites a page — or skips itContent LabFrom visibility gap to published post — content AI wants to cite
One platform. Every AI conversation, captured.Compare all
Solutions
BrandsSee what AI answers when buyers askAgenciesRun AI visibility across a client portfolioSEO & GEO ProfessionalsAdd GEO to the service you already sell
Pricing
Resources
BlogResearch & playbooks for AI searchResearchPublished papers on AI searchAI Visibility ReportYour AEO/GEO baseline, emailedFree ToolsAudit or generate, no signupCompare GEO/AEO platformsRanqo vs alternatives, head-to-head
Features
Search VisibilityMonitor how AI sees your brand — visibility, position, and sentimentPrompt IntelligenceThe exact questions buyers ask AI — and whether you're in the answerCompetitor BenchmarkingSee who AI recommends instead of you — share of voice, position, and gapsSource AnalyticsThe domains AI trusts when it answers — and how to become oneVisitor AnalyticsSee every AI bot and human visitor — the traffic GA4 missesAction CenterKnow what to do next — every action ranked by impact, tracked to donePage OptimizationSix-dimension audits that show why AI cites a page — or skips itContent LabFrom visibility gap to published post — content AI wants to cite
Solutions
BrandsSee what AI answers when buyers askAgenciesRun AI visibility across a client portfolioSEO & GEO ProfessionalsAdd GEO to the service you already sell
PricingResources
BlogResearch & playbooks for AI searchResearchPublished papers on AI searchAI Visibility ReportYour AEO/GEO baseline, emailedFree ToolsAudit or generate, no signupCompare GEO/AEO platformsRanqo vs alternatives, head-to-head
Book a demoLog inGet Started
Free Tool — No Signup Required

Robots.txt for the AI era.
Allow or block every AI crawler.

Pick which AI bots can read your site. Generate a clean robots.txt with rules for every major crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more.

Default behavior

AI Training Crawlers

Used to build future AI models. Block to opt out of training.

GPTBot
OpenAI's training crawler — used to build future ChatGPT models.
ClaudeBot
Anthropic's general-purpose web crawler used to improve Claude.
anthropic-ai
Anthropic's legacy training token. Keep alongside ClaudeBot for full coverage.
Google-Extended
Token controlling whether Bard / Gemini can use your content. Doesn't affect Google Search.
Applebot-Extended
Token controlling whether Apple Intelligence can train on your content.
Bytespider
ByteDance's aggressive AI training crawler. Common block target.
CCBot
Common Crawl powers training datasets for many AI models including older GPTs.
Meta-ExternalAgent
Meta's AI training crawler for Llama and Meta AI products.
xAI
xAI's web crawler used for training Grok.

AI Assistant Fetchers

Fetch your URL on demand when a user asks AI about your page.

ChatGPT-User
Fetches pages on demand when a ChatGPT user asks about your URL.
Claude-User
Fetches pages on demand when a Claude user asks about your URL.
Perplexity-User
Fetches pages on demand for Perplexity user queries.
DuckAssistBot
Fetches pages for DuckDuckGo's AI-assisted answers.

AI Search Indexes

Index your content for AI-powered answer engines.

OAI-SearchBot
Indexes pages for ChatGPT Search results.
PerplexityBot
Indexes pages for Perplexity's answer engine.

Search Engine Crawlers

Standard search crawlers. Blocking removes you from search.

Googlebot
Google's main search crawler. Blocking this removes you from Google Search.
Applebot
Apple's main crawler for Siri and Spotlight Suggestions.
Bingbot
Bing's search crawler. Powers Bing Search and Microsoft Copilot grounding.
Your robots.txt is ready

Copy or download your file

Drop this at the root of your site as /robots.txt.

Bots crawl by default. No bots are explicitly blocked.
/robots.txt
# Generated with Ranqo (https://ranqo.ai/free-tools/robots-txt-generator)

# Default rule for all bots
User-agent: *
Allow: /

Need to make changes? Tweak the form above — your file updates live.

Where to place this file

The file must be served at https://yoursite.com/robots.txt — bots only check the root.

  • Static sites · S3 · Vercel · Netlify · Cloudflare Pages

    Upload robots.txt to your project root. Most static hosts serve files from the root automatically.

  • Next.js · Nuxt · Astro · Remix

    Save the file as public/robots.txt. The framework will serve it at /robots.txt on the next deploy.

  • WordPress

    Use Yoast SEO or Rank Math's built-in robots.txt editor (Tools → File Editor), or upload via FTP to your site root. Don't place it inside /wp-content/.

  • Webflow · Squarespace · Wix · Shopify

    Find the SEO settings panel in your site builder — most platforms have a dedicated robots.txt editor. Paste the contents above and save.

Verify after deploy: visit https://yoursite.com/robots.txt in your browser. You should see this exact content. If you see a 404 or your site's HTML, the file isn't at the root yet.

Want to know if your blocks are working?

robots.txt is the start. Sign up free to track real AI citations and confirm which bots are actually reaching your pages — every day.

Why Robots.txt Generator matters

AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended pull data from your site by default, for training, for answer grounding, or both. Without explicit rules you are handing your content over without deciding to. This generator lets you opt each of eighteen bots in or out individually, with control over which paths they can reach.

How it works

  1. 01

    Decide bot by bot

    Eighteen crawlers, grouped by what they actually do: training collectors, assistant fetchers, AI search indexers, and classic search engines. Nothing about your site is fetched or scanned, so there is no waiting and nothing leaves your browser.

  2. 02

    Add path rules if you need them

    Block a bot from your pricing pages while leaving your blog open, or the reverse. Most sites want a blanket answer; some want a scalpel.

  3. 03

    Copy the file to your root

    The output is a standard robots.txt with one user-agent block per bot. Serve it at /robots.txt and the rules take effect on the crawlers' next visit.

What it checks

Training crawlers
GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Bytespider, CCBot, Meta-ExternalAgent, and xAI. These collect content for model training.
Assistant fetchers
ChatGPT-User, Claude-User, Perplexity-User, and DuckAssistBot. These fetch a page because a person asked about it right now, which is the traffic you almost certainly want to allow.
AI search indexers
OAI-SearchBot and PerplexityBot, which build the indexes those products answer from. Block these and you opt out of being cited.
Classic search engines
Googlebot, Bingbot, and Applebot are listed so you can see them beside the AI bots and avoid the mistake of blocking your search rankings while aiming at a training collector.
[ FAQ ]

Robots.txt Generator,
answered.

Not necessarily, and the distinction is the whole point of the categories. Blocking GPTBot or Google-Extended opts you out of training. Blocking OAI-SearchBot or PerplexityBot opts you out of being retrieved and cited in live answers, which is the thing you are presumably trying to win. Blocking Google-Extended does not affect your Google Search ranking; blocking Googlebot does.
The major documented ones from OpenAI, Anthropic, Perplexity, Google, and Apple publish their tokens and state that they honour it. Compliance is voluntary and unverifiable from your side. robots.txt is a stated preference, not an access control, so anything that must not be public needs authentication rather than a directive.
Only if you replace the file wholesale. The generated output covers the crawlers listed here. If you already have rules for Googlebot, sitemap declarations, or your own disallows, merge the new user-agent blocks into your existing file rather than pasting over it.
For most brands trying to be found: allow the assistant fetchers and AI search indexers, and make training a deliberate decision. Publishers with licensing revenue usually block training collectors. Software and services companies usually allow everything, because being in the answer is worth more than the content is.
Run your URL through the Crawler Inspector. It fetches your page as each bot and reads your live robots.txt, so you see which bots are allowed and which are blocked rather than trusting that the file parsed the way you intended.
Ranqo

Be the Source AI Cites.

Track, analyze, and improve your brand's presence across AI search engines.

Tracked across
ChatGPTChatGPT
GeminiGemini
PerplexityPerplexity
Google AI OverviewsAI Overviews
ClaudeClaude
GrokGrok
G25.0Read our reviewsRanqo approved on SaasHub

Product

  • Search Visibility
  • Prompt Intelligence
  • Competitor Benchmarking
  • Source Analytics
  • Visitor Analytics
  • Action Center
  • Page Optimization
  • Content Lab

Solutions

  • All Solutions
  • Brands
  • Agencies
  • SEO & GEO Professionals

Resources

  • Blog
  • Research
  • AI Visibility Report
  • Compare

Company

  • About
  • Pricing
  • Book a demo
  • Contact

Legal

  • Privacy
  • Terms
  • Cookies
  • RanqoBot

Free Tools

  • All Free Tools
  • AI Visibility Checker
  • AI Readiness Score
  • LLMs.txt Generator
  • Robots.txt Generator
  • Page Token Inspector
  • AI Content Grader
  • AI Search Crawler Inspector
  • AI Query Fan-Out Generator

© 2026 Ranqo. All rights reserved.