Robots.txt for the AI era.
Allow or block every AI crawler.
Pick which AI bots can read your site. Generate a clean robots.txt with rules for every major crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more.
Default behavior
AI Training Crawlers
Used to build future AI models. Block to opt out of training.
AI Assistant Fetchers
Fetch your URL on demand when a user asks AI about your page.
AI Search Indexes
Index your content for AI-powered answer engines.
Search Engine Crawlers
Standard search crawlers. Blocking removes you from search.
Copy or download your file
Drop this at the root of your site as /robots.txt.
# Generated with Ranqo (https://ranqo.ai/free-tools/robots-txt-generator) # Default rule for all bots User-agent: * Allow: /
Need to make changes? Tweak the form above — your file updates live.
Where to place this file
The file must be served at https://yoursite.com/robots.txt — bots only check the root.
- Static sites · S3 · Vercel · Netlify · Cloudflare Pages
Upload robots.txt to your project root. Most static hosts serve files from the root automatically.
- Next.js · Nuxt · Astro · Remix
Save the file as public/robots.txt. The framework will serve it at /robots.txt on the next deploy.
- WordPress
Use Yoast SEO or Rank Math's built-in robots.txt editor (Tools → File Editor), or upload via FTP to your site root. Don't place it inside /wp-content/.
- Webflow · Squarespace · Wix · Shopify
Find the SEO settings panel in your site builder — most platforms have a dedicated robots.txt editor. Paste the contents above and save.
Want to know if your blocks are working?
robots.txt is the start. Sign up free to track real AI citations and confirm which bots are actually reaching your pages — every day.
Why Robots.txt Generator matters
AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended pull data from your site by default — for training, for answer grounding, or both. Without explicit rules, you're handing your content over without consent. This generator lets you opt every major AI bot in or out individually, with fine-grained control over which paths they can access and which they can't.