Free Tool — No Signup Required

AI Search Crawler Inspector

Fetch any URL as ChatGPT, Claude, Perplexity, and Google. See exactly which AI bots can read your page — and which are blocked.

Why AI Search Crawler Inspector matters

If GPTBot, ClaudeBot, or PerplexityBot cannot reach your page, no amount of SEO will get you cited. Cloudflare challenges, a stray robots.txt directive, a JavaScript-only render, and a 403 served only to bots all kill AI visibility silently, and none of them show up in a browser. This tool fetches your URL as every bot on the list plus an ordinary browser, side by side, and shows you who actually got the page.

How it works

01
We read your robots.txt
Your live file is fetched and parsed the way a crawler parses it, matching each bot's token against your user-agent blocks with wildcard fallback. Rules people believe they wrote and rules that actually apply are frequently different.
02
We fetch your page as each bot, and as a browser
Every crawler your robots.txt admits gets a real request carrying its published user-agent string, one at a time and spaced out so we do not trip your own rate limits. One it already turns away is reported from the file rather than contacted. A Chrome request goes first as the baseline. Anything that does not answer 200 is asked again a few seconds later, because a first refusal is sometimes our request rate rather than your policy.
03
We compare each bot against the browser
Status code, response size, how much readable text came back against how much the browser got, and the robots.txt verdict. The browser is the part that matters: it is the only thing that separates a rule aimed at bots from a page that answers nobody.

What it checks

HTTP status per bot: A 200 for your browser and a 403 for GPTBot is a common and completely invisible failure. Here the two sit in the same table.
Was there content in the response: A 200 is not a pass. We count the readable words a bot got and compare them with what a browser got, which catches a page served thinner to crawlers than to visitors without failing a short page that is genuinely short. We also request your page the way a crawler does, without running JavaScript, and say so when the HTML that comes back is too thin to read — that one is about the page rather than about any single bot.
Whether a browser gets in at all: If Chrome is refused too, nothing below it can be read as a rule about AI. We say so rather than reporting your site as invisible to ChatGPT on evidence that does not support it.
Robots.txt verdict: Whether each bot is allowed at that specific path, evaluated against your live file rather than assumed from the file's intent.
Training-only tokens: Google-Extended and Applebot-Extended never fetch anything themselves; they are robots.txt tokens that govern training use. We report their directive and say so, rather than inventing a fetch result.

FAQ

AI Search Crawler Inspector,
answered.

18. 16 get a live fetch: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Claude-User, Googlebot, PerplexityBot, Perplexity-User, Applebot, Bytespider, CCBot, Bingbot, DuckAssistBot, Meta-ExternalAgent and xAI. 2 are reported from robots.txt alone, because Google-Extended and Applebot-Extended are training directives rather than crawlers. An ordinary browser request runs alongside them as the baseline.

Two usual causes. Either the page renders its content in JavaScript, so the HTML arrives as an empty shell and the crawler never runs the script, or a CDN served a bot-challenge interstitial with a 200 status. Both look fine to a status check and both mean your content did not arrive. We flag the first as a note about the page, because our own baseline request does not run JavaScript either, and we refuse a challenge page's title rather than reporting it as yours.

It depends which bot. A refusal to an assistant fetcher or an AI search indexer means you are not citable through that product, which is worth fixing. A refusal to a training collector may be exactly what you configured on purpose. Read the category before you read the status. Note that a refusal does not always arrive as a 403: shared Apache hosting commonly turns a bot away with a 406, which looks like a content-negotiation error and is really a filter reading your user-agent.

It caps your ceiling, so it is the first thing to rule out. But an unblocked site with nothing quotable on it will also score low. Run this to eliminate the mechanical explanation, then run the Content Grader on the pages you want cited.

Re-run this immediately to confirm the file parses the way you meant. Whether the crawlers act on it is up to their own schedules, which are not published and vary by product, so treat re-crawl timing as unknown rather than assuming a day or a week.

Free Tool — No Signup Required

AI Search Crawler Inspector

Fetch any URL as ChatGPT, Claude, Perplexity, and Google. See exactly which AI bots can read your page — and which are blocked.

Why AI Search Crawler Inspector matters

How it works

01
We read your robots.txt
Your live file is fetched and parsed the way a crawler parses it, matching each bot's token against your user-agent blocks with wildcard fallback. Rules people believe they wrote and rules that actually apply are frequently different.
02
We fetch your page as each bot, and as a browser
Every crawler your robots.txt admits gets a real request carrying its published user-agent string, one at a time and spaced out so we do not trip your own rate limits. One it already turns away is reported from the file rather than contacted. A Chrome request goes first as the baseline. Anything that does not answer 200 is asked again a few seconds later, because a first refusal is sometimes our request rate rather than your policy.
03
We compare each bot against the browser
Status code, response size, how much readable text came back against how much the browser got, and the robots.txt verdict. The browser is the part that matters: it is the only thing that separates a rule aimed at bots from a page that answers nobody.

What it checks

HTTP status per bot: A 200 for your browser and a 403 for GPTBot is a common and completely invisible failure. Here the two sit in the same table.
Was there content in the response: A 200 is not a pass. We count the readable words a bot got and compare them with what a browser got, which catches a page served thinner to crawlers than to visitors without failing a short page that is genuinely short. We also request your page the way a crawler does, without running JavaScript, and say so when the HTML that comes back is too thin to read — that one is about the page rather than about any single bot.
Whether a browser gets in at all: If Chrome is refused too, nothing below it can be read as a rule about AI. We say so rather than reporting your site as invisible to ChatGPT on evidence that does not support it.
Robots.txt verdict: Whether each bot is allowed at that specific path, evaluated against your live file rather than assumed from the file's intent.
Training-only tokens: Google-Extended and Applebot-Extended never fetch anything themselves; they are robots.txt tokens that govern training use. We report their directive and say so, rather than inventing a fetch result.

FAQ

AI Search Crawler Inspector

Why AI Search Crawler Inspector matters

How it works

We read your robots.txt

We fetch your page as each bot, and as a browser

We compare each bot against the browser

What it checks

AI Search Crawler Inspector,answered.

AI Search Crawler Inspector

Why AI Search Crawler Inspector matters

How it works

We read your robots.txt

We fetch your page as each bot, and as a browser

We compare each bot against the browser

What it checks

AI Search Crawler Inspector,answered.

AI Search Crawler Inspector,
answered.

AI Search Crawler Inspector,
answered.