Crawlers

Crawlers come up in Ranqo in two ways. Ranqo has its own crawler, RanqoBot, which fetches pages for some features. And two features work with the AI crawlers that visit your site: Site Access tests whether they can fetch it, and Site Tracking counts their real visits. Each of those two features keeps its own list, and this page says which is which.

RanqoBot#

RanqoBot is the crawler Ranqo uses whenever it follows links or fetches pages in volume. Its User-Agent contains the token RanqoBot and a link to ranqo.ai/bot, the page for site owners that describes exactly what it does. If you found RanqoBot in your access logs, that page is the full answer.

Nearly all of its requests trace to someone using Ranqo:

• Page discovery. When a Ranqo customer audits a site, RanqoBot follows internal links on that one site to find pages worth auditing.
• Outbound link checks. When an audited page links to another site, RanqoBot checks the link still works, for up to 50 links per audit.
• Citation sources. When an AI engine cites a page without giving its title, RanqoBot reads the title so the publisher can be named.
• Site Access. When Site Access checks a Ranqo customer's site, RanqoBot reads its robots.txt, sitemap and llms.txt.
• Research samples. The exception: when Ranqo measures public pages for a study it publishes, RanqoBot fetches a few pages per site, and we keep measurements about them rather than their text.
• robots.txt. Before any of the above, and before Site Access or the AI Crawler Inspector fetches a page, it reads the site's robots.txt.

Allow or block RanqoBot#

RanqoBot obeys robots.txt, including Allow rules, wildcards and patterns anchored to the end of a path. A group that names RanqoBot takes precedence over your * group. To block it entirely:

User-agent: RanqoBot
Disallow: /

Ranqo keeps a copy of each site's robots.txt for 12 hours, so a change can take that long to apply. Refusing the User-Agent at your server or CDN takes effect at once. The RanqoBot page has more examples and the full detail.

Other ways Ranqo fetches a page#

Not every request from Ranqo is RanqoBot, and a robots.txt rule for RanqoBot does not stop the others:

• One page on request. When someone pastes a URL into a page audit or most free tools, Ranqo fetches that single page with an ordinary browser User-Agent, follows no links, and queues nothing.
• The free llms.txt generator reads your homepage and sitemap under its own User-Agent, RanqoLlmsTxtBot.
• Gemini's citation links. Gemini returns each source as a link on Google's redirect service. Ranqo asks that service where the link points, under its own User-Agent, Ranqo-Citation-Resolver, and never requests the cited page. It does not read the service's robots.txt, which disallows those links for crawlers: each link is one Google's own API returned with the answer.
• Each crawler's own User-Agent. Site Access and the free AI Crawler Inspector send GPTBot's, ClaudeBot's and the other crawlers' own User-Agents, to show how your server answers each one. These requests come from Ranqo, not from the AI vendors, and neither tool runs if your robots.txt disallows RanqoBot. Anthropic does not publish its crawlers' User-Agents, so the ClaudeBot and Claude-User checks send the string Site Tracking sees most often on visits verified as Anthropic's, and the Claude-SearchBot check sends one modelled on them, because no verified visit has carried one yet.

The RanqoBot page lists every User-Agent Ranqo sends.

Ranqo's requests in Site Tracking#

If your site runs Site Tracking, your install reports Ranqo's own requests like any other visit. Site Access checks your site on its own after your tracking runs, at most once every 7 days, so these requests arrive whether or not you open it.

• Site Access and the AI Crawler Inspector send each crawler's User-Agent from Ranqo's servers. On the Traffic page, those requests appear as visits from the crawlers they name, except DuckAssistBot, MistralAI-Index and MistralAI-Training, which Site Tracking's catalog does not list, so they count as Other bots. Where Ranqo has a crawler's published IP ranges, the visit is marked unverified, because Ranqo's servers are not in them. The browser request each check compares against counts as Other bots too, because its User-Agent claims an outdated Chrome.
• RanqoBot and RanqoLlmsTxtBot match no bot in the catalog either, so their requests, including RanqoBot's reads of robots.txt, sitemaps and llms.txt, count as Other bots. So do the single-page fetches Ranqo makes with a browser User-Agent, which claims an outdated Chrome, and the sitemap checks that go with them.

If you block RanqoBot on your own site#

If you are a Ranqo customer and your robots.txt disallows RanqoBot:

• Site Access reads your robots.txt and then stops before fetching any page, says so, and shows the verdict Unknown.
• Page discovery cannot follow links on your site to find pages to audit.
• An audit of a single URL you enter still runs, because that one fetch is not RanqoBot.

Which crawler list belongs to which feature#

The two lists overlap but are not the same, because the jobs differ. Site Access needs the crawlers that decide whether AI can read you. Site Tracking also has to recognise traffic that is not AI at all, so it is not counted as people. The lists are kept separately, so a crawler Site Access tests is not always one Site Tracking recognises: DuckAssistBot, MistralAI-Index and MistralAI-Training are in the first list only, and so are the two robots.txt permissions, Google-Extended and Applebot-Extended.

The crawlers Site Access tests#

A Site Access check fetches your homepage, and a few pages picked from your sitemap (up to 4 pages in all), as each of the 20 crawlers marked fetchable below, and as a regular browser to compare against. Some entries are permissions only: robots.txt tokens that a vendor honours but no crawler sends as a User-Agent. For those, Site Access reports what your robots.txt says, because that is the whole answer.

The User-Agent Site Access sends for each crawler is an approximation modelled on that crawler's own, and contains its token.

The Crawlers tab in Site Access groups them by what they are for:

• Live answers: assistants that fetch a page while answering someone, and AI search indexes. A block here can take you out of AI answers.
• Search: traditional search engine crawlers.
• Training: crawlers that collect data to train future models. Blocking one is never reported as a blocker: none of the AI answers Ranqo tracks fetch through a training crawler, so it does not change whether those answers cite you.

See Site Access for what each result means.

The bots Site Tracking recognises#

Site Tracking checks the User-Agent of every visit your server reports against its own catalog, and puts each recognised bot in one of these groups on the Traffic page:

• Training: crawlers collecting data for AI model training.
• Indexing: AI search indexes that decide what an assistant can cite.
• Agentic: fetches by an AI assistant acting on someone's request.
• Search Engine: traditional search engine crawlers.
• Social Preview: link-preview generators from chat and social apps, and automated browsers such as performance audits and headless test browsers.

Google-Extended and Applebot-Extended are not in the catalog. They are robots.txt permissions that no crawler sends as a User-Agent (see the Site Access table above), so a visit carrying either name is someone else using it, and Site Tracking treats it like any visit that matches no bot.

A visit whose User-Agent matches no bot counts as a person only when it comes from a web browser on a version an up-to-date browser could send (Site Tracking's human visits explains how that is decided): Human (AI Referral) when an AI assistant sent it, and Human (Direct) otherwise. Anything else counts as Other bots, including crawlers the catalog does not list, such as RanqoBot, DuckAssistBot, MistralAI-Index, MistralAI-Training and Googlebot-Image, and scrapers that send an old browser's User-Agent.

For a bot marked IP verifiable in the table, Ranqo checks each visit against the IP addresses its operator publishes. On the Traffic page, the Verified column of Agents detected shows the share of a bot's checked visits that came from those addresses. A visit that claims to be a bot from anywhere else may be someone else using its name. Automated browsers cannot be verified this way, because anyone can run them.

See Site Tracking for installing it and how visits are counted.