RanqoBot
Ranqo's page-audit fetcher
If you found this URL in your access log, this page is the whole answer. RanqoBot fetches pages on behalf of a Ranqo customer, and it obeys robots.txt.
User-Agent
RanqoBot/1.0 (+https://ranqo.ai/bot)The token to match in robots.txt is RanqoBot. There is no fixed IP range to allowlist, so match the User-Agent.
What it fetches, and why
Three things, each triggered by a person using Ranqo.
Page discovery
A Ranqo customer points an audit at a site, and we follow internal links from that page to find the pages worth auditing.
Follows links within one site only. Obeys robots.txt. 15s timeout per page.
Outbound link validation
A page being audited links out to your site, and we check that the link still resolves.
HEAD, falling back to GET. At most 50 links per audit, cached 24 hours.
Citation source resolution
An AI engine cited your page in an answer, and we read its title, or unwrap the redirect proxy the engine used, so we can name you as the publisher.
GET. We read the first 16KB for a title tag, or discard the body entirely when only the URL is needed. Cached 30 days.
What it does not do
It does not crawl uninvited
RanqoBot reaches a site because a Ranqo customer pointed an audit at it, because a page being audited links to it, or because an AI engine cited it. It does not go looking for sites on its own.
It does not build a search index
Nothing RanqoBot fetches is stored as page content. The link check keeps a status code; citation resolution keeps a URL and a title. Response bodies are otherwise discarded.
It does not train models
Ranqo does not train or fine-tune models, and no content RanqoBot fetches is used as training data by Ranqo or passed to a model provider for that purpose.
It does not run on a schedule
Every request traces to a person: an audit someone started, or an AI answer that cited a source. No customer activity against your site means no requests.
Allowing and blocking
RanqoBot reads your robots.txt before each request and obeys it, including Allow, wildcards and end-anchored patterns.
Block it entirely
User-agent: RanqoBot
Disallow: /Block part of the site
User-agent: RanqoBot
Disallow: /members/
Allow: /members/public/A rule naming RanqoBot takes precedence over your wildcard group, so a site that blocks * and allows us is honoured, and so is the reverse. We cache robots.txt for 12 hours, so a change takes up to that long to take effect. If your robots.txt is unreachable when we ask, we treat that as permission rather than refusal, because the alternative silently empties a customer's report every time a host has a bad minute.
Denying the User-Agent at your server or CDN works too, and takes effect immediately. Links we are not allowed to fetch are reported to the customer as blocked, not broken, so refusing us never makes your site look dead in someone else's audit.
On-request page fetches
RanqoBot is not the only way Ranqo reaches a page, and the rest of this page would be misleading without saying so.
Some Ranqo features fetch a single page the moment someone asks for it, and those requests do not carry the RanqoBot token, so a robots.txt rule naming us will not stop them. Here is every other User-Agent we send:
- An ordinary browser User-Agent
- One page someone pasted into an audit or a free tool. Rendered like a browser, nothing followed, nothing queued.
- RanqoCrawlerInspector/1.0
- Reads your robots.txt when someone runs our AI Crawler Inspector against a URL.
- GPTBot, ClaudeBot, PerplexityBot and others
- The same Crawler Inspector then requests that one URL once per bot, sending each bot's own User-Agent, to show how your server answers each of them. These come from Ranqo, not from OpenAI or Anthropic.
- Mozilla/5.0 (compatible; RanqoLlmsTxtBot/1.0)
- Reads your homepage and sitemap when someone generates an llms.txt for your site.
- Ranqo-IP-Range-Refresher/1.0 (+https://ranqo.ai)
- Fetches published IP-range files from AI vendors so we can verify bot traffic. Never points at a customer site.
The third entry is the one worth reading twice: our AI Crawler Inspector exists to show you how your own server answers each AI bot, so it sends each bot's User-Agent from our infrastructure. If you see GPTBot in your logs from an IP that is not OpenAI's, that may be us, and it fetched one URL once.
We draw the line there because none of these crawl: one URL, requested by a person, no links followed, nothing queued. Everything that does follow links, or runs in volume across your site, goes through RanqoBot and is covered by the rules above.
If you would rather Ranqo did not fetch your pages at all, the User-Agent block at your server or CDN is the reliable control. If that is not practical for you, write to us.
Something looks wrong?
If RanqoBot is doing something this page does not describe, we want to know. Include a few log lines with timestamps and we will trace them.