Site Access

Site Access answers one question: when an AI engine sends its crawler to your site, does it get the page? It is the one measurement in Ranqo that can see a rule on your server or CDN that turns crawlers away. Open it from Site Access in the Analyze section of the sidebar.

What a check does#

A check fetches your homepage once as a regular browser and once as each of the 20 crawlers in Ranqo's catalog, sending each crawler's own User-Agent. Then it compares what each crawler got with what the browser got.

• The browser is the control. It is what separates "this site blocks crawlers" from "this site was down". Without a browser answer, nothing is reported as a block.
• Requests are spaced out, never sent at once, and anything other than a 200 is asked again a few seconds later before it is recorded. Firing every request together would trip rate limits and report a block that Ranqo caused. If the check runs out of time before that second ask, the crawler shows as Not confirmed and never counts as a block. A 429 is the one exception: it is reported as rate-limited either way.
• Deeper pages are checked too. When Ranqo can read your sitemap, it also fetches pages from it as every crawler, one from each of its biggest sections, up to 4 pages in all counting your homepage. On a large sitemap the sections are counted over the first pages Ranqo reads from it (highest priority first, then in the order the sitemap lists them), so they can differ from the counts on the Sitemap tab. A refusal on the homepage that a deeper page contradicts is reported as unknown, not as a site-wide block.
• The files that govern crawlers are read fresh: robots.txt (and what it says to each crawler for your homepage), your sitemap (whether it is reachable and declared in robots.txt), and llms.txt (whether a real file is served, not an HTML page at that address).

A check takes about a minute. See AI crawlers for the full catalog and what each crawler is for.

The verdict: a state, not a score#

Site Access deliberately has no 0 to 100 score. Measured across live sites, results cluster at the two ends, completely clear or hard blocked, and a score would blur the only distinction that matters. The banner at the top states one of four verdicts:

Under the verdict, Blockers and Worth a look count the findings of each kind, and Requests made is how many fetches the check sent.

Blockers, warnings and notices#

Each finding has a severity. Only blockers make the verdict Blocked.

Blockers are anything that keeps AI answers or search engines from reading your site:

• An assistant or an AI search crawler was refused by your server while a browser got the page. When Googlebot and Bingbot got in at the same time, the refusal is a rule about who is asking, usually a CDN or firewall bot filter.
• Your robots.txt disallows an assistant, an AI search crawler or a search engine. Each kind is reported on its own, for what it does: assistants fetch pages to answer a question someone is asking now, AI search crawlers build the indexes AI answers draw on, and search engines build your search listing.

Worth a look covers results Ranqo cannot call a block with certainty:

• Your server refused every non-browser client, search engines included. Real crawlers arrive from IP ranges their vendors publish, and a filter that checks those may still let them in, while Ranqo sends each User-Agent from its own servers. Check your CDN's bot settings directly.
• Search engines were refused while a browser got the page (for the same reason).
• A training crawler was refused by your server. Worth confirming it is a decision and not a side effect.
• Your robots.txt disallows Google-Extended. Google says this stops your content being used to ground answers in Gemini Apps and Vertex AI, not only to train Gemini. It does not affect your inclusion in Google Search.
• Your server answered 429, which says more about request rate than policy.
• A request timed out or returned a server error, so that crawler is unmeasured, not clear.
• A crawler received a much thinner page than the browser did.
• robots.txt could not be read, no sitemap was reachable, or a request for your sitemap did not complete.
• The check ran out of time before it finished every crawler, either never asking one or never asking it a second time, so those are unmeasured.
• Ranqo could not load your homepage as a browser (the verdict is then Unknown).

Notices are hygiene. They are listed but do not change the verdict: robots.txt disallowing a training crawler other than Google-Extended, robots.txt naming a crawler only by a token its vendor does not list, a sitemap not declared in robots.txt, and a missing llms.txt.

The six tabs#

• Overview. Four cards (Crawler fetch, robots.txt, llms.txt, Sitemap), each opening its tab, then What we found: every finding with an explanation and a button to open the tab and crawler group it concerns. When your sitemap is not declared in robots.txt and Ranqo knows its address, the finding carries a Copy fix button with the exact Sitemap: line to add. Ranqo offers it only when it could read your robots.txt; when it could not, it cannot see whether the file already lists your sitemap. On a clear site, the panel says so and points you to the Action Center, because the visibility gap is somewhere else.
• Crawlers. Who can fetch your site: one row per crawler with its status, response time and word count beside the browser's, filterable by All, Live answers, Search and Training. A crawler the check could not finish reads Not checked (never asked) or Not confirmed (asked once, not asked again before the time limit). Beyond the homepage shows the deeper pages, and Edge uses response timing to say whether a refusal came from your CDN edge or your application, when the timing is clear enough to tell.
• robots.txt. Your file as fetched, annotated line by line, with What it tells each kind of crawler for your homepage.
• llms.txt. The file Ranqo found and the checks it ran on it. No engine has published that it reads llms.txt, so this is always a notice.
• Sitemap. Whether it is reachable and declared, what is in it, and a sample of its pages, with the ones the check fetched marked. When robots.txt could not be read, whether it declares your sitemap reads as unknown, and only the standard sitemap addresses were tried.
• History. One row per completed check, with the verdict and a note on what changed since the previous one. The note also says when Ranqo adds or removes a crawler or changes how it checks one, when a check ran out of time before finishing a crawler or asked different deeper pages, and when it could not finish reading robots.txt, your sitemap or llms.txt because the request timed out or failed before your server answered. Without robots.txt, Ranqo cannot see which sitemaps it lists, so the sitemap reads as unfinished too. Those are changes on Ranqo's side, not your site's: a note that names only those reads in grey, and one that also names a change on your site lists that change first and reads in ink. A crawler or file a check could not finish is compared next time against the last check that did. Up to 26 checks are kept.

When checks run#

Site Access runs on its own after one of your brand's tracking runs, at most once every 7 days. To check again after a fix, press Re-check at the top of the page.

Re-checks have a cooldown of 60 minutes per brand, and only one check can run against a domain at a time. That is a limit on how often Ranqo hits your server, not a plan allowance: Site Access does not count toward any monthly limit.

When a check finds a blocker, it also appears in This week's signals on your Home page. When an automatic check finds a crawler newly blocked (your robots.txt now disallows it, your server now refuses it where the last check that finished it got the page or was only rate-limited, or the deeper page that served it now refuses it too), or finds a blocker on a brand's first check or on a crawler no earlier check managed to finish, Ranqo also emails you about it. Turn these emails off with Alerts in Account settings.

What to do with a blocker#

1. Open the finding's tab to see exactly which crawlers were refused and what your server answered.
2. If robots.txt is the cause, edit the rule for that crawler. If your server or CDN refused it, look for a bot-management or firewall rule that matches the crawler's User-Agent.
3. Press Re-check once the change is live, and confirm the finding is gone.

Clearing a blocker is a precondition for being read at all. Everything else on this page is hygiene, and Ranqo does not claim it will move your visibility.