How to Verify Googlebot, GPTBot and ClaudeBot in Your Logs: 12 Checks to Catch Fakes
Between late July and late August 2026, GreyNoise saw 824 addresses send AI crawler names, and none sat in the ranges the operators publish. A user agent is a claim anyone can type. Most operators let you check their crawlers by DNS, an address list or a signature. Here are 12 checks, a script for most of them, and what to do with a fake.
TL;DR
- AI crawler names are borrowed all the time, and the fakes ask for keys and password files, not pages, so treat each name in your logs as a claim to check.
- Where an operator publishes lists, match each bot against its own: an address on ChatGPT-User's list proves nothing about GPTBot, and one next to a listed range proves nothing at all.
- Download every list again each day and keep the last good copy: lists change without notice, and a list's date does not say when its ranges changed.
- Block a forged request on the name it claims plus the failed check, never on the user agent alone, and replace every key in any credentials file it was served.
To verify a crawler in your logs, check where the request came from, not what it calls itself. Take the client IP address and run the check the crawler's operator publishes. For Googlebot, Bingbot, Applebot and CCBot, that is a reverse DNS lookup followed by a forward lookup. For GPTBot, ClaudeBot, PerplexityBot and most other AI crawlers, it is a match against the address list the operator publishes. An address that fails the check is forged.
Two names are always forged: Google-Extended and Applebot-Extended, because no crawler sends them. Some names can't be verified at all, because their operator publishes no way to check them. Meta's crawlers are the biggest example.
The checks matter because crawler names are borrowed all the time. Between 28 July and 23 August 2026, GreyNoise saw six AI crawler names arrive from 824 addresses, and not one of them sat in the ranges published by the four companies behind those names. The traffic asked for passwords and keys, not pages.
This guide sets out the proof each operator offers, checked against its own documentation and files on 11 November 2026, then 12 checks in order, a short script that runs checks 3 to 9, and what to do with each verdict.
How Often Crawler Names Are Forged
A user agent is a line of text the client writes about itself. It costs nothing to copy, and AI crawler names are worth copying: many sites wave them through. There is also more real AI traffic to hide among. Cloudflare says daily requests from AI agents on its network grew by more than 1,700% over the past year. Three measurements show how often the names lie. Each covers a different set of sites, so their rates don't compare with each other.
GreyNoise: 824 addresses, none of them real crawlers
GreyNoise records traffic arriving at its sensors. Between 28 July and 23 August 2026, six crawler names belonging to Anthropic, OpenAI, Google and Perplexity arrived from 824 addresses spread over 795 separate /24 networks, so there was no single network to block. GreyNoise checked every address against the four companies' published lists, and Amazon's too. None matched. All 263,849 sessions carrying Google-Extended were forged, since no Google crawler sends that name.
The forged names never requested /robots.txt. They asked for .env files, cloud access keys, private keys and password stores. Over the same weeks, /robots.txt was the path Anthropic's real crawler requested most, 12% of its traffic. GreyNoise is careful to say it saw requests arrive, not whether any file was returned.
0 of 824
Addresses sending AI crawler names that sat in the operators' published lists, 28 July to 23 August 2026 (GreyNoise)
Fastly: millions of fake requests in one day
Fastly looked at one day, 23 September 2026, across its customers using bot management. About 6.3 million requests from nearly 10,200 addresses claimed to be AI crawlers and fetchers while probing for private files. The path /.env alone was probed by nearly 1,800 addresses. Fastly also says most AI bot traffic on its network is what it claims to be. The fakes borrow the names because, in Fastly's words, some sites treat AI bots "with more leniency": an allowlist entry, or a looser rate limit.
HUMAN: 1 in 18 requests
HUMAN measured two weeks of its customers' traffic for 16 AI crawlers and published the result on 9 September 2025. Of the requests using an AI crawler's user agent, 5.7% were spoofed, or 1 in 18. ChatGPT-User was the most spoofed name, at 1 in 6 requests.
The fakes were built to pass lazy checks. Most fake ChatGPT-User traffic came from the same Microsoft Azure network as the real traffic, and some came from addresses just outside OpenAI's ranges. HUMAN saw more than 14,000 requests from 40.84.221.0, close to OpenAI's 40.84.221.208/28, which does not contain it. A check on the network, or on a nearby address, lets those fakes through. Only the exact list catches them.
Which Proof Each Crawler Offers
Operators prove their crawlers in three ways, the same three Cloudflare accepts when it marks a bot as verified: a reverse DNS name, a published address list, or a cryptographic signature. Google, Bing, Apple and Common Crawl offer DNS and a list. OpenAI, Anthropic, Perplexity, Amazon, DuckDuckGo and Mistral offer lists. A few AI agents sign their requests. ByteDance lists ten ranges on a web page, and Meta publishes nothing to check against.
How each operator lets you verify its crawlers
From each operator's own documentation and files, checked 11 November 2026. DNS means a reverse lookup that ends in the name shown, confirmed by a forward lookup.
| Crawler | Operator | Proof | Check against |
|---|---|---|---|
| Googlebot and Google's other common crawlers | DNS or list | Host ends in googlebot.comcommon-crawlers.json | |
| Google-Agent | List; some requests signed | user-triggered-agents.jsonSigned as agent.bot.goog | |
| Google-Extended | Never sent | A robots.txt token only | |
| Bingbot | DNS or list | Host ends in search.msn.combingbot.json | |
| GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot | List, one per bot | gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json | |
| ChatGPT Work's Cloud browser | Signature | Signature-Agent "https://chatgpt.com" | |
| ClaudeBot, Claude-User, Claude-SearchBot | List | bots.json | |
| Claude-Agent | List | .well-known/sailAnthropic's hosted browser | |
| PerplexityBot, Perplexity-User | List, one per bot | perplexitybot.json, perplexity-user.json | |
| Applebot | DNS or list | Host ends in applebot.apple.comapplebot.json | |
| Applebot-Extended | Never sent | A robots.txt token only | |
| Amazonbot, Amzn-SearchBot, Amzn-User | List, one per bot | Amazonbot, Amzn-SearchBot, Amzn-UserPublished as web pages | |
| CCBot | DNS (IPv4) or list | Host ends in crawl.commoncrawl.orgccbot.json | |
| DuckAssistBot, DuckDuckBot | List | duckassistbot.json, duckduckbot.jsonThe two files were identical | |
| MistralAI-User, MistralAI-Index | List, one per bot | mistralai-user-ips.json, mistralai-index-ips.json | |
| MistralAI-Training | None published | Nothing to check against | |
| Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer, Meta-ExternalAds, FacebookExternalHit | None published | Nothing to check against | |
| Bytespider | List, on a web page | Ten /24 ranges on its pageReverse DNS ends in crawl.bytedance.com, with no forward record |
Two things in the table catch people out. First, where OpenAI, Perplexity, Amazon and Mistral publish addresses, each bot has a list of its own, and an address on one bot's list proves nothing about another. Perplexity's guidance is to "match each User-Agent against its own IP ranges". Second, the fetchers that act when a person asks, such as ChatGPT-User, Perplexity-User and Meta-ExternalFetcher, may not follow robots.txt at all. Verifying them tells you who sent the request, not whether your rules allowed it.
Checks 1 to 3: Before You Look Up an Address
1. Log the address that really connected
Behind a CDN or a load balancer, your server sees the proxy's address, not the crawler's. Log the client address your proxy passes on: Cloudflare sends it in CF-Connecting-IP. Trust that header only when your server accepts connections from Cloudflare alone, since anyone who reaches your server directly can send it. If you read X-Forwarded-For, the same holds: trust only the entries your own proxies added, because MDN warns that any part a trusted proxy did not add "may be spoofed". Log the user agent, the time and the path beside the address.
2. Read the user agent as a claim
A request that says GPTBot claims to be GPTBot until its address passes OpenAI's list. Find every place where a crawler name grants access or skips a control, such as an allowlist, a rate limit exception or a firewall rule that waves AI bots through, and put the address check behind it. Those are the doors the fakes in Fastly's and GreyNoise's data were looking for.
3. Count Google-Extended and Applebot-Extended as forged
These two names exist only in robots.txt. Google says Google-Extended "doesn't have a separate HTTP request user agent string", and Apple says Applebot-Extended "does not crawl webpages". Each one controls how content that Google's or Apple's ordinary crawlers already fetched may be used for AI. A request with either name in its user agent is forged, whatever its address.
Checks 4 to 6: Google and Bing
4. Verify Googlebot with a reverse and a forward lookup
Run a reverse DNS lookup on the address, check the name it returns, then run a forward lookup on that name and make sure it returns the address you started with. This is Google's own example:
$ host 66.249.66.1 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. $ host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 66.249.66.1
Google's page lists three domains, googlebot.com, google.com and googleusercontent.com, because it covers every kind of Google fetch. For a request that claims to be Googlebot, require a name ending in googlebot.com. A name under googleusercontent.com can be a fetch made by a site running on Google Cloud for one of its own users. The forward lookup is the step that proves anything: whoever controls an address can make its reverse DNS say googlebot.com, but only Google can make a googlebot.com name point back to that address.
5. Or match the right one of Google's five lists
For checks at scale, Google publishes five lists, one for each kind of fetch. On 11 November 2026 they held 317 ranges for common crawlers such as Googlebot, 272 for special-case crawlers such as AdsBot, 1,088 and 504 for fetches people trigger through Google Cloud sites and through Google itself, and 20 for agents such as Google-Agent. Match each name against its own list. The old address of the Googlebot list, a googlebot.json under /search/apis/, now redirects to common-crawlers.json, so code that still uses it works only while it follows redirects.
6. Verify Bingbot by DNS first
Bing's method is the same: the reverse lookup must end in search.msn.com, and the forward lookup must return the address you started with. Bing also offers a Verify Bingbot tool and publishes bingbot.json with the instruction "Refresh this list daily as we can change this list any time." The file we downloaded on 11 November 2026 said it was generated on 3 January 2024. Run DNS first, and if an address passes DNS but is missing from the file, believe DNS. A section of Bing's own guide is headed "Don't Use Hardcoded IP Addresses or Address Ranges".
Checks 7 to 9: OpenAI, Anthropic and the Rest
OpenAI, Anthropic and Perplexity publish no DNS name for their crawlers. Their address lists are the only check, so the lists have to be exact and current.
7. Match each OpenAI bot against its own list
OpenAI runs four bots with four lists: GPTBot (18 ranges), OAI-SearchBot (39), ChatGPT-User (237) and OAI-AdsBot (2), which only visits pages submitted as ads in ChatGPT. Match the address against the list for the bot the user agent names. GPTBot and OAI-SearchBot may add a robots.txt marker to their user agent when they fetch that file, which helps when your logs leave out paths. HUMAN's 40.84.221.0 is the trap to remember: it sits close to two of ChatGPT-User's ranges, inside the same Azure network, and is in neither.
8. Match Anthropic's bots against bots.json
ClaudeBot, Claude-User and Claude-SearchBot share one list, bots.json. On 11 November 2026 it held 1 /22 block and 13 single addresses, and Anthropic says an address on it "indicates that the crawler is coming from Anthropic". The network around those addresses proves nothing: the block is carved out of Amazon's address space and the single addresses belong to Google, so a rule that trusts traffic from Amazon or Google Cloud would pass anyone renting a server there.
Anthropic's hosted browser, whose user agent carries the token Claude-Agent, has a separate list of 9 addresses at claude.com/.well-known/sail, written in a new format based on TOML rather than JSON. Anthropic's advice for it applies to every name in this guide: anyone can send the same user agent, so check the address before you conclude a request is genuine.
9. Use the published method for every other crawler
- Perplexity: PerplexityBot and Perplexity-User have a list each. Perplexity-User "generally ignores robots.txt rules", because a person asked for the fetch.
- Apple: Applebot resolves under applebot.apple.com, or match applebot.json.
- Common Crawl: CCBot resolves under crawl.commoncrawl.org over IPv4. Over IPv6, where it has no reverse DNS yet, match its list. Common Crawl says it is "aware of crawlers falsely identifying themselves as CCBot".
- Amazon: Amazonbot, Amzn-SearchBot and Amzn-User have a list each, published as a web page rather than a plain file.
- DuckDuckGo: DuckAssistBot and DuckDuckBot each link a list. On 11 November 2026 the two files were identical: 486 addresses.
- Mistral: MistralAI-User and MistralAI-Index have a list each; MistralAI-Training has none.
- ByteDance: ByteDance's page lists ten /24 ranges for Bytespider and says its hostnames end in bytedance.com. On 12 November those names had no forward record, so a reverse lookup alone proves nothing. Match the ten ranges.
- Meta: Meta's crawler page names five crawlers and gives no list, DNS name or signature for any of them. Requests under these names are unverifiable, whatever they look like.
Checks 10 to 12: Signatures, Behaviour and Fresh Lists
10. Check the signature where an agent signs
A few AI agents now sign their requests using HTTP Message Signatures, the standard behind Web Bot Auth. A signed request carries Signature, Signature-Input and Signature-Agent headers, and the operator publishes its public keys at a fixed address. ChatGPT Work's Cloud browser signs with the Signature-Agent https://chatgpt.com. Google signs some requests from Google-Agent as https://agent.bot.goog, and says it does not sign every request, so keep the address check as the fallback.
Keys change, and their dates won't warn you. On 11 and 12 November, OpenAI's key directory held the same single key, each time marked to expire a week after we fetched it. Fetch the directory each time you verify rather than pinning a key. Writing your own verifier is rarely worth it: OpenAI lists several CDN and bot-management providers that check its signatures for you, and Google says major CDNs and firewalls support the protocol.
11. Use behaviour as a second signal
Real crawlers read robots.txt and fetch pages. GreyNoise's fakes never requested robots.txt and went straight for files such as /.env and /.aws/credentials, which no crawler has a reason to ask for. Behaviour can't verify a request, since a careful fake can fetch robots.txt too, but it makes a good alarm. As GreyNoise advises, alert on any request for /.env, /.aws/credentials or /.git/config, whatever name it carries. Judge robots.txt reads over several days, because real crawlers cache the file.
12. Re-fetch every list every day
Lists change without warning, and a list's date says when the file was built, not when its ranges changed. Google rebuilds its five every day: its common-crawler list held the same 317 ranges on 11 and 12 November under a new date each time. Bing's file still said 3 January 2024. A list copied into a firewall rule last quarter will sooner or later block a real crawler or pass a fake, which is why GreyNoise advises giving each list "an owner and a refetch schedule". Fetch each one daily, and keep the last good copy when a download fails.
How long ago each address list says it was generated
Days between the date inside each list and 11 November 2026, when we downloaded it.
- Google, all five lists
- 1 day ago10 November 2026
- OpenAI ChatGPT-User
- 5 days ago6 November 2026
- Perplexity, both lists
- 14 days ago28 October 2026
- Anthropic Claude-Agent
- 20 days ago22 October 2026
- Anthropic bots.json
- 22 days ago20 October 2026
- OpenAI GPTBot
- 50 days ago22 September 2026
- Apple Applebot
- 57 days ago15 September 2026
- Amazonbot, Amzn-SearchBot
- 64 days ago8 September 2026
- DuckDuckGo
- 71 days ago1 September 2026
- Common Crawl CCBot
- 92 days ago11 August 2026
- OpenAI OAI-AdsBot
- 183 days ago12 May 2026
- MistralAI-Index
- 206 days ago19 April 2026
- OpenAI OAI-SearchBot
- 313 days ago2 January 2026
- Amazon Amzn-User
- 372 days ago4 November 2025
- MistralAI-User
- 630 days ago19 February 2025
- Bing Bingbot
- 1,043 days ago3 January 2024
A list's date is when the operator last built the file, not when its ranges changed. Google rebuilds its five every day, so its date is always recent. Fetch every list on a schedule, because nothing tells you when the next change lands.
After you add a rule, confirm the real crawler still gets through. In Google Search Console, run Test live URL in URL Inspection: the fetch comes from Google-InspectionTool, one of Google's own crawlers, so if the test fails where it passed before, your new rule is the likely cause.
A Script That Runs the Checks
This Python script runs checks 3 to 9 for one crawler name and any number of addresses. Save it as verify_crawler.py. It uses only the standard library, so it runs on the Python 3.9 that Apple's Command Line Tools install on a Mac, and it downloads each list once per run. We tested it on 14 November 2026 with Python 3.9 and 3.13.
#!/usr/bin/env python3
"""Check whether addresses from your logs belong to the crawler they claim to be.
Usage: python3 verify_crawler.py <crawler> <ip> [<ip> ...]
Example: python3 verify_crawler.py gptbot 132.196.86.5
"""
import functools
import http.clientimport ipaddress
import json
import socket
import sys
import urllib.request
# Operators that publish a reverse DNS name for the crawler.
DNS = {
"googlebot": ".googlebot.com",
"google-inspectiontool": ".googlebot.com",
"bingbot": ".search.msn.com",
"applebot": ".applebot.apple.com",
"ccbot": ".crawl.commoncrawl.org",
}
# Operators that publish an address list for the crawler.
G = "https://developers.google.com/static/crawling/ipranges/"
LISTS = {
"googlebot": G + "common-crawlers.json",
"google-inspectiontool": G + "common-crawlers.json",
"google-agent": G + "user-triggered-agents.json",
"bingbot": "https://www.bing.com/toolbox/bingbot.json",
"gptbot": "https://openai.com/gptbot.json",
"oai-searchbot": "https://openai.com/searchbot.json",
"chatgpt-user": "https://openai.com/chatgpt-user.json",
"oai-adsbot": "https://openai.com/adsbot.json",
"claudebot": "https://claude.com/crawling/bots.json",
"claude-user": "https://claude.com/crawling/bots.json",
"claude-searchbot": "https://claude.com/crawling/bots.json",
"perplexitybot": "https://www.perplexity.ai/perplexitybot.json",
"perplexity-user": "https://www.perplexity.ai/perplexity-user.json",
"applebot": "https://search.developer.apple.com/applebot.json",
"ccbot": "https://index.commoncrawl.org/ccbot.json",
"duckassistbot": "https://duckduckgo.com/duckassistbot.json",
"duckduckbot": "https://duckduckgo.com/duckduckbot.json",
"mistralai-user": "https://mistral.ai/mistralai-user-ips.json",
"mistralai-index": "https://mistral.ai/mistralai-index-ips.json",
}
# Names that only appear in robots.txt. No crawler sends them.
TOKENS_ONLY = {"google-extended", "applebot-extended"}
# Crawlers whose operator publishes no way to check them.
NO_METHOD = {
"meta-externalagent", "meta-externalfetcher", "meta-webindexer",
"meta-externalads", "facebookexternalhit", "mistralai-training",
}
def parse(ip):
"""The logged address, with an IPv4 address written in IPv6 form unwrapped."""
address = ipaddress.ip_address(ip)
return getattr(address, "ipv4_mapped", None) or address
def dns_check(address, suffix):
"""True or False when DNS decides, None when it can't."""
try:
host = socket.gethostbyaddr(str(address))[0].rstrip(".").lower()
except OSError:
return None
if not host.endswith(suffix):
return False
try:
forward = {parse(a[4][0]) for a in socket.getaddrinfo(host, None)}
except (OSError, ValueError):
return None
return address in forward
@functools.lru_cache(maxsize=None)
def networks(url):
"""Download a list once per run. None when it can't be read."""
request = urllib.request.Request(url, headers={"User-Agent": "verify-crawler/1.0"})
try:
with urllib.request.urlopen(request, timeout=15) as response:
prefixes = json.load(response)["prefixes"]
return [
ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"), strict=False)
for p in prefixes
]
except (OSError, ValueError, KeyError, TypeError, AttributeError, http.client.HTTPException):
return None
def verdict(crawler, ip):
if crawler in TOKENS_ONLY:
return "forged: no crawler sends this name"
try:
address = parse(ip)
except ValueError:
return "not an IP address"
if crawler in DNS:
result = dns_check(address, DNS[crawler])
if result is not None:
return "verified by DNS" if result else "forged: DNS does not match"
if crawler in LISTS:
ranges = networks(LISTS[crawler])
if ranges is None:
return "unverifiable: could not read the operator's list"
if any(address in network for network in ranges):
return "verified by the operator's list"
return "forged: not in the operator's list"
if crawler in NO_METHOD:
return "unverifiable: the operator publishes no way to check this name"
return "unverifiable: this script has no check for this name"
if __name__ == "__main__":
if len(sys.argv) < 3:
sys.exit(__doc__)
name = sys.argv[1].lower()
for ip in sys.argv[2:]:
print(f"{ip:<16} {name:<16} {verdict(name, ip)}")Pass the crawler name as the script spells it, then the addresses from your log. This is its real output on 14 November 2026, including a Googlebot address written the way dual-stack servers log it and HUMAN's fake ChatGPT-User address:
$ python3 verify_crawler.py googlebot 66.249.66.1 ::ffff:66.249.66.1 8.8.8.8 66.249.66.1 googlebot verified by DNS ::ffff:66.249.66.1 googlebot verified by DNS 8.8.8.8 googlebot forged: DNS does not match $ python3 verify_crawler.py chatgpt-user 40.84.221.210 40.84.221.0 40.84.221.210 chatgpt-user verified by the operator's list 40.84.221.0 chatgpt-user forged: not in the operator's list $ python3 verify_crawler.py google-extended 66.249.66.1 66.249.66.1 google-extended forged: no crawler sends this name $ python3 verify_crawler.py meta-externalagent 203.0.113.7 203.0.113.7 meta-externalagent unverifiable: the operator publishes no way to check this name
For a whole log, pull out each pair of crawler name and address first, then run the script once per name. It checks addresses, not signatures, and it skips Amazon's lists, ByteDance's ranges and Anthropic's Claude-Agent file, which are not plain JSON.
What to Do With Each Verdict
The three verdicts and what to do
Every request that names a crawler ends in one of these.
| Verdict | What it means | What to do |
|---|---|---|
| Verified | The address passes the check the operator publishes: its DNS name, its list or its signature. | Treat it as that crawler. Your robots.txt decides what it may read, and it stays in your crawler reports. |
| Forged | The name is one no crawler sends, or the address fails the operator's check. | Block or challenge it at your CDN or firewall, keyed on the name it claims and the failed check. Keep it out of crawler reports and see what it asked for. |
| Unverifiable | The operator publishes no method, or the check could not run. | Give it nothing a user agent alone would earn: no allowlist entry and no looser rate limit. Judge it by what it requests. |
When the request is forged
Block or challenge it at your CDN or firewall, using the name it claims together with the failed check, never the user agent alone. A rule keyed on the claim and the check keeps working when the fake moves to new addresses, as Fastly notes of its own rule. Take forged requests out of your crawler reports, or your count of GPTBot or ClaudeBot visits measures the fakes too. Then look at what they asked for. If they requested /.env, /.git/config or cloud credentials, check whether any of those paths returned 200. If one did, move the file out of the web root and replace every key it held.
When the crawler is real but unwanted
A verified crawler you don't want is a robots.txt decision, not a verification problem. Disallow it in robots.txt, which the crawlers that follow robots.txt honour. For a fetcher that acts for a person and may ignore robots.txt, block its published addresses instead. Keep the two questions apart: who is asking, then what you allow.
When it can't be verified
Give unverifiable names, such as Meta's crawlers and MistralAI-Training, nothing a user agent alone would earn: no allowlist entry and no looser rate limit. Judge them by what they request, as you would any other client.
How Ranqo Does It
We built Site Tracking to run the address check for you. It records the page requests your site reports to Ranqo, and for crawlers from Google, Bing, OpenAI, Anthropic, Perplexity and Common Crawl it checks the visiting address against the list the operator publishes, using copies Ranqo downloads every day. On the Traffic page, the Verified Bots card shows the share of checkable bot visits that matched, and Agents detected lists each bot with whether its visits were verified. Bots Ranqo has no address list for, such as Applebot and Meta's crawlers, are left out of that share rather than counted as suspect.
One kind of failed check is expected. Crawler test tools send a crawler's user agent from their own servers, so their requests fail these checks by design. Ranqo's Site Access checks and the AI Crawler Inspector work the same way, and Site Tracking marks those visits unverified. To see what each AI crawler gets back from a page on your site, run the free AI Crawler Inspector. To check which crawlers can reach your site at all, see our crawler-access guide.
See which crawlers really visit your site
Site Tracking records crawler visits to your pages, checks each one from Google, Bing, OpenAI, Anthropic, Perplexity or Common Crawl against that operator's list and shows the share that passed, beside the people AI assistants send you. For the rules side, see how to control AI crawlers with robots.txt and how agentic browsers reach your site.
Start a 7-day trialWritten by
Nisha Kumari
Nisha Kumari is Co-Founder at Ranqo, where she leads growth strategy and client acquisition. With a background in digital marketing and financial management, she specializes in SEO, Generative Engine Optimization, and helping brands build visibility across AI platforms.
Share this article
Related articles
Is Your Site Blocking AI Crawlers? A Ten-Point Check After Cloudflare's September Change
Most sites are not blocking AI crawlers, and most that are did not decide to. Refusing the crawler that trains a model costs nothing in AI answers; refusing the one that fetches pages at answer time is what removes you. What Cloudflare changed in September, which bot is which, and a ten-point check to run on your own site.
AI.txt vs Llms.txt vs Robots.txt: The Complete AI Crawler Control Guide for 2026
Most articles about AI crawlers ask the wrong question: whether to block them. The strategic question is which crawlers should train your model, which should retrieve from you for citation, and where licensing replaces both. This guide covers the three control files (robots.txt, llms.txt, AI.txt), the AI crawler taxonomy, the crawl-to-referral economics that should drive your decisions (Perplexity 4,454:1 vs Anthropic 500.5:1 over the four weeks to 21 November 2026), the Perplexity stealth-crawling case study, and industry-specific decision frameworks. Every claim is verified against published sources.
How to Optimize for the Agentic Browser Era
Adobe Analytics reported a 4,700% YoY surge in AI-driven traffic to US retail sites in July 2025, and your client-side analytics doesn't see most of it. Comet launched July, Atlas in October, and the agent is now a third class of visitor with its own retrieval pattern. Here's the playbook for the agentic browser era.