Why Every AI Visibility Tool Shows You a Different Number (Data From 102,025 Responses)

Marketers noticed the inconsistency before the vendors explained it. Digiday reported in May 2026 that agencies are growing wary of expensive AI visibility tools for exactly this reason. As Paul Dyer, CEO of the agency /prompt, put it: "If you use three different tools and give them the same prompts, you get three different answers." Another agency leader in the same piece dismissed the category as "just a benchmarker," not a source of truth; a third called it "extremely early days."

They are right to be skeptical, and right that the tools are benchmarkers rather than oracles. Where the conventional wisdom goes wrong is the implied conclusion -- that because the numbers disagree, they must be worthless. The disagreement is not noise obscuring a single true value. There is no single true value. Visibility in AI search is a distribution across engines, prompts, and time, and a tool's headline number is one particular way of collapsing that distribution into a digit. Different collapses, different digits.

This is the biggest and most underappreciated source of disagreement. ChatGPT, Claude, Perplexity, Gemini, and Grok are different models, with different training data, different search back-ends, and different willingness to name brands. Ask each of them the same unbranded category question -- "what's the best payment platform for startups?" -- and they surface brands at genuinely different rates. On the first run across our 102-brand cohort, unbranded recognition was 22.1% on ChatGPT, 18.7% on Gemini, 23.9% on Perplexity, and 51.5% on Claude.

Read that chart as a warning about tools, not a leaderboard of engines. A tool that tracks only ChatGPT and a tool that weights Claude heavily will report visibility numbers more than twice as far apart as each other -- for the identical brand, on the identical prompts -- purely because of their engine coverage and blending choices. (One honest caveat on our own data: Claude and Grok are Agency-tier engines in the study, so they ran on far fewer brands, skewed toward higher-stature ones. Claude's 51.5% partly reflects which brands it saw, not the engine alone -- which is itself another reason two tools with different brand samples would disagree.)

It gets worse over time, because the engines don't just differ -- they diverge. Here is a single mid-market brand we track, five consecutive weeks, same prompts throughout. Its recognition on ChatGPT fell 25 points while its recognition on Perplexity rose 25 points, and on Claude it barely moved.

One brand can't prove a trend, and we show this as an illustration rather than evidence. But it makes the mechanism concrete: a client watching a ChatGPT-only dashboard that month would have seen a crisis; the same client on a Perplexity dashboard would have seen a breakout. Neither tool is lying. They are watching different windows onto the same brand.

Even the background rate of change differs by engine. Across brands that ran no deliberate intervention during our window, unbranded visibility drifted down measurably on ChatGPT and Perplexity while staying statistically flat on the others.

This is the "no-intervention baseline" -- it does not prove visibility can't be improved (almost no brand in the cohort actively tried), only that the engines move at different speeds when left alone. Independent research points the same way: an eMarketer analysis reported by Search Engine Land found that "between 40% and 60% of cited sources change month-to-month" across Google AI Mode and ChatGPT. When the underlying answers churn that much, two tools sampling on different days will legitimately report different numbers.

Even within a single engine, the same prompt does not always return the same answer. This isn't a bug in the tools -- it's a property of the models. Academic work has shown that LLMs produce varying outputs even at temperature 0, the setting that is supposed to be deterministic; the authors measured accuracy swings of up to 15% across runs that were expected to be identical. Add live web search on top -- where the retrieved pages themselves change between requests -- and run-to-run variation is guaranteed.

We measured how much of that variation is real signal versus noise. Most of AI visibility is actually stable: the majority of brand-prompt-engine combinations are either always-mentioned or never-mentioned, run after run. But a meaningful minority -- roughly a fifth of cells -- aren't reliably always- or never-mentioned, and a smaller core genuinely flips run to run; sentiment is noisier still. The practical consequence for tool disagreement: a tool that samples a prompt once and a tool that samples it ten times will report different numbers for the volatile cells, and the single-sample tool will look more confident than it has any right to be. We unpack the full stability breakdown -- and why a single measurement is a coin flip -- in the case against measuring once. The peer-reviewed argument that visibility should be treated as a distribution, not a single observation, lands in the same place.

Suppose you fixed the first two causes -- same engines, same prompts, plenty of samples. Two tools could still report different numbers, because "visibility" is not one metric. It is a stack of definitional choices, and each one can halve or double the result without anything changing in reality.

Mention rate vs citation rate

Does the tool count every time an engine names your brand in its answer text, or only when it attaches a linked citation to a source? These are different graphs -- the model recommends far more brands than it links to, and engines vary enormously in how often they cite at all. A mention-based tool and a citation-based tool are measuring two different things and will never match. We covered why the link graph and the citation graph diverge separately.

The denominator, position weighting, and aggregation

Share of voice is three more decisions before it is a number: who is in the competitive set (the denominator), whether a first mention outweighs a fifth (position weighting), and whether per-engine scores get blended into one figure (aggregation). In our own dashboards, swapping a hand-picked five-competitor set for the full category set moves a brand's share of voice by 3x on identical answers. The three decisions that change the number is the full breakdown -- the point here is simply that two vendors making different choices at each fork will land on different numbers honestly.

The wrong move is to keep shopping for the tool whose number feels right. The right move is to stop treating any single number as ground truth and change what you ask of a tool:

Track per engine, not one blended score

A single "visibility score" blended across five engines hides the exact thing that matters -- that you might be strong on Perplexity and invisible on ChatGPT in the same week. Insist on a per-engine view. If your buyers live on ChatGPT, a great Perplexity number is cold comfort.

Read trends, not absolutes

Because the absolute number is definition-dependent and the target drifts, the level is less trustworthy than the direction measured consistently. A tool that samples the same prompts the same way every week gives you a reliable trend line even if its absolute level differs from a competitor's. Pick one methodology and stick with it rather than comparing snapshots across tools.

Make the vendor show its method

The tools worth paying for can tell you exactly how they sample and what they count. The ones to avoid wave a confident number and won't explain it. Ask -- including us -- how many times they run each prompt, whether web search is on, which engines are covered, and whether they count mentions or citations. If the answers are vague, the number is theatre.

The category's critics are right that no tool can hand you a single authoritative visibility number. But the conclusion isn't that measurement is hopeless -- it's that you were asking for the wrong deliverable. A brand's presence in AI answers genuinely is different on every engine, genuinely does move week to week, and genuinely depends on how you define a mention. The disagreement between tools is a faithful reflection of that reality, not a failure to capture it.

Used that way, the numbers are useful precisely because they vary. The per-engine spread tells you where to focus. The week-over-week direction tells you whether your work is landing. The gap between a mention count and a citation count tells you whether you're being talked about or linked to. A tool that pretends all of that collapses into one clean digit is the one you should distrust.

Three different tools, three different answers -- and, correctly understood, all three can be right at once. They are measuring a moving, multi-engine, definition-dependent target, and the honest way to read them is per engine, on a cadence, with the methodology written down. Chase the trend, not the digit.