Your AI Visibility Score Moves 15 Points for No Reason

What We Measured

81 brands with at least 3 completed tracking runs each, 851 runs in total, and 27,803 individual results behind them. For every pair of consecutive runs we took the difference in the reported metric. No fix shipped in between, no prompt set edited, nothing done. Whatever moved, moved by itself.

One detail matters more than it looks. The metric is computed through the same function the product uses to show a customer their score, not through a cleaner formula written for the analysis. A hand-rolled version would disagree with what people actually see on screen, and then the noise floor would describe a number nobody is looking at.

The Noise Floor

Read the bars as: nineteen times out of twenty, a run-to-run change smaller than this is nothing. The overall score clears 14.5 points on its own often enough that a 10-point gain is not evidence of anything. The typical move is smaller, a standard deviation of 7.4 points, but the typical move is not the one you have to defend in a meeting.

We round these up into the thresholds the product enforces: 15 points overall, 20 for a single engine, 27 for a category. Below the line, a measured lift is not shown as a number at all. Showing one would turn a running total of sampling variance into something that looks like progress.

Narrower Is Noisier, Not Cleaner

The result that surprised us: the more specific the cut, the worse the noise. One category moves 1.9× as far as overall score does, 27.3 points against 14.5.

That runs against the instinct that a narrower slice is a sharper instrument. The mechanism is just sample size: a category holds a handful of prompts rather than the whole set, so a single answer flipping moves the average much further. The same applies to a single-engine view, which lands in between.

The practical consequence is uncomfortable. Category and per-engine breakdowns are the views people naturally drill into when they want to know what is working, and they are the views where a number is least likely to mean anything. It is one reason we publish a per-kind table rather than one global threshold.

The Individual Answers Are Fairly Stable

Here is the part that reframes the whole thing. Take a single prompt on a single engine and ask whether the verdict flipped between runs. It flipped 6.4% of the time across 24,467 comparable pairs. That is a fairly steady measurement.

So the answers are not chaotic. The score is. A visibility score is an average over a modest number of those cells, and averaging a small number of noisy-ish things produces something that jumps around more than any of its parts. The instability is arithmetic, not the engines being fickle.

It also tells you where to look. If the cells are stable and the score is not, then the useful question is never "did my number go up". It is which specific prompts changed hands, and whether they stayed changed.

The Two-Run Rule

A mention you have seen exactly once survives to the next run 54.3% of the time. A mention seen twice survives 96.1% of the time. That is a 41.8-point gap, and it is the single most useful thing in this post.

A first sighting is close to a coin flip. It is not a win, it is a candidate. The second consecutive sighting is what turns it into something you can report, plan around, or tell a client about. The rule that falls out is blunt: never count a new mention until you have seen it twice in a row.

The reverse holds just as strongly and is easier to act on. A prompt you have lost in two consecutive runs comes back on its own only 3.4% of the time. Those are not going to fix themselves, which makes them the cleanest work queue you have.

What a Ten-Point Gain Actually Means

Take a brand sitting at 30% visibility. A month later it reads 40%. That is the exact scenario an agency puts in a monthly report, and it is the scenario the measurement cannot support.

Ten points is below the 15-point band, which means a move that size happens on its own often enough that it carries no information. It might be real. The measurement simply cannot tell you which, and no amount of confidence in the underlying work changes that. Reported without a caveat, it is a claim the data does not make.

What would settle it is not a bigger number but another run. If the brand reads 40% again next cycle, and the specific prompts that gained are the same ones, the picture changes completely: two consecutive readings of the same thing is a different piece of evidence from one reading twice as large. This is the difference between measuring something and hoping.

There is an uncomfortable corollary for anyone reporting monthly on a weekly cadence. Four runs is not many samples, and a month is short enough that most genuine content work has not landed yet. The honest monthly report often says the number moved and we cannot yet attribute it, alongside the list of prompts that changed hands. That reads as weaker than a confident percentage, and it is worth considerably more, because the alternative is claiming a win that reverses next month.

The same logic cuts the other way and is easier to sell. A month where the score fell eight points is not a month where anything went wrong. Being able to say so, and to point at a threshold you set in advance rather than one chosen after seeing the result, is most of what separates a measurement from a narrative.

What to Do With This

• 01Put a threshold on your reporting before you look at the number. Ours is 15 points overall and 27 for a category. Anything smaller is "no detectable change", not a small win.
• 02Count runs, not weeks. Two consecutive runs is the minimum before a new mention is real, and no amount of calendar time substitutes for a second measurement.
• 03Track which prompts changed, not just the headline. The cells are stable enough to be worth reading individually, which the score is not.
• 04Prefer ratios to levels where you can. Share of voice moves less than a raw visibility level, because a run that returns more names inflates the denominator too.
• 05Be suspicious of a tool that reports a two-point improvement with a straight face. That is also part of why two tools disagree about the same brand on the same day.

We wrote the qualitative version of this argument months ago, in a post about why one measurement is never enough. It leaned on the published study and made the case without a number for how much movement to expect. This is that number.