Ask an AI assistant the same question twice and you can get two different answers. That makes every AI visibility score an estimate — and most are built on far too few answers to support the conclusions drawn from them.
Large language models sample their output. The same question, asked the same way, can name your brand in one answer and leave it out of the next. So “does ChatGPT recommend us?” has no yes-or-no answer; it has a rate. You estimate that rate by asking many times and counting — and like any estimate from a sample, it comes with a margin of error.
Each row shows a brand observed in half of its sampled answers, and the 95% confidence interval for its true visibility (Wilson score interval, which behaves properly at small samples). The last column is the textbook ± margin at 50%, for comparison.
| Answers sampled | Observed | 95% range for true visibility | Simple margin |
|---|---|---|---|
| 5 | 2 of 5 (40%) | 12%–77% | ±44 pts |
| 10 | 5 of 10 (50%) | 24%–76% | ±31 pts |
| 15 | 7 of 15 (47%) | 25%–70% | ±25 pts |
| 25 | 12 of 25 (48%) | 30%–67% | ±20 pts |
| 50 | 25 of 50 (50%) | 37%–63% | ±14 pts |
| 100 | 50 of 100 (50%) | 40%–60% | ±10 pts |
| 200 | 100 of 200 (50%) | 43%–57% | ±7 pts |
| 400 | 200 of 400 (50%) | 45%–55% | ±5 pts |
Five answers is a common size for a quick check — including our own free check. If a brand’s true visibility is exactly 50%, these are the odds of each score:
| Score | 0/5 | 1/5 | 2/5 | 3/5 | 4/5 | 5/5 |
|---|---|---|---|---|---|---|
| Probability | 3.1% | 15.6% | 31.2% | 31.2% | 15.6% | 3.1% |
A five-answer check is good at one thing: telling you whether you appear at all. If you score 0/5, your true visibility is probably low (the 95% upper bound is 43%). It is not good at telling 40% from 60%, or this week from last.
Comparing two periods is harder than measuring one, because both readings are noisy. The table gives the smallest change you could reliably detect (80% power, 5% significance, around 50% visibility) with the same number of answers in each period.
| Answers per period | Smallest detectable change |
|---|---|
| 5 | 89 pts |
| 15 | 51 pts |
| 25 | 40 pts |
| 50 | 28 pts |
| 100 | 20 pts |
| 200 | 14 pts |
| 400 | 10 pts |
So a dashboard that reports “+8 points this week” from 25 answers a week is mostly reporting noise. Trend lines over many weeks, or pooled across many questions, are far more trustworthy than any single week-over-week delta.
The math above assumes every answer is an independent draw. They aren’t quite: repeated answers to the same question are correlated, so 5 questions asked 3 times each carries less information than 15 different questions. Spread your sample across many distinct buyer questions rather than repeating a few, and treat the ranges above as the best case.
Our tracked keywords ask each of five frozen questions three times per run for exactly this reason, and our methodology shows the sample behind every number. For the formulas behind the metrics themselves, see how to calculate AI share of voice.
You are welcome to quote and chart these figures with a link back. Suggested citation: Mentioned, “How Many AI Answers Do You Need to Measure Visibility? The Margin-of-Error Math,” October 25, 2026, https://mentioned.at/learn/ai-visibility-margin-of-error. Data and text are published under CC BY 4.0.