How to Test Whether a GEO Change Worked

Key findings

A single before-and-after check is mostly noise: AI answers vary from run to run, and models change underneath you.
Use target questions the change should affect and control questions it shouldn’t, and compare how each moved.
To detect a 10-point lift from a 30% baseline you need about 356 answers before and 356 after; a 20-point lift needs about 93 each.
Plan the test before you make the change: write down the questions, the models, the sample size and what result would count as a win.

Why before-and-after misleads

Three things move an AI visibility number besides your work:

Sampling noise. The same question can name you in one answer and not the next. A small sample swings by tens of points on its own — see the margin-of-error tables.
Model changes. Providers update models and retrieval without notice. A jump the week after your launch may be theirs, not yours.
The market. A competitor’s funding news or a viral review moves answers in your category whatever you do.

A good test separates your effect from all three.

The design: target and control questions

Pick target questions your change should plausibly affect. If you published a page comparing you with a rival, the targets are questions where buyers compare the two.
Pick control questions in the same category that the change should not affect — different use cases, different buyer segments.
Freeze both sets before the change. Rewording a question mid-test changes what you are measuring.
Measure before: ask every question many times on each model you care about, over a week or two.
Make the change, then wait long enough for it to be crawled and indexed. Answers that use live web search can react within days; anything that depends on model training takes far longer.
Measure after with the same questions, models and sample size.
Compare the changes: the effect is (target after − target before) − (control after − control before). A model update that lifts everything lifts the controls too, and cancels out.

This is a difference-in-differences design, the same logic economists use when they can’t run a clean experiment. It doesn’t prove causation, but it rules out the most common false wins.

How many answers you need

Answers needed in each period (before and after) to detect a lift of the given size, by starting visibility. 80% power, 5% two-sided significance, two-proportion test.

Starting visibility	+5 pts	+10 pts	+15 pts	+20 pts
5%	435	141	76	49
10%	686	199	100	62
20%	1,094	294	138	82
30%	1,377	356	163	93
50%	1,565	388	170	93

Two things stand out. Small lifts are expensive to prove — a 5-point gain needs well over a thousand answers per period at typical baselines. And brands starting near zero can detect a lift with fewer answers, because going from 5% to 15% is a large relative move.

The table assumes independent answers. Repeated answers to the same question are correlated, so spread the sample across as many distinct target questions as you can, and treat these numbers as minimums.

A worked plan

Decision	Example
Change	Publish a comparison page against your largest rival and get it listed on two review sites
Target questions	10 questions buyers ask when choosing between you and that rival
Control questions	10 questions about an unrelated use case in the same category
Models	ChatGPT and Gemini, reported separately
Sample	Each question asked 20 times per model per period: 200 target answers per model per period
Win condition	Target lifts at least 10 points more than control on either model
Timing	Two weeks before; change; wait two weeks; two weeks after

With 200 answers per period this plan can reliably detect a lift of roughly 13–14 points around a 30% baseline. If you expect less, ask more times or wait for a longer after-period.

Common mistakes

Choosing questions after you see the results. Decide the target set first, or you will find the questions that happened to move.
Averaging across models. A real lift on one model can vanish in the average. Report each model, and decide in advance which counts.
Stopping early. Checking every day and stopping on the first good day inflates false wins. Fix the end date up front.
Calling a null a failure. “No detectable change at this sample size” is not “no effect”. Say which lift the test could have detected.

For the metrics themselves, see how to calculate AI share of voice. Mentioned keeps each tracked keyword’s questions frozen and re-asks each one three times per run, so the before-and-after series exists without extra setup.

Cite this

You are welcome to quote and chart these figures with a link back. Suggested citation: Mentioned, “How to Test Whether a GEO Change Actually Worked,” October 25, 2026, https://mentioned.at/learn/how-to-test-geo-changes. Data and text are published under CC BY 4.0.

Run a free check