One changed AI answer is not a result yet

After editing a page, it is easy to see a new answer and attribute it to the edit. At the same time, sources, search mode, model, dialogue context, or random generation may have changed. The first new answer is therefore a useful lead for testing, not a final metric.

The record should begin with an observation: what changed, where, when, and under which settings. Only then add an inference and a hypothesis about the cause. If conditions are unknown, say so; unknown must not become zero or confirmation.

The Answer Index compared 480 identical queries in Google AI Overviews, ChatGPT, Perplexity, and Gemini. In the described English-language API slice, average cited-domain overlap for the closest pair, Perplexity and Google AI Overviews, was 18.1%. It was not a test of user interfaces, Claude, or Alice, and it does not explain why sources were selected.

The practical meaning is limited but useful: do not transfer one service’s result to another. If ChatGPT changed after an edit, that does not justify saying overall AI visibility changed. Service, mode, and search access must remain separate measurement interfaces and modes.

Create an affected-query group: the new information should be relevant to each query. Keep control queries that do not depend on the edited passage. Set the number of repeats and the before/after window in advance; do not expand them after an inconvenient result.

Each run passport stores service, mode, web access, shown model or version, language, region, account, date, full query, full answer, and visible sources. The same query text does not make two runs comparable when the other conditions differ.

Record whether the edited page appeared among visible sources; whether AI transferred the edited fact correctly; whether it named or recommended the brand in the intended scenario; and whether it gave a useful next-step link. These are different events. A link is not the same as a source link, a source link is not a visit, and a mention is not a recommendation.

State matching counts and total counts. “Three of ten affected queries received the intended fact” is more useful than “visibility grew.” Also record how many runs were unknown or could not be archived.

If the affected group changes while the control group stays stable, the edit-effect hypothesis becomes more interesting. Even that does not automatically prove causation: index, demand, or service interface may have changed simultaneously. A control reduces explanations; it does not eliminate them.

Preserve complete answers, source URLs, and the page version. If a result cannot be repeated or checked, keep it as an observation in the editorial log rather than a universal recommendation.

Do not claim overall visibility from one service. Do not attribute an effect to one sentence when page structure, external sources, and technical access changed together. Do not treat a bot visit as proof that a page was cited.

The correct next step is to repeat the predefined measurement, update the conditions log, and show a mixed result honestly. For GeoAeoAle, the value of an experiment is a checkable trail, including failure.