Why an API cannot replace auditing recommendations in the consumer AI interface

This guide is for teams trying to understand what a person sees when an AI service recommends a product. The measurement unit is not a vague visibility score but an observable recommendation under a specified setup: service, mode, country, language, date, and account state. An API request is a different measurement object. It can improve automation and repeatability, but it cannot automatically stand in for the consumer interface.

Keep five events separate. A service may name a product: recommendation or mention. It may show a link: a visible source or path, not proof of a click. A person may visit the page: traffic. Only a confirmed target action is a conversion. A correct link does not prove a purchase, and a bot request does not prove that a page entered an answer.

Write the reader job before running the audit: for example, “which products and sources does an ordinary user receive for the same prompt?” If the question is “which service is best,” define whether you mean product stability, factual accuracy, source quality, or post-answer action. These are different outcomes and each needs its own base count.

• Interface and API are separate conditions.
• Recommendation, link, visit, and order are separate events.
• One screenshot is one observation, not a starting measurement rate.

Section sources:[1] Lucas G. Uberti-Bona Marin et al. / arXiv

The paper used 117 physical-product prompts. Each prompt was run three times in five conditions, producing 1,755 observations; 1,536 yielded an analyzable response. The three consumer-facing conditions were the ChatGPT and Gemini web interfaces and AI Overviews in Google Search; the other two were programmatic calls to ChatGPT and Gemini. The prompts were mostly English, and the browser was configured for English in the Netherlands in September 2026.

The study also compared interface and API calls close in time, but that does not make them the same environment. The interface model or mode could be hidden or changed by the service; the API used specified models and parameters. Therefore “the interface differed” is an observation for these conditions. “The API caused the difference” would be causal and is not supported by the design.

Do not expand the population to which the results apply. The findings concern physical products, selected languages, locations, dates, and models. They do not measure apps, services, Google Play, App Store, RuStore, Russian-language prompts, or another region. That is a boundary of the evidence, not a formatting detail.

• 117 prompts × 3 repeats × 5 conditions = 1,755 observations.
• 1,536 observations produced an analyzable response.
• Physical products, mostly English prompts, the Netherlands, September 2026.

Section sources:[1] Lucas G. Uberti-Bona Marin et al. / arXiv

A single run cannot tell you whether you observed a typical result or a draw from a variable process. In the study, “best” framing differed across repeats for 27–40% of prompts depending on the interface. This does not mean the service is always random. It means a single answer does not measure stability.

The authors compared sets of named products. Mean Jaccard similarity was 0.178 for ChatGPT, 0.287 for Gemini, and 0.421 for AI Overviews. The coefficient concerns product-set overlap, not clicks or product quality. Another repeat count, language, or user state may produce a different value.

Predefine the repeat count and do not select the most convenient answer from a series. Store every full answer and count how often a product appeared, was called best, or received a visible link. “8 of 30 runs” is more transparent than “often.”

• Stability belongs to a series, not one screenshot.
• This overlap score compares sets; it does not explain causes.
• Store inconvenient repeats too.

Section sources:[1] Lucas G. Uberti-Bona Marin et al. / arXiv

An answer may name a product without a link, show a link without recommending the product, or name a product and link to a page that does not support the wording. The audit table therefore needs separate fields: product named, brand named, “best” framing, visible link, exact destination, and whether the page supports the claim.

Across repeated answers, displayed-domain overlap was 26.0% for ChatGPT, 29.8% for Gemini, and 45.9% for AI Overviews. Exact-page overlap was 18.1%, 28.4%, and 41.4%. These are visible-source measurements, not search rank or traffic. Domain overlap is also not content overlap: two pages on one domain may support different claims.

Mark “source shown,” “page supports claim,” and “user visited” separately. If the service does not expose returned-but-hidden pages, record unknown. Do not replace unknown with a log-based guess or similar wording.

• A mention is not a shown source link; a shown link is not a visit.
• A displayed domain and an exact page are different units.
• Unknown is better than an invented source origin.

Section sources:[1] Lucas G. Uberti-Bona Marin et al. / arXiv

Run two parallel protocols. For the consumer interface, record the visible mode, sign-in state, model when known, language, country, date, full answer, and every link. For the API, record the specific method, model, settings, client-library version, system instruction when applicable, randomness settings, and the full response. Do not collapse either record into “AI answered.”

The study found mean displayed-domain overlap between interface and API conditions of 12.0% for ChatGPT and 14.8% for Gemini. That is a strong reason not to treat an API observation as a stand-in for what users see. But the numbers do not say why conditions differed: model, settings, tools, personalization, output format, and snapshot timing are all plausible explanations.

Compare like with like. First estimate the share of runs naming the exact product, then runs showing a visible link, then links whose page supports the claim. If model and interface change together, call the result a comparison of two conditions rather than a one-cause experiment.

• An API improves repeatable calls but does not automatically copy the interface.
• Store configuration beside every answer.
• Do not make a causal claim from an interface difference.

Section sources:[1] Lucas G. Uberti-Bona Marin et al. / arXiv

Give every run its own row. Record the prompt identifier and repeat number, date and time, language, country, consumer interface or API, visible mode, known model, full answer, named products and brands, recommendation wording, displayed addresses, and page status. For each address, note whether it is an exact product page, a brand site, a review, a marketplace, a reference page, or another source.

Then record each outcome separately: explicit recommendation, mention only, link shown to the user, opened page supporting the specific claim, observed page visit, and confirmed target action. A zero in one column must not be copied into another.

Publish each count together with the total cases: “product named in 18 of 30 runs,” “exact link shown in 9 of 30,” “page supported 6 of 9 checked links.” If a page cannot be opened, mark it “not measured,” not “unsupported.” This keeps the audit reproducible and prevents missing data from becoming a negative result.

• One row equals one prompt run in one condition.
• Keep the full response, not only the winning brand.
• “Unknown” and “not measured” are necessary honest states.

Section sources:[1] Lucas G. Uberti-Bona Marin et al. / arXiv

You can conclude that, in the selected series, answers, product sets, and visible sources were not fully stable, and that the consumer interface did not match the API on displayed domains. You can use this to design monitoring: repeat calls, store conditions, and separate reader-visible outcomes from server requests.

You cannot conclude that a particular site displaced a competitor, that publishing a page caused a recommendation, that a bot visited because of an answer, or that all services use one shared domain-selection formula. A page matching an answer can support text, but it is not a record of hidden reasoning. The links between source, mention, and conversion need separate evidence.

For a next step, start with a small control set: 20–30 prompts, one language setting, three repeats per condition, and a predefined recording scheme. Repeat the same procedure later. A change in product-name share is an observation. Its cause requires checking model version, service mode, data, prompts, and other factors that changed at the same time.

• The result describes conditions and a period, not universal AI behavior.
• Ranking position, shown link, page visit, and conversion require separate evidence.
• A repeated control is more informative than a single attractive answer.

Section sources:[1] Lucas G. Uberti-Bona Marin et al. / arXiv