{
  "@context": "https://schema.org",
  "@type": "Article",
  "schema_version": "1.1",
  "content_item_id": "guide-interface-api-product-recommendation-audit.en",
  "translation_group_id": "guide-interface-api-product-recommendation-audit",
  "locale": "en",
  "type": "guide",
  "section": "knowledge",
  "slug": "interface-api-product-recommendation-audit",
  "title": "Why an API cannot replace auditing recommendations in the consumer AI interface",
  "description": "A practical protocol for comparing consumer interfaces and APIs, repeated recommendations, visible sources, visits, and conversions without treating one run as a stable result.",
  "direct_answer": "One consumer-interface answer and one run through an application programming interface (API) do not establish a stable product recommendation. Repeat the same prompt in the same consumer interface, repeat it separately through the API, record named products and visible links, and compare the resulting rates. A study of 117 mostly English physical-product prompts found that “best” framing changed across repeats for 27–40% of prompts, while interface and API conditions produced different displayed domains. That observation does not prove that the API itself caused the difference: models, settings, and user-facing interfaces also differed.",
  "sections": [
    {
      "heading": "1. What this protocol measures",
      "paragraphs": [
        "This guide is for teams trying to understand what a person sees when an AI service recommends a product. The measurement unit is not a vague visibility score but an observable recommendation under a specified setup: service, mode, country, language, date, and account state. An API request is a different measurement object. It can improve automation and repeatability, but it cannot automatically stand in for the consumer interface.",
        "Keep five events separate. A service may name a product: recommendation or mention. It may show a link: a visible source or path, not proof of a click. A person may visit the page: traffic. Only a confirmed target action is a conversion. A correct link does not prove a purchase, and a bot request does not prove that a page entered an answer.",
        "Write the reader job before running the audit: for example, “which products and sources does an ordinary user receive for the same prompt?” If the question is “which service is best,” define whether you mean product stability, factual accuracy, source quality, or post-answer action. These are different outcomes and each needs its own base count."
      ],
      "bullets": [
        "Interface and API are separate conditions.",
        "Recommendation, link, visit, and order are separate events.",
        "One screenshot is one observation, not a starting measurement rate."
      ],
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ]
    },
    {
      "heading": "2. How the primary study was designed",
      "paragraphs": [
        "The paper used 117 physical-product prompts. Each prompt was run three times in five conditions, producing 1,755 observations; 1,536 yielded an analyzable response. The three consumer-facing conditions were the ChatGPT and Gemini web interfaces and AI Overviews in Google Search; the other two were programmatic calls to ChatGPT and Gemini. The prompts were mostly English, and the browser was configured for English in the Netherlands in September 2026.",
        "The study also compared interface and API calls close in time, but that does not make them the same environment. The interface model or mode could be hidden or changed by the service; the API used specified models and parameters. Therefore “the interface differed” is an observation for these conditions. “The API caused the difference” would be causal and is not supported by the design.",
        "Do not expand the population to which the results apply. The findings concern physical products, selected languages, locations, dates, and models. They do not measure apps, services, Google Play, App Store, RuStore, Russian-language prompts, or another region. That is a boundary of the evidence, not a formatting detail."
      ],
      "bullets": [
        "117 prompts × 3 repeats × 5 conditions = 1,755 observations.",
        "1,536 observations produced an analyzable response.",
        "Physical products, mostly English prompts, the Netherlands, September 2026."
      ],
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ]
    },
    {
      "heading": "3. Why repeating a prompt is mandatory",
      "paragraphs": [
        "A single run cannot tell you whether you observed a typical result or a draw from a variable process. In the study, “best” framing differed across repeats for 27–40% of prompts depending on the interface. This does not mean the service is always random. It means a single answer does not measure stability.",
        "The authors compared sets of named products. Mean Jaccard similarity was 0.178 for ChatGPT, 0.287 for Gemini, and 0.421 for AI Overviews. The coefficient concerns product-set overlap, not clicks or product quality. Another repeat count, language, or user state may produce a different value.",
        "Predefine the repeat count and do not select the most convenient answer from a series. Store every full answer and count how often a product appeared, was called best, or received a visible link. “8 of 30 runs” is more transparent than “often.”"
      ],
      "bullets": [
        "Stability belongs to a series, not one screenshot.",
        "This overlap score compares sets; it does not explain causes.",
        "Store inconvenient repeats too."
      ],
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ]
    },
    {
      "heading": "4. A recommendation and a source are different events",
      "paragraphs": [
        "An answer may name a product without a link, show a link without recommending the product, or name a product and link to a page that does not support the wording. The audit table therefore needs separate fields: product named, brand named, “best” framing, visible link, exact destination, and whether the page supports the claim.",
        "Across repeated answers, displayed-domain overlap was 26.0% for ChatGPT, 29.8% for Gemini, and 45.9% for AI Overviews. Exact-page overlap was 18.1%, 28.4%, and 41.4%. These are visible-source measurements, not search rank or traffic. Domain overlap is also not content overlap: two pages on one domain may support different claims.",
        "Mark “source shown,” “page supports claim,” and “user visited” separately. If the service does not expose returned-but-hidden pages, record unknown. Do not replace unknown with a log-based guess or similar wording."
      ],
      "bullets": [
        "A mention is not a shown source link; a shown link is not a visit.",
        "A displayed domain and an exact page are different units.",
        "Unknown is better than an invented source origin."
      ],
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ]
    },
    {
      "heading": "5. How to compare a consumer interface with an API",
      "paragraphs": [
        "Run two parallel protocols. For the consumer interface, record the visible mode, sign-in state, model when known, language, country, date, full answer, and every link. For the API, record the specific method, model, settings, client-library version, system instruction when applicable, randomness settings, and the full response. Do not collapse either record into “AI answered.”",
        "The study found mean displayed-domain overlap between interface and API conditions of 12.0% for ChatGPT and 14.8% for Gemini. That is a strong reason not to treat an API observation as a stand-in for what users see. But the numbers do not say why conditions differed: model, settings, tools, personalization, output format, and snapshot timing are all plausible explanations.",
        "Compare like with like. First estimate the share of runs naming the exact product, then runs showing a visible link, then links whose page supports the claim. If model and interface change together, call the result a comparison of two conditions rather than a one-cause experiment."
      ],
      "bullets": [
        "An API improves repeatable calls but does not automatically copy the interface.",
        "Store configuration beside every answer.",
        "Do not make a causal claim from an interface difference."
      ],
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ]
    },
    {
      "heading": "6. A working audit table",
      "paragraphs": [
        "Give every run its own row. Record the prompt identifier and repeat number, date and time, language, country, consumer interface or API, visible mode, known model, full answer, named products and brands, recommendation wording, displayed addresses, and page status. For each address, note whether it is an exact product page, a brand site, a review, a marketplace, a reference page, or another source.",
        "Then record each outcome separately: explicit recommendation, mention only, link shown to the user, opened page supporting the specific claim, observed page visit, and confirmed target action. A zero in one column must not be copied into another.",
        "Publish each count together with the total cases: “product named in 18 of 30 runs,” “exact link shown in 9 of 30,” “page supported 6 of 9 checked links.” If a page cannot be opened, mark it “not measured,” not “unsupported.” This keeps the audit reproducible and prevents missing data from becoming a negative result."
      ],
      "bullets": [
        "One row equals one prompt run in one condition.",
        "Keep the full response, not only the winning brand.",
        "“Unknown” and “not measured” are necessary honest states."
      ],
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ]
    },
    {
      "heading": "7. What the evidence can and cannot say",
      "paragraphs": [
        "You can conclude that, in the selected series, answers, product sets, and visible sources were not fully stable, and that the consumer interface did not match the API on displayed domains. You can use this to design monitoring: repeat calls, store conditions, and separate reader-visible outcomes from server requests.",
        "You cannot conclude that a particular site displaced a competitor, that publishing a page caused a recommendation, that a bot visited because of an answer, or that all services use one shared domain-selection formula. A page matching an answer can support text, but it is not a record of hidden reasoning. The links between source, mention, and conversion need separate evidence.",
        "For a next step, start with a small control set: 20–30 prompts, one language setting, three repeats per condition, and a predefined recording scheme. Repeat the same procedure later. A change in product-name share is an observation. Its cause requires checking model version, service mode, data, prompts, and other factors that changed at the same time."
      ],
      "bullets": [
        "The result describes conditions and a period, not universal AI behavior.",
        "Ranking position, shown link, page visit, and conversion require separate evidence.",
        "A repeated control is more informative than a single attractive answer."
      ],
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ]
    }
  ],
  "published_at": "2026-09-29",
  "modified_at": "2026-10-01",
  "data_through": "2026-09-16",
  "next_review_at": "2026-10-31",
  "author": "GeoAeoAle Editorial",
  "origin": "editorial",
  "topics": [
    "recommendation-audit",
    "consumer-interface",
    "api-comparison",
    "repeatability",
    "source-attribution"
  ],
  "publisher": "GeoAeoAle Editorial",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "canonical_url": "https://geoaeoale.com/en/knowledge/interface-api-product-recommendation-audit/",
  "claims": [
    {
      "claim_id": "study-scope",
      "text": "The authors tested 117 physical-product prompts three times in five conditions: 1,755 observations, of which 1,536 produced an analyzable response.",
      "status": "observed",
      "confidence": "high",
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    },
    {
      "claim_id": "recommendation-framing",
      "text": "At least one product or brand was recommended in 700 of 836 consumer-interface responses. Among those product-recommending responses, first-person recommendation shares were 79% for ChatGPT, 7% for Gemini, and 2% for AI Overviews.",
      "status": "observed",
      "confidence": "high",
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    },
    {
      "claim_id": "repeat-instability",
      "text": "Best framing differed across repeats for 27–40% of prompts by interface; mean product-set Jaccard similarity was 0.178 for ChatGPT, 0.287 for Gemini, and 0.421 for AI Overviews.",
      "status": "observed",
      "confidence": "high",
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    },
    {
      "claim_id": "source-instability",
      "text": "Displayed-domain overlap across repeats was 26.0% for ChatGPT, 29.8% for Gemini, and 45.9% for AI Overviews; exact-page overlap was 18.1%, 28.4%, and 41.4% respectively.",
      "status": "observed",
      "confidence": "high",
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    },
    {
      "claim_id": "interface-api",
      "text": "Mean displayed-domain overlap between interface and API conditions was 12.0% for ChatGPT and 14.8% for Gemini in this study.",
      "status": "observed",
      "confidence": "high",
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    },
    {
      "claim_id": "separate-conditions",
      "text": "Interface and API should be treated as separate audit conditions, not as a clean test of API causation: the study used different models and settings and does not identify a causal mechanism.",
      "status": "inference",
      "confidence": "high",
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    },
    {
      "claim_id": "audit-protocol",
      "text": "A reliable audit counts product recommendation, brand mention, visible link, page visit, and conversion separately; none proves the next event by itself.",
      "status": "inference",
      "confidence": "high",
      "source_ids": [
        "arxiv-ai-product-recommendations-2026-09-16"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    }
  ],
  "sources": [
    {
      "source_id": "arxiv-ai-product-recommendations-2026-09-16",
      "canonical_url": "https://arxiv.org/html/2609.18729v1",
      "title": "If I Had to Buy Just ONE: Galaxy S26 Ultra: Auditing AI-Generated Product Recommendations",
      "publisher": "Lucas G. Uberti-Bona Marin et al. / arXiv",
      "source_type": "primary-research",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-10-01",
      "sha256": "a522da6a1591b18e6e20077777d5ded7b15ed4c99fb706c71d5be4f266cac5be",
      "license": "CC BY-SA 4.0",
      "visibility": "public"
    }
  ],
  "related_slugs": [
    "ai-commerce-measurement-chain",
    "visible-links-search-record",
    "measuring-ai-visibility"
  ],
  "distribution_links": [
    {
      "kind": "telegram",
      "url": "https://t.me/geoaeoale/26",
      "label": "Short version on Telegram"
    }
  ],
  "limitations": [
    "This is version 1 of a preprint submitted 16 September 2026 and may change after review.",
    "The sample covers physical products, not services, apps, Google Play, App Store, or RuStore.",
    "Prompts were mostly English and the browser was configured for English in the Netherlands; transfer to Russian, another region, or another date is unproven.",
    "Interface/API differences cannot be attributed to API alone because models, modes, and parameters differed.",
    "A visible source does not measure a click, visit, order, or causal page influence.",
    "The paper's values are not market share and do not forecast future ranking or whether an AI service will show a source link."
  ],
  "corrections": []
}