How to test whether source identity changes an AI agent's choice

Displayed source identity means the URL, domain, or platform name that an AI agent sees beside a candidate. It is not the same as the candidate's content: its description, price, conditions, limits, and other properties. A source effect exists when the selection changes after the source label changes while the candidate content remains fixed.

This distinction matters when auditing visibility in generative search because a well-written page and a familiar publisher are two different signals. The preprint supports measuring them separately, but it does not justify calling any familiar domain a guaranteed winner. Preferences in the study depended on the model, domain, and experimental conditions.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University

The editorial desk extracted the main parameters into a visible evidence record. The study used 4,822 requests: 1,500 WebShop shopping requests, 786 requests from the expanded HotelQuEST accommodation set, and 2,536 ScholarGym scholarly-search requests. Twelve models handled them in a shared agent search-and-selection setup.

These are not 4,822 people, live consumer-product sessions, or traffic measurements. They are benchmark requests from three datasets. The agents received a result title, text snippet, and URL rather than opening and reading the complete page. The evidence record therefore describes this research environment, not the whole AI-search market.

• 12 agent models across three selection domains.
• 4,822 benchmark requests: 1,500 + 786 + 2,536.
• Median inversion rates of 68% versus 2%; a 19% neutral-source starting measurement.
• Source-identity swap: fixed content, two source assignments, and both display orders.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University

A raw selection share cannot reveal a source effect: one website may supply better-matching candidates or appear in a more favorable position. The authors therefore paired candidates from different sources only when they satisfied the same explicit requirements. They rotated result lists so each candidate could be compared at the same display position.

Requirement satisfaction was judged from the title and text snippet with URLs and source names removed. This reduces the risk that the quality score already contains a domain preference. A pairwise statistical model then accounted for which other sources each domain had been compared against. The method is more informative than ranking domains by their unadjusted selection counts.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University

The paper calls a choice an inversion when the agent selects the less satisfying of two candidates. When that candidate satisfied one fewer explicit requirement but came from a preferred source, the median inversion rate across model-domain combinations was 68%. With the sources reversed, the median was 2%. For two neutral sources, the median was 19%.

The base used for these percentages is not all 4,822 requests. It is the set of pairwise comparisons in which the candidates differed by exactly one satisfied requirement and the agent selected exactly one of them. Each plotted point rested on a median of about 2,000 such comparisons. These percentages are neither click-through rates nor purchase probabilities.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University

The most useful part for practitioners is the controlled identity swap. The researchers took two candidates that satisfied the same requirements, preserved their titles and content, and changed the displayed source assignment. Each pair appeared four times: the authentic assignment in both orders and the swapped assignment in both orders.

The same content was selected more often when it displayed the preferred-source identity across all 12 models and all three domains. The effect was positive in every model-domain cell, although not every individual cell reached the same significance threshold. That is why a repeated controlled comparison is stronger than one striking example.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University

A separate experiment replayed the same result lists under three conditions: source information hidden, URL visible, or URL plus source names visible in the text. Full disclosure raised the average preference score by 5.8 percentage points for preferred sources and lowered it by 5.9 points for dispreferred sources relative to the hidden condition.

Most of the widening appeared as soon as the URL was visible: in nine of eleven models, that step accounted for more than 80% of the total widening. This replay measures what happens when source information becomes visible. The identity swap instead tests whether the source label changes selection for fixed content. A useful audit should not substitute one test for the other.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University

A practical diagnostic starts with two comparable descriptions, not with a search for a supposedly powerful domain. List the prompt's explicit requirements and independently mark which candidate satisfies each one. Freeze the exact text and its hash so the content stays identical throughout the diagnostic.

Show the pair as A-B and B-A with the original source labels. Then swap only the labels and repeat both orders. Do not run a fresh web search between these four presentations because new results would introduce another difference. This is a diagnostic for one model and setup, not a complete reproduction of the academic study.

• Original labels: A-source 1 and B-source 2; order A-B.
• Original labels: A-source 1 and B-source 2; order B-A.
• Swapped labels: A-source 2 and B-source 1; order A-B.
• Swapped labels: A-source 2 and B-source 1; order B-A.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University

For every presentation, preserve the run ID, date and time, model and version, exact prompt, search environment, both candidate texts, their order, displayed URLs, and final selection. Add the explicit requirement list and two independent satisfaction judgments. Without these fields, a source effect cannot be separated from a position effect or a difference in the description.

Report six events separately: a candidate was exposed, its source identity was visible, the candidate was selected, an exact link appeared in the answer, a visit occurred, and a downstream action occurred. The preprint studies selection inside an agent environment. It does not measure visible links, visits, or conversions, so those last three fields require different evidence.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University

If changing only the displayed source repeatedly changes the choice while content and order are balanced, it supports a source effect in that specific test. The conclusion does not automatically transfer to another model, domain, language, region, or search environment. It should not be described as a proven ranking factor.

A zero result also does not prove that source identity never matters. The sample may be too small, the chosen sources may not differ for that model, or run settings may reduce stability. Report raw selection counts, the total number of repeats, and an uncertainty interval. One answer change is an example, not a measurement.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University

This is version one of a preprint, not a peer-reviewed standard. The study uses a text-based agent environment, one search setup, three request collections, and partial use of a language-model judge. The authors validated the judge on a human-labeled sample, but that does not remove every classification error.

The defensible lesson is narrower than saying brands must publish on strong websites. Search-optimization teams should make pages complete and explicit; communications teams should build verifiable external evidence; analysts should separate content quality from the source effect. None of those actions guarantees a mention, exact link, visit, or sale.

Section sources:[1] Jonghyun Song, Haewon Park, Jeonghoon Shim, Woojung Song, and Yohan Jo; Seoul National University