How to compare AI search across languages without confusing language with platform

The same request intent in two languages does not create two identical experiences with only one word changed. A search engine may use a different pool of available sites, show an AI overview differently, and respond to interface language, country, account state, and collection time. Without those conditions, the phrase 'language changed the result' hides several possible causes.

The useful comparison unit is a specific platform-language condition collected under a recorded setup. Describe what the interface showed before proposing an explanation. This order keeps the result testable: another researcher knows what to repeat, and a site owner does not receive a false promise that translating a page alone will change its visibility.

• Observation: what the interface showed under recorded conditions.
• Inference: a bounded explanation across several observations.
• Hypothesis: a possible cause that still needs a separate test.

Section sources:[1] Yibo Li et al. / arXiv

The authors took 500 English requests from MS MARCO and translated them into Simplified Chinese with Google machine translation. They then tested four conditions: Baidu Chinese, Baidu English, Google Chinese, and Google English. Two missing Google records left 1,998 observations in total: 500, 500, 499, and 499.

Collection ran from 20 June to 10 July 2026 from Milan through Chrome and Playwright. The researcher was signed in to Google, whose interface was English; Baidu used a Chinese interface. Incognito mode, a virtual private network, and a proxy were not used. These details are part of the finding, not an unimportant technical footnote.

• Base: 500 user intents.
• Request languages: English and Simplified Chinese.
• Platforms: Baidu and Google.
• Total: 1,998 observations across four conditions.

Section sources:[1] Yibo Li et al. / arXiv

In the Baidu Chinese condition, an overview appeared in 410 of 500 runs, or 82%. In Baidu English it appeared in only 20 of 500 runs, or 4%. Google produced an overview in 487 of 499 Chinese runs and 484 of 499 English runs: 97.6% and 97.0%. These rates describe feature availability under the tested conditions, not answer quality.

The authors excluded Baidu English from the later topic and source analysis because twenty successful cases are not comparable with groups containing more than four hundred answers. This is an important reporting decision. Errors, feature absence, answers without visible hosts, and a measured zero cannot share one code. Show how many runs were planned, completed, and eligible for each calculation.

• Baidu Chinese: 410 of 500.
• Baidu English: 20 of 500; not treated as a full group later.
• Google Chinese: 487 of 499.
• Google English: 484 of 499.

Section sources:[1] Yibo Li et al. / arXiv

For each condition, the researchers measured how strongly exposure clustered around a small number of hosts. A Gini coefficient is zero for an even distribution and approaches one as concentration rises. Baidu Chinese scored 0.895, and the most frequent host received 27.1% of appearances. Google Chinese scored 0.500 and 6.6%; Google English scored 0.462 and 5.5%.

A different measure asks whether the lists contain the same hosts. Jaccard overlap divides the number of hosts shared by two lists by the number of all unique hosts in both. It was 0.034 for Google Chinese versus English, 0.032 for Baidu Chinese versus Google Chinese, and 0.008 for Baidu Chinese versus Google English. Low overlap says nothing about which list is better and does not identify a cause.

• Concentration shows how often leading hosts repeat.
• Overlap shows what share of unique hosts two lists have in common.
• Neither measure captures visits, sales, or support for an individual statement.

Section sources:[1] Yibo Li et al. / arXiv

For 331 intents with answers available in all three main conditions, the authors measured semantic similarity on a zero-to-one scale. The median was 0.813 for Google Chinese versus Baidu Chinese, 0.791 for Google Chinese versus Google English, and 0.701 for Google English versus Baidu Chinese. This indicates substantial textual similarity, but not equal facts or evidence.

At the same time, the aggregate visible-host lists barely overlapped. The two checks therefore cannot replace each other: similar wording does not show that the engines presented the same sites, while a shared host does not prove equal support for every statement. A practical audit needs separate fields for answer text, exact addresses shown to the reader, and whether an opened page supports the nearby statement.

• Text similarity is not a factual-accuracy check.
• A host list does not map every answer statement to a page.
• A shown link does not establish a user visit.

Section sources:[1] Yibo Li et al. / arXiv

The authors extracted hosts from links visible beside an AI overview and removed repeated appearances of the same host within one answer. They did not observe the full list of pages found, hidden search steps, or the internal reason for choosing a source. The accurate statement is therefore 'the host was shown to the reader,' not 'the system used only these sources.'

The same boundary matters to a site owner. A bot request in a server log means that software received a page at a certain time. A displayed link means that an address appeared in one interface. A brand mention, support for a statement, search position, and a human visit are four more events. They cannot be merged into one visibility number or reconstructed from one another.

• Keep the full answer and exact address of every shown page.
• Check which statement the opened page actually supports.
• When the platform's internal record is unavailable, mark the full search path as unknown.

Section sources:[1] Yibo Li et al. / arXiv

Freeze the original request set, translation rules, and allowed differences in terminology before collection. For each run, save the platform, request and interface languages, country, account state, product setting, date, full answer, and every displayed address. Repeat every request the same predeclared number of times. Give errors, empty answers, missing AI overviews, and answers without links different labels.

After collection, compare only like-for-like groups. Report overview availability, number of visible hosts, concentration of leading hosts, list overlap, answer similarity, and support for key statements separately. Give an absolute count beside every percentage, such as 41 of 50 rather than only 82%. Then list excluded runs and explain why each was left out.

• Do not change the request set between languages without recording it.
• Do not combine different product settings in one rate.
• Preserve the original text, translation, and matching rule.
• Publish conditions, exclusions, and the next test with the result.

Section sources:[1] Yibo Li et al. / arXiv

The supported conclusion is narrow: in one Milan collection, four platform-language conditions differed in AI-overview availability and visible hosts; similar answer wording did not imply similar aggregate host lists. The study is useful as a model for measuring several sides of an answer. It does not isolate language from platform, available search environment, interface, or other conditions.

It cannot establish that translating a page raises its chance of appearing, that Google is always more diverse than Baidu, or that Russian requests will reproduce these numbers. The study did not test Russian, Yandex, user visits, sales, or ranking formulas. The honest next step is a preregistered repeat in the target country and on the target platforms, preserving the request set and counting rules.

• Do not transfer the result to Russian and Yandex without a new measurement.
• Do not call an observed difference a causal language effect.
• Do not promise more links, traffic, or search visibility after translation.

Section sources:[1] Yibo Li et al. / arXiv