{ "@context": "https://schema.org", "@type": "Article", "schema_version": "1.1", "content_item_id": "guide-visible-links-search-record.en", "translation_group_id": "guide-visible-links-search-record", "locale": "en", "type": "guide", "section": "knowledge", "slug": "visible-links-search-record", "title": "Why links under an AI answer do not reveal the full search record", "description": "A practical way to distinguish visible source links, pages returned to an AI service, and unsupported claims without presenting the visible answer as a complete activity log.", "direct_answer": "No. The links shown under an AI answer are not a complete record of the service's search activity. A primary study of records from ChatGPT, Claude, and Grok found that some claims were supported by pages returned during search but never linked in the answer. Across two study settings, such cases covered 14% to 53% of all claims. This does not let anyone guess the hidden source behind a particular live answer, and it does not prove that a page affected rank. Use separate fields for the claim text, the link shown to the reader, any returned page present in an available search record, and the resulting site visit. When the internal record is unavailable, mark that part as unknown. No visible link means only that no link was shown; it does not prove that the service did not search, use a page, or already know the information.", "sections": [ { "heading": "1. Separate what was shown, returned, and left unknown", "paragraphs": [ "An ordinary reader sees a finished answer and a few source links. That is the first evidence layer: the source name, page address, and the statement beside which the link appears. It lets a reviewer check whether the open page supports the nearby statement. It says nothing about pages that the service may have found, read, and then omitted. Saying that the system used only the displayed sources is too strong unless the researcher has a separate search record.", "A second layer exists only when a researcher has a record of the searches and returned pages. It may contain more page addresses than the finished answer and allows the content of each page to be compared with individual claims. A third layer stays unknown: private selection rules, internal scores, training information, and steps that were never included in the accessible record. A useful audit keeps this gap as an unknown value instead of filling it with a guess." ], "bullets": [ "Shown to the reader: answer text and visible source links.", "Available to the researcher: search requests and returned pages, when the service included them in an export.", "Unknown: internal weights, every page considered, and the exact reason for a choice." ], "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ] }, { "heading": "2. What the researchers actually studied", "paragraphs": [ "The paper dated 16 September 2026 combines two observation methods. First, the authors used conversations voluntarily donated by users. The set contains 171,264 conversations from 613 people across ChatGPT, Claude, Grok, and DeepSeek. The exports included prompts, search requests, returned pages, displayed links, and answers. This gives the researchers more evidence than a finished interface, but the set still depends on the people who agreed to donate their data.", "Second, the same set of 1,000 de-identified prompts was sent to GPT-5.3-chat, Claude Sonnet 4.6, Grok-4.3, and DeepSeek-v4-flash through programming interfaces. Reusing the prompts reduces differences between user groups and supports comparison among the selected models under stated conditions. It remains a snapshot of the tested versions and conditions reported in the 16 September 2026 preprint. The result should not be extended to every future application version or to another language without a new test." ], "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ] }, { "heading": "3. What the 14% to 53% range means", "paragraphs": [ "The authors split answers into individual factual claims and checked whether page text supported each one. They first examined the source link beside a claim, then other links elsewhere in the same answer, and finally pages returned during search that were never displayed. For ChatGPT, Claude, and Grok, the last group supported 14% to 53% of all claims across different services and both study settings. This is a share of claims, not a share of sites, prompts, answers, or visits.", "The range is broad because the services, models, and settings differed. It cannot become a market average or a promise for one domain. The published statement does not include DeepSeek in this range, so adding a fourth service to the number would be unsupported. The defensible conclusion is narrower and more useful: in the studied data, the visible link list did not include every returned page whose text supported an answer claim." ], "bullets": [ "Counted cases: claims supported by a returned page that was not shown.", "Total count: all extracted claims in the corresponding answer group.", "Not measured by this number: visits, sales, page rank, or the causal effect of publishing a page." ], "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ] }, { "heading": "4. Why matching text does not prove influence", "paragraphs": [ "When a returned page contains a fact similar to an answer claim, it provides textual support, but it is not a recording of the model's private reasoning. The same fact may appear on several pages, in earlier conversation turns, or in the model's learned knowledge. The authors state that they cannot see hidden reasoning, ranking formulas, or proprietary search processes. Even their unusually rich records do not fully explain the cause of every generated phrase.", "The support check was not entirely manual. An automated judge compared claims with page content, and people reviewed a random set of 30 examples. The three reviewers reached 78.2% agreement with each other; the automated result matched the combined human label in 93.1% of cases. This is useful validation, but 30 examples cannot remove every possible error. A local audit should retain an unclear category instead of forcing every row into supported or unsupported." ], "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ] }, { "heading": "5. A practical protocol for a site owner", "paragraphs": [ "A site owner normally does not receive the internal search record of ChatGPT, Claude, or Grok. A repeatable audit must therefore start with observable events. For every run, save the exact prompt, date, language, country, service mode, complete answer, and all displayed page addresses. Break the answer into short claims and open each displayed source. Record whether the page supports the nearby wording fully, partly, not at all, or whether the result is unclear.", "Server logs may sit beside this record, but they answer a different question. They can show that a program requested a page at a certain time and received a response code. They cannot show that the page entered a particular answer. When a service does not disclose the pages returned during search, the field for returned but unshown pages must stay unknown. Similar wording or a bot request at roughly the same time is not enough to fill it." ], "bullets": [ "Claim: a short answer fragment that can be checked on its own.", "Displayed source link: the page address visible to a reader in that answer.", "Support: full, partial, none, or unclear.", "Server page request: a separate event with a date and response code.", "Returned but unshown page: record it only when the service provides an accessible search record; otherwise use unknown." ], "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ] }, { "heading": "6. How to review one answer without inventing a cause", "paragraphs": [ "Imagine an answer saying, ‘The service suits small teams and keeps a change history,’ while showing only a pricing page. Split the sentence into two claims. The pricing page may support the team-size condition while saying nothing about a change history. Mark the first claim as supported by the displayed link and the second as not supported by the displayed link. That does not make the second claim fabricated; it may come from another returned page, earlier conversation context, or learned knowledge.", "If an accessible search record contains documentation for the change history, the second claim can be marked as supported by a returned but unshown page. Without that record, use ‘origin unknown.’ The two cases must remain separate in a report: a displayed link describes what the reader received, while a returned page is an additional researcher observation. Neither event alone proves a site visit or that the page caused the service to choose the product." ], "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ] }, { "heading": "7. How to compare dates and service versions", "paragraphs": [ "Use the same prompt set, language, country, mode, and repeat count in each period. Count answers that name the subject, answers that show any source link, answers that link to the target domain, and claims that the displayed page actually supports as separate outcomes. Show the underlying count beside every share, such as ‘a useful link appeared in 8 of 40 runs.’ This shows whether the reader-visible result changed without pretending to explain why it changed.", "A new service version, a different user group, an interface change, or an updated page may all change at once. ‘More links appeared after publication’ is an observation. ‘Publication caused the increase’ requires a separate test with a control page and a comparable period. The 2026 study shows why a visible-link count cannot stand for the complete search process: some supporting pages may remain outside the finished answer." ], "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ] }, { "heading": "8. What decision the evidence can support", "paragraphs": [ "A useful audit result does not have to be positive. When a displayed page supports the claim, preserve the answer and continue monitoring. When the link does not support the wording, record a source-quality problem. When the brand is named without a link, record a mention without a visible source. When a bot requests the page but the page does not appear in an answer, keep the two events independent. This record suggests the next test without creating an invented cause.", "The editorial lesson is not to guess private rules. A page should name its subject clearly, separate facts from advice, state the relevant period, and place primary evidence beside important claims. That makes the page easier for people and systems to check, but it does not promise a link. The Amani et al. study supports a narrower point: visible links are incomplete as a description of search activity. It does not reveal ranking formulas, give site owners access to hidden returned pages, or guarantee visits after an edit." ], "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ] } ], "published_at": "2026-09-25", "modified_at": "2026-09-27", "data_through": "2026-09-27", "next_review_at": "2026-10-27", "author": "GeoAeoAle Editorial", "origin": "editorial", "topics": [ "source-links", "search-records", "answer-audit", "measurement" ], "publisher": "GeoAeoAle Editorial", "license": "https://creativecommons.org/licenses/by/4.0/", "canonical_url": "https://geoaeoale.com/en/knowledge/visible-links-search-record/", "claims": [ { "claim_id": "visible-links-incomplete-record", "text": "Links shown in an answer reveal only the reader-visible part of the service's activity and should not be described as a complete record of found or used pages.", "status": "inference", "confidence": "high", "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ], "publication_status": "public", "confidentiality": "public" }, { "claim_id": "study-observational-scope", "text": "The study examined 171,264 user-donated conversations from 613 participants across ChatGPT, Claude, Grok, and DeepSeek.", "status": "observed", "confidence": "high", "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ], "publication_status": "public", "confidentiality": "public" }, { "claim_id": "study-controlled-scope", "text": "The authors also sent the same set of 1,000 de-identified prompts to four selected models through the services' programming interfaces.", "status": "observed", "confidence": "high", "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ], "publication_status": "public", "confidentiality": "public" }, { "claim_id": "uncited-returned-support", "text": "For ChatGPT, Claude, and Grok, pages returned in search but not linked in the answer supported 14% to 53% of all extracted claims across the observational and controlled settings; the authors did not apply that range to DeepSeek.", "status": "observed", "confidence": "high", "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ], "publication_status": "public", "confidentiality": "public" }, { "claim_id": "entailment-validation", "text": "On a validation set of 30 randomly selected examples reviewed by three annotators, the automated support judgment agreed with the combined human label in 93.1% of cases, while agreement among people was 78.2%.", "status": "observed", "confidence": "high", "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ], "publication_status": "public", "confidentiality": "public" }, { "claim_id": "three-layer-audit", "text": "A practical audit should separate three layers: what the reader can see, what an available search record contains, and what remains unknown.", "status": "inference", "confidence": "high", "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ], "publication_status": "public", "confidentiality": "public" }, { "claim_id": "no-ranking-or-causal-proof", "text": "The study does not disclose ranking formulas or prove that changing a page will cause a display, mention, source link, or site visit.", "status": "inference", "confidence": "high", "source_ids": [ "arxiv-agentic-search-lifecycle-2026-09-16" ], "publication_status": "public", "confidentiality": "public" } ], "sources": [ { "source_id": "arxiv-agentic-search-lifecycle-2026-09-16", "canonical_url": "https://arxiv.org/html/2609.19244v1", "title": "Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses", "publisher": "Mahsa Amani et al. / arXiv", "source_type": "primary-research", "locale": "en", "published_at": null, "checked_at": "2026-09-27", "sha256": "7c33c4399a2f53d35fc6dcc447d177c4e03af7b5333df9b9ef7dd6b5027a1b10", "license": "CC BY 4.0", "visibility": "public" } ], "related_slugs": [ "measuring-ai-visibility", "ai-citation", "source-attribution", "chatgpt-search", "claude-search" ], "distribution_links": [ { "kind": "telegram", "url": "https://t.me/geoaeoale/24", "label": "Short version on Telegram" } ], "limitations": [ "The work is version 1 of a preprint submitted on 16 September 2026 and may change after review.", "Donated conversations do not represent every user, country, language, subscription tier, or service mode.", "The controlled study covers four named models and one set of 1,000 prompts; later versions may behave differently.", "The 14% to 53% range applies to claims in ChatGPT, Claude, and Grok and must not be extended to DeepSeek or the whole market.", "The automated support check was validated on 30 randomly selected examples, so some classification errors remain possible.", "The protocol separates observable events but cannot expose the internal search process or guarantee rank, a source link, or a site visit." ], "corrections": [] }