Why an AI-bot visit is not a citation

The question is: which part of the path from a web request to an AI answer can a site owner observe? We use fetch → index signal → retrieval → mention → citation → referral. This is a measurement framework, not a claim about private operator architecture. The study synthesizes public OpenAI, Perplexity, and Google documents. There is no matched answer sample across systems, so it provides no ranking, share, or causal effect. Its purpose is to show which artifact is required before making each stronger claim.

The practical meaning of the boundary is that a visit log cannot answer why a page was cited or not cited. Each stage therefore needs its own artifact and confidence level. We record access rules where an operator publishes them and a displayed link where a user can see one. We do not turn private stages into a page-quality score. This helps both editors and analysts: it identifies the next check to run instead of hiding uncertainty behind one composite number.

Section sources:[1] OpenAI Developers

Fetch is an HTTP request and server response. An index signal is an observable indication of discovery or search eligibility, such as a console record; a log does not create one. Retrieval is the private selection of a document for an answer and is usually unavailable. Mention means a saved answer names a page, brand, or entity. Citation means the answer contains a verifiable link or attribution to that source. Referral means a user arrived through the link or referrer. Events may align in time, but they do not prove one another: a crawler visit does not become a citation.

Treat these events as independent observation surfaces rather than a linear funnel. A fetch may precede an index update; a citation may appear without a fresh visible fetch; a referral depends on human behavior after the answer. This does not mean there is no relationship, but it forbids a causal claim without a matched test. Every number should name the operator, URL, period, collection method, and event definition. Otherwise the single word “visibility” mixes technical access with a user outcome.

Section sources:[2] Perplexity Docs

A fetch needs an access-log event with time, URL, status, and response size. A verified bot needs a matching User-Agent, official IP range, and saved list version. Perplexity publishes separate PerplexityBot and Perplexity-User roles and endpoints; OpenAI points to an official JSON range list. A citation needs a full-answer archive or screenshot with visible link, URL, and time. A referral needs an independent analytics or server visit. Suitable public proof of retrieval is usually absent: record unknown rather than deriving it from a graph.

The matrix works only when original materials are preserved. A log needs the request line, status, and classifier version; a console needs a dated capture or export; a citation needs the full answer and exact URL; a referral needs an independent visit record. Link every file to a claim_id and page version. If an artifact cannot be reopened, its strength falls. This matters during corrections: new copy must not silently rewrite the meaning of an earlier observation.

Section sources:[3] Google Search Central

Start by distrusting the User-Agent: it is easy to spoof. Preserve the raw string, extract the IP, and check it against the operator’s current official JSON. When rDNS is used, perform reverse DNS and forward-confirmed reverse DNS: the name should point to the expected domain and the reverse record should return the original IP. Record time, list version, and outcome. Do not merge ranges for different roles: user-triggered fetching and search crawling are distinct signals. On any mismatch, keep the classification claimed or unknown.

The verification result should be reproducible, not merely an analytics label. Preserve the raw request, DNS-check time, official list used, and final status. Sample several requests from one crawler and several URLs; one match does not show stability. When records disagree, do not raise confidence manually. An unknown classification is more honest than calling it a bot from one User-Agent line, and it avoids a false link between a technical visit and a future answer.

Section sources:[1] OpenAI Developers

Example: `2026-09-11T10:14:02Z | /en/research/crawler-access-vs-citation/ | 200 | UA=PerplexityBot/1.0 | IP=verified | list=perplexitybot.json@2026-09-11 | event=fetch`. It supports only “PerplexityBot fetched the HTML.” False positives include a script copying the UA, a CDN request mistaken for a visitor, a cached repeat, or an answer linking to the domain but not this page. OAI-SearchBot, GPTBot, and Perplexity-User must not be treated as interchangeable.

Build the permitted conclusion step by step. First verify request identity; then confirm that the server returned the intended URL and HTML; then independently find a matching interface answer and check whether its link points to this page. Without the second artifact, the result is fetch only. Even temporal alignment remains correlation: a crawler may fetch the page while the answer comes from another index or previously stored context.

Section sources:[1] OpenAI Developers

Define URLs and window; enable JSON logs; save official IP endpoints daily with a hash; classify unknown → claimed → verified; inspect a random sample; preregister prompts; save full answers, mode, region, and links; count fetch, mention, citation, and referral separately; publish aggregates with a clear denominator. Next test: ten matched questions, three repetitions per surface, and 30 days. When fetch and citation align, report correlation, not causation.

For a working team, create a table in advance: surface, URL, exact prompt, locale, region, mode, timestamp, HTTP event, answer, displayed source, and evidence level. Repeat the panel under the same conditions and publish absolute values beside rates. If conditions change, start a new slice instead of merging it with the old one. The method then becomes portable: another editor can repeat the procedure and see which conclusions are facts and which remain hypotheses.

Section sources:[1] OpenAI Developers

Record a result as a combination of surface, query, time, and artifact. Then ask separately whether there was a request, HTML access, a page in the answer, and a link supporting the claim. If the last answer is unknown, do not use “citation.”

Keep three report layers separate: observed is a preserved fact; inference interprets several facts; hypothesis is an explanation requiring a test. This allows updates without losing history.

Section sources:[1] OpenAI Developers[2] Perplexity Docs

For each run preserve run_id, canonical URL, page hash, surface, prompt, locale, region, mode, timestamp, raw response, parsed links, bot identity, HTTP status, source match, and decision. An empty decision means unknown, not zero.

Store the decision snapshot with its inputs: when a source changes, the earlier result remains available for comparison. This turns the audit into a versioned procedure rather than a one-off number.

Section sources:[1] OpenAI Developers[2] Perplexity Docs