ChatGPT Search: discovery, sources, and observability limits

ChatGPT Search can combine an answer with links to web sources. OpenAI’s public documentation does not provide a deterministic appearance formula, so access, source selection, mention, and source link must be tracked as separate events.

Section sources:[1] OpenAI Developers[2] Google Search Central

OpenAI describes OAI-SearchBot as the crawler for surfacing sites in search results, GPTBot as a crawler for potential training use, and ChatGPT-User as a user-triggered page request. The roles have different purposes. robots.txt expresses policy, but permission is only a prerequisite for a request.

• Record user-agent, time, URL, status, and size.
• Separate search and training policy.
• Confirm bots with user-agent and official network data.

Section sources:[1] OpenAI Developers[2] Google Search Central

A response link is strong public evidence that a page was selected. An OAI-SearchBot visit is weaker: it proves a request, not use of the material. Experiments should preserve question, language and region, date, full answer, and distinguish direct source link from a prior visit.

Section sources:[1] OpenAI Developers[2] Google Search Central

There is no general publisher dashboard exposing every ChatGPT Search source selection or source link. Logs can classify documented roles when user-agent and published IP ranges agree. Actual AI visibility needs a separate prompt-to-answer-to-links observation set.

Section sources:[1] OpenAI Developers[2] Google Search Central

robots.txt is a policy signal, not access control. Other clients can page request public pages. Protect private material with authentication. GeoAeoAle should record policies for OAI-SearchBot, GPTBot, and ChatGPT-User separately.

Section sources:[1] OpenAI Developers[2] Google Search Central

OpenAI does not disclose the complete source selection index, ranking weights, query fan-out, thresholds, freshness windows, or link-selection reasons. A bot visit establishes none of these internal states.

Section sources:[1] OpenAI Developers[2] Google Search Central

Test the URL with JavaScript disabled, then inspect status, preferred page address, visible text, links, robots, and sitemap. Validate bot IP/rDNS. Repeat a fixed fixed prompt sets, archive answers and links, and report sample size and uncertainty.

Section sources:[1] OpenAI Developers[2] Google Search Central

Review every 30 days and whenever OpenAI documentation changes. This is a public-source operational guide, not a promise of discovery or source link.

Section sources:[1] OpenAI Developers[2] Google Search Central

A line from OAI-SearchBot confirms a request to a URL under that documented role. It does not prove that the text reached a user answer. That conclusion needs a separate prompt-to-answer-to-link record, stored with date, language and region, and the full response.

Section sources:[1] OpenAI Developers[2] Google Search Central

First record robots.txt policy separately for OAI-SearchBot, GPTBot, and ChatGPT-User. Then test ordinary HTML delivery and page access. Finally repeat the same fixed prompt sets, distinguishing a brand mention, a link to the page, and no source link.

Section sources:[1] OpenAI Developers[2] Google Search Central

For every run, preserve the exact prompt, date, language, region or interface setting, full answer, and every displayed link. Keep log time, URL, user-agent, status, and IP/rDNS verification alongside it. This supports the statements “the page was requested by OAI-SearchBot” or “the page was cited in this answer.” Do not substitute the first for the second, and do not infer ChatGPT Search ranking from GPTBot activity.

Section sources:[1] OpenAI Developers[2] Google Search Central

If an answer contains a link to the article, record it as a source link in that particular observation, not as a permanent property of the domain. If the link disappears in the next run, that is a result change, not proof that the page became “worse.” Keep no-link answers as well: they help measure observation coverage, but they do not reveal the internal reason for selection. This log makes the conclusion reviewable by another editor and shows where public observation ends and the system’s unknown internal logic begins.

Section sources:[1] OpenAI Developers[2] Google Search Central