ChatGPT Search: discovery, sources, and observability limits

ChatGPT Search can combine an answer with links to web sources. OpenAI’s public documentation does not provide a deterministic appearance formula, so access, retrieval, mention, and citation must be tracked as separate events.

Section sources:[1] OpenAI Developers[2] Google Search Central

OpenAI describes OAI-SearchBot as the crawler for surfacing sites in search results, GPTBot as a crawler for potential training use, and ChatGPT-User as a user-triggered fetch. The roles have different purposes. robots.txt expresses policy, but permission is only a prerequisite for a request.

• Record user-agent, time, URL, status, and size.
• Separate search and training policy.
• Confirm bots with user-agent and official network data.

Section sources:[1] OpenAI Developers[2] Google Search Central

A response link is strong public evidence that a page was selected. An OAI-SearchBot visit is weaker: it proves a request, not use of the material. Experiments should preserve question, locale, date, full answer, and distinguish direct citation from a prior visit.

Section sources:[1] OpenAI Developers[2] Google Search Central

There is no general publisher dashboard exposing every ChatGPT Search retrieval or citation. Logs can classify documented roles when user-agent and published IP ranges agree. Actual AI visibility needs a separate prompt-to-answer-to-links observation set.

Section sources:[1] OpenAI Developers[2] Google Search Central

robots.txt is a policy signal, not access control. Other clients can fetch public pages. Protect private material with authentication. GeoAeoAle should record policies for OAI-SearchBot, GPTBot, and ChatGPT-User separately.

Section sources:[1] OpenAI Developers[2] Google Search Central

OpenAI does not disclose the complete retrieval index, ranking weights, query fan-out, thresholds, freshness windows, or link-selection reasons. A bot visit establishes none of these internal states.

Section sources:[1] OpenAI Developers[2] Google Search Central

Test the URL with JavaScript disabled, then inspect status, canonical, visible text, links, robots, and sitemap. Validate bot IP/rDNS. Repeat a fixed prompt panel, archive answers and links, and report sample size and uncertainty.

Section sources:[1] OpenAI Developers[2] Google Search Central

Review every 30 days and whenever OpenAI documentation changes. This is a public-source operational guide, not a promise of discovery or citation.

Section sources:[1] OpenAI Developers[2] Google Search Central

A line from OAI-SearchBot confirms a request to a URL under that documented role. It does not prove that the text reached a user answer. That conclusion needs a separate prompt-to-answer-to-link record, stored with date, locale, and the full response.

Section sources:[1] OpenAI Developers[2] Google Search Central

First record robots.txt policy separately for OAI-SearchBot, GPTBot, and ChatGPT-User. Then test ordinary HTML delivery and page access. Finally repeat the same question panel, distinguishing a brand mention, a link to the page, and no citation.

Section sources:[1] OpenAI Developers[2] Google Search Central

For every run, preserve the exact prompt, date, language, region or interface setting, full answer, and every displayed link. Keep log time, URL, user-agent, status, and IP/rDNS verification alongside it. This supports the statements “the page was requested by OAI-SearchBot” or “the page was cited in this answer.” Do not substitute the first for the second, and do not infer ChatGPT Search ranking from GPTBot activity.

Section sources:[1] OpenAI Developers[2] Google Search Central

If an answer contains a link to the article, record it as a citation in that particular observation, not as a permanent property of the domain. If the link disappears in the next run, that is a result change, not proof that the page became “worse.” Keep no-link answers as well: they help measure observation coverage, but they do not reveal the internal reason for selection. This log makes the conclusion reviewable by another editor and shows where public observation ends and the system’s unknown internal logic begins.

Section sources:[1] OpenAI Developers[2] Google Search Central