Crawler

A crawler follows URLs, receives HTTP responses, and may pass content to a later system stage. One provider can operate bots for different purposes, such as search, model training, or user-initiated fetches, so the provider name alone is not a sufficient classification.

Section sources:[1] Google Search Central[2] OpenAI Developers

A crawler is not the same as a human browser, an indexer, or a retrieval component. The server sees the request, status, and headers, but usually cannot see whether the system stored the document, selected it for a query, or exposed it to a user.

Section sources:[1] Google Search Central[2] OpenAI Developers

A request with the OAI-SearchBot User-Agent and HTTP 200 shows that a client using that header received the page. Calling it a verified OpenAI bot also requires the official address-verification procedure; even then, the event remains a fetch rather than a citation.

Record time, URL, status, response size, User-Agent, and a privacy-preserving network identifier. Check the provider's official bot list, CIDR data, or forward-confirmed reverse DNS. Classify spoofed or unverified automation separately.

No visit in a short window does not mean a system is unaware of the page; it may have fetched it earlier, through another URL, or through another component. A visit also proves neither storage, positive quality assessment, nor answer use.