Weekly AI search research: crawler access follows several policies

This issue examines the practical mistake of allowing all AI bots or blocking all AI bots. A public site may be relevant to search, answer engines, training collection, agents, advertising systems, and monitoring, each with different purposes and policies. OpenAI and Google documentation name and describe some crawler classes but do not reveal the full path from crawl to answer. The result here is a decision matrix, not a promise of indexing.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central

OpenAI describes separate crawlers for search, user-requested actions, and training-data collection. A site owner may let the search crawler discover pages, block collection for training, and decide separately whether the system may open a page for a user's request. The crawler name in a request is not enough: use published IP ranges or reverse-domain checks when the operator provides them, because anyone can copy a name into a request header. Keep a change history and explain the purpose of every rule.

Section sources:[1] OpenAI[2] OpenAI Developers

Google describes robots.txt as a way to tell crawlers which paths they may crawl. It is not a password, does not remove a previously known URL, and does not guarantee that a search system will not retain a headline or link. A disallowed path may still be available to an ordinary user when no authorization exists. Private material needs server-side access control. Public pages should be checked with HTTP status, preferred page address, sitemap, noindex, and observed crawling. Otherwise one robots.txt line is incorrectly presented as a complete visibility policy.

Section sources:[3] Google Search Central[4] Google Search Central

These four decisions should not be combined into one switch. A search crawler helps discover pages; training-data collection follows a separate policy or license; opening a page for a user relates to that specific request; an advertising crawler checks ad placement. Allowing one type does not mean the system will link to the site. Blocking another does not mean the domain disappears from every interface. Public documentation should use a simple table: purpose, crawler name, permission, and what that permission does not prove.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

Testing starts with a list of page addresses and expected response codes. For each operator, preserve the claimed crawler name, date, IP verification result, page address, server response code, redirect sequence, response size, and blocking directives. Test the page separately in an ordinary browser and through an authorized test path. Do not send private snapshots or client addresses into the public publishing process. If a content-delivery network returns a protective challenge, that is an access observation, not proof of an operator-level prohibition. Repeat the test after every policy change and preserve a list of differences.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

The most dangerous error is treating the absence of an observed visit as proof that a site is absent from AI answers. Do not trust a crawler name without verifying its IP address, treat permission in robots.txt as a crawl guarantee, mix OpenAI SearchBot with GPTBot, expose a private source in a companion JSON file, or combine technical access and actual source links in one score. Mark every statement as an observation, an inference, or a hypothesis. When evidence is insufficient, report 'no data' rather than zero.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

GeoAeoAle’s public policy should be explicit: list access classes, purpose, technical rule, and the boundary of the conclusion. Content can be available to search and users without exposing private artifacts or promising source link. Review is required when documentation, IP ranges, CDN policy, or crawler classes change. October 11, 2026 is a control point, not decorative freshness.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

The registry should store more than an allow-or-block decision. It also needs the rule's purpose, check date, and observed result. For one page address, keep separate rows for a search crawler, opening the page for a user, and training-data collection. Record whether the operator was verified through official IP ranges or a reverse-domain check. When the content-delivery network, site protection, or robots.txt changes, do not silently overwrite the old record. The history lets readers compare stated rules with actual access and see what was tested, when, how, and which conclusion is still unavailable.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

Before publication, an editor should answer four questions. Which access purpose is being described: search, training, a user request, an agent, or advertising? Which operator and path are supported by documentation? What was actually observed in HTTP or the interface, and what remains unknown? What review date is needed if policy or infrastructure changes? Record the answers beside the claim rather than hiding them in a technical appendix. This protects readers from a common mistake: turning robots.txt permission into a promise of indexing or treating one failed request as proof that a page is absent everywhere. It also makes the weekly issue useful to an engineer: the text yields a test, an expected result, and a stopping condition.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

Public documentation describes access boundaries but does not reveal the entire internal path used to select sources. A responsible publication therefore records the exact wording, check date, and scope of each document. Where the documentation is silent, we use unknown and propose a reproducible test. This keeps the issue useful without turning an assumption about crawler behavior into a fact.