Weekly GEO research: crawl access is several different policies

This issue examines the practical mistake of allowing all AI bots or blocking all AI bots. A public site may be relevant to search, answer engines, training collection, agents, advertising systems, and monitoring, each with different purposes and policies. OpenAI and Google documentation name and describe some crawler classes but do not reveal the full path from crawl to answer. The result here is a decision matrix, not a promise of indexing.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central

OpenAI describes search, user-requested, and training scenarios separately. A site owner can therefore make different choices: allow a search crawler for discovery, disallow a training collector, restrict agent fetch, or leave it open for a specific experience. A User-Agent string alone is insufficient: verify published IP ranges or reverse DNS where the operator provides them, and do not trust an arbitrary header. Policies should be versioned and explain the purpose of every rule.

Section sources:[1] OpenAI[2] OpenAI Developers

Google describes robots.txt as a way to tell crawlers which paths they may crawl. It is not a password, does not remove a previously known URL, and does not guarantee that a search system will not retain a headline or link. A disallowed path may still be available to an ordinary user when no authorization exists. Private material needs server-side access control. Public pages should be checked with HTTP status, canonical, sitemap, noindex, and observed crawling. Otherwise one robots.txt line is incorrectly presented as a complete visibility policy.

Section sources:[3] Google Search Central[4] Google Search Central

Four decisions should not be combined into one switch. Search crawling concerns discovery and index signals; training concerns a separate data-use policy or license; user fetch concerns a specific human request; an advertising bot concerns ad placement evaluation. Allowing one class does not mean the site will be cited. Blocking another does not mean the domain disappears from every surface. Public documentation should show a table of purpose, crawler, permission, and what the permission does not prove.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

Testing starts with a URL inventory and expected statuses. For each operator, preserve User-Agent, date, IP verification, URL, HTTP code, redirect chain, response size, and blocking directives. Test the page separately in an ordinary browser and through an authorized test path. Do not send private snapshots or client URLs into a public pipeline. If a CDN returns a challenge, that is an access observation, not proof of an operator-level prohibition. Repeat after every policy change and preserve the diff.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

The most dangerous error is treating no observed visit as proof of no AI presence. Other errors include trusting a User-Agent without IP verification, treating robots.txt permission as a crawl guarantee, mixing OpenAI SearchBot with GPTBot, exposing a private source through a sidecar, and using one score for technical access and observed citation. Every statement needs a level: observed, inference, or hypothesis. When data is insufficient, the correct value is unknown, not zero.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

GeoAeoAle’s public policy should be explicit: list access classes, purpose, technical rule, and the boundary of the conclusion. Content can be available to search and users without exposing private artifacts or promising citation. Review is required when documentation, IP ranges, CDN policy, or crawler classes change. October 11, 2026 is a control point, not decorative freshness.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

The registry should store more than allow or deny: it needs the rule’s purpose, check date, and observed result. For one URL, keep separate rows for a search crawler, user-requested fetch, and training collection; record whether the operator was verified through published IP ranges or reverse DNS. When a CDN, WAF, or robots.txt changes, do not overwrite the old record without a history. This makes it possible to compare intended policy with actual HTTP access and prevents a loud conclusion based on one header. In a public article, the scheme turns a technical decision into a reproducible procedure: readers can see what was tested, when, how, and which conclusion remains unavailable.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

Before publication, an editor should answer four questions. Which access purpose is being described: search, training, a user request, an agent, or advertising? Which operator and path are supported by documentation? What was actually observed in HTTP or the interface, and what remains unknown? What review date is needed if policy or infrastructure changes? Record the answers beside the claim rather than hiding them in a technical appendix. This protects readers from a common mistake: turning robots.txt permission into a promise of indexing or treating one failed request as proof that a page is absent everywhere. It also makes the weekly issue useful to an engineer: the text yields a test, an expected result, and a stopping condition.

Section sources:[1] OpenAI[2] OpenAI Developers[3] Google Search Central[4] Google Search Central

Public documentation describes access boundaries but does not reveal the entire internal path used to select sources. A responsible publication therefore records the exact wording, check date, and scope of each document. Where the documentation is silent, we use unknown and propose a reproducible test. This keeps the issue useful without turning an assumption about crawler behavior into a fact.