{
  "@context": "https://schema.org",
  "@type": "Report",
  "schema_version": "1.1",
  "content_item_id": "weekly-research-2026-08-03.en",
  "translation_group_id": "weekly-research-2026-08-03",
  "locale": "en",
  "type": "research",
  "section": "research",
  "slug": "weekly-research-2026-08-03",
  "title": "Weekly GEO research: crawl access is several different policies",
  "description": "How to separate search, training, agent, and advertising access to a site.",
  "direct_answer": "A site-access policy should describe a specific crawler and purpose. Allowing or blocking one User-Agent does not establish a general result for search indexing, training, agent fetches, or advertising.",
  "sections": [
    {
      "heading": "Research question and market",
      "paragraphs": [
        "This issue examines the practical mistake of allowing all AI bots or blocking all AI bots. A public site may be relevant to search, answer engines, training collection, agents, advertising systems, and monitoring, each with different purposes and policies. OpenAI and Google documentation name and describe some crawler classes but do not reveal the full path from crawl to answer. The result here is a decision matrix, not a promise of indexing."
      ],
      "source_ids": [
        "openai-publishers-faq-2026-08-03",
        "openai-bots",
        "google-robots"
      ]
    },
    {
      "heading": "OpenAI access classes",
      "paragraphs": [
        "OpenAI describes search, user-requested, and training scenarios separately. A site owner can therefore make different choices: allow a search crawler for discovery, disallow a training collector, restrict agent fetch, or leave it open for a specific experience. A User-Agent string alone is insufficient: verify published IP ranges or reverse DNS where the operator provides them, and do not trust an arbitrary header. Policies should be versioned and explain the purpose of every rule."
      ],
      "source_ids": [
        "openai-publishers-faq-2026-08-03",
        "openai-bots"
      ]
    },
    {
      "heading": "robots.txt and its limits",
      "paragraphs": [
        "Google describes robots.txt as a way to tell crawlers which paths they may crawl. It is not a password, does not remove a previously known URL, and does not guarantee that a search system will not retain a headline or link. A disallowed path may still be available to an ordinary user when no authorization exists. Private material needs server-side access control. Public pages should be checked with HTTP status, canonical, sitemap, noindex, and observed crawling. Otherwise one robots.txt line is incorrectly presented as a complete visibility policy."
      ],
      "source_ids": [
        "google-robots",
        "google-crawl"
      ]
    },
    {
      "heading": "Search, training, agent, and ads",
      "paragraphs": [
        "Four decisions should not be combined into one switch. Search crawling concerns discovery and index signals; training concerns a separate data-use policy or license; user fetch concerns a specific human request; an advertising bot concerns ad placement evaluation. Allowing one class does not mean the site will be cited. Blocking another does not mean the domain disappears from every surface. Public documentation should show a table of purpose, crawler, permission, and what the permission does not prove."
      ],
      "source_ids": [
        "openai-publishers-faq-2026-08-03",
        "openai-bots",
        "google-robots",
        "google-crawl"
      ]
    },
    {
      "heading": "How to test in practice",
      "paragraphs": [
        "Testing starts with a URL inventory and expected statuses. For each operator, preserve User-Agent, date, IP verification, URL, HTTP code, redirect chain, response size, and blocking directives. Test the page separately in an ordinary browser and through an authorized test path. Do not send private snapshots or client URLs into a public pipeline. If a CDN returns a challenge, that is an access observation, not proof of an operator-level prohibition. Repeat after every policy change and preserve the diff."
      ],
      "source_ids": [
        "openai-publishers-faq-2026-08-03",
        "openai-bots",
        "google-robots",
        "google-crawl"
      ]
    },
    {
      "heading": "Interpretation failures",
      "paragraphs": [
        "The most dangerous error is treating no observed visit as proof of no AI presence. Other errors include trusting a User-Agent without IP verification, treating robots.txt permission as a crawl guarantee, mixing OpenAI SearchBot with GPTBot, exposing a private source through a sidecar, and using one score for technical access and observed citation. Every statement needs a level: observed, inference, or hypothesis. When data is insufficient, the correct value is unknown, not zero."
      ],
      "source_ids": [
        "openai-publishers-faq-2026-08-03",
        "openai-bots",
        "google-robots",
        "google-crawl"
      ]
    },
    {
      "heading": "Conclusion and review",
      "paragraphs": [
        "GeoAeoAle’s public policy should be explicit: list access classes, purpose, technical rule, and the boundary of the conclusion. Content can be available to search and users without exposing private artifacts or promising citation. Review is required when documentation, IP ranges, CDN policy, or crawler classes change. October 11, 2026 is a control point, not decorative freshness."
      ],
      "source_ids": [
        "openai-publishers-faq-2026-08-03",
        "openai-bots",
        "google-robots",
        "google-crawl"
      ]
    },
    {
      "heading": "A decision for the editorial registry",
      "paragraphs": [
        "The registry should store more than allow or deny: it needs the rule’s purpose, check date, and observed result. For one URL, keep separate rows for a search crawler, user-requested fetch, and training collection; record whether the operator was verified through published IP ranges or reverse DNS. When a CDN, WAF, or robots.txt changes, do not overwrite the old record without a history. This makes it possible to compare intended policy with actual HTTP access and prevents a loud conclusion based on one header. In a public article, the scheme turns a technical decision into a reproducible procedure: readers can see what was tested, when, how, and which conclusion remains unavailable."
      ],
      "source_ids": [
        "openai-publishers-faq-2026-08-03",
        "openai-bots",
        "google-robots",
        "google-crawl"
      ]
    },
    {
      "heading": "Pre-publication checklist",
      "paragraphs": [
        "Before publication, an editor should answer four questions. Which access purpose is being described: search, training, a user request, an agent, or advertising? Which operator and path are supported by documentation? What was actually observed in HTTP or the interface, and what remains unknown? What review date is needed if policy or infrastructure changes? Record the answers beside the claim rather than hiding them in a technical appendix. This protects readers from a common mistake: turning robots.txt permission into a promise of indexing or treating one failed request as proof that a page is absent everywhere. It also makes the weekly issue useful to an engineer: the text yields a test, an expected result, and a stopping condition."
      ],
      "source_ids": [
        "openai-publishers-faq-2026-08-03",
        "openai-bots",
        "google-robots",
        "google-crawl"
      ]
    },
    {
      "heading": "Editorial replication",
      "paragraphs": [
        "Public documentation describes access boundaries but does not reveal the entire internal path used to select sources. A responsible publication therefore records the exact wording, check date, and scope of each document. Where the documentation is silent, we use unknown and propose a reproducible test. This keeps the issue useful without turning an assumption about crawler behavior into a fact."
      ]
    }
  ],
  "published_at": "2026-08-03",
  "modified_at": "2026-08-03",
  "data_through": "2026-08-03",
  "next_review_at": "2026-10-11",
  "author": "GeoAeoAle Editorial",
  "origin": "editorial",
  "topics": [
    "weekly-research"
  ],
  "publisher": "GeoAeoAle Editorial",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "canonical_url": "https://geoaeoale.com/en/research/weekly-research-2026-08-03/",
  "claims": [
    {
      "claim_id": "wr0803-policy",
      "text": "Different OpenAI crawler classes have different purposes and require separate policies.",
      "status": "observed",
      "confidence": "high",
      "source_ids": [
        "openai-publishers-faq-2026-08-03",
        "openai-bots"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    },
    {
      "claim_id": "wr0803-boundary",
      "text": "robots.txt provides crawl instructions but is not authorization or a guarantee of indexing.",
      "status": "observed",
      "confidence": "high",
      "source_ids": [
        "google-robots",
        "google-crawl"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    }
  ],
  "sources": [
    {
      "source_id": "openai-publishers-faq-2026-08-03",
      "canonical_url": "https://help.openai.com/en/articles/12627856-publishers-and-developers-faq",
      "title": "Publishers and developers FAQ",
      "publisher": "OpenAI",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "521cc5d325ad3c0c80501178613bf585b2449af4b68f100eb942b83f3ee1110a",
      "license": "Source terms apply",
      "visibility": "public"
    },
    {
      "source_id": "openai-bots",
      "canonical_url": "https://developers.openai.com/api/docs/bots",
      "title": "Overview of OpenAI crawlers",
      "publisher": "OpenAI Developers",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "ccbdef3018bd08dceaacb7fe0ea07a2020d25e201ab84aa44625827aac925440",
      "license": "Source terms apply",
      "visibility": "public"
    },
    {
      "source_id": "google-robots",
      "canonical_url": "https://developers.google.com/search/docs/crawling-indexing/robots/intro",
      "title": "Introduction to robots.txt",
      "publisher": "Google Search Central",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "549549689880625483238eea25463220d4ca54d96bfde30360343966cb4ac6ea",
      "license": "Source terms apply",
      "visibility": "public"
    },
    {
      "source_id": "google-crawl",
      "canonical_url": "https://developers.google.com/search/docs/essentials/technical",
      "title": "Google Search technical requirements",
      "publisher": "Google Search Central",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "5fff9bd0dd8ef5fe14fdbc1b7debbf79ef3b8b6b507b21b87ac5e62b2ba25b30",
      "license": "Source terms apply",
      "visibility": "public"
    }
  ],
  "related_slugs": [
    "crawler",
    "robots-txt",
    "indexability"
  ],
  "limitations": [
    "Documentation describes stated access rules but not internal source selection; a non-observation should be treated as unknown rather than zero."
  ],
  "corrections": []
}