{
  "@context": "https://schema.org",
  "@type": "DefinedTerm",
  "schema_version": "1.1",
  "content_item_id": "dictionary-crawler.en",
  "translation_group_id": "dictionary-crawler",
  "locale": "en",
  "type": "glossary",
  "section": "dictionary",
  "slug": "crawler",
  "title": "Crawler",
  "description": "A crawler is an automated client that requests URLs for discovery, refresh, or another documented purpose. A matching User-Agent alone is not enough to verify its identity, and a successful request proves only that the server returned a response, not indexing, retrieval, or citation.",
  "direct_answer": "A crawler is an automated client that requests URLs for discovery, refresh, or another documented purpose. A matching User-Agent alone is not enough to verify its identity, and a successful request proves only that the server returned a response, not indexing, retrieval, or citation.",
  "sections": [
    {
      "heading": "Working meaning",
      "paragraphs": [
        "A crawler follows URLs, receives HTTP responses, and may pass content to a later system stage. One provider can operate bots for different purposes, such as search, model training, or user-initiated fetches, so the provider name alone is not a sufficient classification."
      ],
      "source_ids": [
        "google-crawl",
        "openai-bots"
      ]
    },
    {
      "heading": "How to distinguish",
      "paragraphs": [
        "A crawler is not the same as a human browser, an indexer, or a retrieval component. The server sees the request, status, and headers, but usually cannot see whether the system stored the document, selected it for a query, or exposed it to a user."
      ],
      "source_ids": [
        "google-crawl",
        "openai-bots"
      ]
    },
    {
      "heading": "Example",
      "paragraphs": [
        "A request with the OAI-SearchBot User-Agent and HTTP 200 shows that a client using that header received the page. Calling it a verified OpenAI bot also requires the official address-verification procedure; even then, the event remains a fetch rather than a citation."
      ]
    },
    {
      "heading": "Testing",
      "paragraphs": [
        "Record time, URL, status, response size, User-Agent, and a privacy-preserving network identifier. Check the provider's official bot list, CIDR data, or forward-confirmed reverse DNS. Classify spoofed or unverified automation separately."
      ]
    },
    {
      "heading": "Interpretation limit",
      "paragraphs": [
        "No visit in a short window does not mean a system is unaware of the page; it may have fetched it earlier, through another URL, or through another component. A visit also proves neither storage, positive quality assessment, nor answer use."
      ]
    }
  ],
  "published_at": "2026-09-11",
  "modified_at": "2026-09-11",
  "data_through": "2026-09-11",
  "next_review_at": "2026-10-11",
  "author": "GeoAeoAle Editorial",
  "origin": "editorial",
  "publisher": "GeoAeoAle Editorial",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "canonical_url": "https://geoaeoale.com/en/dictionary/crawler/",
  "claims": [
    {
      "claim_id": "crawler-scope",
      "text": "A verified crawler visit requires a claimed User-Agent plus an official IP range or validated reverse DNS when the platform publishes such a method.",
      "status": "inference",
      "confidence": "medium",
      "source_ids": [
        "google-crawl",
        "openai-bots"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    }
  ],
  "sources": [
    {
      "source_id": "google-crawl",
      "canonical_url": "https://developers.google.com/search/docs/essentials/technical",
      "title": "Google Search technical requirements",
      "publisher": "Google Search Central",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "5fff9bd0dd8ef5fe14fdbc1b7debbf79ef3b8b6b507b21b87ac5e62b2ba25b30",
      "license": "Source terms apply",
      "visibility": "public"
    },
    {
      "source_id": "openai-bots",
      "canonical_url": "https://developers.openai.com/api/docs/bots",
      "title": "Overview of OpenAI crawlers",
      "publisher": "OpenAI Developers",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "ccbdef3018bd08dceaacb7fe0ea07a2020d25e201ab84aa44625827aac925440",
      "license": "Source terms apply",
      "visibility": "public"
    }
  ],
  "related_slugs": [
    "ai-visibility",
    "retrieval",
    "source-attribution"
  ],
  "limitations": [
    "Identity verification depends on provider-published mechanisms and complete logs. Proxies, caches, and incomplete IP lists can leave an event classified only as probable or suspicious automation."
  ],
  "corrections": []
}