{
  "@context": "https://schema.org",
  "@type": "DefinedTerm",
  "schema_version": "1.1",
  "content_item_id": "dictionary-robots-txt.en",
  "translation_group_id": "dictionary-robots-txt",
  "locale": "en",
  "type": "glossary",
  "section": "dictionary",
  "slug": "robots-txt",
  "title": "robots.txt",
  "description": "Which requests robots.txt governs and why it does not protect private data.",
  "direct_answer": "robots.txt is a public rule file at /robots.txt where a site owner declares directives for named User-Agents and paths. It governs voluntarily compliant crawling but is not authorization, does not guarantee URL removal from an index, and does not conceal the address.",
  "sections": [
    {
      "heading": "Working meaning",
      "paragraphs": [
        "A robot may first request /robots.txt, select a rule group for its User-Agent, and apply Allow or Disallow to URL paths. Behavior when the file is unavailable, supported directives, and conflict handling are defined by the named crawler's documentation."
      ],
      "source_ids": [
        "google-robots",
        "openai-bots"
      ]
    },
    {
      "heading": "How to distinguish",
      "paragraphs": [
        "robots.txt concerns crawling, meta robots and X-Robots-Tag concern handling of a particular response, and login or network controls govern access. A sitemap can be declared in robots.txt but remains a separate URL-discovery mechanism."
      ],
      "source_ids": [
        "google-robots",
        "openai-bots"
      ]
    },
    {
      "heading": "Example",
      "paragraphs": [
        "The rule Disallow: /private/ asks the selected crawler not to request that path. A person or noncompliant bot can still open the URL when the server requires no authentication, so putting a secret document there is unsafe."
      ]
    },
    {
      "heading": "Testing",
      "paragraphs": [
        "Open the file as plain text and inspect its HTTP status, syntax, User-Agent groups, and path matching for every important crawler. Then request protected routes externally and confirm that authentication works independently of the file."
      ]
    },
    {
      "heading": "Interpretation limit",
      "paragraphs": [
        "An allow rule does not make a crawler visit, while a disallow rule does not remove a previously known URL from every system. Bots support different directives, so rules for one User-Agent cannot automatically be generalized to another."
      ]
    }
  ],
  "published_at": "2026-09-11",
  "modified_at": "2026-09-11",
  "data_through": "2026-09-11",
  "next_review_at": "2026-10-11",
  "author": "GeoAeoAle Editorial",
  "origin": "editorial",
  "publisher": "GeoAeoAle Editorial",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "canonical_url": "https://geoaeoale.com/en/dictionary/robots-txt/",
  "claims": [
    {
      "claim_id": "robots-txt-scope",
      "text": "A private route must be protected with authentication; a robots.txt disallow rule only asks compliant crawlers not to fetch the path.",
      "status": "inference",
      "confidence": "medium",
      "source_ids": [
        "google-robots",
        "openai-bots"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    }
  ],
  "sources": [
    {
      "source_id": "google-robots",
      "canonical_url": "https://developers.google.com/search/docs/crawling-indexing/robots/intro",
      "title": "Introduction to robots.txt",
      "publisher": "Google Search Central",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "549549689880625483238eea25463220d4ca54d96bfde30360343966cb4ac6ea",
      "license": "Source terms apply",
      "visibility": "public"
    },
    {
      "source_id": "openai-bots",
      "canonical_url": "https://developers.openai.com/api/docs/bots",
      "title": "Overview of OpenAI crawlers",
      "publisher": "OpenAI Developers",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "ccbdef3018bd08dceaacb7fe0ea07a2020d25e201ab84aa44625827aac925440",
      "license": "Source terms apply",
      "visibility": "public"
    }
  ],
  "related_slugs": [
    "ai-visibility",
    "retrieval",
    "source-attribution"
  ],
  "limitations": [
    "Before changing the file, check the official documentation for each named crawler and the effect on existing search features. A mistake in a broad User-Agent group can unintentionally block useful crawling."
  ],
  "corrections": []
}