robots.txt

A robot may first request /robots.txt, select a rule group for its User-Agent, and apply Allow or Disallow to URL paths. Behavior when the file is unavailable, supported directives, and conflict handling are defined by the named crawler's documentation.

Section sources:[1] Google Search Central[2] OpenAI Developers

robots.txt concerns crawling, meta robots and X-Robots-Tag concern handling of a particular response, and login or network controls govern access. A sitemap can be declared in robots.txt but remains a separate URL-discovery mechanism.

Section sources:[1] Google Search Central[2] OpenAI Developers

The rule Disallow: /private/ asks the selected crawler not to request that path. A person or noncompliant bot can still open the URL when the server requires no authentication, so putting a secret document there is unsafe.

Open the file as plain text and inspect its HTTP status, syntax, User-Agent groups, and path matching for every important crawler. Then request protected routes externally and confirm that authentication works independently of the file.

An allow rule does not make a crawler visit, while a disallow rule does not remove a previously known URL from every system. Bots support different directives, so rules for one User-Agent cannot automatically be generalized to another.