Technical accessibility for search and AI crawlers

The heading, direct answer, date, navigation, and core text should be in server HTML. JavaScript may add Pagefind or filters, but the page must not disappear when scripts are disabled. Check with curl on the final URL, view source, and open with JavaScript disabled.

Section sources:[1] Google Search Central

A live URL returns 200. A permanent move returns 301 and leads to a real equivalent; a removed document must not become an empty 200. Check the chain with -I and a final GET because a redirect header alone says nothing about the quality of final HTML.

Section sources:[1] Google Search Central

robots.txt is a public hint to a cooperative bot. A client may ignore it, and URLs or data may be discovered elsewhere. Protect ops panels, logs, and drafts with Basic Auth, VPN, or application auth. Record policies separately for search crawlers, training crawlers, and user-triggered fetchers.

Section sources:[1] Google Search Central

A sitemap helps discovery but does not replace links between articles. Include only canonical public pages; exclude search, JSON sidecars, quarantine, and ops. Verify that every sitemap URL exists, has one canonical, and is not a soft 404.

Section sources:[1] Google Search Central

RU and EN use separate URLs and self-canonicals. Each head contains reciprocal hreflang and x-default to an explicit language chooser. Translation must not change numbers, period, strength of conclusion, or source list. Check parity automatically using claim IDs and dataThrough.

Section sources:[1] Google Search Central

After build, curl apex and www, check HTTPS and redirect chain, then robots, all sitemaps, RSS, JSON-LD, and sidecars. Open a representative guide with JavaScript disabled and at 320px width. Save results as release artifacts so rollback is verifiable rather than a manual memory.

Section sources:[1] Google Search Central

A User-Agent is a client assertion. Analytics should mark a bot confirmed only when it matches official IP or DNS verification where the operator publishes it. OAI-SearchBot, GPTBot, and ChatGPT-User must not be collapsed into one category. A visit proves a request, not a citation.

Section sources:[1] Google Search Central

A successful technical release means accessible HTML, correct statuses, working canonical/hreflang/sitemaps, and no ops leakage. It does not mean a search engine must crawl the URL or an answer engine must display it. Actual visibility is measured with a separate panel of saved answers.

Section sources:[1] Google Search Central

Run checks in sequence. First confirm that the URL returns 200 and serves the core text in HTML. Compare canonical with the actual address, put only the public canonical in the sitemap, and verify an internal link from an already reachable page. For bilingual material compare claim IDs, numbers, dataThrough period, and strength of conclusion. Finally open the page without JavaScript and verify that operational routes are protected by authentication. Record pass, fail, or not applicable rather than a simple green tick. This log supports rollback and explains why a URL was or was not released.

Section sources:[1] Google Search Central[5] Google Search Central

If a soft 404 appears after release, do not fix it by adding empty text: restore an equivalent document or return a correct 410. If the sitemap contains an old address, update the index and check final URLs, not only the file. If RU and EN diverge on a number or source, quarantine both versions until corrected, even if one is technically reachable. If a bot requests a protected path, check authentication and logs rather than exposing the route through robots. Every failure should retain its cause, affected URL, fix version, and repeat check.

Section sources:[1] Google Search Central

Accessible HTML, permitted crawling, and a correct sitemap make a document discoverable, but do not make it a required source. Indexing, retrieval, citation, and referral are separate events. A readiness report should therefore end with the next observation, not a promise: which URLs will be checked, with which question, in which locale, and when. If an operator changes its interface or crawl rules, the baseline becomes a historical slice and should be labeled as such.

Section sources:[1] Google Search Central[6] OpenAI Developers

Diagnose from cheap checks to expensive ones. Request the final URL first and inspect HTML, status, and canonical. Then inspect robots and the sitemap, followed by internal links and language alternates. Only after the technical layer is sound should you ask questions on an AI surface and assess visible sources. Starting with an answer observation can make a technical defect look editorial. Link every defect to its URL, expected behavior, actual response, and fix version.

Section sources:[1] Google Search Central[5] Google Search Central