Experiment: distinguishing HTML churn from documentation change

The question is: how can raw HTML churn be distinguished from a meaningful documentation change on an open URL? Raw churn changes bytes without changing the extracted claim or answer structure. A meaningful change adds, removes, or corrects content, an entity, link, date, or limitation in a way that could change reader understanding or answer extraction. Fix this definition before inspecting results.

Section sources:[2] Google Search Central

The pilot uses only public documents, such as Google Search Central pages and open crawler documentation. For each URL record UTC timestamp, HTTP status, content type, final URL, raw-body size, hash, and saved snapshot. Exclude private client pages, cookies, tokens, personal data, and internal logs. Publish the URL list and manifest together with the normalization code.

Section sources:[1] Google Search Central

Fetch HTML with an honestly identified User-Agent and store response headers separately. Check robots.txt and source terms; do not bypass restrictions. For replication retain URL, command or script, date, timezone, redirect chain, and checksum. A snapshot is usable when another researcher can fetch the same public address and understand which inputs produced the diff.

Section sources:[3] OpenAI Developers

Compare four layers: raw bytes; HTML after removing dynamic timestamps, nonces, tracking parameters, and explicit utility nodes; visible text preserving headings, lists, and links; and a structured set of entities, claims, dates, and outbound URLs. Version deterministic removal rules. Do not delete everything unfamiliar: a normalizer that erases a real sentence hides meaningful change.

Section sources:[2] Google Search Central

This item reports no numeric result: the pilot result is a reproducible way to generate candidates, not a validated proportion. For each diff, two reviewers assign noise, meaningful, or unclear and quote the changed fragment. Resolve disagreements with a third reviewer or a prewritten rule. Separately mark changes visible only in raw bytes and changes that survive every normalization layer.

Section sources:[2] Google Search Central

Without repeated collection, churn frequency and classifier accuracy cannot be estimated. Rendering, experiments, geography, rate limits, CDNs, and personalization may produce different snapshots. Removing a timestamp can wrongly remove an important date. A hash proves byte difference, not meaning. Even a meaningful documentation change does not prove that an AI surface reindexed it or changed citation behavior.

Section sources:[3] OpenAI Developers

1) Copy the manifest of public URLs. 2) Run the collector in two preassigned UTC windows. 3) Check status, redirects, and checksums. 4) Apply the pinned normalizer version. 5) Produce diffs at four layers. 6) Conduct independent manual labeling and save decisions. 7) Publish raw snapshots, normalized views, code, dependency versions, and an error log. You can then add control pages and estimate agreement, but numbers exist only after execution.

Section sources:[2] Google Search Central

Split collection into immutable steps. The collector stores the response and manifest; the parser extracts headings, text, links, and dates; the normalizer creates a rule-versioned view; the differ creates candidates; reviewers assign labels. Every step records tool version, input hash, and time. If a URL is unavailable, do not silently replace it with a new address: keep the gap visible. Re-running the same manifest should be idempotent and should not create duplicate snapshots.

Section sources:[2] Google Search Central[3] OpenAI Developers

Before analysing live changes, create a small fixture set of HTML files. Add a timestamp, nonce, and tracking parameter to one; change a sentence, heading, link, and date in another; give a third a dynamic menu. Predefine expected output: noise is removed while semantic changes remain. Test dangerous omissions as well as candidate precision. Every normalization-rule change gets a new version and a rerun of the fixture set.

Section sources:[2] Google Search Central

In the publication separate four outcomes: byte difference, post-normalization difference, manual label, and possible reader or retrieval impact. The last item cannot be inferred automatically from a diff. Show the changed fragment, URL, timestamp, and layer where it appeared. If reviewers disagree, keep unclear instead of choosing a convenient label. This format helps editors decide which page to read and verify without simulating precision the method does not have.

Section sources:[3] OpenAI Developers[4] Microsoft Learn

A minimal replication package contains the manifest, raw responses, normalized text, layered diffs, configuration, and decision log. Store a hash, timestamp, and schema version for every file. Do not replace a snapshot with a re-saved page: that is a new artifact with a new provenance chain. If the publication includes documentation excerpts, check the licence and quotation scope. List unavailable URLs, parser errors, and manual fixes. The next researcher should see not only the final picture but also where the method might have failed.

Section sources:[2] Google Search Central[3] OpenAI Developers