Experiment: distinguishing HTML churn from documentation change

The experiment separates technical noise from a change in meaning. Noise is a difference in source code that does not change a visible claim, date, link, or document structure. A meaningful change adds, removes, or corrects a fact, named subject, link, date, or limitation and could therefore change a reader's understanding. Write this rule before reviewing results so the judgement is not adjusted to fit a convenient example.

Section sources:[2] Google Search Central

Use public documents only, such as Google Search Central pages and official bot documentation. For each page, record the UTC collection time, server response code, file type, final address after redirects, response size, checksum, and saved copy. Exclude private client material, cookies, access tokens, personal data, and internal logs. Publish the address list and a description of each saved copy together with the method.

Section sources:[1] Google Search Central

Download the page code with software that identifies itself honestly, and store the server response headers separately. Check robots.txt and the source terms; do not bypass restrictions. To repeat the experiment, retain the page address, command or script, date, time zone, redirect chain, and file checksum. A saved copy is useful when another researcher can open the same public address and understand which inputs produced the difference.

Section sources:[3] OpenAI Developers

First compare the complete files. Then repeat the comparison after removing known technical values such as build times, one-time identifiers, and tracking tags. At the third level, compare visible text while preserving headings, lists, and links. At the fourth, compare named subjects, claims, dates, and external links separately. Every removal rule should produce the same result when repeated and should have a version number. Do not remove everything unfamiliar: a rule that erases a real sentence can hide an important change.

Section sources:[2] Google Search Central

This article does not report a percentage of meaningful changes. Its result is a reproducible way to find passages that need review. Two editors independently label each difference as technical noise, a meaningful change, or unclear, and quote the changed passage. A third editor or a rule written before the review resolves disagreements. Mark differences that appear only in the raw file separately from those that remain after every cleaning step.

Section sources:[2] Google Search Central

Without repeated copies, we cannot estimate how often technical noise appears or how accurately the rule separates it from meaningful changes. Dynamic loading, page experiments, user country, request limits, delivery networks, and personalization can produce different copies of the same page. Automatically removing a time value may also erase an important date. A checksum proves that files differ, but not what the difference means. Even a confirmed documentation change does not prove that an AI service refreshed its data or changed the source link it shows.

Section sources:[3] OpenAI Developers

1) Use the published page list. 2) Save the pages in two preassigned UTC time windows. 3) Check server response codes, redirects, and file checksums. 4) Apply the same version of the cleaning rules. 5) Compare the four data levels. 6) Run an independent manual review and save the editors' decisions. 7) Publish the original copies, cleaned versions, code, dependency versions, and error log. You can then add control pages and calculate how often reviewers agree. Numbers exist only after the experiment runs.

Section sources:[2] Google Search Central

First, one program saves the page and a description of the copy. A separate step extracts headings, text, links, and dates. The third step applies cleaning rules, the fourth finds differences, and editors review them. Every step records the tool version, checksum of the input file, and run time. If a page is unavailable, do not silently substitute a new address; keep the gap visible. Repeating the same page list should not create unnecessary duplicate copies.

Section sources:[2] Google Search Central[3] OpenAI Developers

Before analysing live pages, create a small set of control files. Add a build time, one-time identifier, and tracking tag to one. Change a sentence, heading, link, and important date in another. Give a third a dynamic menu. Record the expected result in advance: technical noise disappears while changes in meaning remain. Check dangerous omissions as well as detected differences. Every rule change gets a new version number and another run against the control set.

Section sources:[2] Google Search Central

Report four results separately: whether the original files differ; whether the difference remains after cleaning; how editors judged it; and whether it could affect a reader's understanding or a system's source choice. The last point cannot be inferred automatically from a file difference. Show the changed passage, page address, time, and comparison level where it appeared. If reviewers disagree, leave the result unclear. This format identifies the page that needs review without pretending the method is more precise than it is.

Section sources:[3] OpenAI Developers[4] Microsoft Learn

The minimum package for repeating the experiment includes the page list, original server responses, cleaned text, comparison at all four levels, settings, and the editors' decision log. Store a checksum, collection time, and format version for every file. Do not replace an old copy with a newly saved page; that is a new file with a new history. If you publish a documentation excerpt, check its licence and the permitted quotation length. List unavailable pages, text-extraction errors, and manual fixes. The next researcher should see both the final result and the places where the method could have failed.

Section sources:[2] Google Search Central[3] OpenAI Developers