{
  "@context": "https://schema.org",
  "@type": "Report",
  "schema_version": "1.1",
  "content_item_id": "research-source-change-monitoring-experiment.en",
  "translation_group_id": "research-source-change-monitoring-experiment",
  "locale": "en",
  "type": "research",
  "section": "research",
  "slug": "source-change-monitoring-experiment",
  "title": "Experiment: distinguishing HTML churn from documentation change",
  "description": "A reproducible pilot on open source snapshots without invented numeric results.",
  "direct_answer": "To determine whether documentation meaning changed, a raw HTML diff is insufficient. Snapshot an open URL, normalize noisy elements, compare extracted text and structure, then manually review candidate meaningful fragments. This item reports a methodology and pilot protocol, not a quantitative result: numeric observations must be collected by actually running the experiment.",
  "sections": [
    {
      "heading": "1. Question and signal distinction",
      "paragraphs": [
        "The question is: how can raw HTML churn be distinguished from a meaningful documentation change on an open URL? Raw churn changes bytes without changing the extracted claim or answer structure. A meaningful change adds, removes, or corrects content, an entity, link, date, or limitation in a way that could change reader understanding or answer extraction. Fix this definition before inspecting results."
      ],
      "source_ids": [
        "google-crawl"
      ]
    },
    {
      "heading": "2. Open-data sample",
      "paragraphs": [
        "The pilot uses only public documents, such as Google Search Central pages and open crawler documentation. For each URL record UTC timestamp, HTTP status, content type, final URL, raw-body size, hash, and saved snapshot. Exclude private client pages, cookies, tokens, personal data, and internal logs. Publish the URL list and manifest together with the normalization code."
      ],
      "source_ids": [
        "google-ai-search"
      ]
    },
    {
      "heading": "3. Snapshot method",
      "paragraphs": [
        "Fetch HTML with an honestly identified User-Agent and store response headers separately. Check robots.txt and source terms; do not bypass restrictions. For replication retain URL, command or script, date, timezone, redirect chain, and checksum. A snapshot is usable when another researcher can fetch the same public address and understand which inputs produced the diff."
      ],
      "source_ids": [
        "openai-bots"
      ]
    },
    {
      "heading": "4. Normalization and comparison layers",
      "paragraphs": [
        "Compare four layers: raw bytes; HTML after removing dynamic timestamps, nonces, tracking parameters, and explicit utility nodes; visible text preserving headings, lists, and links; and a structured set of entities, claims, dates, and outbound URLs. Version deterministic removal rules. Do not delete everything unfamiliar: a normalizer that erases a real sentence hides meaningful change."
      ],
      "source_ids": [
        "google-crawl"
      ]
    },
    {
      "heading": "5. Pilot result and manual truth",
      "paragraphs": [
        "This item reports no numeric result: the pilot result is a reproducible way to generate candidates, not a validated proportion. For each diff, two reviewers assign noise, meaningful, or unclear and quote the changed fragment. Resolve disagreements with a third reviewer or a prewritten rule. Separately mark changes visible only in raw bytes and changes that survive every normalization layer."
      ],
      "source_ids": [
        "google-crawl"
      ]
    },
    {
      "heading": "6. Limitations",
      "paragraphs": [
        "Without repeated collection, churn frequency and classifier accuracy cannot be estimated. Rendering, experiments, geography, rate limits, CDNs, and personalization may produce different snapshots. Removing a timestamp can wrongly remove an important date. A hash proves byte difference, not meaning. Even a meaningful documentation change does not prove that an AI surface reindexed it or changed citation behavior."
      ],
      "source_ids": [
        "openai-bots"
      ]
    },
    {
      "heading": "7. Replication and extension",
      "paragraphs": [
        "1) Copy the manifest of public URLs. 2) Run the collector in two preassigned UTC windows. 3) Check status, redirects, and checksums. 4) Apply the pinned normalizer version. 5) Produce diffs at four layers. 6) Conduct independent manual labeling and save decisions. 7) Publish raw snapshots, normalized views, code, dependency versions, and an error log. You can then add control pages and estimate agreement, but numbers exist only after execution."
      ],
      "source_ids": [
        "google-crawl"
      ]
    },
    {
      "heading": "8. A practical validation pipeline",
      "paragraphs": [
        "Split collection into immutable steps. The collector stores the response and manifest; the parser extracts headings, text, links, and dates; the normalizer creates a rule-versioned view; the differ creates candidates; reviewers assign labels. Every step records tool version, input hash, and time. If a URL is unavailable, do not silently replace it with a new address: keep the gap visible. Re-running the same manifest should be idempotent and should not create duplicate snapshots."
      ],
      "source_ids": [
        "google-crawl",
        "openai-bots"
      ]
    },
    {
      "heading": "9. How to test the normalizer",
      "paragraphs": [
        "Before analysing live changes, create a small fixture set of HTML files. Add a timestamp, nonce, and tracking parameter to one; change a sentence, heading, link, and date in another; give a third a dynamic menu. Predefine expected output: noise is removed while semantic changes remain. Test dangerous omissions as well as candidate precision. Every normalization-rule change gets a new version and a rerun of the fixture set."
      ],
      "source_ids": [
        "google-crawl"
      ]
    },
    {
      "heading": "10. Interpretation and publication",
      "paragraphs": [
        "In the publication separate four outcomes: byte difference, post-normalization difference, manual label, and possible reader or retrieval impact. The last item cannot be inferred automatically from a diff. Show the changed fragment, URL, timestamp, and layer where it appeared. If reviewers disagree, keep unclear instead of choosing a convenient label. This format helps editors decide which page to read and verify without simulating precision the method does not have."
      ],
      "source_ids": [
        "openai-bots",
        "bing-public-web"
      ]
    },
    {
      "heading": "11. Artifacts to preserve",
      "paragraphs": [
        "A minimal replication package contains the manifest, raw responses, normalized text, layered diffs, configuration, and decision log. Store a hash, timestamp, and schema version for every file. Do not replace a snapshot with a re-saved page: that is a new artifact with a new provenance chain. If the publication includes documentation excerpts, check the licence and quotation scope. List unavailable URLs, parser errors, and manual fixes. The next researcher should see not only the final picture but also where the method might have failed."
      ],
      "source_ids": [
        "google-crawl",
        "openai-bots"
      ]
    }
  ],
  "published_at": "2026-09-11",
  "modified_at": "2026-09-11",
  "data_through": "2026-09-11",
  "next_review_at": "2026-10-11",
  "author": "GeoAeoAle Editorial",
  "origin": "editorial",
  "publisher": "GeoAeoAle Editorial",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "canonical_url": "https://geoaeoale.com/en/research/source-change-monitoring-experiment/",
  "claims": [
    {
      "claim_id": "scm-html-noise",
      "text": "Raw HTML can change because of templates, build time, or utility elements without a meaning change.",
      "status": "hypothesis",
      "confidence": "medium",
      "source_ids": [
        "google-crawl"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    },
    {
      "claim_id": "scm-pilot-status",
      "text": "The reported result is a pilot methodology; no quantitative result is claimed.",
      "status": "observed",
      "confidence": "high",
      "source_ids": [
        "google-crawl",
        "openai-bots"
      ],
      "publication_status": "public",
      "confidentiality": "public"
    }
  ],
  "sources": [
    {
      "source_id": "google-ai-search",
      "canonical_url": "https://developers.google.com/search/docs/appearance/ai-features",
      "title": "Top ways to ensure your content performs well in Google's AI experiences",
      "publisher": "Google Search Central",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "c6b267ed42c26ee63151c87d27de45d8882a5b7dc77c3d3dece510b21d63c300",
      "license": "Source terms apply",
      "visibility": "public"
    },
    {
      "source_id": "google-crawl",
      "canonical_url": "https://developers.google.com/search/docs/essentials/technical",
      "title": "Google Search technical requirements",
      "publisher": "Google Search Central",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "5fff9bd0dd8ef5fe14fdbc1b7debbf79ef3b8b6b507b21b87ac5e62b2ba25b30",
      "license": "Source terms apply",
      "visibility": "public"
    },
    {
      "source_id": "openai-bots",
      "canonical_url": "https://developers.openai.com/api/docs/bots",
      "title": "Overview of OpenAI crawlers",
      "publisher": "OpenAI Developers",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "ccbdef3018bd08dceaacb7fe0ea07a2020d25e201ab84aa44625827aac925440",
      "license": "Source terms apply",
      "visibility": "public"
    },
    {
      "source_id": "bing-public-web",
      "canonical_url": "https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/generative-ai-public-websites",
      "title": "Generative answers over public websites",
      "publisher": "Microsoft Learn",
      "source_type": "official",
      "locale": "en",
      "published_at": null,
      "checked_at": "2026-09-11",
      "sha256": "a35496cfe1965d3f1f1862cda5855d6f4825c8316b1ccb5c38722916e94e96d2",
      "license": "Source terms apply",
      "visibility": "public"
    }
  ],
  "related_slugs": [
    "measuring-ai-visibility",
    "evidence-content-entities-citability",
    "how-ai-search-finds-sources"
  ],
  "limitations": [
    "This is a pilot protocol with no claimed numeric results; public-source availability and terms may change."
  ],
  "corrections": []
}