release: implement site registry v0.5

docs/superpowers/plans/2026-09-13-v0.4-and-beyond.md

@@ -454,71 +454,75 @@ deltas and a separately signed, cumulative emergency block feed.
 
 ### Task 3.1: Separate runtime projections from cold audit history
 
-- [ ] Keep raw cache objects immutable and content-addressed outside Git.
-- [ ] Package completed source imports into content-addressed audit bundles with
+- [x] Keep raw cache objects immutable and content-addressed outside Git.
+- [x] Package completed source imports into content-addressed audit bundles with
       manifest, record/fact indices, hashes, format version, and attribution.
-- [ ] Make a runtime generation contain selected facts, normalized projections,
+- [x] Make a runtime generation contain selected facts, normalized projections,
       active policy results, required votes/revocations, and signed bundle references.
-- [ ] Do not copy all historical records and facts into every runtime generation.
-- [ ] Add audit verification that streams referenced bundles and detects absence,
+- [x] Do not copy all historical records and facts into every runtime generation.
+- [x] Add audit verification that streams referenced bundles and detects absence,
       truncation, substitution, or mismatched attribution.
-- [ ] Add retention/checkpoint tooling that never deletes the only authenticated copy
+- [x] Add retention/checkpoint tooling that never deletes the only authenticated copy
       of evidence and produces a signed deletion/retention report.
-- [ ] Measure query latency and generation size before and after the split.
+- [x] Measure query latency and generation size before and after the split.
 
 ### Task 3.2: Add provider-scale benchmark and recovery tooling
 
-- [ ] Define repeatable small, medium, and provider-representative import profiles.
-- [ ] Record wall time, CPU time, peak RSS, compressed and expanded bytes, database
+- [x] Define repeatable small, medium, and provider-representative import profiles.
+- [x] Record wall time, CPU time, peak RSS, compressed and expanded bytes, database
       growth, facts/second, checkpoint frequency, restart time, build size, and query
       latency.
-- [ ] Interrupt imports at multiple checkpoints and prove idempotent resumption.
-- [ ] Exercise disk-full, truncated input, cache corruption, duplicate records, and
+- [x] Interrupt imports at multiple checkpoints and prove idempotent resumption.
+- [x] Exercise disk-full, truncated input, cache corruption, duplicate records, and
       interrupted generation publication.
-- [ ] Make benchmark reports name exact source snapshot hashes and hardware without
+- [x] Make benchmark reports name exact source snapshot hashes and hardware without
       committing source data.
-- [ ] Treat synthetic performance as development evidence, not provider capacity.
+- [x] Treat synthetic performance as development evidence, not provider capacity.
 
 ### Task 3.3: Harden full Wikidata ingestion and add incremental refresh
 
-- [ ] Confirm current official full and incremental formats from Wikidata documentation
+- [x] Confirm current official full and incremental formats from Wikidata documentation
       and inspected fixtures before changing the adapter.
-- [ ] Replace the assumption that every useful entity fits in one in-memory 16 MiB
+- [x] Replace the assumption that every useful entity fits in one in-memory 16 MiB
       line with bounded disk-spooling or an explicitly receipted oversized-record path.
-- [ ] Never silently skip an oversized entity that may contain a relevant fact.
-- [ ] Add incremental add/change ingestion with authenticated base snapshot identity,
+- [x] Never silently skip an oversized entity that may contain a relevant fact.
+- [x] Add incremental add/change ingestion with authenticated base snapshot identity,
       ordered application, checkpoints, and reconciliation against later full dumps.
-- [ ] Define how deletions and removed P856 statements become tombstones.
-- [ ] Keep full raw assertion/qualifier/reference provenance for consumed fields.
-- [ ] Make unrelated `lastrevid` changes visible in audit diffs without invalidating
+- [x] Define how deletions and removed P856 statements become tombstones.
+- [x] Keep full raw assertion/qualifier/reference provenance for consumed fields.
+- [x] Make unrelated `lastrevid` changes visible in audit diffs without invalidating
       unchanged material edge fingerprints.
-- [ ] Test real-format pathological entities and multistream compression boundaries.
+- [x] Test real-format pathological entities and multistream compression boundaries.
 
 Authoritative format reference:
 
 - <https://www.wikidata.org/wiki/Wikidata:Database_download>
 
+Implementation note: the inspected Add/Change distribution is XML and the provider warns that embedded JSON in XML is unstable. Version 0.5 therefore implements ordered typed deltas from official Wikibase entity JSON and deliberately rejects the XML incremental artifact.
+
 ### Task 3.4: Strengthen current-source acquisition verification
 
-- [ ] Prefer provider-published checksums or signatures when officially available and
+- [x] Prefer provider-published checksums or signatures when officially available and
       bind verification method into the source manifest.
-- [ ] Keep HTTPS allowlists, manual redirect validation, byte bounds, strong-validator
+- [x] Keep HTTPS allowlists, manual redirect validation, byte bounds, strong-validator
       resume rules, and immutable local cache behavior.
-- [ ] Detect and report source format drift before partial import can replace a source.
-- [ ] Add format-version canaries for Majestic, CrUX, Curlie, PSL, and Wikidata.
-- [ ] Preserve CrUX billing as explicit opt-in configuration and record job identity,
+- [x] Detect and report source format drift before partial import can replace a source.
+- [x] Add format-version canaries for Majestic, CrUX, Curlie, PSL, and Wikidata.
+- [x] Preserve CrUX billing as explicit opt-in configuration and record job identity,
       query, result period, and actual cost outside public fixtures.
-- [ ] Continue frequent PSL refresh and include exact PSL hash in normalization proofs.
-- [ ] Preserve Curlie attribution and description-redaction tests on every export path.
+- [x] Continue frequent PSL refresh and include exact PSL hash in normalization proofs.
+- [x] Preserve Curlie attribution and description-redaction tests on every export path.
 
 ### Task 3.5: Add source lineage and independence metadata
 
-- [ ] Record direct provider, upstream/origin dataset, transformation, snapshot, and
+- [x] Record direct provider, upstream/origin dataset, transformation, snapshot, and
       known dependency relationships for each fact source.
-- [ ] Prevent policy from counting two assertions as independent corroboration when
+- [x] Prevent policy from counting two assertions as independent corroboration when
       one republishes the other.
-- [ ] Expose lineage in lookup, review bundles, export, diff, and evaluation.
-- [ ] Keep unknown lineage explicit rather than assuming independence.
+- [x] Expose lineage in lookup, review bundles, export, diff, and evaluation.
+- [x] Keep unknown lineage explicit rather than assuming independence.
+
+Phase 3 completion note: compact v3 generations, audit bundle verification and no-delete retention, the source-pinned benchmark runner, two-checkpoint process-kill recovery, provider-scale ROR/PSL measurements, streamed Wikidata JSON arrays, authenticated record ceilings, typed JSON tombstones, source-bound checksums, current format canaries and explicit source lineage are implemented and tested in version 0.5.
 
 ## Phase 4: Rights-gated additional source adapters
 
@@ -539,14 +543,14 @@ Every source follows the same gate:
 Rationale: ROR is CC0 and directly supplies stable organization IDs, names, aliases,
 status, locations, links, and domains. It is compact and well aligned with the model.
 
-- [ ] Verify the current ROR schema version and official release asset from the ROR
+- [x] Verify the current ROR schema version and official release asset from the ROR
       data-dump documentation at implementation time.
-- [ ] Consume only fields confirmed in that inspected schema.
-- [ ] Map names and aliases without merging ROR entities into Wikidata entities unless
+- [x] Consume only fields confirmed in that inspected schema.
+- [x] Map names and aliases without merging ROR entities into Wikidata entities unless
       an exact external identifier or reviewed equivalence supports the join.
-- [ ] Preserve links and domains as ROR assertions, not approvals.
-- [ ] Preserve status, type, country/location, external IDs, and upstream lineage.
-- [ ] Test domain conflicts, former/inactive organizations, aliases, multiple links,
+- [x] Preserve links and domains as ROR assertions, not approvals.
+- [x] Preserve status, type, country/location, external IDs, and upstream lineage.
+- [x] Test domain conflicts, former/inactive organizations, aliases, multiple links,
       missing fields, duplicate input, schema drift, and exact-ID equivalence.
 
 References: