release: implement site registry v0.5

docs/ARCHITECTURE.md

@@ -11,7 +11,9 @@ flowchart LR
   I --> W[(Writer store)]
   O[Bounded candidate observer] --> B[Immutable observation batches]
   B --> W
-  W --> G[Candidate generation]
+  W --> A[Content-addressed cold audit bundles]
+  W --> G[Compact candidate generation]
+  A -. digest references .-> G
   G --> Q[Evidence bundles and review queue]
   Q --> V[Signed reviewer votes]
   V --> W
@@ -24,12 +26,19 @@ flowchart LR
 
 ## Data boundaries
 
-`sources`, `records`, and `facts` preserve provider-native evidence. Typed
+The writer's `sources`, `records`, and `facts` preserve provider-native evidence. Typed
 coverage selects a coherent active set without deleting older snapshots.
 `selected_sources` and `active_records` record that derivation. Names, entities,
 web properties, edges, popularity, and rejected facts are deterministic
 projections.
 
+A compact build packages every completed source into a content-addressed cold
+JSONL bundle before removing superseded records and unprojected fact history from
+the runtime copy. `COMPLETE.json` binds every bundle and the runtime database.
+Normal queries need only the compact generation. Audit export, verification and
+retention checkpoints stream the external bundle store and fail on absence,
+substitution, truncation, malformed order, or mismatched source/attribution.
+
 Names, website edges, and entity equivalences have separate material
 fingerprints. Adding an alias therefore cannot inherit an approved destination,
 and changing unrelated entity metadata does not invalidate an unchanged website
@@ -72,7 +81,7 @@ unlinked snapshot before opening them.
 | `source`, `store`, adapters | Manifest validation and streaming source-specific import |
 | `coverage` | Full/partition/delta graph validation and active-record masking |
 | `normalize` | Deterministic URL, hostname, registrable-domain, suffix, and name normalization |
-| `build` | Canonical immutable generation and receipt creation |
+| `build`, `audit` | Canonical compact generation, cold history, verification and retention |
 | `bundle`, `policy`, `vote` | Review evidence, policy epochs, authenticated quorum, revocation |
 | `observer`, `observation`, `queue` | Candidate-only collection, replay/import, reverse lookup, drift and queues |
 | `query`, `resolution`, `identity`, `catalog` | Audit lookup, equivalence, resolution, statistics |

docs/BENCHMARKING.md

@@ -0,0 +1,80 @@
+# Benchmarking and recovery
+
+`scripts/benchmark.py` runs one import, build, audit verification, or evaluation
+command without a shell and creates a new evidence directory. `REPORT.json`
+records exact source-manifest and source-object hashes, hardware, wall/CPU time,
+peak RSS, database/record/fact growth, hashes of stdout/stderr, optional measured
+artifacts, and a hash of the argument vector. Source bytes are never copied into
+the report. The command refuses to overwrite an existing directory.
+
+Use three profiles consistently:
+
+- `small`: synthetic or source-shaped fixtures used for correctness and rapid
+  regression checks;
+- `medium`: a documented provider subset large enough to exercise checkpoints,
+  bounds and projection behavior; and
+- `provider`: one complete current official provider object in its native format.
+
+Example import measurement:
+
+```bash
+python3 scripts/benchmark.py --profile provider \
+  --output /data/benchmarks/ror-v2.12-import \
+  --source-manifest /data/cache/ror/source.json \
+  --database /data/ror-writer.sqlite --expanded-bytes 362619018 -- \
+  argand-site-registry import --database /data/ror-writer.sqlite \
+    --input /data/cache/ror/SHA256 --manifest /data/cache/ror/source.json
+```
+
+For build measurements, label exact regular files with repeated
+`--measured-path NAME=PATH`. Run evaluation with at least 1,000 representative
+cases in one process so authenticated startup/copy time remains separate from
+the report's native per-query p50/p95. Store reports outside Git because commands
+and snapshots can reveal private operational structure. A report establishes one
+machine and object only; synthetic numbers are never provider-capacity evidence.
+
+## Version 0.5 provider canary
+
+The 2026-09-13 canary used official ROR release `v2.12-2026-08-25`, Zenodo record
+`22099990`. The compressed object was 36,246,232 bytes, SHA-256
+`5779c7baf71771fd8ea829201e7bd4343a3c68ff36c595f480b3a00292f78931`, and
+matched provider MD5 `ce8807691455d4ada3216c31408e9e1a`. Its JSON and CSV
+members expanded to 362,619,018 bytes.
+
+The final streaming import produced 137,398 records and 914,439 facts in 62.30
+seconds (9.47 user, 13.24 system), with 25,032 KiB peak RSS and about 14,678
+facts/second. It grew the PSL-initialized writer by 828,682,240 bytes to
+829,505,536 bytes. Reimport replayed the complete pinned source in 0.025 seconds
+without changing any record, fact, or database byte count.
+
+A compact build combining that ROR source with the current PSL projected 137,398
+entities, 131,591 properties and 133,398 edges. The pre-compaction database was
+1,345,097,728 bytes; the runtime database was 824,705,024 bytes, a 38.688%
+reduction. The initial compact build took 58.36 seconds with 45,140 KiB peak RSS;
+a second build reused the same bundles and produced byte-identical database and
+receipt bytes. The two cold bundles totaled 686,468,048 bytes and held 137,399
+records plus 914,440 facts. Warm-cache full streaming audit verification took
+3.03 seconds with 24,076 KiB peak RSS.
+
+A 1,000-case safe-abstention evaluation passed 1,000/1,000 with native reusable
+reader latency of 72 microseconds p50 and 96 microseconds p95. The warm-cache full
+process took 0.83 seconds, including authentication and the private copy of the
+786.5 MiB SQLite file. Serving deployments must reuse the verified `Registry` and
+must measure cold startup on their own storage.
+
+Hardware: Linux 6.18.51-1-lts x86_64, AMD Ryzen 9 7900X (12 cores, 24 logical
+CPUs), 61 GiB RAM, ext4. These numbers are a single local canary and not a public
+service capacity claim.
+
+## Recovery matrix
+
+The acceptance suite sends real process termination after two distinct committed
+checkpoints and proves the same import completes exactly once on restart. The
+provider canary also killed the final binary at a 2,304-record checkpoint via a
+bounded-runner fault, recovered its hot journal, and reached the exact clean-import
+record/fact totals. Other tests cover handled parser failure rollback, SQLite growth-cap
+(`SQLITE_FULL`) behavior, truncated/compressed inputs, mutated cache objects,
+duplicate records and JSON keys, archive-member drift, missing/truncated/substituted
+cold bundles, existing publication targets, failed atomic writes, and unchanged
+previous generations after update failure. Provider objects and benchmark output
+remain outside the repository.

docs/CONSUMERS.md

@@ -34,8 +34,8 @@ source attribution and application-specific malware/content policy.
 
 ## Current contracts
 
-Code version 0.4.0 uses writer schema 5 and `argand.site-rules/v4`.
-`COMPLETE.json` uses `argand.site-registry/v2` and binds:
+Code version 0.5.0 uses writer schema 5 and `argand.site-rules/v4`.
+New compact `COMPLETE.json` files use `argand.site-registry/v3` and bind:
 
 - authenticated `registry.sqlite` bytes;
 - the active source coverage graph and exact PSL source;
@@ -43,12 +43,23 @@ Code version 0.4.0 uses writer schema 5 and `argand.site-rules/v4`.
 - decision-time policy;
 - source license and machine-readable attribution files; and
 - entity, property, edge and rejection counts.
+- every cold audit bundle's digest, size, counts, source, selection state,
+  attribution digest and coverage digest.
 
 Normal `export` uses `argand.site-export/v2`, contains selected active nonrejected
 facts, and marks every assertion `active`. `export-audit` uses the same schema with
-mode `audit` and includes superseded/rejected states and rejection reasons. Neither
+mode `audit` for v2 generations. Compact generations require their external audit
+store and emit `argand.site-export/v3` after verifying every referenced bundle.
+Both include superseded/rejected states and rejection reasons. Neither
 contains a list of resolver-approved routes.
 
+Keep the audit store separate from the runtime deployment, replicate it by exact
+object digest, and run `verify-audit` before retention attestations or audit export.
+Runtime resolution verifies the compact generation receipt and database without
+opening cold history. A missing audit object therefore does not silently change a
+query, but it is an audit/retention failure and blocks any claim of complete
+provenance. Retention checkpoints never authorize deletion.
+
 Votes use `argand.site-vote/v1` and the OpenSSH namespace
 `argand-site-registry-vote`. Consumers compile them under the exact receipt-bound
 policy and trusted query time. Unknown schema or rule versions fail closed.
@@ -77,7 +88,8 @@ resolution policy, expiry, revocation continuity or signature verification.
 
 V1 source manifests remain readable as isolated legacy provider/scope streams.
 A deliberate legacy-compatible build policy can replay v0.3 reviews, but strict
-0.4 builds require votes and reviewer trust. Current readers accept v2 receipts;
+0.4 builds require votes and reviewer trust. Current readers accept v2 and compact
+v3 receipts;
 rollback checks may open pinned v1 receipts with rules v1-v3 only to compare
 revocation history.
 
@@ -93,7 +105,7 @@ The last recorded downstream integration replaced Argand's embedded crate with
 signed v0.3.0 revision `ac8282093d8a815c6227cff86e1f40714d510bcd` at Argand
 commit `d9dfd1585ce21d9c4136bcc24fa01fe3bfb8ed6e`.
 
-Version 0.4 is handed off as a signed standalone revision. Argand should update its
+Version 0.5 is handed off as a signed standalone revision. Argand should update its
 full Git `rev` in a separate coordinated source/build window, compare contract
 changes, and rerun navigation compiler, native resolver, API, abstention,
 revocation and clean-process gates. Changing the code dependency does not activate

docs/EVALUATION.md

@@ -1,7 +1,9 @@
 # Resolver evaluation
 
 `evaluate` streams authored JSONL judgments through the same pinned native reader
-used by consumers. It reports correctness and p50/p95 latency. Pass `--at` for a
+used by consumers. Its v2 report includes every active source snapshot and declared
+lineage (explicit `null` for legacy unknown lineage), plus correctness and p50/p95
+latency. Pass `--at` for a
 reproducible policy clock; approval expiry and future votes otherwise depend on
 current time. Inputs are bounded to 16 MiB, lines to 64 KiB, and case IDs must be
 unique.

docs/FORMATS.md

@@ -1,13 +1,19 @@
 # Versioned formats
 
-Version 0.4 uses writer schema 5 and `argand.site-rules/v4`. Schema identifiers
+Version 0.5 uses writer schema 5 and `argand.site-rules/v4`. Schema identifiers
 are independent from the crate version. Unknown schemas and rules fail closed.
 
 | Artifact | Current schema | Purpose |
 | --- | --- | --- |
-| Source manifest | `argand.site-source/v2` | Exact source object plus typed coverage |
-| Generation receipt | `argand.site-registry/v2` | Hash-bound immutable generation contract |
+| Source manifest | `argand.site-source/v3` | Exact source object, integrity proof, lineage, parser bound and typed coverage |
+| Generation receipt | `argand.site-registry/v3` | Compact runtime plus content-addressed audit references |
 | Active/audit JSONL | `argand.site-export/v2` | Source-bearing assertions with selection state |
+| Compact audit JSONL | `argand.site-export/v3` | Verified external history with bundle references and tombstones |
+| Cold audit bundle | `argand.site-audit-bundle/v2` | Source manifest and ordered record/fact history |
+| Audit verification | `argand.site-audit-verification/v1` | Streaming bundle verification totals |
+| Audit retention checkpoint | `argand.site-audit-retention/v1` | Signed no-delete retention set |
+| Benchmark report | `argand.site-benchmark/v1` | Source-pinned operation and resource evidence |
+| Evaluation report | `argand.site-evaluation/v2` | Resolver judgments, latency, and active source lineage |
 | Evidence bundle | `argand.site-evidence-bundle/v1` | Exact evidence signed by reviewer votes |
 | Vote | `argand.site-vote/v1` | Authenticated approve/revoke decision |
 | Review policy | `argand.site-policy/v1` | Threshold, groups, revocation, and separation rules |
@@ -18,12 +24,30 @@ are independent from the crate version. Unknown schemas and rules fail closed.
 | Active pointer | `argand.site-current/v2` | Signed generation pin plus revocation continuity |
 | Diff | `argand.site-diff/v4` | Typed change stream across generations |
 
-Source manifest v2 coverage is one of `full`, `partition`, or `delta`. A delta
+Source manifest v3 adds explicit `integrity`, `lineage`, and an optional
+`maximum_record_bytes` parser ceiling to v2 typed coverage. Integrity always
+contains the locally computed SHA-256 and may also carry a source-bound provider
+checksum. Lineage names the direct provider, upstream datasets, transformations,
+and a conservative independence group. A missing legacy lineage serializes as
+unknown and never becomes evidence of independence.
+
+Coverage is one of `full`, `partition`, or `delta`. A delta
 names its exact base, positive consecutive sequence, and superseded source IDs.
 A full source cannot compose with active partitions. Overlap, gaps, cycles,
 missing bases, cross-provider supersession, and mixed legacy/typed frontiers fail
 the build.
 
+Generation v3 uses `runtime_layout: "compact-v1"`. Its local SQLite database keeps
+the selected source manifests, active facts and projections, review/policy state,
+and the evidence needed by runtime queries. Each complete source history is first
+written to an external JSONL object. The receipt binds its SHA-256, length, source,
+record/fact counts, selected state, attribution digest and coverage digest.
+`verify-audit` re-hashes and parses every object before an audit export or retention
+checkpoint. The separate retention signature namespace is
+`argand-site-registry-audit-retention`; every checkpoint contains
+`deletion_authorized: false`. Generation v2 remains supported with embedded
+history for compatibility.
+
 The normal export emits selected active facts only and excludes rejected facts.
 Every assertion has `selection_state: "active"`. Audit export includes active,
 superseded, and rejected facts; rejected assertions include their reason. Both

docs/INDEX.md

@@ -11,10 +11,15 @@
 - [Trust](TRUST.md): enforced checks and publisher/consumer responsibilities.
 - [Publishing](PUBLISHING.md): reviewer keys, candidate acceptance and activation.
 - [Evaluation](EVALUATION.md): bounded JSONL judgments and result interpretation.
+- [Benchmarking and recovery](BENCHMARKING.md): repeatable profiles, evidence schema,
+  provider canary, and failure matrix.
+- [Source candidates](SOURCE-CANDIDATES.md): admitted, held, correlated, and excluded providers.
 - [Releasing](RELEASING.md): CI, source signing and archive verification.
 - [Validation](VALIDATION.md): independent builds and native acceptance evidence.
 - [Version 0.4 security review](SECURITY-REVIEW-0.4.md): threat boundaries,
   resolved findings and residual operator responsibilities.
+- [Version 0.5 security review](SECURITY-REVIEW-0.5.md): compact audit storage,
+  provider checksums, ZIP parsing, lineage and benchmark tooling.
 - [Contributing](../CONTRIBUTING.md), [governance](../GOVERNANCE.md),
   [security](../SECURITY.md): proposals, decisions and incidents.
 - [Extraction design](superpowers/specs/2026-09-12-standalone-design.md) and

docs/SECURITY-REVIEW-0.5.md

@@ -0,0 +1,105 @@
+# Version 0.5 security review
+
+Date: 2026-09-13. Scope: all changes from signed version 0.4.0 through the 0.5
+implementation, including source manifest v3, ROR ZIP acquisition/import, streamed
+Wikidata JSON, compact generations, external audit bundles, retention signatures,
+evaluation lineage, benchmark tooling, dependencies and operator documentation.
+
+No unresolved critical, high, or medium-severity finding remained at release
+validation. This is a source review and adversarial test result, not a claim that
+an imported URL is safe or that a future public registry has been reviewed.
+
+## Boundaries reviewed
+
+- Provider responses, manifests, JSON/CSV/XML/ZIP/archive members, source records,
+  names, URLs, lineages and audit-store entries are untrusted input.
+- A source assertion cannot authorize a route. Name, edge and equivalence authority
+  still comes from policy-qualified reviewer signatures; release authority remains
+  a separate publisher signature and activation step.
+- The cache, writer, audit store, reviewer trust file and keys are operator-owned
+  local resources. Consumer trust starts with an independently obtained receipt
+  pin or publisher trust root.
+- The project ships no hosted API, browser rendering surface, credential store,
+  production key, preapproved dataset, automatic signer, or automatic activation.
+
+## Findings fixed
+
+| ID | Severity before fix | Resolution |
+| --- | --- | --- |
+| SR-05-01 | Medium | Audit-store discovery previously used the 512 MiB fact-line ceiling for an untrusted unmatched header. Header reads now stop at 1 MiB before JSON materialization; facts remain bounded and content-authenticated. |
+| SR-05-02 | Medium | SQLite can auto-rollback a transaction on `SQLITE_FULL`; an unconditional second rollback could return before cleanup and leave a committed incomplete checkpoint. Recovery now detects autocommit, always attempts source discard and growth-ceiling restoration, and has a real growth-cap regression. |
+| SR-05-03 | Medium | A syntactically valid compact receipt could omit a selected audit reference while still naming other bundles. Opening now validates every source declaration, exact PSL identity, receipt-to-database selected-source equality, and exact equality between selected bundle references and active receipt sources. |
+| SR-05-04 | Medium | Initial ROR ZIP limits trusted member size metadata. JSON and CSV decompression now have independent actual-byte ceilings in addition to record, count, database-growth, member-count, compression-method, encryption and name checks. |
+| SR-05-05 | Medium | Foreign keys were disabled during an indexed compaction transaction and the first error on rollback could bypass restoration. Every exit now attempts rollback as needed and restores foreign-key enforcement before returning. Publication still requires `integrity_check` and a zero-row `foreign_key_check`. |
+| SR-05-06 | Low | The first benchmark stdout cap applied `RLIMIT_FSIZE` to the whole child and could kill SQLite writes. The final runner uses bounded stdout/stderr pipes, kills the isolated process group only on output overflow, records the condition, and never limits dataset/database files. The induced hot journal was recovered and the import resumed to the clean-import totals. |
+
+The review also capped authenticated custom records at 256 MiB so generated cold
+bundle lines remain within their verifier ceiling. Removed Wikidata statements
+carry their exact superseded source IDs in tombstone exports. ROR inactive and
+withdrawn website assertions remain auditable but ineligible.
+
+## Acquisition and archive checks
+
+Source URLs use an HTTPS provider allowlist, reject credentials/fragments/custom
+ports and revalidate every redirect. Resumption requires a strong unchanged ETag,
+exact final URL, matching total and Content-Range. Downloads have caller-supplied
+byte ceilings and immutable completion receipts. Provider checksum evidence is
+accepted only for reviewed source-bound locations: the matching Zenodo record for
+ROR or the same Wikimedia dump directory for Wikidata. ROR's MD5 is used only to
+match Zenodo's published field; the local content identity and all downstream
+references use SHA-256.
+
+The ROR reader accepts exactly one version-matched JSON and CSV member, Stored or
+Deflated, with no path components, encryption or extra members. It streams schema
+2.1 JSON with duplicate-key rejection and fails on unknown top-level fields. Every
+source import re-hashes the exact no-follow input descriptor after parsing and
+publishes completion only after all records commit.
+
+## Compact history and signing checks
+
+Cold bundles use digest-derived names and are created without replacement. A
+reference binds exact bytes, source declaration, selected state, record/fact
+counts, coverage and attribution. Verification opens with `O_NOFOLLOW`, checks
+length and SHA-256 on the same descriptor, then parses bounded JSONL in required
+record-before-fact order. Audit export and retention checkpoint first verify every
+object. Retention signatures use the distinct
+`argand-site-registry-audit-retention` namespace, and their authenticated document
+always requires `deletion_authorized: false`; the software provides no deletion
+command.
+
+Runtime compaction occurs only after all bundles are durable and verified. It
+uses an indexed exact fact keep-set, commits a deterministic runtime copy, restores
+foreign keys, then runs integrity and foreign-key checks before `COMPLETE.json` is
+created. Readers authenticate and copy SQLite into a private unlinked file before
+opening immutable mode, preserving the existing sidecar and post-open mutation
+defenses.
+
+## Automated and manual evidence
+
+- Workspace formatting, locked all-target checks, Clippy with warnings denied,
+  strict rustdoc, Rust/Python tests, consumer parity, reproducible source packaging
+  and extracted-source acceptance are release gates.
+- `cargo audit --deny warnings` loaded 1,243 RustSec advisories and reported no
+  finding across 277 locked dependency nodes.
+- The repository and diff contain no private key, token, password, dataset,
+  production review, or generated registry. SQL values remain parameterized;
+  dynamic SQL fragments are fixed internal table/predicate vocabularies.
+- Adversarial tests cover duplicate JSON keys/records, schema/member drift,
+  symlinks, truncation/substitution, oversized records, malformed coverage,
+  process termination at multiple checkpoints, SQLite growth exhaustion,
+  incomplete update/publication, receipt bundle omission, signature tampering,
+  reviewer/publisher separation, revocation continuity and rollback.
+
+## Residual operator responsibilities
+
+Protect cache, writer and audit directories from untrusted local writers and
+replicate cold objects by exact digest. A deliberately configured 256 MiB record
+can require comparable memory; choose the smallest inspected limit and enforce
+process/storage quotas around provider jobs. Measure cold authenticated startup on
+deployment storage, because every registry open copies the full runtime database.
+
+Obtain provider checksums and publisher/reviewer trust roots independently. A valid
+hash or signature authenticates bytes and a decision, not ownership, current site
+safety, commercial fitness, or reviewer competence. Continue malware/content,
+drift, expiry, revocation, backup/restore and incident controls before any public
+dataset or Argand route is activated.

docs/SOURCE-CANDIDATES.md

@@ -0,0 +1,23 @@
+# Source admission status
+
+This registry supports a provider only after current rights, distribution,
+schema, lineage, attribution, bounded import, and no-auto-approval behavior have
+all been reviewed. A research entry below is not permission to ingest it.
+
+| Source | Status | Decision and next gate |
+| --- | --- | --- |
+| ROR | Admitted in 0.5 | The official CC0 schema 2.1 ZIP is streamed with exact Zenodo checksum evidence. Organization websites remain assertions; inactive/withdrawn edges are ineligible. GeoNames location lineage is explicit. |
+| MusicBrainz | Next adapter; held | The [official download documentation](https://musicbrainz.org/doc/MusicBrainz_Database/Download) identifies the core `mbdump.tar.bz2` snapshot as CC0. The live replication/edit/statistics material with noncommercial terms is excluded. Admission still needs a current core snapshot/checksum canary and a bounded relational-table adapter for documented [URL relationships](https://musicbrainz.org/doc/Style/Relationships/URLs). |
+| GND | Research hold | The [DNB open-data distribution](https://data.dnb.de/opendata/) must be checked at implementation time for the exact file license, current JSON-LD/RDF predicates, checksum and useful homepage coverage. Stop the adapter if explicit homepage coverage does not justify it. |
+| ORCID public data | Research hold | Its self-declared links need an individuals-only privacy, impersonation and volatility policy in addition to the [public-file terms](https://info.orcid.org/public-data-file-use-policy/). It could never auto-approve a route. |
+| OpenAlex institutions | Correlated-source hold | Institution metadata can inherit ROR. Any future use must declare ROR upstream and cannot count as independent website corroboration. See the [institution source documentation](https://help.openalex.org/data/institutions/). |
+| OpenStreetMap | License-architecture hold | No ingestion until an ODbL-compatible attribution, database-right and redistribution design is accepted. See the [OSMF license FAQ](https://osmfoundation.org/wiki/Licence_and_Legal_FAQ). |
+| Government/corporate registries | Jurisdiction hold | Review one jurisdiction and exact field at a time. Stable identifiers may support crosswalks; the registry cannot infer a website absent an authoritative field. |
+| DNS, RDAP, certificate transparency, package registries, web crawl data | Observation-only research | Exact commercial reuse terms and retention rules must be approved first. These sources describe current infrastructure and cannot establish entity ownership alone. |
+| Open Library and unresolved-rights sources | Excluded | Keep excluded until the underlying data rights and redistribution obligations are clear enough for commercial reuse. |
+
+Cloudflare Radar, default Tranco, Cisco Umbrella, arbitrary mirrors, and any
+provider without verified commercial-reuse rights remain unsupported. Proposals
+must update `LICENSE_SOURCES.md`, add an allowlisted source/format pair and
+source-native test fixtures, and demonstrate that the adapter cannot create or
+renew an approved navigation route.

docs/TRUST.md

@@ -7,9 +7,10 @@ reach consumers.
 
 ## What the implementation enforces
 
-Source adapters accept only documented providers and formats. Manifests bind the
+Source adapters accept only documented providers and formats. V3 manifests bind the
 exact object, origin URL, source-native snapshot, license, retrieval time, byte
-length, digest, and typed coverage. Full, partition and delta graphs reject gaps,
+length, local digest, source-bound provider checksum where available, per-record
+parser ceiling, typed coverage and declared lineage. Full, partition and delta graphs reject gaps,
 cycles, overlap, cross-provider replacement and ambiguous active branches. Failed
 or incomplete imports cannot replace complete evidence.
 
@@ -19,6 +20,13 @@ entities. Source-specific popularity stays separate from identity. Every fact
 keeps source, source identifier, selector, license, retrieval time, confidence and
 raw evidence needed for audit.
 
+Lineage declares the direct provider, known upstream datasets, provider
+transformations and a conservative independence group. ROR location metadata, for
+example, records GeoNames upstream. Unknown legacy lineage stays unknown. Source
+assertion count is not reviewer quorum: the current policy grants authority only
+to authenticated reviewer identities, independent groups and physical keys, so
+two providers cannot manufacture approval by copying one upstream claim.
+
 Names, entity-to-property edges and entity equivalences have independent material
 fingerprints. Under the reference policy, `resolve` needs two independent votes
 for the matched name and two for the selected edge. Reviewer groups and physical
@@ -57,13 +65,20 @@ and malware-policy results without naming a vendor. No such provider is built in
 an operator must verify commercial-reuse terms and preserve its exact source and
 rights declaration before importing those records.
 
-Generations bind authenticated SQLite bytes, selected coverage, policy, reviewer
+Compact generations bind authenticated SQLite bytes, selected coverage, policy, reviewer
 trust bytes, licenses, attribution, and decision-time rules into `COMPLETE.json`.
 Readers verify the receipt pin and copy the database into a private unlinked file
 before SQLite opens it. Release signing and activation reverify every stored
 signature. The strict policy rejects a publisher identity or physical key used for
 any reviewer vote.
 
+Complete historical records and facts live in content-addressed cold bundles bound
+by the same receipt. Their verifier hashes the exact no-follow file descriptor and
+checks bytes, JSON order, source identity, selection state, counts, coverage and
+attribution. Runtime lookup cannot weaken when cold storage is offline; audit
+export and retention fail closed. A separately signed no-delete checkpoint makes
+the retained object set explicit without adding deletion authority.
+
 ## What a publisher must establish
 
 The software verifies evidence integrity and decision authorization. A publisher

docs/adr/0006-source-lineage.md

@@ -1,15 +1,17 @@
 # ADR 0006: Source lineage and independence
 
-Status: Accepted design; implementation scheduled for 0.5.
+Status: Implemented in 0.5.
 
 Corroboration must describe the direct provider, upstream dataset, transformation,
 and snapshot. Two providers that copied the same upstream assertion do not count
 as independent evidence merely because their URLs differ. Unknown lineage stays
 unknown.
 
-Version 0.4 preserves provider-native provenance and never combines popularity or
-same-domain evidence into ownership confidence. Version 0.5 will add explicit
-lineage fields and independence-aware corroboration without rewriting history.
+Version 0.5 adds explicit source-manifest lineage and exposes it in evidence and
+evaluation without rewriting legacy history. It preserves provider-native
+provenance and never combines popularity or same-domain evidence into ownership
+confidence. Current admission quorum is made only from authenticated reviewers,
+groups and physical keys, so copied source assertions cannot increase authority.
 
 ## Rejected alternatives
 

docs/superpowers/plans/2026-09-13-v0.4-and-beyond.md

@@ -454,71 +454,75 @@ deltas and a separately signed, cumulative emergency block feed.
 
 ### Task 3.1: Separate runtime projections from cold audit history
 
-- [ ] Keep raw cache objects immutable and content-addressed outside Git.
-- [ ] Package completed source imports into content-addressed audit bundles with
+- [x] Keep raw cache objects immutable and content-addressed outside Git.
+- [x] Package completed source imports into content-addressed audit bundles with
       manifest, record/fact indices, hashes, format version, and attribution.
-- [ ] Make a runtime generation contain selected facts, normalized projections,
+- [x] Make a runtime generation contain selected facts, normalized projections,
       active policy results, required votes/revocations, and signed bundle references.
-- [ ] Do not copy all historical records and facts into every runtime generation.
-- [ ] Add audit verification that streams referenced bundles and detects absence,
+- [x] Do not copy all historical records and facts into every runtime generation.
+- [x] Add audit verification that streams referenced bundles and detects absence,
       truncation, substitution, or mismatched attribution.
-- [ ] Add retention/checkpoint tooling that never deletes the only authenticated copy
+- [x] Add retention/checkpoint tooling that never deletes the only authenticated copy
       of evidence and produces a signed deletion/retention report.
-- [ ] Measure query latency and generation size before and after the split.
+- [x] Measure query latency and generation size before and after the split.
 
 ### Task 3.2: Add provider-scale benchmark and recovery tooling
 
-- [ ] Define repeatable small, medium, and provider-representative import profiles.
-- [ ] Record wall time, CPU time, peak RSS, compressed and expanded bytes, database
+- [x] Define repeatable small, medium, and provider-representative import profiles.
+- [x] Record wall time, CPU time, peak RSS, compressed and expanded bytes, database
       growth, facts/second, checkpoint frequency, restart time, build size, and query
       latency.
-- [ ] Interrupt imports at multiple checkpoints and prove idempotent resumption.
-- [ ] Exercise disk-full, truncated input, cache corruption, duplicate records, and
+- [x] Interrupt imports at multiple checkpoints and prove idempotent resumption.
+- [x] Exercise disk-full, truncated input, cache corruption, duplicate records, and
       interrupted generation publication.
-- [ ] Make benchmark reports name exact source snapshot hashes and hardware without
+- [x] Make benchmark reports name exact source snapshot hashes and hardware without
       committing source data.
-- [ ] Treat synthetic performance as development evidence, not provider capacity.
+- [x] Treat synthetic performance as development evidence, not provider capacity.
 
 ### Task 3.3: Harden full Wikidata ingestion and add incremental refresh
 
-- [ ] Confirm current official full and incremental formats from Wikidata documentation
+- [x] Confirm current official full and incremental formats from Wikidata documentation
       and inspected fixtures before changing the adapter.
-- [ ] Replace the assumption that every useful entity fits in one in-memory 16 MiB
+- [x] Replace the assumption that every useful entity fits in one in-memory 16 MiB
       line with bounded disk-spooling or an explicitly receipted oversized-record path.
-- [ ] Never silently skip an oversized entity that may contain a relevant fact.
-- [ ] Add incremental add/change ingestion with authenticated base snapshot identity,
+- [x] Never silently skip an oversized entity that may contain a relevant fact.
+- [x] Add incremental add/change ingestion with authenticated base snapshot identity,
       ordered application, checkpoints, and reconciliation against later full dumps.
-- [ ] Define how deletions and removed P856 statements become tombstones.
-- [ ] Keep full raw assertion/qualifier/reference provenance for consumed fields.
-- [ ] Make unrelated `lastrevid` changes visible in audit diffs without invalidating
+- [x] Define how deletions and removed P856 statements become tombstones.
+- [x] Keep full raw assertion/qualifier/reference provenance for consumed fields.
+- [x] Make unrelated `lastrevid` changes visible in audit diffs without invalidating
       unchanged material edge fingerprints.
-- [ ] Test real-format pathological entities and multistream compression boundaries.
+- [x] Test real-format pathological entities and multistream compression boundaries.
 
 Authoritative format reference:
 
 - <https://www.wikidata.org/wiki/Wikidata:Database_download>
 
+Implementation note: the inspected Add/Change distribution is XML and the provider warns that embedded JSON in XML is unstable. Version 0.5 therefore implements ordered typed deltas from official Wikibase entity JSON and deliberately rejects the XML incremental artifact.
+
 ### Task 3.4: Strengthen current-source acquisition verification
 
-- [ ] Prefer provider-published checksums or signatures when officially available and
+- [x] Prefer provider-published checksums or signatures when officially available and
       bind verification method into the source manifest.
-- [ ] Keep HTTPS allowlists, manual redirect validation, byte bounds, strong-validator
+- [x] Keep HTTPS allowlists, manual redirect validation, byte bounds, strong-validator
       resume rules, and immutable local cache behavior.
-- [ ] Detect and report source format drift before partial import can replace a source.
-- [ ] Add format-version canaries for Majestic, CrUX, Curlie, PSL, and Wikidata.
-- [ ] Preserve CrUX billing as explicit opt-in configuration and record job identity,
+- [x] Detect and report source format drift before partial import can replace a source.
+- [x] Add format-version canaries for Majestic, CrUX, Curlie, PSL, and Wikidata.
+- [x] Preserve CrUX billing as explicit opt-in configuration and record job identity,
       query, result period, and actual cost outside public fixtures.
-- [ ] Continue frequent PSL refresh and include exact PSL hash in normalization proofs.
-- [ ] Preserve Curlie attribution and description-redaction tests on every export path.
+- [x] Continue frequent PSL refresh and include exact PSL hash in normalization proofs.
+- [x] Preserve Curlie attribution and description-redaction tests on every export path.
 
 ### Task 3.5: Add source lineage and independence metadata
 
-- [ ] Record direct provider, upstream/origin dataset, transformation, snapshot, and
+- [x] Record direct provider, upstream/origin dataset, transformation, snapshot, and
       known dependency relationships for each fact source.
-- [ ] Prevent policy from counting two assertions as independent corroboration when
+- [x] Prevent policy from counting two assertions as independent corroboration when
       one republishes the other.
-- [ ] Expose lineage in lookup, review bundles, export, diff, and evaluation.
-- [ ] Keep unknown lineage explicit rather than assuming independence.
+- [x] Expose lineage in lookup, review bundles, export, diff, and evaluation.
+- [x] Keep unknown lineage explicit rather than assuming independence.
+
+Phase 3 completion note: compact v3 generations, audit bundle verification and no-delete retention, the source-pinned benchmark runner, two-checkpoint process-kill recovery, provider-scale ROR/PSL measurements, streamed Wikidata JSON arrays, authenticated record ceilings, typed JSON tombstones, source-bound checksums, current format canaries and explicit source lineage are implemented and tested in version 0.5.
 
 ## Phase 4: Rights-gated additional source adapters
 
@@ -539,14 +543,14 @@ Every source follows the same gate:
 Rationale: ROR is CC0 and directly supplies stable organization IDs, names, aliases,
 status, locations, links, and domains. It is compact and well aligned with the model.
 
-- [ ] Verify the current ROR schema version and official release asset from the ROR
+- [x] Verify the current ROR schema version and official release asset from the ROR
       data-dump documentation at implementation time.
-- [ ] Consume only fields confirmed in that inspected schema.
-- [ ] Map names and aliases without merging ROR entities into Wikidata entities unless
+- [x] Consume only fields confirmed in that inspected schema.
+- [x] Map names and aliases without merging ROR entities into Wikidata entities unless
       an exact external identifier or reviewed equivalence supports the join.
-- [ ] Preserve links and domains as ROR assertions, not approvals.
-- [ ] Preserve status, type, country/location, external IDs, and upstream lineage.
-- [ ] Test domain conflicts, former/inactive organizations, aliases, multiple links,
+- [x] Preserve links and domains as ROR assertions, not approvals.
+- [x] Preserve status, type, country/location, external IDs, and upstream lineage.
+- [x] Test domain conflicts, former/inactive organizations, aliases, multiple links,
       missing fields, duplicate input, schema drift, and exact-ID equivalence.
 
 References: