nicweyand/argand-site-registry
1
0
Fork 0
Code Issues Pull requests Projects Releases 7 Packages Wiki Activity Actions

release: implement site registry v0.4 trust pipeline

Browse source
This commit is contained in:
Nic Weyand 2026-09-13 12:22:05 -04:00
commit e26efc19fa
Signed by: nicweyand
SSH key fingerprint: SHA256:2te+ycJIQON/Wo/dH6+ZkFSQ4HnHWpetV2azx9E65dQ
67 changed files with 10698 additions and 640 deletions
Download patch file Download diff file Expand all files Collapse all files

83
docs/ARCHITECTURE.md Normal file
View file

@ -0,0 +1,83 @@
# Architecture
Argand Site Registry is a local Rust library and CLI around a mutable SQLite
writer and immutable, content-pinned reader generations. Source acquisition,
evidence normalization, editorial decisions, and release authority remain
separate.
```mermaid
flowchart LR
S[Allowlisted source objects] --> I[Streaming adapters]
I --> W[(Writer store)]
O[Bounded candidate observer] --> B[Immutable observation batches]
B --> W
W --> G[Candidate generation]
G --> Q[Evidence bundles and review queue]
Q --> V[Signed reviewer votes]
V --> W
W --> R[Reviewed generation]
R --> P[Separate publisher signature]
P --> C[Verified lookup and resolve]
R --> E[Signed cumulative revocation feed]
E --> C
```
## Data boundaries
`sources`, `records`, and `facts` preserve provider-native evidence. Typed
coverage selects a coherent active set without deleting older snapshots.
`selected_sources` and `active_records` record that derivation. Names, entities,
web properties, edges, popularity, and rejected facts are deterministic
projections.
Names, website edges, and entity equivalences have separate material
fingerprints. Adding an alias therefore cannot inherit an approved destination,
and changing unrelated entity metadata does not invalidate an unchanged website
edge. Every evidence bundle contains the exact current assertion, provenance, and
attached observations that a vote signs.
Observation batches are append-only. Redirects, canonicals, hreflang, JSON-LD
`sameAs`, sitemaps, country selectors, HTTP state, public DNS-set hashes, TLS
certificate hashes, and bounded failures remain observations. They never create
an entity, ownership edge, role, or approval.
Votes are exact signed JSON documents. The writer records its own `accepted_at`,
the SSH signature, physical public-key digest, evidence-bundle digest, policy
digest, scope, expiry, and any explicitly superseded revocation IDs. Policy
compilation counts independent identities, groups, and physical keys. The
reference policy requires two independent approvals and makes one revocation
sticky.
## Trust boundaries
- Source HTTPS and a content digest establish what was imported, not whether the
assertion is true.
- Reviewer signatures establish who made an exact decision, not site safety.
- Publisher signatures authenticate a complete generation, not every assertion.
- `lookup` is an audit surface. `resolve` is the policy-enforced navigation
surface.
- Automated acquisition, observation, and builds stop at candidates. They have no
review, publisher, or activation authority.
Strict generation receipts bind database bytes, license and attribution files,
selected coverage, the review policy, reviewer trust-root bytes, and the trusted
acceptance-time rule. Readers copy authenticated SQLite bytes into a private
unlinked snapshot before opening them.
## Main modules
| Module | Responsibility |
| --- | --- |
| `download`, `crux` | Allowlisted, bounded source acquisition and resumable cache |
| `source`, `store`, adapters | Manifest validation and streaming source-specific import |
| `coverage` | Full/partition/delta graph validation and active-record masking |
| `normalize` | Deterministic URL, hostname, registrable-domain, suffix, and name normalization |
| `build` | Canonical immutable generation and receipt creation |
| `bundle`, `policy`, `vote` | Review evidence, policy epochs, authenticated quorum, revocation |
| `observer`, `observation`, `queue` | Candidate-only collection, replay/import, reverse lookup, drift and queues |
| `query`, `resolution`, `identity`, `catalog` | Audit lookup, equivalence, resolution, statistics |
| `release`, `revocation`, `generation`, `ssh` | Export, signature verification, emergency overlays, activation, rollback protection |
The database schema lives in ordered migrations. JSON and receipt contracts have
their own schema strings and fail closed on unknown versions. See
[FORMATS.md](FORMATS.md) and [MIGRATING-0.4.md](MIGRATING-0.4.md).

135
docs/CONSUMERS.md
View file

@ -1,65 +1,100 @@
# Consumer and compatibility contract
## Rust library
## Use the policy-enforced reader
Use `release::verify_signed(generation, publisher_signers, publisher_identity,
reviewer_signers)` once per production generation and reuse the returned reader.
`Registry::open(generation, trusted_pin)` is the lower-level path when the pin
distributor is also trusted for the complete review decision. `lookup(query,
limit)` returns evidence and complete ambiguity counts; `resolve_explained` returns a reviewed candidate or
a typed abstention reason with counts. Exact reverse views cover entity IDs,
URLs/domains, popularity and Curlie categories. Check the compiled example and API
docs for exact types. `selection_context` binds the full alternative set for
downstream query review. Preserve returned provenance, scopes, counts and attribution.
For a production generation, call
`release::verify_signed(generation, publisher_signers, publisher_identity,
reviewer_signers)` once and reuse the returned `Registry`. `Registry::open` is the
lower-level API for deployments that already trust an exact `COMPLETE.json`
SHA-256 for the whole publication decision.
For local integration, point a Cargo dependency at
`crates/argand-site-registry` inside an extracted standalone source tree. Once an
upstream repository is published, use its actual Git URL and a full reviewed `rev`
pin. Do not invent a crates.io version or track a mutable branch in production.
Both crates remain in this workspace; the atomic helper is a relative dependency.
`lookup` returns source evidence, candidates, ambiguity counts and attribution.
`resolve_explained` returns either one policy-qualified destination or a typed
abstention. It requires an approved name binding and approved edge under the
receipt's policy. Preserve the full response, especially `destination: null`,
`status`, counts, selected scope, evidence and attribution.
Reverse views cover entity IDs, exact URLs, hostnames, registrable domains,
source-specific popularity, Curlie categories and observations. They are audit
operations and do not imply ownership or admission.
## CLI and other languages
`lookup`, `resolve`, `entity`, `lookup-web`, `popularity`, `category`, `stats`,
`evaluate`, `verify` and the other commands emit JSON. The Python example
passes arguments directly to the native executable, preserving query text and the
entire response. A nonzero exit is an error. `destination: null` is a successful
abstention, not a request to pick the first lookup candidate. Render source names
as untrusted text and satisfy their source-specific attribution requirements.
Every CLI command emits one JSON value to stdout; typed export and diff commands
write bounded JSONL files. A nonzero exit is an error. A successful resolution
with a null destination is an intentional abstention, not an instruction to use
the first lookup result.
Use a bounded process or service wrapper appropriate to your workload. The Python
example has a 60-second timeout and invokes a fresh reader per request; for repeated
low-latency queries, use the reusable Rust reader. No hosted API or Python package
registry publication is claimed by this repository.
The Python example passes an argument array directly to the native CLI and keeps
its 60-second timeout and complete JSON response. Use the Rust reader for repeated
low-latency queries. No hosted API, crates.io release or Python package is claimed.
## SQLite, JSONL and license scope
Render all imported names, categories, URLs and evidence as untrusted data. Apply
source attribution and application-specific malware/content policy.
Distribute `registry.sqlite`, `COMPLETE.json`, `LICENSE_SOURCES.md` and
`ATTRIBUTION.json` together, plus the publisher signature when applicable. The
database contains audit records and source descriptions. Default JSONL export
omits descriptions and includes fact provenance plus an attribution envelope.
It is an assertion export, not a self-contained signed list of admitted routes.
Raw SQL inspection is useful for audit; it does not implement resolution policy.
## Current contracts
Code version 0.3.0 uses schema version 3 and `argand.site-rules/v3`.
It adds reviewer-trust enforcement for consumers, private authenticated SQLite
snapshots, exact-stream import checks, bounded outputs and metadata-bound identity
decisions. Schema/rule contracts remain versioned independently in
receipts. Unsupported contracts fail closed. Pin source releases,
compile consumers and replay fixed fixtures before upgrades. Preserve import and
review history; never mutate complete generations to migrate them.
Code version 0.4.0 uses writer schema 5 and `argand.site-rules/v4`.
`COMPLETE.json` uses `argand.site-registry/v2` and binds:
- authenticated `registry.sqlite` bytes;
- the active source coverage graph and exact PSL source;
- review policy and exact reviewer trust-root bytes;
- decision-time policy;
- source license and machine-readable attribution files; and
- entity, property, edge and rejection counts.
Normal `export` uses `argand.site-export/v2`, contains selected active nonrejected
facts, and marks every assertion `active`. `export-audit` uses the same schema with
mode `audit` and includes superseded/rejected states and rejection reasons. Neither
contains a list of resolver-approved routes.
Votes use `argand.site-vote/v1` and the OpenSSH namespace
`argand-site-registry-vote`. Consumers compile them under the exact receipt-bound
policy and trusted query time. Unknown schema or rule versions fail closed.
Emergency feeds use `argand.site-revocations/v1` and the separate
`argand-site-registry-revocations` namespace. Call `revocation::verify` against
the cached `Registry` and publisher trust root, then use
`resolve_explained_with_revocations`. The CLI accepts the same four feed arguments
on `resolve`. It blocks exact revoked names, edges, and equivalences before route
selection and can choose another approved edge. Verify each replacement against
the previously accepted feed to enforce cumulative continuity. Feeds expire after
seven days. A feed from a newer compatible generation may add blocks to a cached
generation, but reinstatement requires installing the exact full generation so
the consumer can recompute the authenticated superseding quorum.
CLI consumers pass the last feed and signature as `--previous-revocations` and
`--previous-revocation-signature` on subsequent `resolve` calls. The Rust API
passes the last `VerifiedRevocations` to `revocation::verify` before resolution.
Distribute `registry.sqlite`, `COMPLETE.json`, `LICENSE_SOURCES.md`,
`ATTRIBUTION.json`, and `COMPLETE.json.sig` together. Obtain publisher and reviewer
trust roots independently. Raw SQL copies and JSONL extracts do not implement
resolution policy, expiry, revocation continuity or signature verification.
## Compatibility
V1 source manifests remain readable as isolated legacy provider/scope streams.
A deliberate legacy-compatible build policy can replay v0.3 reviews, but strict
0.4 builds require votes and reviewer trust. Current readers accept v2 receipts;
rollback checks may open pinned v1 receipts with rules v1-v3 only to compare
revocation history.
Before upgrading, pin the source release by full signed Git revision, compile the
consumer and replay fixed fixtures. Preserve import, vote, observation and
revocation history. Never mutate a complete generation to migrate it. See
[MIGRATING-0.4.md](MIGRATING-0.4.md) and [FORMATS.md](FORMATS.md).
## Argand integration
UPSTREAM.json pins the exact Argand source baseline, including the beta agent's
selection-context API. The first standalone extraction preserves runtime Rust and
migration bytes. Argand main commit
`d9dfd1585ce21d9c4136bcc24fa01fe3bfb8ed6e` replaced its embedded workspace
crate with this signed `v0.3.0` release at full Git revision
`ac8282093d8a815c6227cff86e1f40714d510bcd`.
`UPSTREAM.json` records the original Argand extraction baseline and file hashes.
The last recorded downstream integration replaced Argand's embedded crate with
signed v0.3.0 revision `ac8282093d8a815c6227cff86e1f40714d510bcd` at Argand
commit `d9dfd1585ce21d9c4136bcc24fa01fe3bfb8ed6e`.
Develop the library here and update Argand through explicit reviewed revision-pin
changes. Each update must compare the old and new contracts and rerun Argand's
navigation compiler and API gates. Preserve existing registry receipts and public
navigation admission; a source dependency change does not activate a registry
generation or approve a destination.
Version 0.4 is handed off as a signed standalone revision. Argand should update its
full Git `rev` in a separate coordinated source/build window, compare contract
changes, and rerun navigation compiler, native resolver, API, abstention,
revocation and clean-process gates. Changing the code dependency does not activate
a registry generation or approve a public destination.

40
docs/EVALUATION.md
View file

@ -1,23 +1,20 @@
# Resolver evaluation
`evaluate` replays authored judgments through the same native resolver used by
consumers. It opens one externally pinned immutable generation, streams up to the
configured case limit and reports correctness plus native p50/p95 query latency.
Pass an explicit `--at` time when the report must be replayable across approval
expiry boundaries; the chosen clock is included in the report.
The input is bounded to 16 MiB, each line to 64 KiB and case IDs must be unique.
Each nonempty JSONL line has this form:
`evaluate` streams authored JSONL judgments through the same pinned native reader
used by consumers. It reports correctness and p50/p95 latency. Pass `--at` for a
reproducible policy clock; approval expiry and future votes otherwise depend on
current time. Inputs are bounded to 16 MiB, lines to 64 KiB, and case IDs must be
unique.
```json
{"id":"facebook-primary","query":"facebook","locale":null,"country":null,"expected_status":"resolved","expected_entity_id":"argand:entity:SOURCE:ID","expected_url":"https://www.facebook.com/"}
```
`locale`, `country`, `expected_entity_id` and `expected_url` are optional. Status
is required and is one of `resolved`, `no_name_match`, `ambiguous_identity`,
`safety_limit_exceeded`, `no_eligible_destination`, `no_active_review`,
`region_mismatch` or `ambiguous_destination`. Expected URLs must be copied from a
pinned registry, including normalization such as a trailing slash.
Optional fields are `locale`, `country`, `expected_entity_id`, and `expected_url`.
Current statuses are `resolved`, `no_name_match`, `ambiguous_identity`,
`no_active_name_review`, `safety_limit_exceeded`, `no_eligible_destination`,
`no_active_review`, `region_mismatch`, and `ambiguous_destination`. Copy expected
URLs from a pinned registry, including normalization such as trailing slash.
```bash
argand-site-registry evaluate --generation /data/registry/reviewed \
@ -27,9 +24,14 @@ argand-site-registry evaluate --generation /data/registry/reviewed \
jq -e '.failed == 0 and .passed == .total' /data/evaluation/report.json
```
Keep the corpus version and digest with the report. Include canonical names,
aliases, Unicode normalization, multiple scripts, every served country/locale,
unknown names, deceptive lookalikes, ambiguous entities, expired/revoked reviews
and tied destinations. Use synthetic or authorized query material; do not commit
private user logs. Latency values are process and hardware measurements. Compare
them only with an equivalent environment and sufficient sample size.
Keep corpus version and digest, registry pin, policy digest, reviewer-trust digest,
process build and hardware with the report. Cover canonical names, aliases,
Unicode normalization, scripts, regions, unknown names, lookalikes, ambiguity,
missing name votes, missing edge votes, correlated reviewers, stale policy,
stale observations, expiry, revocation, sticky supersession and tied routes.
Segment failures by safe abstention and wrong resolved route; any wrong resolved
route is a release blocker.
Use synthetic or authorized queries and do not commit private user logs. Latency is
process and hardware evidence. Compare only equivalent native configurations with
sufficient samples.

61
docs/FORMATS.md Normal file
View file

@ -0,0 +1,61 @@
# Versioned formats
Version 0.4 uses writer schema 5 and `argand.site-rules/v4`. Schema identifiers
are independent from the crate version. Unknown schemas and rules fail closed.
| Artifact | Current schema | Purpose |
| --- | --- | --- |
| Source manifest | `argand.site-source/v2` | Exact source object plus typed coverage |
| Generation receipt | `argand.site-registry/v2` | Hash-bound immutable generation contract |
| Active/audit JSONL | `argand.site-export/v2` | Source-bearing assertions with selection state |
| Evidence bundle | `argand.site-evidence-bundle/v1` | Exact evidence signed by reviewer votes |
| Vote | `argand.site-vote/v1` | Authenticated approve/revoke decision |
| Review policy | `argand.site-policy/v1` | Threshold, groups, revocation, and separation rules |
| Observation batch | `argand.site-observation-source/v1` | Pinned JSONL object and rights |
| Observation | `argand.site-observation/v1` | Normalized subject-bound observation |
| Observer capture | `argand.site-observer-capture/v1` | Cache-only replay input |
| Emergency revocations | `argand.site-revocations/v1` | Cumulative publisher-signed offline block overlay |
| Active pointer | `argand.site-current/v2` | Signed generation pin plus revocation continuity |
| Diff | `argand.site-diff/v4` | Typed change stream across generations |
Source manifest v2 coverage is one of `full`, `partition`, or `delta`. A delta
names its exact base, positive consecutive sequence, and superseded source IDs.
A full source cannot compose with active partitions. Overlap, gaps, cycles,
missing bases, cross-provider supersession, and mixed legacy/typed frontiers fail
the build.
The normal export emits selected active facts only and excludes rejected facts.
Every assertion has `selection_state: "active"`. Audit export includes active,
superseded, and rejected facts; rejected assertions include their reason. Both
modes redact Curlie descriptions by default and neither represents approved
navigation routes. Use `resolve` for admission.
Votes bind the subject kind and fingerprint, evidence-bundle digest, policy
digest, reviewer, decision, reason, asserted review time, optional expiry, exact
role/locale/country scope, and revocation supersession IDs. Exact JSON bytes are
signed with OpenSSH namespace `argand-site-registry-vote`. Writer `accepted_at`
is separate and cannot be supplied by the reviewer.
Policy fields define base thresholds for names, edges, and equivalences; maximum
approval and observation ages; reviewer groups; publisher separation; source and
drift holds; and optional stricter `risk_thresholds` for `source_conflict` and
`dangerous_drift`. Unknown fields and risk classes fail closed.
Emergency feeds retain every authenticated revocation ID across policy epochs,
active state, any explicit superseding quorum, and a refresh deadline no more
than seven days after the effective time. Exact JSON bytes use OpenSSH namespace
`argand-site-registry-revocations`. A feed can overlay a cached generation only
when rules, policy, and reviewer trust-root digests match. Feed continuity rejects
dropped subjects or vote IDs. A feed from another compatible generation may only
add blocks. Removing a block requires the exact full generation so the consumer
can recompute the signed superseding quorum; expired feeds fail closed.
Legacy source manifests remain readable as isolated `(source, scope)` streams.
Legacy 0.3 reviews remain available only through the explicit compatibility
policy. New strict builds use votes and a receipt-bound reviewer trust root.
The contract decisions are recorded in [ADR 0001](adr/0001-typed-source-coverage.md),
[ADR 0002](adr/0002-granular-trust-subjects.md),
[ADR 0003](adr/0003-votes-revocations-and-publishers.md),
[ADR 0004](adr/0004-active-and-audit-views.md), and
[ADR 0005](adr/0005-full-delta-release-identity.md).

15
docs/INDEX.md
View file

@ -3,6 +3,9 @@
- [README](../README.md): build, examples and scope.
- [Operator guide](../crates/argand-site-registry/README.md): all source commands,
schema, reviews, regional resolution, releases and update configuration.
- [Architecture](ARCHITECTURE.md): source, observation, vote, policy and release boundaries.
- [Versioned formats](FORMATS.md): current schema identifiers and compatibility.
- [Migrating to 0.4](MIGRATING-0.4.md): writer migration and trust transition.
- [Source licenses](../crates/argand-site-registry/LICENSE_SOURCES.md): exact terms and attribution.
- [Consumers](CONSUMERS.md): Rust, Python/CLI, data distribution and Argand transition.
- [Trust](TRUST.md): enforced checks and publisher/consumer responsibilities.
@ -10,7 +13,17 @@
- [Evaluation](EVALUATION.md): bounded JSONL judgments and result interpretation.
- [Releasing](RELEASING.md): CI, source signing and archive verification.
- [Validation](VALIDATION.md): independent builds and native acceptance evidence.
- [Version 0.4 security review](SECURITY-REVIEW-0.4.md): threat boundaries,
resolved findings and residual operator responsibilities.
- [Contributing](../CONTRIBUTING.md), [governance](../GOVERNANCE.md),
[security](../SECURITY.md): proposals, decisions and incidents.
- [Extraction design](superpowers/specs/2026-09-12-standalone-design.md) and
[implementation plan](superpowers/plans/2026-09-12-standalone.md).
[initial implementation plan](superpowers/plans/2026-09-12-standalone.md).
- [0.4 and beyond plan](superpowers/plans/2026-09-13-v0.4-and-beyond.md).
- Architecture decisions: [coverage](adr/0001-typed-source-coverage.md),
[trust subjects](adr/0002-granular-trust-subjects.md),
[votes and publishers](adr/0003-votes-revocations-and-publishers.md),
[active/audit views](adr/0004-active-and-audit-views.md),
[release identity](adr/0005-full-delta-release-identity.md),
[source lineage](adr/0006-source-lineage.md), and
[embedding intent](adr/0007-distribution-and-embedding.md).

53
docs/MIGRATING-0.4.md Normal file
View file

@ -0,0 +1,53 @@
# Migrating from 0.3 to 0.4
Back up the writer database, source cache, generations, approvals, and trust files
before upgrading. Do not edit a complete generation in place.
1. Install the 0.4 binary and run `verify` against every retained 0.3 generation
using their existing trusted pins.
2. Open the writer with 0.4. Migrations add authenticated votes and immutable
observation batches, moving `PRAGMA user_version` from 3 to 5 without removing
source or legacy review history.
3. Keep existing v1 source manifests as isolated legacy scopes. Use v2 manifests
with explicit full/partition/delta coverage for new replacement chains.
4. Use the strict reference policy or supply a reviewed policy JSON. Create an
independent OpenSSH reviewer allowed-signers file. Strict `build` requires
`--reviewer-trust`; its exact bytes are bound into the receipt.
5. Use `review-queue` and `evidence` to find each name, edge, and equivalence that
needs votes. Prepare, sign, verify, and append votes, then rebuild.
6. Expect `resolve` to abstain until the selected name and destination edge each
satisfy the new quorum. Audit `lookup` remains available throughout migration.
7. Compare `stats`, `diff`, `export`, and `export-audit`; replay evaluation; then
sign and activate with a publisher identity and key that did not approve.
The legacy compatibility policy exists for controlled replay and transition. It
is not the CLI default and does not provide the reference two-reviewer guarantee.
Legacy approvals remain auditable. A publisher should collect new votes rather
than silently translating old reviews into quorum votes.
## Behavior changes
- Name bindings and website edges now have separate review identities. A new alias
needs its own name votes and cannot inherit a route. An accepted alias update
does not invalidate an unchanged website edge.
- Votes bind the exact review policy. Changing thresholds or reviewer groups makes
earlier votes stale under the new policy; the source evidence is unchanged.
- Reviewer time is advisory. Effective validity starts at the later of signed
`reviewed_at` and writer `accepted_at`, and ends no later than 90 days after
acceptance.
- One retained authenticated revocation is sticky across policy epochs. Ordinary
later approvals do not clear it. Every approval in a fresh quorum must
explicitly supersede every active revocation ID.
- Publisher-signed cumulative feeds can apply new blocks to an older compatible
pinned registry before the complete replacement arrives. Their policy, rules,
reviewer trust, and prior-feed continuity are verified.
- Active export omits superseded and rejected facts. `export-audit` retains them
with explicit state.
- A strict receipt binds selected coverage, review policy, reviewer trust bytes,
and the decision-time contract. Consumers reject unknown contracts.
- Observation imports and observer caches are evidence only. New or changed
observations make existing edge evidence stale and return the item to review.
The 0.4 reader accepts current v2 receipts. Rollback validation can open pinned
v1 receipts using rules v1-v3 only to compare retained revocations. Preserve old
generation directories and pins until every consumer has moved successfully.

140
docs/PUBLISHING.md
View file

@ -1,82 +1,114 @@
# Dataset publisher runbook
This runbook creates a reviewable candidate and a signed activation. Source update,
review and release keys are separate authorities. Scheduled jobs may acquire,
import and build candidates; they do not approve, sign or activate destinations.
Scheduled acquisition and observation jobs create evidence and candidates. Human
reviewers vote on exact bundles. A separate publisher signs and activates an
accepted generation.
## Trust roots and local state
## Prepare trust and state
Keep cache, mutable store, immutable generations, reviewer keys, release keys and
consumer trust files outside the checkout. An OpenSSH reviewer trust file contains
one accepted principal and public key per line:
Keep source caches, captures, writer database, generations, reviewer keys,
publisher keys and consumer trust files outside the checkout. A reviewer
allowed-signers file contains one principal and public key per line. Add OpenSSH
validity options when rotating keys and preserve old keys for retained history.
```text
operator@example.org ssh-ed25519 REVIEWER_PUBLIC_KEY
```
Create a reviewed policy JSON when the reference policy is not appropriate. The
reference policy requires two independent identities, groups and physical keys
for every name, edge and equivalence; sticky revocations and publisher separation
are mandatory. Record actual group membership in `reviewer_groups`. Optional
`risk_thresholds` can require a larger quorum for `source_conflict` or
`dangerous_drift`; blocking those risks remains separately configurable.
For rotation, retain an old public key with an OpenSSH `valid-before` option
covering its signed decision times. New decisions are checked at append time, so
an expired key cannot submit backdated reviews; historical release verification
uses each authenticated `reviewed_at`. Remove a retired key only after no retained
generation or review log depends on it.
## Build and inspect a candidate
Distribute the release publisher public key to consumers through an independent
authenticated channel. Do not put private keys, production trust files or source
datasets in Git or CI. The isolated source CI runner has none of these files.
1. Import only manifests whose source, format, license and typed coverage were
checked. Keep the object and acquisition receipt.
2. Build with `--reviewer-trust`; record the returned receipt pin.
3. Run `verify`, `stats`, `diff`, `export-audit` and the fixed evaluation corpus.
4. Inspect `review-queue`, `evidence`, exact entity/domain reverse lookups and
source-separated popularity.
5. Run candidate observations on a bounded schedule and import their manifests.
Review `drift` and `revocation-candidates`; rebuild after imports.
## Candidate acceptance
Queue viewing is read-only. Observation import changes evidence bundles but never
route state.
Run `update` or the explicit download/import/build commands from the operator guide.
For every candidate generation:
## Create authenticated votes
1. Verify its externally recorded receipt pin with `verify`.
2. Run `stats` and compare source snapshots, selected sources, facts, rejections,
reviews, database bytes and upcoming expiry with the prior accepted generation.
3. Run `diff` against the prior pin. Investigate every source-selection, identity,
name, property, edge, popularity, review and equivalence change.
4. Use `lookup`, `entity`, `lookup-web`, `popularity` and `category` to inspect exact
source evidence and conflicts. Popularity never proves ownership.
5. Replay the maintained evaluation corpus with `evaluate`; require zero judgment
mismatches and compare latency with a documented hardware/process baseline.
## Authenticated decisions
Create a bounded review JSON from the exact candidate fingerprint and evidence.
Sign its exact bytes and append it through the CLI:
Use `prepare-vote` for a name or edge, or `prepare-equivalence-vote` for an exact
entity pair. The command writes canonical JSON containing the current evidence
and policy digests. Review those exact bytes, then sign them:
```bash
ssh-keygen -Y sign -n argand-site-registry-review \
-f /secure/reviewer-key review.json
argand-site-registry review --database /data/registry/import.sqlite \
ssh-keygen -Y sign -n argand-site-registry-vote \
-f /secure/reviewer-one vote.json
argand-site-registry verify-vote \
--generation /data/registry/candidate --pin "$CANDIDATE_PIN" \
--decision review.json --signature review.json.sig \
--decision vote.json --signature vote.json.sig \
--allowed-reviewers /secure/reviewer-allowed-signers \
--identity operator@example.org
--identity reviewer-one
argand-site-registry vote \
--database /data/registry/import.sqlite \
--generation /data/registry/candidate --pin "$CANDIDATE_PIN" \
--decision vote.json --signature vote.json.sig \
--allowed-reviewers /secure/reviewer-allowed-signers \
--identity reviewer-one
```
Identity equivalence decisions use the same signed JSON and reviewer namespace.
Rebuild after appending decisions, then repeat the complete diff and evaluation.
The release command re-verifies every retained reviewer signature against the
current reviewer trust file. Missing, altered or no-longer-trusted proofs stop it.
Repeat with enough independently controlled identities, groups and keys. Use
`equivalence-vote` to append equivalence votes after
`verify-equivalence-vote`. Rebuild and check the compiled decision. Never share
one private key under several reviewer names.
An approval expires within 90 days and begins no earlier than writer acceptance.
A revocation has no expiry. To restore a revoked subject, every new approval in a
complete quorum must list every active revocation ID in `supersedes`.
## Sign and activate
Run all offline gates and the separate network-enabled dependency audit. The
publisher identity and physical key must not have supplied any reviewer vote.
```bash
argand-site-registry sign --generation /data/registry/reviewed \
--pin "$REVIEWED_PIN" --key /secure/release-key \
--allowed-reviewers /secure/reviewer-allowed-signers
--pin "$REVIEWED_PIN" --key /secure/publisher-key \
--allowed-reviewers /secure/reviewer-allowed-signers \
--identity registry-publisher
argand-site-registry activate --generation /data/registry/reviewed \
--current /data/registry/current.json \
--allowed-signers /secure/release-allowed-signers \
--allowed-signers /secure/publisher-allowed-signers \
--allowed-reviewers /secure/reviewer-allowed-signers \
--identity registry-publisher
```
Record the source commit, candidate and accepted receipt pins, typed diff, evaluation
report, reviewer trust-file digest, release signer identity and activation receipt
in an immutable operator log. Activation refuses a rollback that drops a distributed
revocation. Deliver a new current pointer/pin to every consumer and bound their caches.
Record the source-code revision, provider manifest IDs, candidate and reviewed
pins, policy and reviewer-trust digests, diff, evaluation report, publisher
identity, and activation result in an immutable operator log outside Git.
Distribute the complete generation, signature, and independently authenticated
publisher trust root. Confirm every consumer received the new revocation state.
For an incident, append a signed revocation, rebuild with full history, inspect,
evaluate, sign and activate. Preserve the suspect source bytes, generation and proofs.
Follow [SECURITY.md](../SECURITY.md) for private reporting and key compromise.
For an incident, preserve the suspect evidence, append a signed revocation,
rebuild with full history, evaluate, sign and activate. Use
`revocation-candidates` only as review input; it never signs or appends a vote.
To protect pinned consumers before the full generation arrives, publish a small
cumulative feed under the distinct publisher namespace:
```bash
argand-site-registry export-revocations \
--generation /data/registry/revoked --pin "$REVOKED_PIN" \
--effective-at 2026-09-13T00:00:00Z --output /data/revocations.json
argand-site-registry sign-revocations \
--generation /data/registry/revoked --pin "$REVOKED_PIN" \
--input /data/revocations.json --output /data/revocations.json.sig \
--key /secure/publisher-key \
--allowed-reviewers /secure/reviewer-allowed-signers \
--identity registry-publisher
```
Signing recomputes the feed from the pinned generation and re-verifies every
reviewer vote. Distribute the exact feed, signature, and publisher trust root.
When replacing a feed, verify it with the prior feed and signature so a mirror
cannot discard a revocation. Refresh feeds before their seven-day deadline. A
newer compatible feed can block an older pinned generation, but clearing that
block requires installing its exact full generation so the consumer can recompute
the authenticated superseding quorum.
Follow [SECURITY.md](../SECURITY.md) for private disclosure and key compromise.

6
docs/RELEASING.md
View file

@ -87,7 +87,11 @@ receipt; the initial release tool packages source only.
Source releases contain no provider datasets or real approvals. Dataset publishers
follow the operator guide: import, inspect, review, build, diff, sign and activate.
Follow the [publisher runbook](PUBLISHING.md); release signing requires the external
reviewer trust file and re-verifies every stored decision signature.
reviewer trust file and re-verifies every stored decision signature. Version 0.4
strict releases also require `sign --identity`; the publisher identity and
physical key must not have supplied an approval vote. Run
`cargo audit --deny warnings` as a separate network-enabled gate and record the
result in [VALIDATION.md](VALIDATION.md).
The dataset namespace `argand-site-registry` is distinct from the source namespace
above. Keep `LICENSE_SOURCES.md` and `ATTRIBUTION.json` with the database and receipt.
The weekly update example creates candidates. It never approves, renews, signs or

76
docs/SECURITY-REVIEW-0.4.md Normal file
View file

@ -0,0 +1,76 @@
# Version 0.4 security review
Reviewed 2026-09-13 after implementation and before release promotion.
## Scope and trust boundaries
The review covered source-manifest and acquisition validation, typed coverage,
SQLite migrations and projections, observation import and candidate-site network
access, reviewer votes and policy compilation, generation and revocation signing,
consumer verification, CLI argument relationships, systemd isolation, dependency
advisories, source packaging, and accidental credential or dataset disclosure.
The design treats provider bytes, websites, DNS answers, redirects, observation
batches, public proposals, mirrors, and lookup results as untrusted. Reviewer and
publisher private keys, the mutable writer database, consumer trust roots, and the
host operating system remain privileged. A source assertion or observation cannot
approve a route. One reviewer cannot satisfy the reference approval quorum, while
one authenticated revocation can stop an exact subject.
## Findings fixed before release
1. **Cross-generation revocation reinstatement:** a publisher-signed feed could
claim supersession while an older consumer lacked the reviewer votes needed to
recompute it. Cross-generation feeds now only add blocks. Clearing a block
requires the exact full generation containing the authenticated fresh quorum.
2. **Stale emergency feeds:** feeds previously had no artifact deadline and a
verified object could be reused indefinitely. Feeds now expire within seven
days, and freshness is checked both during verification and every resolution.
3. **Resolver continuity:** `verify-revocations` accepted a previous feed, while
`resolve` had no equivalent input. `resolve` now accepts the last feed and
signature and refuses replacements that discard subjects or vote IDs.
4. **Publisher/reviewer separation:** physical-key and identity checks covered
approval voters only. They now cover every reviewer vote, including emergency
revocations, and remove a rejected signature output.
5. **Observation-batch mutation:** completed batch counters, state, and membership
were not all protected by schema triggers. A batch must now be created open,
can complete once only with its exact row count, and cannot accept later rows,
change, or be deleted.
6. **Policy-epoch revocation bypass:** a new policy epoch retained an old signed
revocation in the audit log but excluded it from compilation. All authenticated
revocations now remain sticky across policy epochs until a fresh quorum under
the active policy explicitly supersedes them.
7. **Special-address observation targets:** the outbound filter omitted several
IPv4 and IPv6 special-use ranges. The observer now also blocks IPv4-compatible,
site-local, translation, discard, benchmarking, ORCHID, documentation, and 6to4
destinations before constructing a pinned client.
Regression coverage includes expired verified objects, signed cross-generation
reinstatement attempts, physical reviewer-key reuse by a publisher, immutable
observation batches and late inserts, policy-epoch revocation changes, signature
and feed tampering, redirect loops, private and reserved address ranges,
compressed bodies, malformed markup, extraction caps,
and a deterministic 512-case parser mutation corpus.
## Review result
No known critical, high, or medium security finding remains in the reviewed 0.4
scope. `unsafe` Rust is forbidden workspace-wide. External SSH operations use
argument vectors and descriptor-bound private temporary files. Generation reads
hash a no-follow source into a private unlinked SQLite snapshot before querying.
Network clients disable ambient proxies; source acquisition uses reviewed HTTPS
endpoints and explicit byte limits; the observer pins an entirely public DNS set
per hop and bounds redirects, headers, body bytes, bandwidth, time, and extracted
links.
`cargo audit --deny warnings` scanned 1,243 RustSec advisories across 272 locked
dependencies without a finding. The complete offline gate separately exercises
strict Clippy, documentation, unit and integration tests, native CLI behavior,
consumer parity, source-package defenses, and the synthetic five-source import.
`systemd-analyze verify` accepted the observer service and timer; its only output
was an unrelated warning from the host's installed `arch-audit.service`.
This review authenticates software behavior, not provider truth or a public
dataset. Publishers must protect writer and signing authority, inspect evidence,
retain the last accepted feed, refresh it before expiry, and distribute trust
roots through an independent authenticated channel.

154
docs/TRUST.md
View file

@ -1,73 +1,115 @@
# Trust and evidence policy
Argand Site Registry is designed to fail closed. Imported assertions and crawler
observations become review evidence. Only policy-qualified votes can make a name
or destination resolvable, and only a separately authenticated generation should
reach consumers.
## What the implementation enforces
Inputs use reviewed source adapters and explicit manifests. URL normalization,
PSL parsing and stable identities are deterministic. Fact provenance, conflicting
evidence and source-specific popularity remain separate. Names and similar
hostnames never silently join entities. Invalid URLs and malformed imports fail
validation; a partial import does not replace a complete source selection.
Source adapters accept only documented providers and formats. Manifests bind the
exact object, origin URL, source-native snapshot, license, retrieval time, byte
length, digest, and typed coverage. Full, partition and delta graphs reject gaps,
cycles, overlap, cross-provider replacement and ambiguous active branches. Failed
or incomplete imports cannot replace complete evidence.
Destination and identity decisions bind exact evidence fingerprints, including
names, assertions and normalization context. Reviews expire within 90 days.
Changed evidence invalidates earlier approvals. Resolution abstains on ambiguity,
ties, missing approval or ineligible claims; regional scopes must explicitly
match. Reviewer decisions are signed under a dedicated SSH namespace and the local
writer retains their exact decision/signature bytes in the append-only review log.
URL and domain normalization is deterministic and uses the complete retained
Public Suffix List, including PRIVATE rules. Names and hostnames never merge
entities. Source-specific popularity stays separate from identity. Every fact
keeps source, source identifier, selector, license, retrieval time, confidence and
raw evidence needed for audit.
Generations bind the database, license document and attribution to a completion
receipt. Consumers provide a trusted hash or verify an external publisher key.
Release signing re-verifies every stored decision against an external reviewer
trust file. Activation checks publisher signatures, re-verifies every review
against a separately supplied reviewer trust file and refuses rollback that loses
distributed revocations. Reviewer validity epochs are evaluated at decision time
for retained history while new decisions must pass the trust policy at append
time. Updates build candidates and cannot approve, sign or activate them.
Names, entity-to-property edges and entity equivalences have independent material
fingerprints. Under the reference policy, `resolve` needs two independent votes
for the matched name and two for the selected edge. Reviewer groups and physical
SSH public keys are deduplicated, so aliases for one person or key do not satisfy
quorum. Regional roles require explicit locale or country scope.
A vote signs exact JSON in the `argand-site-registry-vote` namespace and binds the
current assertion, evidence bundle and policy epoch. The writer supplies
`accepted_at`; effective validity starts at the later of acceptance and the
reviewer's time and ends no later than 90 days after acceptance. Future, expired,
wrong-policy, stale-evidence, malformed, untrusted and altered votes do not count.
A retained authenticated revocation is sticky across policy epochs. It blocks
the exact subject until every member of a complete fresh quorum explicitly
supersedes every active revocation ID. Sequence order alone cannot restore a
route. Activation prevents rollback past retained legacy or vote revocations.
A cumulative emergency feed carries the same granular revocation identities under
a distinct publisher signature namespace. Compatible pinned consumers apply it
before name, equivalence, and edge selection. Replacement feeds cannot discard
previous subjects or vote IDs. Cross-generation feeds can only add blocks; a block
can be cleared only against the exact full generation containing the authenticated
superseding quorum. Feed artifacts expire after seven days and must be refreshed.
The observer accepts only an eligible imported edge. It uses public DNS pinning
for every hop, rejects credentials, private/link-local/reserved targets,
nondefault ports and HTTPS downgrade, and bounds time, redirects, response headers,
raw body bytes, bandwidth and extracted links. It rejects compressed bodies and
redirect loops. Capture replay checks the complete chain and
body digest without network access. HTTP, redirect, canonical, hreflang, JSON-LD,
sitemap, country-selector, DNS, TLS and failure records remain observations. They
cannot approve a name, ownership relationship or role.
The observation contract also accepts rights-reviewed domain-registration state
and malware-policy results without naming a vendor. No such provider is built in;
an operator must verify commercial-reuse terms and preserve its exact source and
rights declaration before importing those records.
Generations bind authenticated SQLite bytes, selected coverage, policy, reviewer
trust bytes, licenses, attribution, and decision-time rules into `COMPLETE.json`.
Readers verify the receipt pin and copy the database into a private unlinked file
before SQLite opens it. Release signing and activation reverify every stored
signature. The strict policy rejects a publisher identity or physical key used for
any reviewer vote.
## What a publisher must establish
The reviewer name and evidence locator in a decision are operator assertions.
The CLI authenticates exact decision bytes to an allowed SSH signer and validates
their structure and evidence binding; it does not retrieve the cited evidence or
prove website ownership.
Protect the writer database and signing key with separate operating permissions.
Restrict who can author decisions and require human review before release signing.
The software verifies evidence integrity and decision authorization. A publisher
still has to determine that the source and observation evidence support the exact
entity, URL, relationship and role. TLS, DNS control, a redirect, `sameAs`, ccTLD,
popularity or source confidence alone is insufficient.
Publish dated evidence supporting the exact entity, URL, relationship, role and
country/locale. Prefer independently corroborated primary evidence with immutable
capture identifiers. Record contrary evidence and uncertainty. TLS, DNS control,
registrable-domain spelling, redirects, `sameAs` or popularity alone cannot
establish every identity or role claim. Future crawler observations remain inputs
to review. Confidence values are assertion scores, not calibrated probabilities.
Publishers should:
Choose expiry based on volatility, within the enforced maximum. Do not renew
blindly on a timer. Expired approval should lead to abstention until evidence is
reviewed. Disclose editorial conflicts and use an independent reviewer for a
disputed claim when possible. The implementation is a local single-writer tool
with externally authenticated reviewer keys; it does not provide accounts or an
enforced quorum.
- keep acquisition, writer, reviewer and release authorities separate;
- protect reviewer and publisher keys outside the repository and CI;
- configure real organizational reviewer groups instead of relying only on unique
identity strings;
- examine conflicts and current independent evidence in each review bundle;
- choose shorter expiries for volatile or high-risk routes;
- refresh observation evidence independently from source import cadence;
- review drift and produce signed emergency revocations promptly;
- run diff, evaluation, license and signature gates before release; and
- preserve source objects, generations, trust roots, pins and revocations for
audit and recovery.
Changing policy or reviewer trust produces a different receipt. It does not
silently reinterpret old votes as decisions under the new policy. Removing a key
can also make retained signature verification fail; plan rotations with OpenSSH
validity epochs and immutable history.
## What consumers must preserve
Authenticate a release before opening it. Consumers that rely on reviewer
separation must use `release::verify_signed` or `activate` with independently
distributed publisher and reviewer trust files; a receipt pin alone delegates the
whole release decision to whoever distributed that pin. Keep the full receipt pin
and required attribution with caches and exports. Use `resolve` for reviewed destinations, keep
null as abstention, and enforce application-specific malware/content/navigation
policy separately. A verified signature authenticates the publisher, not the truth
of every assertion. An official website may later be compromised.
Authenticate the publisher and reviewer trust roots independently, or obtain the
full receipt pin over a channel that is already trusted for the complete release
decision. A hash beside an untrusted download authenticates nothing.
Deliver revocations to every active consumer and derived catalogue, and bound cache
lifetimes. The resolver checks review expiry at query time; a detached cached URL
does not recheck itself. Preserve the current review history during rollback.
Copying SQLite rows or the JSONL export into a second resolver can bypass these
checks; use the native API or CLI for admission decisions.
Use `resolve` for navigation. `lookup`, exports, raw SQLite rows and observation
views are audit evidence and can include unreviewed, conflicting, superseded or
malicious claims. Preserve null destinations and typed abstention reasons. Apply
application-specific malware, content and destination policy because a legitimate
site can later be compromised.
## Community changes
Keep license and attribution artifacts with caches and derived datasets. Bound
cache lifetime by vote expiry and revocation delivery. A detached URL copied from
an earlier result no longer performs policy, freshness or rollback checks.
Treat public submissions as untrusted evidence. Do not execute submitted content
or copy review decisions into production automatically. Source allowlist and
attribution changes need both implementation tests and documented rights review.
Reject unknown sources until those checks are complete. Every publisher may apply
stricter admission rules, and must state its actual review and incident procedures.
## Community submissions
Treat public issues, manifests, source files, captures and vote proposals as
untrusted input. They may suggest evidence but cannot write the protected store,
approve, publish or activate a generation. Unknown providers remain unsupported
until current commercial-reuse rights, format, lineage, attribution and tests are
reviewed. Public contribution does not imply production admission.

26
docs/adr/0001-typed-source-coverage.md Normal file
View file

@ -0,0 +1,26 @@
# ADR 0001: Typed source coverage and supersession
Status: Accepted, 2026-09-13.
Source snapshots declare `full`, `partition`, or `delta` coverage. Partitions use
stable disjoint coordinates. Deltas name one exact base, a consecutive sequence,
and every directly superseded object. The build rejects ambiguous frontiers,
cycles, missing bases, cross-source links, overlaps, and mixed legacy/typed active
sets. Older objects and facts remain available for audit.
Free-form scope strings were insufficient to distinguish replacement from
composition. Explicit coverage makes selection reproducible and prevents a
partial object from silently replacing unrelated source data.
## Rejected alternatives
Selecting the newest retrieval time repeats the v0.3 ambiguity and lets clock
skew choose authority. Treating every object as additive retains deleted facts.
Inferring overlap from source URLs or file names is source-specific and unsafe.
## Compatibility
V1 manifests remain isolated by exact provider and scope. A typed object may
replace legacy evidence only by naming its exact manifest ID in `supersedes`.
Readers that do not understand v2 must reject it. See the source-manifest and
coverage contracts in [FORMATS.md](../FORMATS.md).

23
docs/adr/0002-granular-trust-subjects.md Normal file
View file

@ -0,0 +1,23 @@
# ADR 0002: Separate names, website edges, and observations
Status: Accepted, 2026-09-13.
A name binding, entity-to-property edge, and crawler observation have separate
identities and decisions. A resolver must admit the matched name and the selected
website edge. Observations enter evidence bundles but never create ownership,
identity, or regional roles.
This prevents a new alias from inheriting an existing route and avoids invalidating
an unchanged route when unrelated entity metadata changes.
## Rejected alternatives
One decision over the whole entity made harmless label changes invalidate every
route and let an injected alias inherit old authority. Promoting observer signals
directly would turn redirects or self-authored metadata into ownership claims.
## Compatibility
Legacy review rows remain auditable under the explicit compatibility policy.
Strict v0.4 resolution requires separate name and edge votes. V2 edge fingerprints
exclude unrelated entity revision metadata while retaining it in provenance.

38
docs/adr/0003-votes-revocations-and-publishers.md Normal file
View file

@ -0,0 +1,38 @@
# ADR 0003: Signed votes, sticky revocations, and publisher separation
Status: Accepted, 2026-09-13.
Reviews are immutable signed votes compiled under a receipt-bound policy epoch.
The reference policy requires two reviewer identities, two independent groups,
and two physical SSH keys. One revocation blocks the exact subject until every
approval in a fresh quorum explicitly references all active revocation IDs.
Publisher identity and key must be separate from every reviewer vote.
Changing the policy epoch never clears a retained revocation; a fresh quorum
under the new policy must explicitly supersede it.
Trusted writer acceptance time bounds validity. Reviewer timestamps cannot
backdate eligibility or extend an approval beyond 90 days after acceptance.
Risk-class thresholds may raise the base name, edge, or equivalence quorum for
`source_conflict` and `dangerous_drift`. The reference policy also holds those
edges in disputed or probationary state until the underlying risk clears.
Emergency feeds are cumulative, use the separate
`argand-site-registry-revocations` SSH namespace, retain superseded revocation
IDs, and can be applied to a compatible pinned generation before its replacement
arrives. Cross-generation feeds only add blocks. Reinstatement requires the exact
full generation containing the signed superseding quorum, and feed artifacts must
be refreshed at least every seven days.
## Rejected alternatives
Latest-decision-wins lets one later approval erase a revocation. Counting aliases
of one key as separate reviewers does not provide independence. Letting a release
publisher contribute approvals collapses review and publication into one actor.
## Compatibility
Legacy decisions remain available only under the explicit v0.3 compatibility
policy. Strict receipts bind the policy and reviewer trust-root digests. A policy
change creates a new epoch and old approvals become stale rather than being
silently reinterpreted.

29
docs/adr/0004-active-and-audit-views.md Normal file
View file

@ -0,0 +1,29 @@
# ADR 0004: Separate active and audit views
Status: Accepted, 2026-09-13.
Normal export contains only selected, nonrejected facts. Audit export preserves
active, superseded, and rejected facts with an explicit state and rejection
reason. Neither export bypasses resolver policy.
Operational consumers need an unambiguous current evidence view, while
investigators and publishers need conflicting and superseded evidence. One
ambiguous export could be mistaken for an approved route list.
Version 0.4 keeps the complete audit history inside each generation so rollback,
diff, and incident inspection remain self-contained. Separating a compact runtime
projection from content-addressed cold audit bundles is deferred to v0.5 until
size and latency measurements justify the extra recovery surface.
## Rejected alternatives
Deleting superseded facts loses conflict and replacement evidence. Shipping only
the audit view makes accidental use as current state too easy. Splitting storage
before authenticated bundle verification exists risks publishing a runtime index
whose supporting evidence cannot be recovered.
## Compatibility
The normal JSONL envelope is v2 and contains active assertions only. Audit mode
uses the same envelope version with explicit active, superseded, rejected, and
tombstoned rows. V1 consumers must reject the new schema and migrate explicitly.

30
docs/adr/0005-full-delta-release-identity.md Normal file
View file

@ -0,0 +1,30 @@
# ADR 0005: Full and delta release identities
Status: Accepted, 2026-09-13.
Each source object has its own digest. The selected coverage graph, including
every active full, partition, base, and delta object and its precedence, has a
separate digest bound into the generation receipt. A future distributed dataset
delta must name exact base and target generation identities and preserve
revocation continuity.
An unauthenticated `latest` locator can be a convenience pointer, but never the
trust root. Consumers authenticate a full receipt pin or publisher signature.
Version 0.4 implements source-level full/partition/delta identity and cumulative
publisher-signed emergency revocation overlays. General downloadable registry
deltas remain a v0.6 distribution task because they also need mirror-independent
base/target authentication and consumer transaction semantics.
## Rejected alternatives
Mutable releases and unpinned `latest` URLs permit substitution and rollback.
Signing only a compressed archive makes alternate packaging unverifiable. Calling
a source delta a registry delta would hide changes introduced by review policy,
normalization, observations, or another provider.
## Compatibility
V2 receipts bind the selected coverage digest. Current pointers retain every
legacy and vote revocation across activation. Emergency feeds apply only when
their rules, policy, and reviewer-trust digests match the cached generation.

24
docs/adr/0006-source-lineage.md Normal file
View file

@ -0,0 +1,24 @@
# ADR 0006: Source lineage and independence
Status: Accepted design; implementation scheduled for 0.5.
Corroboration must describe the direct provider, upstream dataset, transformation,
and snapshot. Two providers that copied the same upstream assertion do not count
as independent evidence merely because their URLs differ. Unknown lineage stays
unknown.
Version 0.4 preserves provider-native provenance and never combines popularity or
same-domain evidence into ownership confidence. Version 0.5 will add explicit
lineage fields and independence-aware corroboration without rewriting history.
## Rejected alternatives
Counting provider names as independent evidence rewards copied datasets. Guessing
lineage from matching text creates another unsupported inference. Dropping a
source because lineage is unknown destroys useful conflicting evidence.
## Compatibility
The v0.4 policy does not award quorum from source count, so absent lineage cannot
inflate reviewer authority. Future lineage fields must be additive provenance;
old assertions remain byte-identifiable and are never rewritten as independent.

27
docs/adr/0007-distribution-and-embedding.md Normal file
View file

@ -0,0 +1,27 @@
# ADR 0007: Distribution and proprietary embedding
Status: Accepted, 2026-09-13.
The project prioritizes a standalone AGPL-3.0-or-later CLI and Rust library plus
signed data artifacts. Broad proprietary embedding is not a 0.4 goal. Consumers
must assess AGPL obligations for their deployment and comply independently with
every provider data license and attribution term.
The supported integration paths are the native CLI, reusable Rust reader, signed
generation format, and documented subprocess protocol. A future change to code
licensing, dual licensing, hosted APIs, or proprietary linking requires a separate
governance and legal decision; this ADR does not grant one.
## Rejected alternatives
A second permissively licensed verifier crate was considered for v0.4. It would
duplicate format and signature policy before the contracts have deployment data,
and could imply that provider datasets inherit the verifier's license. A network
API would add account, availability, and traffic-trust requirements to an offline
dataset component.
## Compatibility
Rust and subprocess consumers use the same receipt and JSON contracts. Broad
proprietary embedding is outside the supported v0.4 surface. Dataset users must
still follow each source license regardless of how they invoke the verifier.

880
docs/superpowers/plans/2026-09-13-v0.4-and-beyond.md Normal file
View file

@ -0,0 +1,880 @@
# Argand Site Registry v0.4 and Beyond Plan
> **Status:** Version 0.4 phases 0 through 2 implemented and security-reviewed.
> Phases 3 and later remain the sequenced roadmap.
>
> **Baseline:** Clean `main` at `2861337`; runtime behavior is the tagged
> `v0.3.0` release at `ac82820`. The complete v0.3 acceptance suite passes.
>
> **Execution constraint:** Work in this standalone repository only. Do not use
> subagents, edit Argand's main checkout, share its build cache, acquire paid data,
> publish a dataset, sign with production keys, or deploy without the authority
> already established for that specific action.
## Goal
Turn the v0.3 evidence and release substrate into a production-grade, reusable
entity-to-website authority system that can publish a useful signed reference
registry while preserving conflict, provenance, commercial-reuse terms, human
review, abstention, revocation, and deterministic regional resolution.
The implementation must remain useful in two modes:
1. A local publisher builds and reviews its own registry from allowed sources.
2. A consumer verifies and queries a separately distributed signed reference
generation without trusting an unauthenticated download location.
The work is divided into independently releasable stages. Correct active-state
semantics and review controls come before broader ingestion. Provider-scale proof
comes before a public reference dataset. A hosted service remains optional.
## Product thesis
Popularity lists answer which domains receive attention. They do not reliably
answer which entity controls a domain, whether a destination is current, or which
regional property is appropriate. Argand Site Registry should compile independent
source assertions, observed site relationships, and authenticated reviewer votes
into a signed generation that resolves only when policy is satisfied.
The principal product outcome is a signed, explainable answer:
```text
query + locale/country
-> reviewed name-to-entity binding
-> reviewed entity-to-property edge
-> active regional-selection policy
-> URL or typed abstention
```
A larger assertion database is not the goal by itself. Release quality is measured
by resolved-route correctness, useful coverage, freshness, and revocation speed.
## Non-negotiable invariants
- Keep raw inputs, normalized facts, crawler observations, reviewer decisions,
policy decisions, popularity, and release authority logically separate.
- Never infer entity equivalence or website ownership from similar names, domains,
redirects, TLS, DNS, `sameAs`, popularity, or shared upstream data alone.
- Preserve conflicting and rejected evidence with its source-native identity.
- Every imported or derived fact retains source, source identifier, license,
license evidence URL, retrieval timestamp, confidence, and derivation version.
- Every source adapter must be rights-reviewed, format-pinned, streaming, bounded,
reproducible, resumable where feasible, and idempotent.
- Automated updates may download, import, evaluate, and build candidates. They may
not approve, renew, sign, activate, or publish them.
- Resolution remains fail closed. Ambiguity, missing policy, expired evidence,
missing votes, unsupported locale semantics, or conflicting equal candidates
produces a typed abstention.
- Popularity remains source-separated evidence and never becomes an ownership vote.
- Existing v0.3 data remains auditable. Migrations must not reinterpret an old
approval, source scope, or timestamp as if it had been created under a new rule.
- Curlie descriptions remain redacted unless the exact distribution surface meets
its attribution obligations.
- Public submissions are untrusted proposals. They never mutate an active release.
- Cloudflare Radar, default Tranco, Cisco Umbrella, and other unverified sources
remain excluded.
## Target architecture
```mermaid
flowchart LR
A[Rights-reviewed source adapters] --> B[Immutable source manifests and raw cache]
B --> C[Writer store: records, facts, conflicts, tombstones]
C --> D[Normalized assertion graph]
E[Bounded candidate-site observer] --> F[Immutable observation bundles]
F --> D
D --> G[Review queue and evidence diff]
G --> H[Signed reviewer votes]
H --> I[Versioned publisher policy compiler]
I --> J[Immutable active projection]
J --> K[Signed full generation]
J --> L[Signed delta and revocation feed]
K --> M[Verified local library and CLI]
L --> M
M --> N[Exact lookup and regional resolve]
```
The writer store remains append-oriented and audit-capable. Runtime generations
contain the selected facts, active decisions, required proof material, and signed
references to cold audit bundles. They do not duplicate every historical raw row.
## Release sequence
| Release | Purpose | Exit condition |
| --- | --- | --- |
| v0.3.x | Correct active-state and review-time semantics | Historical export is explicit, scope overlap fails closed, review acceptance time is trusted, and legacy behavior is regression-tested. |
| v0.4 | Scalable trust and observation pipeline | Quorum policy, granular name/edge decisions, sticky revocations, structured evidence bundles, and review queues work end to end. |
| v0.5 | Provider-scale and source expansion | Real-format scale canaries pass; incremental Wikidata and the first new rights-approved adapters are reproducible and bounded. |
| v0.6 | Signed public reference registry | A reviewed dataset, full/delta releases, revocation delivery, coverage report, and consumer verification are published independently of source releases. |
| Later | Wider resolver/product surface | Locale fallback, typed destination roles, candidate discovery, platform portability, and an optional read-only service are justified by real use. |
## Phase 0: Freeze contracts and record design decisions
### Task 0.1: Capture the v0.3 compatibility baseline
- [x] Record the exact v0.3 CLI output schemas, generation schema, source-manifest
schema, rules version, review JSON, release receipts, and Python/Rust examples.
- [x] Add golden fixtures for a complete v0.3 generation and a mutable schema-v3
writer database without committing provider data or private keys.
- [x] Verify that current `lookup`, `resolve`, `diff`, `export`, `stats`, signature
verification, rollback refusal, and migrations behave exactly as documented.
- [x] Preserve the current five-source all-synthetic fixture as a compatibility gate.
Likely files:
- `crates/argand-site-registry/tests/common/`
- `crates/argand-site-registry/tests/registry.rs`
- `crates/argand-site-registry/tests/failures.rs`
- `crates/argand-site-registry/tests/cli.rs`
- `docs/VALIDATION.md`
Acceptance:
- A v0.3 reader fixture remains readable or fails with a precise documented version
error after each later schema change.
- No later test can silently regenerate the baseline fixture from new behavior.
### Task 0.2: Write architecture decisions before migrations
- [x] Add an ADR for active source coverage and supersession.
- [x] Add an ADR for separating name bindings, website edges, and observations.
- [x] Add an ADR for reviewer votes, revocation precedence, and publisher policy.
- [x] Add an ADR for runtime projection versus cold audit retention.
- [x] Add an ADR for full and delta dataset release identities.
- [x] Add an ADR for source lineage so copied upstream evidence does not count twice.
- [x] Document whether broad proprietary embedding is a goal. If so, evaluate a
small permissively licensed format/verifier crate without changing the AGPL
publisher engine or source-driven dataset licenses automatically.
Acceptance:
- Every later migration and public contract links to an approved decision.
- The ADRs explain rejected alternatives and compatibility consequences.
## Phase 1: v0.3.x active-state and trust hardening
### Task 1.1: Replace free-form snapshot replacement semantics with typed coverage
Problem: v0.3 selects the latest completed snapshot per exact `(source, scope)`
string. Different strings are treated as additive even when one is a full snapshot
or overlaps another partition.
- [x] Introduce a source-manifest version with explicit coverage semantics:
collection identity, coverage kind, stable partition identity, base snapshot
where applicable, and explicit supersession references.
- [x] Preserve v1 manifest parsing. Map every legacy `(source, scope)` to an isolated
legacy partition so migration does not silently change its active facts.
- [x] Require operators to provide an explicit mapping before a legacy partial
snapshot can join or be replaced by a typed full collection.
- [x] Select one coherent coverage set per source collection:
- a complete full snapshot supersedes older declared partitions;
- disjoint partitions compose only when their identities are explicit;
- deltas require an authenticated base and ordered continuity;
- overlapping or missing coverage relationships abort the build;
- deletions use explicit tombstones and remain auditable.
- [x] Include the complete selected-coverage graph in the build receipt.
- [x] Make `stats` and `diff` report selected, superseded, incomplete, and conflicting
source snapshots separately.
Likely files:
- `crates/argand-site-registry/src/model.rs`
- `crates/argand-site-registry/src/store.rs`
- `crates/argand-site-registry/src/build.rs`
- `crates/argand-site-registry/src/diff.rs`
- `crates/argand-site-registry/src/query.rs`
- `crates/argand-site-registry/src/cli.rs`
- `crates/argand-site-registry/migrations/004.sql`
Tests:
- Latest snapshot replaces an older snapshot in the same partition.
- A declared full snapshot supersedes earlier partitions.
- Two explicit disjoint partitions compose.
- An undeclared full-plus-partial combination fails.
- Overlapping partitions fail.
- Missing delta bases, skipped deltas, and forked delta histories fail.
- Tombstoned facts disappear from the active projection but remain in audit history.
- Failed or partial imports never replace selected complete coverage.
- Reimporting identical manifests and deltas is byte-for-byte idempotent.
### Task 1.2: Split active export from audit-history export
Problem: v0.3 JSONL export emits facts from every complete retained source snapshot,
including superseded snapshots, without a per-row active-state marker.
- [x] Change the normal export contract to emit selected active facts only.
- [x] Version the export envelope and include generation identity, selected source
IDs, coverage-policy version, derivation version, and attribution identity.
- [x] Add explicit selection state to every assertion.
- [x] Add a separate audit export that includes active, superseded, rejected, and
tombstoned facts with replacement links.
- [x] Keep Curlie descriptions redacted by default in both modes.
- [x] Make consumers reject unknown export schema versions.
- [x] Document that neither export bypasses `resolve` admission policy.
Likely files:
- `crates/argand-site-registry/src/release.rs`
- `crates/argand-site-registry/src/cli.rs`
- `crates/argand-site-registry/src/diff.rs`
- `README.md`
- `docs/CONSUMERS.md`
- `docs/TRUST.md`
Tests:
- Superseded facts are absent from active export.
- Audit export retains and labels the same facts.
- Active export and native generation agree exactly on selected source IDs.
- Description redaction and attribution survive both modes.
- Python and Rust consumers refuse unknown versions and preserve null abstention.
### Task 1.3: Bind decisions to trusted acceptance time
Problem: the reviewer signs `reviewed_at`, but the writer does not persist a trusted
append time. That self-declared time influences approval validity and reviewer-key
validity-epoch checks.
- [x] Record `accepted_at` from the writer when exact signature verification and
candidate validation succeed.
- [x] Make approval validity begin no earlier than trusted acceptance.
- [x] Bound effective expiry by both the signed review duration and the enforced
maximum measured from acceptance, preventing backdating or future dating from
extending authority.
- [x] Evaluate reviewer eligibility at acceptance for new decisions. Preserve the
signed claimed decision time for audit, without treating it as trusted time.
- [x] Include acceptance time and the time-policy version in generation receipts.
- [x] Retain legacy approvals for audit but require an explicit migration policy or
fresh decision before they are active under the new rules. Never synthesize a
historical acceptance time. Preserve legacy revocations regardless.
- [x] Test clock skew, future dates, backdates, expired keys, removed keys, offline
signing followed by later acceptance, and reproducible builds at fixed clocks.
Likely files:
- `crates/argand-site-registry/src/review.rs`
- `crates/argand-site-registry/src/query.rs`
- `crates/argand-site-registry/src/resolution.rs`
- `crates/argand-site-registry/src/release.rs`
- `crates/argand-site-registry/migrations/004.sql`
- `docs/TRUST.md`
- `docs/PUBLISHING.md`
### Task 1.4: Make revocation precedence explicit before quorum work
- [x] Treat a valid revocation as sticky for its exact subject.
- [x] Require an explicit signed supersession that references the revocation before
the same destination can become active again.
- [x] Prevent a later ordinary approval from overriding a revocation by sequence
order alone.
- [x] Preserve existing activation rollback checks and strengthen them to compare
revocation identities and supersession relationships.
- [x] Add a machine-readable emergency revocation export suitable for cached
consumers.
Acceptance:
- Reordering, forking, or appending an ordinary approval cannot erase a revocation.
- A deliberately superseded revocation remains visible in lookup, diff, and audit.
- An offline consumer can apply the revocation export before receiving a full build.
### Phase 1 release gate
- [x] Run `cargo fetch --locked` once, then the complete offline `scripts/check.sh`.
- [x] Run migrations from empty, schema v1, v2, and v3 databases.
- [x] Build the same generation twice and compare every output byte.
- [x] Verify source release determinism from a clean signed commit.
- [x] Update changelog, trust docs, format contracts, and migration guidance.
- [x] Run `cargo audit --deny warnings` and record it separately from the offline gate.
- [x] Confirm no provider data, keys, approval logs, or private paths entered Git.
## Phase 2: v0.4 scalable review and structured evidence
### Task 2.1: Separate name-to-entity trust from entity-to-property trust
Problem: v0.3 binds each website approval to the entity's full name set. This stops
a newly injected alias from inheriting a route, but benign alias or label changes
invalidate every website edge. A global Wikidata revision also changes edge identity
when the P856 statement itself did not materially change.
- [x] Create independent material fingerprints for:
- source name or alias assertion to stable entity;
- explicit entity equivalence;
- entity-to-normalized-web-property assertion;
- normalized regional role and scope;
- observation bundle;
- reviewer vote and policy result.
- [x] Bind website-edge identity only to material website evidence: exact source
statement identity/value/rank/relevant qualifiers, normalized URL result, and
derivation rule version.
- [x] Retain source record revision and unrelated entity metadata in provenance and
diffs without making them part of website-edge identity.
- [x] Require `resolve` to satisfy both an admitted name binding and an admitted
website edge. `lookup` continues to expose unreviewed names and edges.
- [x] Ensure a newly imported alias cannot inherit an existing approved route.
- [x] Ensure a benign accepted alias update does not invalidate an unchanged route.
- [x] Give canonical labels, aliases, and source-specific names distinct evidence
identities and policy treatment.
Tests:
- Adding an unreviewed alias never creates a resolvable query.
- Removing or changing an approved alias affects only that binding.
- An unrelated Wikidata `lastrevid` change preserves the route fingerprint.
- A P856 URL, rank, end qualifier, or relevant regional qualifier change invalidates
the route approval.
- Confusable and bidi-control names remain rejected.
- Explicit entity equivalence never carries unstated name or route authority.
### Task 2.2: Replace latest-decision-wins with signed votes and policy compilation
- [x] Represent decisions as immutable signed votes by authenticated reviewer.
- [x] Permit one current vote per signer, subject, decision type, and policy epoch;
preserve superseded votes in the append-only log.
- [x] Add a versioned publisher policy that defines:
- required approval threshold by decision type and risk class;
- revocation threshold and sticky-revocation behavior;
- publisher/reviewer separation;
- reviewer groups or independence constraints where configured;
- maximum approval age and evidence-freshness requirements;
- treatment of source conflicts and unresolved observations.
- [x] Hash the exact policy and selected reviewer trust roots into the release receipt.
- [x] Compile votes deterministically into approved, revoked, expired, disputed,
probationary, or insufficient-review state.
- [x] Keep policy configurable for independent publishers while shipping a strict,
documented reference policy for Argand's own releases.
- [x] Preserve the ability to operate locally without accounts or a network service.
Tests:
- One signer cannot satisfy a two-independent-reviewer policy.
- Duplicate keys or aliases for the same reviewer do not create extra votes.
- Publisher/reviewer separation is enforced when enabled.
- One authorized revocation blocks resolution under the reference policy.
- Conflicting votes produce a disputed state and abstention.
- Changing policy invalidates the old compiled result without changing source facts.
- Release verification fails on policy, signer, vote, or receipt tampering.
### Task 2.3: Turn observation types into a stored evidence pipeline
- [x] Extend the writer schema with immutable observation batches and observations.
- [x] Preserve the existing distinction between observations and ownership claims.
- [x] Add an adapter contract for externally captured observations, including source,
rights declaration, capture ID, retrieval time, content hash, exact selector,
from/to URLs, relation, and confidence.
- [x] Project redirect, canonical, hreflang, JSON-LD `sameAs`, sitemap, and country
selector evidence without automatically creating an entity edge.
- [x] Add observation lookup, reverse lookup, generation diff, and evidence-bundle
output for review.
- [x] Record negative or failed observations with bounded error classes so absence is
not confused with a fetch that never succeeded.
- [x] Define retention that stores hashes and necessary bounded extracts rather than
copied page bodies unless rights and need are established.
Likely files:
- `crates/argand-site-registry/src/observation.rs`
- `crates/argand-site-registry/src/store.rs`
- `crates/argand-site-registry/src/build.rs`
- `crates/argand-site-registry/src/query.rs`
- `crates/argand-site-registry/src/diff.rs`
- `crates/argand-site-registry/migrations/005.sql`
### Task 2.4: Add a bounded candidate-site observer
- [x] Observe only imported/reviewed candidate URLs. Do not start an open-web crawler.
- [x] Enforce scheme, redirect-count, response-size, header-size, decompression,
timeout, DNS-result, address-range, and per-host request bounds.
- [x] Block credentials, local/private/link-local targets, unsafe redirect transitions,
non-HTTP protocols, and host confusion.
- [x] Capture redirect chains, final URL, status, canonical, hreflang, sameAs, sitemap
references, country selectors, DNS answers, and TLS certificate fingerprints as
separate evidence classes.
- [x] Do not treat TLS, DNS, redirects, or site self-assertions as ownership proof.
- [x] Emit immutable observation manifests compatible with Task 2.3.
- [x] Support cache-only replay so parser and policy tests never require the network.
- [x] Keep acquisition cadence, concurrency, bandwidth, and data path configurable.
Security tests:
- SSRF attempts, redirect loops, compression bombs, oversized markup, malformed HTML,
DNS rebinding, mixed encodings, invalid certificates, and cross-scheme redirects.
- Parser fuzz/property tests for headers, hreflang, canonical, JSON-LD, and sitemaps.
- Deterministic replay from captured fixtures with no network access.
### Task 2.5: Build review queues and evidence bundles before a graphical UI
- [x] Add deterministic queue output ordered by risk and material change, not source
popularity alone.
- [x] Queue new routes, new aliases, source conflicts, changed website statements,
redirects across registrable domains, observation drift, expiring approvals,
unresolved regional scopes, and revoked destinations proposed for reinstatement.
- [x] Produce a bounded review bundle containing exact claims, conflicts, observation
diffs, source licenses, capture hashes, requested role, and candidate fingerprint.
- [x] Add commands to prepare a vote, verify exact vote bytes, append it, and show the
compiled policy result.
- [x] Keep the queue read-only and deterministic. Never let viewing evidence mutate
approval state.
- [x] Defer a browser workbench until CLI bundles have proven the workflow.
### Task 2.6: Monitor approved routes and classify drift
- [x] Schedule observation refresh independently from source import cadence.
- [x] Classify material changes: unreachable, cross-domain redirect, DNS/TLS change,
content/canonical shift, domain expiry indicators, malware-policy result, or no
material change.
- [x] Use risk policy to shorten review intervals. Never auto-extend approvals.
- [x] Put materially changed routes into probation or revoke them according to signed
publisher policy; default to abstention where evidence is insufficient.
- [x] Record every transition and make revocation candidates immediately exportable.
- [x] Design malware-feed adapters only after exact commercial-reuse and redistribution
terms are verified. Do not hard-code an unreviewed vendor.
### Phase 2 release gate
- [x] Complete a signed two-reviewer fixture from source assertion through name vote,
edge vote, observation bundle, policy compilation, release, resolution, drift,
revocation, signed delta, and consumer application.
- [x] Prove a malicious contributor cannot submit directly into active state.
- [x] Prove one compromised reviewer cannot approve under the reference policy.
- [x] Prove an emergency revocation reaches a pinned offline consumer without a full
source reimport.
- [x] Preserve all v0.3 conflict, normalization, regional, and rollback tests.
Completion note: the version 0.4 implementation satisfies the Phase 0 through 2
acceptance boundary, including the complete two-reviewer fixture and security
review. The immutable v0.3 contract is frozen by exact signed commit and golden
schema/field coordinates rather than a committed SQLite generation, because this
repository does not admit generated datasets or approval logs. General registry
deltas remain explicitly deferred by ADR 0005; version 0.4 supplies typed source
deltas and a separately signed, cumulative emergency block feed.
## Phase 3: v0.5 storage, scale, and current-source improvements
### Task 3.1: Separate runtime projections from cold audit history
- [ ] Keep raw cache objects immutable and content-addressed outside Git.
- [ ] Package completed source imports into content-addressed audit bundles with
manifest, record/fact indices, hashes, format version, and attribution.
- [ ] Make a runtime generation contain selected facts, normalized projections,
active policy results, required votes/revocations, and signed bundle references.
- [ ] Do not copy all historical records and facts into every runtime generation.
- [ ] Add audit verification that streams referenced bundles and detects absence,
truncation, substitution, or mismatched attribution.
- [ ] Add retention/checkpoint tooling that never deletes the only authenticated copy
of evidence and produces a signed deletion/retention report.
- [ ] Measure query latency and generation size before and after the split.
### Task 3.2: Add provider-scale benchmark and recovery tooling
- [ ] Define repeatable small, medium, and provider-representative import profiles.
- [ ] Record wall time, CPU time, peak RSS, compressed and expanded bytes, database
growth, facts/second, checkpoint frequency, restart time, build size, and query
latency.
- [ ] Interrupt imports at multiple checkpoints and prove idempotent resumption.
- [ ] Exercise disk-full, truncated input, cache corruption, duplicate records, and
interrupted generation publication.
- [ ] Make benchmark reports name exact source snapshot hashes and hardware without
committing source data.
- [ ] Treat synthetic performance as development evidence, not provider capacity.
### Task 3.3: Harden full Wikidata ingestion and add incremental refresh
- [ ] Confirm current official full and incremental formats from Wikidata documentation
and inspected fixtures before changing the adapter.
- [ ] Replace the assumption that every useful entity fits in one in-memory 16 MiB
line with bounded disk-spooling or an explicitly receipted oversized-record path.
- [ ] Never silently skip an oversized entity that may contain a relevant fact.
- [ ] Add incremental add/change ingestion with authenticated base snapshot identity,
ordered application, checkpoints, and reconciliation against later full dumps.
- [ ] Define how deletions and removed P856 statements become tombstones.
- [ ] Keep full raw assertion/qualifier/reference provenance for consumed fields.
- [ ] Make unrelated `lastrevid` changes visible in audit diffs without invalidating
unchanged material edge fingerprints.
- [ ] Test real-format pathological entities and multistream compression boundaries.
Authoritative format reference:
- <https://www.wikidata.org/wiki/Wikidata:Database_download>
### Task 3.4: Strengthen current-source acquisition verification
- [ ] Prefer provider-published checksums or signatures when officially available and
bind verification method into the source manifest.
- [ ] Keep HTTPS allowlists, manual redirect validation, byte bounds, strong-validator
resume rules, and immutable local cache behavior.
- [ ] Detect and report source format drift before partial import can replace a source.
- [ ] Add format-version canaries for Majestic, CrUX, Curlie, PSL, and Wikidata.
- [ ] Preserve CrUX billing as explicit opt-in configuration and record job identity,
query, result period, and actual cost outside public fixtures.
- [ ] Continue frequent PSL refresh and include exact PSL hash in normalization proofs.
- [ ] Preserve Curlie attribution and description-redaction tests on every export path.
### Task 3.5: Add source lineage and independence metadata
- [ ] Record direct provider, upstream/origin dataset, transformation, snapshot, and
known dependency relationships for each fact source.
- [ ] Prevent policy from counting two assertions as independent corroboration when
one republishes the other.
- [ ] Expose lineage in lookup, review bundles, export, diff, and evaluation.
- [ ] Keep unknown lineage explicit rather than assuming independence.
## Phase 4: Rights-gated additional source adapters
Every source follows the same gate:
1. Verify authoritative download, schema, update cadence, license, attribution,
redistribution, database-right, and commercial-use documentation.
2. Record the exact decision and URLs in `LICENSE_SOURCES.md`.
3. Inspect current official fixtures. Do not infer fields from third-party examples.
4. Add a source enum/format only with a streaming adapter and bounded failure tests.
5. Preserve native IDs, raw relevant records, selectors, lineage, and confidence.
6. Import into a separate logical source layer.
7. Demonstrate that the source cannot auto-create an approved route.
8. Run deterministic, idempotent, interrupted, malformed, and conflict fixtures.
### Task 4.1: Add ROR first
Rationale: ROR is CC0 and directly supplies stable organization IDs, names, aliases,
status, locations, links, and domains. It is compact and well aligned with the model.
- [ ] Verify the current ROR schema version and official release asset from the ROR
data-dump documentation at implementation time.
- [ ] Consume only fields confirmed in that inspected schema.
- [ ] Map names and aliases without merging ROR entities into Wikidata entities unless
an exact external identifier or reviewed equivalence supports the join.
- [ ] Preserve links and domains as ROR assertions, not approvals.
- [ ] Preserve status, type, country/location, external IDs, and upstream lineage.
- [ ] Test domain conflicts, former/inactive organizations, aliases, multiple links,
missing fields, duplicate input, schema drift, and exact-ID equivalence.
References:
- <https://ror.readme.io/docs/data-dump>
- <https://ror.readme.io/docs/ror-data-structure>
- <https://ror.readme.io/docs/fields>
### Task 4.2: Add MusicBrainz core snapshots second
- [ ] Use only the CC0 core database snapshot and its verified checksums/signatures.
- [ ] Do not use the CC BY-NC-SA live replication feed in the commercial-safe default
pipeline.
- [ ] Consume exact URL entities and documented URL relationship types, including
official-homepage and ended-state metadata.
- [ ] Keep artists, labels, places, and events separate by stable MusicBrainz ID.
- [ ] Preserve relationship begin/end dates and link types as material evidence.
- [ ] Treat community-curated URLs as assertions requiring normal Argand review.
- [ ] Test removed URLs, ended relationships, entity redirects/merges, duplicate URLs,
malicious/taken-over sites, and snapshot replacement.
References:
- <https://musicbrainz.org/doc/MusicBrainz_Database/Download>
- <https://musicbrainz.org/doc/Style/Relationships/URLs>
- <https://musicbrainz.org/doc/Live_Data_Feed>
### Task 4.3: Validate GND as the third adapter
- [ ] Verify the exact current CC0 declaration for the selected GND files.
- [ ] Inspect current JSON-LD or RDF schema and confirm homepage predicates before
implementing an adapter.
- [ ] Preserve authority IDs, preferred/variant names, types, countries, external IDs,
and exact homepage assertions where present.
- [ ] Keep its regional and language focus visible in provenance and evaluation.
- [ ] Stop after the rights/schema spike if homepage coverage does not justify the
adapter cost.
Reference:
- <https://data.dnb.de/opendata/>
### Task 4.4: Keep lower-priority candidates behind explicit holds
- [ ] ORCID: research a low-confidence individuals-only adapter. Its public file is
CC0, but links are self-declared and require privacy, impersonation, and
volatility policy. Never auto-approve. Reference:
<https://info.orcid.org/public-data-file-use-policy/>.
- [ ] OpenAlex: use only for research-activity/popularity metadata if useful. Record
ROR as upstream lineage and never count its institution website as independent
corroboration. Reference: <https://help.openalex.org/data/institutions/>.
- [ ] OpenStreetMap: do not ingest until an ODbL-compatible distribution and
attribution architecture is approved. Reference:
<https://osmfoundation.org/wiki/Licence_and_Legal_FAQ>.
- [ ] Government/corporate registries: assess jurisdiction by jurisdiction. Prefer
stable identity crosswalks; do not infer a website where no authoritative field
exists.
- [ ] DNS, RDAP, certificate transparency, package registries, and web crawl data:
evaluate as observation sources only after exact terms are verified.
- [ ] Open Library and other sources with unresolved underlying rights remain excluded.
### Phase 4 acceptance
- Each admitted source has authoritative license evidence, current fixture evidence,
exact fields consumed, attribution behavior, source lineage, and format-drift tests.
- Full source-specific provenance appears in lookup, review bundles, export, and diff.
- Removing any new adapter leaves existing source identities and release verification
deterministic.
- No new source changes an existing edge's approval merely by corroborating it.
## Phase 5: Resolver, evaluation, and consumer improvements
### Task 5.1: Canonicalize locale and country semantics
- [ ] Add standards-based BCP 47 parsing/canonicalization after reviewing the chosen
library and current specification behavior.
- [ ] Define deterministic precedence for exact locale/country, country-only,
language-parent, and global-primary candidates.
- [ ] Abstain on equal candidates at the same specificity.
- [ ] Preserve the requested and normalized locale in the explanation envelope.
- [ ] Add tests for `en-GB`, `en`, script subtags, case, deprecated aliases, malformed
tags, country-only properties, multi-country sites, and conflicting scopes.
### Task 5.2: Extend property roles without weakening default navigation
- [ ] Define a versioned role vocabulary covering at least global primary, regional
primary, product, support, developer, careers, login, status, and other reviewed
roles justified by real cases.
- [ ] Keep default `resolve` restricted to the requested navigation role.
- [ ] Require role-specific evidence and votes; a support site cannot become primary
because it shares a domain.
- [ ] Preserve unknown source roles as evidence without admitting them.
### Task 5.3: Add candidate discovery separately from resolution
- [ ] Add prefix/fuzzy discovery only as an audit/candidate operation.
- [ ] Preserve exact normalized matching for final admission.
- [ ] Return candidate score components and ambiguity rather than hiding a rewrite.
- [ ] Never let similarity bypass reviewed name-to-entity bindings.
- [ ] Test homographs, typosquatting, short names, multilingual aliases, and popular
entities with colliding names.
### Task 5.4: Expand evaluation into a release gate
- [ ] Extend judgments to cover expected abstention reasons, selected name binding,
route edge, regional precedence, and policy state.
- [ ] Report resolved-route errors separately from safe abstentions.
- [ ] Measure coverage, abstention taxonomy, active-evidence age, expiring approvals,
conflict rate, reviewer agreement, review turnaround, observation drift, and
revocation propagation.
- [ ] Add adversarial suites for source poisoning, alias injection, correlated sources,
malicious redirects, domain takeover, compromised reviewer, compromised
publisher, stale cache, rollback, and policy downgrade.
- [ ] Segment evaluation by source, entity type, language, country, popularity band,
destination role, and evidence age without collapsing source signals.
- [ ] Establish release thresholds only after an audited baseline exists. Any known
wrong resolved destination is a release blocker for the reference dataset.
## Phase 6: v0.6 signed public reference dataset
### Task 6.1: Define reference publisher governance
- [ ] Publish reviewer eligibility, independence, conflict-of-interest, evidence,
expiry, appeals, correction, key rotation, incident, and emergency-revocation
policies.
- [ ] Separate source maintainers, reviewers, and release publishers where practical.
- [ ] Publish the exact policy hash and reviewer trust roots with each release.
- [ ] Define a transparent proposal process in which contributors submit signed
evidence bundles rather than direct active-dataset edits.
- [ ] Record disputed claims and abstain until policy is met.
- [ ] Publish change logs and correction history without exposing sensitive reviewer
material unnecessarily.
### Task 6.2: Build a deliberately bounded starter registry
- [ ] Select a high-value initial coverage set using source-separated popularity and
declared entity classes only for prioritization.
- [ ] Publish the selection methodology and its biases.
- [ ] Review name bindings, entity equivalences, website edges, regional roles, and
current observations under the reference policy.
- [ ] Do not claim comprehensive web, country, language, or entity-type coverage.
- [ ] Require every resolvable destination to have current votes, unexpired evidence,
complete provenance, and an active monitoring schedule.
- [ ] Run the full evaluation, diff, license, source-lineage, and release gates.
### Task 6.3: Publish full releases, deltas, and revocations
- [ ] Keep dataset releases separate from source-code releases and raw provider cache.
- [ ] Produce a deterministic full generation, receipt, attribution bundle, policy,
reviewer trust roots, evaluation report, coverage report, and detached publisher
signature.
- [ ] Produce ordered, signed deltas bound to exact base and target generation IDs.
- [ ] Publish a small signed revocation feed with monotonic continuity and rollback
protection.
- [ ] Make full and delta application atomic and recoverable after interruption.
- [ ] Provide a documented mirror-independent verification procedure.
- [ ] Never make an unauthenticated `latest` URL the trust root. A convenience current
pointer must itself be signed and rollback protected.
### Task 6.4: Make immediate use simple
- [ ] Add a beginner workflow that downloads a release, verifies its publisher and
policy, pins it, and resolves `facebook` locally.
- [ ] Show an entity with multiple reviewed regional properties and an abstention.
- [ ] Provide identical native CLI, Rust library, and Python subprocess examples.
- [ ] Add a machine-readable capability/version command.
- [ ] Explain code license separately from each dataset source license and attribution.
- [ ] Provide update and emergency-revocation examples for systemd and cron.
### Reference release acceptance
- Every resolved route is reproducible from selected source facts, name/edge votes,
policy, and observations contained in or authenticated by the release.
- Every conflict, rejection, abstention, and supersession remains inspectable.
- An untrusted mirror cannot substitute a generation, policy, reviewer set, delta, or
revocation feed without verification failure.
- A clean consumer machine can verify and query the release without provider
credentials, reviewer keys, publisher private keys, or the mutable writer database.
- The coverage and evaluation reports distinguish implemented capability from actual
reviewed data coverage.
## Phase 7: Optional service and portability
### Task 7.1: Cross-platform immutable generation reads
- [ ] Implement equivalent no-follow, regular-file, bounded-copy, hash-before-open,
and private-temporary-file behavior for macOS and Windows.
- [ ] Add platform CI only on isolated runners with no release authority.
- [ ] Preserve Linux behavior and reject platforms without a safe implementation.
### Task 7.2: Read-only service only after the dataset proves useful
- [ ] Expose the existing verified query envelope through a minimal read-only local
HTTP or Unix-socket service if consumer demand justifies it.
- [ ] Pin one generation per process/request context and swap only after full signature,
policy, reviewer, and rollback verification.
- [ ] Bound query size, output size, concurrency, and time.
- [ ] Preserve typed abstention and explanation; do not add silent query rewriting.
- [ ] Keep publication, review, acquisition, and private keys out of the serving process.
## Operational security and supply-chain work
- [ ] Add property tests or fuzz targets for URL/domain normalization, PSL behavior,
JSON duplicate keys, each parser, coverage selection, observation parsing,
signed-delta application, and locale handling.
- [ ] Add dependency vulnerability and license-policy checks to a network-enabled,
isolated scheduled workflow. Preserve the offline acceptance gate separately.
- [ ] Pin and document CI images and external actions by immutable identity.
- [ ] Verify official provider checksums/signatures where available.
- [ ] Add threat-model cases for reviewer-key theft, publisher-key theft, source
compromise, malicious proposal bundles, cache substitution, stale mirrors,
rollback, denial of service, and domain takeover.
- [ ] Add key rotation and emergency response drills using disposable test keys.
- [ ] Keep signed receipts and validation artifacts outside the source tree unless they
are deliberately public, non-sensitive release evidence.
## Documentation deliverables
- [ ] Keep the README focused on the first successful verified lookup.
- [ ] Add an architecture document explaining assertion, observation, decision, policy,
generation, delta, and consumer boundaries.
- [ ] Expand `LICENSE_SOURCES.md` for every admitted source with exact fields consumed,
source URLs, licenses, attribution, redistribution, update cadence, and lineage.
- [ ] Document active versus audit export semantics.
- [ ] Document migrations and legacy decision handling.
- [ ] Publish the reference review and incident policies.
- [ ] Publish provider-scale measurements without implying serving or corpus coverage.
- [ ] Maintain a source-candidate table showing approved, research, held, and rejected
sources with the reason for each state.
## SWOT-driven checks
### Preserve strengths
- Deterministic, source-separated facts and normalization.
- Fail-closed resolution and explicit abstention.
- Signed reviews/releases and rollback-safe revocations.
- Complete licenses, attribution, provenance, and conflict evidence.
- Streaming, bounded, resumable, idempotent imports.
- Meaningful adversarial and cross-consumer tests.
### Correct weaknesses
- Publish an immediately usable dataset after trust and scale gates pass.
- Replace one-latest-decision semantics with policy-compiled votes.
- Reduce benign review churn without allowing alias inheritance.
- Separate active runtime state from unbounded audit history.
- Add provider-scale proof, monitoring, queues, and richer metrics.
- Improve regional semantics and platform support after the core release.
### Capture opportunities
- Establish an open evidence-bundle and signed-generation interchange format.
- Let independent publishers share facts while choosing different trust policies.
- Supply explainable navigation authority to search engines, assistants, browsers,
bookmarks, enterprise catalogs, and safety products.
- Publish an evaluation benchmark for entity-to-site resolution and abstention.
- Use ROR, MusicBrainz, and GND to expand high-quality vertical coverage.
### Mitigate threats
- Compromised reviewers: quorum, independence, expiry, and sticky revocation.
- Compromised publishers: independent reviewer roots and consumer policy verification.
- Domain takeover: active observation, probation, and rapid revocation delivery.
- Source poisoning: no auto-approval, source lineage, evidence diffs, and review queues.
- Correlated evidence: upstream lineage and independence-aware policy.
- License drift: exact rights gates and fail-closed format/license changes.
- Cache/mirror staleness: signed continuity, expiry, and rollback protection.
- Governance capture: public policies, disputes, appeals, and transparent changes.
## Explicit deferrals
Do not prioritize these before a reviewed reference dataset exists:
- A general hosted API or multi-tenant account system.
- ML or LLM approval of ownership or regional roles.
- Open-web crawling.
- Fuzzy matching in the final resolver.
- Additional popularity feeds merely to enlarge source count.
- Sources with unclear commercial reuse, database rights, or redistribution terms.
- ODbL data without an approved packaging architecture.
- A graphical review UI before deterministic CLI bundles and queues are proven.
- Automatic approval renewal based on unchanged popularity, TLS, DNS, or redirects.
## Definition of complete
The roadmap is complete when:
1. Source coverage and exports cannot confuse historical facts with active facts.
2. Review authority is bound to trusted acceptance time and a signed policy.
3. Name bindings and website edges change independently without creating inherited
authority.
4. Multiple authenticated reviewers and sticky revocations protect the reference
policy from one compromised contributor or reviewer.
5. Candidate-site observations are reproducible, bounded, rights-declared evidence
and material drift reaches reviewers promptly.
6. Runtime generations remain compact while complete audit evidence stays verifiable.
7. Current real source formats import and resume within documented modest-hardware
bounds, with no silent oversized-record loss.
8. Every new source has verified commercial-reuse rights, exact consumed fields,
lineage, attribution, and adversarial tests.
9. Regional resolution uses deterministic locale/country precedence and abstains on
unresolved ambiguity.
10. A separately distributed signed reference dataset produces useful reviewed
results, including `facebook -> Facebook -> facebook.com` and multiple regional
properties, from a clean consumer machine.
11. Full releases, deltas, and revocations authenticate independently of mirrors and
refuse rollback.
12. Published evaluation and coverage reports state what is actually reviewed,
current, relevant, and resolvable without conflating implementation with data.
13. Formatter, compiler checks, strict lints, documentation, unit/integration tests,
adversarial tests, source-release determinism, provider-scale canaries, and a
clean final diff all pass for every release boundary.
## First implementation slice after approval
Begin only with Phase 0 and Phase 1. Do not combine the initial migration with the
crawler, new sources, public data acquisition, or dataset publication.
The first concrete change set should contain:
1. Frozen v0.3 compatibility fixtures.
2. Approved active-coverage and export ADRs.
3. Typed source coverage with conservative legacy behavior.
4. Active-only export plus explicit audit export.
5. Trusted review acceptance time and sticky revocation semantics.
6. Schema/rules migration, documentation, and complete offline acceptance evidence.
Review that diff and release it before beginning granular name/edge votes or crawler
observations. This keeps the highest-risk semantic corrections small enough to audit
and makes every later phase build on an unambiguous active registry.