nicweyand/argand-site-registry
1
0
Fork 0
Code Issues Pull requests Projects Releases 7 Packages Wiki Activity Actions

release: implement site registry v0.4 trust pipeline

Browse source
This commit is contained in:
Nic Weyand 2026-09-13 12:22:05 -04:00
commit e26efc19fa
Signed by: nicweyand
SSH key fingerprint: SHA256:2te+ycJIQON/Wo/dH6+ZkFSQ4HnHWpetV2azx9E65dQ
67 changed files with 10698 additions and 640 deletions
Download patch file Download diff file Expand all files Collapse all files

2
crates/argand-site-registry/Cargo.toml
View file

@ -10,6 +10,7 @@ rust-version.workspace = true
[dependencies]
anyhow.workspace = true
argand-atomic = { path = "../argand-atomic" }
base64 = "0.22.1"
bzip2 = "0.6.1"
chrono.workspace = true
clap.workspace = true
@ -19,6 +20,7 @@ libc.workspace = true
publicsuffix = "=2.3.0"
reqwest.workspace = true
rusqlite = { version = "=0.40.2", features = ["bundled"] }
scraper = "0.27.0"
serde.workspace = true
serde_json.workspace = true
sha2.workspace = true

8
crates/argand-site-registry/LICENSE_SOURCES.md
View file

@ -1,6 +1,6 @@
# Site Registry source licenses
Reviewed against the primary distribution and licensing pages on 2026-09-12.
Reviewed against the primary distribution and licensing pages on 2026-09-13.
The Rust code uses the workspace's **AGPL-3.0-or-later** license. Imported data keeps
its own licenses; neither the code license nor a merged export relicenses it.
Commercial reuse is supported subject to the following obligations. A provider's
@ -13,6 +13,7 @@ listing is evidence of an assertion, not a guarantee of ownership or safety.
| Chrome UX Report (CrUX), Google | [CC BY 4.0 International](https://creativecommons.org/licenses/by/4.0/), [`CC-BY-4.0`](https://developer.chrome.com/docs/crux/methodology) | [Monthly BigQuery dataset](https://developer.chrome.com/docs/crux/bigquery/): `origin`, `experimental.popularity.rank`, observation month, optional audience-country dataset code. The adapter produces `origin,rank,yyyymm,country_code` CSV. Rank is a coarse bucket, not a precise visit count. Audience country is not website jurisdiction. No API key or OAuth token is retained. |
| Curlie | [CC BY 3.0 Unported](https://creativecommons.org/licenses/by/3.0/), [`CC-BY-3.0`](https://curlie.org/docs/en/license.html), including the attribution placement prescribed on that page | [Format documentation](https://curlie.org/docs/en/rdf.html), [official download redirect](https://curlie.org/directory-dl), currently [Passau-hosted archive](https://share.innkube.fim.uni-passau.de/curlie-rdf/curlie-rdf-all.tar.gz). Despite its RDF name, the current archive contains **literal TSV**. Content: URL, title, description, category ID. Structure: category ID, full category path, entry count, description, latitude, longitude. Archive notices are retained. |
| Public Suffix List contributors | [Mozilla Public License 2.0](https://mozilla.org/MPL/2.0/), [`MPL-2.0`](https://publicsuffix.org/list/public_suffix_list.dat) | [Official list](https://publicsuffix.org/list/public_suffix_list.dat). All ICANN and PRIVATE rules, wildcard/exception rules, version/commit comments and notices. Used for hostname, registrable-domain and public-suffix derivations. Download at most once per day. |
| Argand candidate observer | [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/), `CC0-1.0` | Local host-side observations authored by the registry publisher: HTTP status and redirect targets, canonical/hreflang/JSON-LD/sitemap/country-selector targets, public DNS-set hash, TLS leaf-certificate hash, bounded failure class, content hash and selectors. These records describe a capture; they do not incorporate page prose or prove ownership. |
## Attribution and distribution
@ -47,6 +48,11 @@ listing is evidence of an assertion, not a guarantee of ownership or safety.
and `facts`; exports include the PSL fact and original download locator.
Changes to covered PSL source files must remain available under MPL 2.0.
The Rust `publicsuffix` parser is MIT/Apache-2.0; that is separate from the list.
* **Argand observer:** locally produced observation metadata is dedicated under
CC0. The fetched page remains subject to its own rights. The default observer
stores only a bounded body in the local replay cache and emits normalized link,
status, hash and failure metadata. Publishers control and document retention of
local cache bodies; generated registry releases do not contain them.
Every generation contains this document and `ATTRIBUTION.json`, both hash-bound
by its signed completion receipt. Exports carry source manifests, licenses,

372
crates/argand-site-registry/README.md
View file

@ -53,28 +53,39 @@ of current source sizes. Increase a cap only after checking available storage.
export ARGAND_SITE_DATA="$HOME/.local/share/argand-site-registry"
mkdir -p "$ARGAND_SITE_DATA"
cat > "$ARGAND_SITE_DATA/full-coverage.json" <<'JSON'
{"collection":"default","kind":"full","partition":null,"base":null,"sequence":null,"supersedes":[]}
JSON
cat > "$ARGAND_SITE_DATA/wikidata-selection-coverage.json" <<'JSON'
{"collection":"entity-selections","kind":"partition","partition":"facebook-amazon","base":null,"sequence":null,"supersedes":[]}
JSON
argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
--source psl --format psl-text \
--url https://publicsuffix.org/list/public_suffix_list.dat \
--snapshot "$(date -u +%F)" --scope full --maximum-bytes 1000000 \
--coverage "$ARGAND_SITE_DATA/full-coverage.json" \
> "$ARGAND_SITE_DATA/psl-download.json"
argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
--source wikidata --format wikidata-entities \
--url 'https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q355%7CQ3884&format=json&maxlag=5' \
--snapshot "$(date -u +%F)" --scope selection:facebook-amazon \
--coverage "$ARGAND_SITE_DATA/wikidata-selection-coverage.json" \
--maximum-bytes 5000000 > "$ARGAND_SITE_DATA/wikidata-download.json"
argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
--source majestic --format majestic-csv \
--url https://downloads.majestic.com/majestic_million.csv \
--snapshot "$(date -u +%F)" --scope full --maximum-bytes 250000000 \
--coverage "$ARGAND_SITE_DATA/full-coverage.json" \
> "$ARGAND_SITE_DATA/majestic-download.json"
argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
--source curlie --format curlie-tar-gz \
--url https://curlie.org/directory-dl \
--snapshot "$(date -u +%F)" --scope full --maximum-bytes 1000000000 \
--coverage "$ARGAND_SITE_DATA/full-coverage.json" \
> "$ARGAND_SITE_DATA/curlie-download.json"
for source in psl wikidata majestic curlie; do
@ -84,6 +95,13 @@ for source in psl wikidata majestic curlie; do
done
```
The reusable `default` collection is safe because coverage graphs are separated
by provider. For a replacement, copy the preceding manifest ID into the new
coverage object's `supersedes` array. For a delta, also set `kind: "delta"`,
`base` to that ID, and a consecutive positive `sequence`. Use distinct stable
`partition` names only for provider-declared disjoint subsets. The build rejects
ambiguous or overlapping typed coverage.
For a full Wikidata dump, select a real dump URL from the official
[download index](https://dumps.wikimedia.org/wikidatawiki/entities/), then use
`--format wikidata-dump --compression gzip` (or `bzip2`) and `--scope full`.
@ -134,7 +152,15 @@ Create `crux-request.json` with your project and explicit limits:
"month": "202608",
"country": null,
"maximum_bytes_billed": 1000000000,
"maximum_output_bytes": 500000000
"maximum_output_bytes": 500000000,
"coverage": {
"collection": "monthly-origins",
"kind": "partition",
"partition": "global",
"base": null,
"sequence": null,
"supersedes": []
}
}
```
@ -158,224 +184,192 @@ exact CSV projection can instead use `manifest --source crux --format crux-csv
retrieval time, query/snapshot identity and appropriate `monthly:YYYYMM:country`
scope. The token is never written into a manifest.
## Build, look up and review
Keep a partition coordinate stable across refreshes, such as `global` or `GB`.
The month belongs in the source snapshot identity. With scheduled typed
supersession enabled, the next month then replaces the same audience partition
instead of accumulating stale popularity facts.
## Build, inspect and review
The strict default requires an independently maintained OpenSSH reviewer trust
file and two independent approvals for each name, edge, and equivalence. Build a
candidate and inspect its deterministic work queue:
```bash
argand-site-registry build --database "$ARGAND_SITE_DATA/import.sqlite" \
--output "$ARGAND_SITE_DATA/generation-1" > "$ARGAND_SITE_DATA/build-1.json"
export ARGAND_SITE_PIN="$(jq -r .pin "$ARGAND_SITE_DATA/build-1.json")"
argand-site-registry lookup --generation "$ARGAND_SITE_DATA/generation-1" \
--output "$ARGAND_SITE_DATA/candidate" \
--reviewer-trust /secure/reviewer-allowed-signers \
> "$ARGAND_SITE_DATA/candidate.json"
export ARGAND_SITE_PIN="$(jq -r .pin "$ARGAND_SITE_DATA/candidate.json")"
argand-site-registry lookup --generation "$ARGAND_SITE_DATA/candidate" \
--pin "$ARGAND_SITE_PIN" --query facebook
argand-site-registry lookup --generation "$ARGAND_SITE_DATA/generation-1" \
--pin "$ARGAND_SITE_PIN" --query amazon --limit 100
argand-site-registry review-queue --generation "$ARGAND_SITE_DATA/candidate" \
--pin "$ARGAND_SITE_PIN" --at 2026-09-13T00:00:00Z
```
The real-source acceptance run produced:
A source-shaped Facebook result retains `https://www.facebook.com/`, its
Wikidata Q355 identity, the `facebook.com` registrable domain, all relevant source
facts, and source-separated popularity. An entity can carry `example.com`,
`example.co.uk`, and `example.de` only when source evidence attaches each property
to that exact entity. PSL parsing returns `example.co.uk`, not `co.uk`, as the
registrable domain.
| Query | Entity | Example properties | Registrable domains |
| --- | --- | --- | --- |
| `facebook` | Facebook (Q355) | `https://www.facebook.com/`, `https://m.facebook.com/` | `facebook.com` |
| `amazon` | Amazon (Q3884) | `https://www.amazon.com/`, `https://www.amazon.co.uk/`, `https://www.amazon.de/` | `amazon.com`, `amazon.co.uk`, `amazon.de` |
These properties were asserted on the **same Wikidata entity**. Hostname
resemblance did not establish the relationship. Imported qualifiers remain in
`evidence` and `property_scopes`; unknown locale/country remains null. Names,
aliases and entity metadata carry their own fact-level source declarations.
Regional locale/country and role are explicit reviewed assertions. They are
separate from entity headquarters, ccTLD spelling and CrUX audience country.
Inspect the full statements, references, names/aliases, hostname spelling and
independent current ownership/role evidence. A review JSON has this shape:
```json
{
"fingerprint": "COPY_THE_EXACT_64_CHARACTER_FINGERPRINT_FROM_LOOKUP",
"decision": "approve",
"reviewer": "operator identity",
"reason": "How entity ownership and this exact destination role were verified",
"evidence": "An immutable capture identifier or evidence digest",
"reviewed_at": "2026-09-12T12:00:00Z",
"expires_at": "2026-10-12T12:00:00Z",
"role": "regional",
"locale": "",
"country": "GB"
}
```
Replace the example evidence and dates; approvals expire within 90 days. Use
`role: "primary"` for the independently verified default and `country: "DE"`
for a separately verified German regional property. A country-scoped review
can leave locale empty. If both are specified, both must match the request.
Sign the exact decision bytes under the dedicated reviewer namespace. The reviewer
identity must match the JSON and an independently maintained OpenSSH allowed-signers
file. The release signing key may be separate from reviewer keys.
Prepare canonical vote JSON for the exact queued name or edge. The command fills
the current evidence-bundle and policy digests. Never hand-copy an earlier digest.
```bash
ssh-keygen -Y sign -n argand-site-registry-review \
-f /secure/reviewer-key review.json
argand-site-registry review --database "$ARGAND_SITE_DATA/import.sqlite" \
--generation "$ARGAND_SITE_DATA/generation-1" --pin "$ARGAND_SITE_PIN" \
--decision review.json --signature review.json.sig \
--allowed-reviewers /secure/reviewer-allowed-signers \
--identity operator@example.org
argand-site-registry build --database "$ARGAND_SITE_DATA/import.sqlite" \
--output "$ARGAND_SITE_DATA/generation-2" > "$ARGAND_SITE_DATA/build-2.json"
export ARGAND_SITE_PIN="$(jq -r .pin "$ARGAND_SITE_DATA/build-2.json")"
argand-site-registry resolve --generation "$ARGAND_SITE_DATA/generation-2" \
--pin "$ARGAND_SITE_PIN" --query amazon --country GB
argand-site-registry prepare-vote \
--generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
--subject-kind edge --fingerprint "$EDGE_FINGERPRINT" \
--decision approve --reviewer reviewer-one \
--reason "Verified entity, URL, and exact role from retained evidence" \
--reviewed-at 2026-09-13T00:00:00Z \
--expires-at 2026-10-13T00:00:00Z --role primary \
--output /secure/edge-vote.json
ssh-keygen -Y sign -n argand-site-registry-vote \
-f /secure/reviewer-one /secure/edge-vote.json
argand-site-registry verify-vote \
--generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
--decision /secure/edge-vote.json --signature /secure/edge-vote.json.sig \
--allowed-reviewers /secure/reviewer-allowed-signers --identity reviewer-one
argand-site-registry vote --database "$ARGAND_SITE_DATA/import.sqlite" \
--generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
--decision /secure/edge-vote.json --signature /secure/edge-vote.json.sig \
--allowed-reviewers /secure/reviewer-allowed-signers --identity reviewer-one
```
After the corresponding real reviews, GB selects the reviewed UK property;
DE selects the reviewed German property; otherwise an explicitly reviewed
primary may be used. Unknown, expired, tied or entity-ambiguous requests return
`"destination": null` with `status` and rejection counts. Result limits never hide
ambiguity. Name/alias changes,
changed statements/revisions, URLs or normalization evidence invalidate reviews.
Deprecated, end-dated and non-value statements remain audit evidence and cannot
be admitted. An unchanged PSL file with a new retrieval time preserves reviews.
Repeat with another identity, group, and physical key. Prepare separate name votes
for the exact label or alias used by the query. A regional edge approval uses
`--role regional` plus `--country GB`, `--locale en-GB`, or both. A primary edge
is an unscoped global fallback. If both country and locale are set, both must
match. The resolver abstains on missing quorum, ambiguity, expiry, revocation,
stale evidence, stale policy, or equal destinations.
The writer assigns `accepted_at`. Effective validity begins at the later of that
time and signed `reviewed_at`, and ends no later than 90 days after acceptance.
A revocation has no expiry. Every approval in a new quorum must pass each active
revocation ID with `--supersedes`; ordinary later approvals remain blocked.
### Observe an existing candidate
The observer is candidate-only and does not grant approval. It pins public DNS per
hop, rejects private/link-local addresses, credentials, nondefault ports, HTTPS
downgrade, compressed response bodies, oversized headers/bodies and more than five redirects. Store its cache
outside Git, replay it without network access, import the immutable batch, and
rebuild:
```bash
argand-site-registry observe \
--generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
--fingerprint "$EDGE_FINGERPRINT" \
--capture "$ARGAND_SITE_DATA/captures/run-1" \
--output "$ARGAND_SITE_DATA/observations/run-1.jsonl" \
--manifest-output "$ARGAND_SITE_DATA/observations/run-1.source.json"
argand-site-registry observation-import \
--database "$ARGAND_SITE_DATA/import.sqlite" \
--generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
--input "$ARGAND_SITE_DATA/observations/run-1.jsonl" \
--manifest "$ARGAND_SITE_DATA/observations/run-1.source.json"
```
`observe-replay` reproduces JSONL from the cache without a request.
`observations` looks up an exact subject, `observation-lookup` searches exact URLs
or domains, and `drift` compares the latest two batches. A new observation changes
the evidence bundle and requires fresh review; it never auto-renews a vote.
For scheduled runs, `{timestamp}` in the three `observe` output paths expands once
to a nanosecond UTC coordinate. The template service and timer in `examples/`
serialize enabled fingerprints with a runtime lock, cap response bandwidth, and
run one candidate per process; operators still import and
review the produced batch separately.
## Release, export, update and recovery
When two providers describe the same navigational entity, `lookup` deliberately
shows both source IDs. Connect them only after reviewing their identities:
Preview an entity pair, then use the dedicated equivalence vote commands. Two
similar names or domains remain separate until the equivalence quorum passes:
```bash
argand-site-registry equivalence --generation "$ARGAND_SITE_DATA/generation-2" \
--pin "$ARGAND_SITE_PIN" --left SOURCE_ENTITY_ID --right OTHER_SOURCE_ENTITY_ID
argand-site-registry equivalence --generation "$ARGAND_SITE_DATA/candidate" \
--pin "$ARGAND_SITE_PIN" --left SOURCE_ENTITY_ID --right OTHER_ENTITY_ID
argand-site-registry prepare-equivalence-vote \
--generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
--left SOURCE_ENTITY_ID --right OTHER_ENTITY_ID --decision approve \
--reviewer reviewer-one --reason "Same entity under both source IDs" \
--reviewed-at 2026-09-13T00:00:00Z \
--expires-at 2026-10-13T00:00:00Z --output /secure/equivalence-vote.json
```
Use the returned fingerprint in a review JSON with `role: "unspecified"`, empty
locale/country, a reason, immutable identity evidence and an expiry. Sign
`identity-review.json` with the reviewer namespace and repeat the command with
`--database`, `--decision`, `--signature`, `--allowed-reviewers` and `--identity`,
then rebuild. `resolve` follows only active, explicitly
reviewed equivalences and includes their provenance. Each destination still
needs its own review. The original IDs, raw ambiguity counts and conflicting
assertions remain visible. Changed names or website assertions invalidate the
identity decision; identity revocations use the same append-only release log.
Operator-authored decisions are published under CC0-1.0, separately from source
data licenses.
Each generation contains `registry.sqlite`, `LICENSE_SOURCES.md`,
`ATTRIBUTION.json` and a hash-binding `COMPLETE.json`. Distribute all four together.
Keep the import database, review history and cached source bytes for recovery.
The SQLite file includes raw relevant records and descriptions for audit; public
consumers must obey the source attribution requirements. `export` streams
source-bearing JSONL and omits descriptions by default:
After appending complete quorums, rebuild and inspect `stats`, `diff`,
`review-queue`, and `evaluate`. `export` writes selected, nonrejected facts.
`export-audit` also writes superseded and rejected facts. Both preserve provenance
and attribution, redact Curlie descriptions by default, and are evidence exports
rather than admitted-route lists.
```bash
argand-site-registry export --generation "$ARGAND_SITE_DATA/generation-2" \
--pin "$ARGAND_SITE_PIN" --output "$ARGAND_SITE_DATA/assertions.jsonl"
argand-site-registry sign --generation "$ARGAND_SITE_DATA/generation-2" \
--pin "$ARGAND_SITE_PIN" --key /secure/registry-signing-key \
--allowed-reviewers /secure/reviewer-allowed-signers
argand-site-registry activate --generation "$ARGAND_SITE_DATA/generation-2" \
argand-site-registry export --generation "$ARGAND_SITE_DATA/reviewed" \
--pin "$REVIEWED_PIN" --output "$ARGAND_SITE_DATA/active.jsonl"
argand-site-registry export-audit --generation "$ARGAND_SITE_DATA/reviewed" \
--pin "$REVIEWED_PIN" --output "$ARGAND_SITE_DATA/audit.jsonl"
argand-site-registry sign --generation "$ARGAND_SITE_DATA/reviewed" \
--pin "$REVIEWED_PIN" --key /secure/publisher-key \
--allowed-reviewers /secure/reviewer-allowed-signers \
--identity registry-publisher
argand-site-registry activate --generation "$ARGAND_SITE_DATA/reviewed" \
--current "$ARGAND_SITE_DATA/current.json" \
--allowed-signers /secure/registry-allowed-signers \
--allowed-signers /secure/publisher-allowed-signers \
--allowed-reviewers /secure/reviewer-allowed-signers \
--identity registry-publisher
```
Before signing, the CLI replays every stored reviewer signature against the supplied
reviewer trust file and rejects missing, forged or altered proofs. Use an existing
operator-controlled SSH release key. The external release allowed-signers
file follows OpenSSH syntax: `registry-publisher ssh-ed25519 PUBLIC_KEY`. Neither
keys nor the trust file should come from the downloaded dataset. Consumers can
open `Registry::open(path, trusted_receipt_sha256)` once and reuse its indexed
queries, or verify both a publisher and the retained reviewer proofs with
`release::verify_signed` first. A hash proves
integrity only relative to a trusted pin. Signature verification authenticates
the publisher, not the truth of a source assertion.
Strict signing rejects a publisher identity or physical key used for any reviewer vote.
Activation re-verifies the publisher, reviewer trust digest, every retained vote,
and revocation continuity. Rollback is allowed only when the target retains every
distributed legacy and vote revocation.
`diff --old PATH --old-pin HASH --new PATH --new-pin HASH` streams typed added,
removed and changed source selections, entities, names, properties, edges,
popularity observations, reviews and equivalences. `verify --generation PATH
--pin HASH` checks every artifact
bound by the receipt. To revoke, append a review with `decision: "revoke"`, then
rebuild, sign and activate. Re-activating an older signed generation supports
rollback **only if it retains every distributed revocation**. Otherwise rebuild
the older source selection with the current review log; never edit generations.
For emergency delivery, `export-revocations` creates a cumulative feed at an
explicit time. `sign-revocations` recomputes it from the pinned generation before
using the publisher's separate SSH key. `verify-revocations` checks its signature,
compatibility, effective time, seven-day refresh deadline, and optional prior-feed
continuity. Cross-generation feeds can add blocks but cannot restore a route. A consumer may
pass `--revocations`, `--revocation-signature`, `--allowed-publishers`, and
`--publisher-identity` to `resolve` so the signed block applies before a full
replacement generation is installed. After bootstrap, also pass the last accepted
feed through `--previous-revocations` and `--previous-revocation-signature` to
reject a replacement that drops retained revocations.
The [update configuration](examples/update.toml) and [systemd service/timer](examples/)
provide weekly candidate refreshes without a resident daemon. Set absolute paths
and an installed executable path. TOML paths do not expand environment variables.
`update --config /etc/argand-site-registry.toml` downloads/imports all configured
sources and builds only after they succeed. The same inputs and review log reuse
the same generation. A nonzero exit is a failed refresh; the active pointer stays
intact. Failed `pending-*` builds can be inspected before explicitly removing
that incomplete directory. Acquisition, import and update operations take local
locks; use one writer and keep old complete generations for rollback.
Downloads permit only the reviewed HTTPS source endpoints, validate each
redirect, bound bytes and bind range resumes to strong ETags. Chunked/validatorless
responses safely restart on interruption. PSL network attempts are limited to
once per 24 hours per cache. CrUX is opt-in and may use `{previous_month}` in
update configuration; monthly data may not yet be published on the first day.
Wikidata source snapshot labels support `{date}` and `{month}` in scheduled
downloads. No scheduled job signs, approves, renews approvals or activates links.
The [update configuration](examples/update.toml) and
systemd examples refresh candidates without a resident daemon. Its explicit
`auto_supersede_typed_snapshots = true` setting replaces only the current frontier
with the same provider, collection and full/partition coordinate; deltas and
coverage-layout changes still require exact operator-supplied bases. Update jobs may
download, import and build. Schedule observation commands independently according
to risk. Use separate cache/capture/output paths and process concurrency limits;
one observer invocation handles one candidate with explicit body, redirect and
time bounds. No scheduled command approves, renews, signs or activates.
## Storage and operating limits
Migrations `migrations/001.sql`, `002.sql` and `003.sql` own schema version 3. `sources`, `records` and
`facts` preserve snapshot/native IDs, licenses, retrieval times and confidence;
`reviews` and their cryptographic `review_auth` proofs are append-only. Complete source selection is latest retrieval time per
provider/scope, with digest as the deterministic tie break. Use the **same scope**
for a replacement snapshot, and separate scopes for deliberate independent
selections. History and conflicts remain stored. A failed source cannot replace
a complete one. Avoid overlapping full/partial scopes unless both evidences are
intended to remain active.
Migrations 001 through 005 own writer schema 5. Source manifests v2 declare typed
full, partition, or delta coverage. Deltas name an exact base, consecutive sequence
and superseded object. Ambiguous coverage, overlap, cycles, gaps, cross-provider
supersession and duplicate native records fail the build. V1 manifests remain
isolated by provider and scope for compatibility.
Derived tables are `selected_sources`, `entities`, `names`, `properties`, `edges`,
`popularity` and `rejected`. Entity IDs derive from source/native IDs; URL IDs
derive from strict normalized URLs. Equal source entities merge across snapshots
and equal URLs share a property. Explicit `equivalences` connect reviewed
cross-source identities while preserving both IDs. Names are never an identity
join. The canonical label rule prefers labels, then English,
then language/text order. Original labels/aliases are kept. Popularity has its
own source, target, observation period and audience scope and never creates an
ownership edge. All derivations bind input fact IDs, PSL identity and the
`argand.site-rules/v3` contract through their generation receipt. A v1 or v2 writer store
migrates in place while retaining review history. Any legacy unauthenticated
decision makes release signing fail closed; start a reviewed v3 store from the
pinned source inputs rather than deleting historical decisions.
Each generation contains `registry.sqlite`, `LICENSE_SOURCES.md`,
`ATTRIBUTION.json`, and `COMPLETE.json`, plus an optional publisher signature. The
receipt binds database bytes, source selection, policy, reviewer trust, licenses,
attribution, and decision-time contract. Keep source objects, writer state,
generations, pins, trust files and signatures for recovery.
Imports use transactions of 256 relevant records with durable replay checkpoints.
Restart replays the compressed stream and skips committed records. The exact open
descriptor stream is hashed, and any integrity or resource failure removes all
partially committed rows for that source. Large source records are capped at 16 MiB;
imports also have finite expanded-byte, record and database-growth ceilings. Override
them with `import --maximum-expanded-bytes`, `--maximum-records` and
`--maximum-database-growth-bytes` when a reviewed source requires different bounds.
SQLite has an 8 MiB page cache and disk-backed sorts.
Builds stream a canonical sorted copy and never load the full registry into RAM.
Names and entity metadata exposed by lookup are capped at 256 facts each, with
uncapped totals; the complete assertions remain available in the database/export.
Lookup returns at most 100 edges, reports all pre-limit ambiguity counts and caps
aggregate serialized candidate data at 64 MiB. Typed diff events are capped at
16 MiB and the full stream at 1 GiB; descriptions are redacted and the header
contains source attribution.
The full store/history and each generation consume disk; there is no automatic
pruning. These bounds are not a full-dump throughput claim.
Imports are streaming, transactional, resumable and idempotent. Defaults bound
expanded bytes, record size/count, database growth, query output and diff output.
Use command-line overrides only after checking the real object and local capacity.
Raw datasets, credentials, signing keys and production review logs do not belong
in this repository.
The `observation` module validates and deterministically normalizes future crawler
evidence for redirects, canonical links, hreflang, JSON-LD sameAs, sitemaps and
country selectors, with capture IDs, hashes, rights and confidence. It does not
crawl, admit a new source or automatically infer ownership.
Use `entity`, `lookup-web`, `popularity`, `category` and `stats` for reverse/audit
views. Curlie descriptions remain redacted on the category surface. The
`evaluate` command accepts bounded JSONL judgments; see the repository
[evaluation guide](../../docs/EVALUATION.md) and [publisher runbook](../../docs/PUBLISHING.md).
Source vandalism, compromised publishers and a domain changing ownership cannot
be eliminated by hashes or popularity. Review expiration, exact evidence binding,
signed releases, explicit revocations and conservative abstention contain those
risks. Protect the writer database, signing key and consumer trust configuration.
Do not feed raw `lookup` candidates straight into an automatic redirect consumer.
If an import fails, fix the input/format or reuse the matching original source
manifest, then rerun the same import. Do not edit digests to make corrupted data
pass. A source host/schema change needs an adapter review. HTTP 403/429 is a
source-access failure; reuse an authorized retained snapshot or retry according
to the provider's policy. A null resolution means evidence/review is missing,
expired or ambiguous; `lookup` explains which assertions are involved.
`lookup`, reverse lookup, popularity and categories are audit surfaces. Popularity,
TLS, DNS, redirects, `sameAs`, ccTLD spelling and source confidence do not prove
ownership or safety. Use `resolve` for navigation and keep a null destination as
an intentional abstention. See [TRUST.md](../../docs/TRUST.md),
[PUBLISHING.md](../../docs/PUBLISHING.md), and
[LICENSE_SOURCES.md](LICENSE_SOURCES.md).

39
crates/argand-site-registry/examples/argand-site-registry-observe@.service Normal file
View file

@ -0,0 +1,39 @@
# By Nic Weyand! One bounded candidate fingerprint per instance; no approval authority.
[Unit]
Description=Observe Argand Site Registry candidate %i
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
User=argand-site-registry
Group=argand-site-registry
StateDirectory=argand-site-registry
RuntimeDirectory=argand-site-registry-observer
EnvironmentFile=/etc/argand-site-registry-observer.env
ExecStartPre=/usr/bin/mkdir -p /var/lib/argand-site-registry/captures /var/lib/argand-site-registry/observations
ExecStart=/usr/bin/flock --nonblock /run/argand-site-registry-observer/observer.lock /usr/local/bin/argand-site-registry observe --generation ${ARGAND_REGISTRY_GENERATION} --pin ${ARGAND_REGISTRY_PIN} --fingerprint %i --capture /var/lib/argand-site-registry/captures/%i-{timestamp} --output /var/lib/argand-site-registry/observations/%i-{timestamp}.jsonl --manifest-output /var/lib/argand-site-registry/observations/%i-{timestamp}.source.json --maximum-body-bytes 2097152 --maximum-redirects 5 --timeout-seconds 20 --maximum-bytes-per-second 1048576
UMask=0077
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=true
ProtectClock=true
ProtectControlGroups=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProcSubset=pid
ReadWritePaths=/var/lib/argand-site-registry
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
CapabilityBoundingSet=
AmbientCapabilities=
SystemCallArchitectures=native
TimeoutStartSec=3min

12
crates/argand-site-registry/examples/argand-site-registry-observe@.timer Normal file
View file

@ -0,0 +1,12 @@
# By Nic Weyand! Enable only for explicitly reviewed candidate fingerprints.
[Unit]
Description=Refresh Argand Site Registry observation %i daily
[Timer]
OnCalendar=daily
RandomizedDelaySec=2h
Persistent=true
Unit=argand-site-registry-observe@%i.service
[Install]
WantedBy=timers.target

3
crates/argand-site-registry/examples/observer.env Normal file
View file

@ -0,0 +1,3 @@
# Refresh these public candidate coordinates before each scheduled run.
ARGAND_REGISTRY_GENERATION=/var/lib/argand-site-registry/generations/candidate-RECEIPT_PIN
ARGAND_REGISTRY_PIN=RECEIPT_PIN

26
crates/argand-site-registry/examples/update.toml
View file

@ -2,6 +2,10 @@
cache = "/var/lib/argand-site-registry/cache"
database = "/var/lib/argand-site-registry/import.sqlite"
generations = "/var/lib/argand-site-registry/generations"
reviewer_trust = "/etc/argand-site-registry/reviewer-allowed-signers"
# This authorizes each scheduled full/partition download to supersede the exact
# current frontier for the same provider, collection and partition coordinate.
auto_supersede_typed_snapshots = true
[[downloads]]
source = "psl"
@ -10,6 +14,10 @@ url = "https://publicsuffix.org/list/public_suffix_list.dat"
snapshot = "{date}"
scope = "full"
maximum_bytes = 1000000
[downloads.coverage]
collection = "default"
kind = "full"
supersedes = []
[[downloads]]
source = "wikidata"
@ -18,6 +26,11 @@ url = "https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q355%7CQ3884&
snapshot = "{date}"
scope = "selection:facebook-amazon"
maximum_bytes = 5000000
[downloads.coverage]
collection = "entity-selections"
kind = "partition"
partition = "facebook-amazon"
supersedes = []
[[downloads]]
source = "majestic"
@ -26,6 +39,10 @@ url = "https://downloads.majestic.com/majestic_million.csv"
snapshot = "{date}"
scope = "full"
maximum_bytes = 250000000
[downloads.coverage]
collection = "default"
kind = "full"
supersedes = []
[[downloads]]
source = "curlie"
@ -34,6 +51,10 @@ url = "https://curlie.org/directory-dl"
snapshot = "{date}"
scope = "full"
maximum_bytes = 1000000000
[downloads.coverage]
collection = "default"
kind = "full"
supersedes = []
# Optional pinned acquisitions; repeat [[inputs]] for each source.
# [[inputs]]
@ -48,3 +69,8 @@ maximum_bytes = 1000000000
# country = "GB"
# maximum_bytes_billed = 1000000000
# maximum_output_bytes = 500000000
# [crux.coverage]
# collection = "monthly-origins"
# kind = "partition"
# partition = "GB"
# supersedes = []

37
crates/argand-site-registry/migrations/004.sql Normal file
View file

@ -0,0 +1,37 @@
-- By Nic Weyand! ADR 0002/0003: granular accepted-time, authenticated reviewer votes.
CREATE TABLE votes (
sequence INTEGER PRIMARY KEY,
id TEXT NOT NULL UNIQUE,
fingerprint TEXT NOT NULL,
subject_kind TEXT NOT NULL CHECK(subject_kind IN ('name','edge','equivalence')),
decision TEXT NOT NULL CHECK(decision IN ('approve','revoke')),
reviewer TEXT NOT NULL,
reason TEXT NOT NULL,
evidence_bundle TEXT NOT NULL,
policy_sha256 TEXT NOT NULL,
reviewed_at TEXT NOT NULL,
expires_at TEXT,
role TEXT NOT NULL,
locale TEXT NOT NULL,
country TEXT NOT NULL,
supersedes_json TEXT NOT NULL,
accepted_at TEXT NOT NULL,
document_json BLOB NOT NULL
) STRICT;
CREATE INDEX vote_subject ON votes(subject_kind,fingerprint,sequence);
CREATE INDEX vote_reviewer ON votes(reviewer,sequence);
CREATE TABLE vote_auth (
sequence INTEGER PRIMARY KEY REFERENCES votes(sequence),
signer TEXT NOT NULL,
signature_sha256 TEXT NOT NULL,
namespace TEXT NOT NULL CHECK(namespace='argand-site-registry-vote'),
decision_sha256 TEXT NOT NULL,
key_sha256 TEXT NOT NULL,
signature BLOB NOT NULL
) STRICT;
CREATE TRIGGER vote_no_update BEFORE UPDATE ON votes BEGIN SELECT RAISE(ABORT,'votes are append-only'); END;
CREATE TRIGGER vote_no_delete BEFORE DELETE ON votes BEGIN SELECT RAISE(ABORT,'votes are append-only'); END;
CREATE TRIGGER vote_auth_no_update BEFORE UPDATE ON vote_auth BEGIN SELECT RAISE(ABORT,'vote authentication is immutable'); END;
CREATE TRIGGER vote_auth_no_delete BEFORE DELETE ON vote_auth BEGIN SELECT RAISE(ABORT,'vote authentication is immutable'); END;
UPDATE registry_metadata SET rules='argand.site-rules/v4' WHERE singleton=1;
PRAGMA user_version=4;

25
crates/argand-site-registry/migrations/005.sql Normal file
View file

@ -0,0 +1,25 @@
-- By Nic Weyand! ADR 0002: store immutable, subject-bound observation batches.
CREATE TABLE observation_batches (
id TEXT PRIMARY KEY,
source TEXT NOT NULL,
retrieved_at TEXT NOT NULL,
manifest_json TEXT NOT NULL,
records INTEGER NOT NULL DEFAULT 0 CHECK(records>=0),
complete INTEGER NOT NULL DEFAULT 0 CHECK(complete IN (0,1))
) STRICT;
CREATE TABLE observations (
fingerprint TEXT PRIMARY KEY,
batch_id TEXT NOT NULL REFERENCES observation_batches(id),
subject_kind TEXT NOT NULL CHECK(subject_kind IN ('name','edge','equivalence')),
subject_fingerprint TEXT NOT NULL,
document_json TEXT NOT NULL
) STRICT;
CREATE INDEX observation_subject ON observations(subject_kind,subject_fingerprint,fingerprint);
CREATE TRIGGER observation_batch_open_insert BEFORE INSERT ON observation_batches WHEN NEW.records<>0 OR NEW.complete<>0 BEGIN SELECT RAISE(ABORT,'observation batch must be created open'); END;
CREATE TRIGGER observation_batch_no_update BEFORE UPDATE OF id,source,retrieved_at,manifest_json ON observation_batches BEGIN SELECT RAISE(ABORT,'observation batch identity is immutable'); END;
CREATE TRIGGER observation_batch_finish_once BEFORE UPDATE OF records,complete ON observation_batches WHEN OLD.complete<>0 OR NEW.complete<>1 OR NEW.records<=0 OR NEW.records<>(SELECT count(*) FROM observations WHERE batch_id=OLD.id) BEGIN SELECT RAISE(ABORT,'observation batch can complete only once with its exact record count'); END;
CREATE TRIGGER observation_batch_no_delete BEFORE DELETE ON observation_batches BEGIN SELECT RAISE(ABORT,'observation batches are immutable'); END;
CREATE TRIGGER observation_insert_open BEFORE INSERT ON observations WHEN NOT EXISTS(SELECT 1 FROM observation_batches WHERE id=NEW.batch_id AND complete=0) BEGIN SELECT RAISE(ABORT,'observations require an open batch'); END;
CREATE TRIGGER observation_no_update BEFORE UPDATE ON observations BEGIN SELECT RAISE(ABORT,'observations are immutable'); END;
CREATE TRIGGER observation_no_delete BEFORE DELETE ON observations BEGIN SELECT RAISE(ABORT,'observations are immutable'); END;
PRAGMA user_version=5;

176
crates/argand-site-registry/src/build.rs
View file

@ -16,11 +16,9 @@ use std::{
};
const PROJECTIONS: &str = "
CREATE TABLE selected_sources(id TEXT PRIMARY KEY REFERENCES sources(id)) STRICT;
INSERT INTO selected_sources SELECT id FROM (
SELECT id,row_number() OVER(PARTITION BY source,scope ORDER BY retrieved_at DESC,id DESC) AS position
FROM sources WHERE complete=1) WHERE position=1;
CREATE TABLE names(entity TEXT NOT NULL,key TEXT NOT NULL,text TEXT NOT NULL,language TEXT NOT NULL,kind TEXT NOT NULL,fact TEXT NOT NULL REFERENCES facts(id),PRIMARY KEY(entity,fact)) STRICT;
CREATE TABLE selected_sources(id TEXT PRIMARY KEY REFERENCES sources(id),precedence INTEGER NOT NULL CHECK(precedence>=0),partition TEXT NOT NULL) STRICT;
CREATE TABLE active_records(source_id TEXT NOT NULL,ordinal INTEGER NOT NULL,PRIMARY KEY(source_id,ordinal),FOREIGN KEY(source_id,ordinal) REFERENCES records(source_id,ordinal)) STRICT;
CREATE TABLE names(entity TEXT NOT NULL,key TEXT NOT NULL,text TEXT NOT NULL,language TEXT NOT NULL,kind TEXT NOT NULL,fact TEXT NOT NULL REFERENCES facts(id),fingerprint TEXT NOT NULL UNIQUE,PRIMARY KEY(entity,fact)) STRICT;
CREATE INDEX name_lookup ON names(key,entity);
CREATE TABLE entities(id TEXT PRIMARY KEY,canonical_name TEXT NOT NULL,names_fingerprint TEXT NOT NULL) STRICT;
CREATE TABLE properties(id TEXT PRIMARY KEY,url TEXT NOT NULL UNIQUE,hostname TEXT NOT NULL,domain TEXT NOT NULL,suffix TEXT NOT NULL,derived_json TEXT NOT NULL) STRICT;
@ -30,13 +28,14 @@ CREATE INDEX edge_entity ON edges(entity,property);
CREATE TABLE popularity(fact TEXT PRIMARY KEY REFERENCES facts(id),source TEXT NOT NULL,target TEXT NOT NULL,hostname TEXT NOT NULL,domain TEXT NOT NULL,value TEXT NOT NULL,derived_json TEXT NOT NULL) STRICT;
CREATE INDEX popularity_host ON popularity(hostname,source);
CREATE TABLE rejected(fact TEXT PRIMARY KEY REFERENCES facts(id),reason TEXT NOT NULL) STRICT;
CREATE TABLE review_policy(singleton INTEGER PRIMARY KEY CHECK(singleton=1),id TEXT NOT NULL,document TEXT NOT NULL) STRICT;
";
/// Completion receipt. Authenticity needs an external digest or trusted signature.
#[derive(Clone, Debug, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct Receipt {
/// `argand.site-registry/v1`.
/// `argand.site-registry/v2`.
pub schema: String,
/// Parser/derivation contract.
pub rules: String,
@ -50,6 +49,21 @@ pub struct Receipt {
pub psl_source: String,
/// Active source snapshot declarations.
pub sources: Vec<SourceManifest>,
/// Exact policy compiled by consumers.
#[serde(default)]
pub review_policy: crate::policy::ReviewPolicy,
/// Digest of the canonical review-policy JSON.
#[serde(default)]
pub review_policy_sha256: String,
/// Digest of exact allowed-reviewer file bytes for strict policies.
#[serde(default)]
pub reviewer_trust_sha256: String,
/// Digest of the complete selected source coverage graph.
#[serde(default)]
pub coverage_sha256: String,
/// Trusted writer-acceptance and bounded-expiry contract.
#[serde(default)]
pub decision_time_policy: String,
/// Distinct entity count.
pub entities: u64,
/// Strict URL identity count.
@ -65,13 +79,55 @@ pub struct Receipt {
/// # Errors
/// Rejects existing destinations, incomplete PSL, corrupt stores, and I/O failures.
pub fn build(db: &Connection, output: &Path) -> anyhow::Result<Receipt> {
build_with_policy(db, output, &crate::policy::ReviewPolicy::reference())
}
/// Builds a generation under an explicit, receipt-authenticated review policy.
///
/// # Errors
/// Rejects invalid policy, existing destinations, corrupt stores, and I/O failures.
pub fn build_with_policy(
db: &Connection,
output: &Path,
policy: &crate::policy::ReviewPolicy,
) -> anyhow::Result<Receipt> {
build_with_policy_and_trust(db, output, policy, None)
}
/// Builds with an external reviewer trust root bound into the immutable receipt.
///
/// # Errors
/// Rejects strict policies without trust roots, malformed roots, invalid policy,
/// existing destinations, corrupt stores, and I/O failures.
pub fn build_with_policy_and_trust(
db: &Connection,
output: &Path,
policy: &crate::policy::ReviewPolicy,
reviewer_trust: Option<&Path>,
) -> anyhow::Result<Receipt> {
policy.validate()?;
let reviewer_trust_sha256 = reviewer_trust
.map(|path| crate::ssh::sealed_input(path, 1024 * 1024))
.transpose()?
.map(|input| crate::digest(&input.bytes))
.unwrap_or_default();
ensure!(
policy.allow_legacy_reviews || crate::model::valid_digest(&reviewer_trust_sha256),
"strict review policy requires an exact reviewer trust root"
);
fs::create_dir(output).context("generation destination must not exist")?;
let path = output.join("registry.sqlite");
let snapshot = store::open(&path)?;
copy_canonical(db, &snapshot)?;
snapshot.execute_batch(PROJECTIONS)?;
let (psl_id,text): (String,String)=snapshot.query_row("SELECT f.source_id,f.value FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.predicate='psl' ORDER BY f.source_id LIMIT 1",[],|r| Ok((r.get(0)?,r.get(1)?))).context("import a complete PSL snapshot first")?;
let psl_count:u64=snapshot.query_row("SELECT count(*) FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.predicate='psl'",[],|r|store::unsigned(r,0))?;
crate::coverage::project(&snapshot)?;
let policy_id = policy.id()?;
snapshot.execute(
"INSERT INTO review_policy VALUES(1,?1,?2)",
params![policy_id, serde_json::to_string(policy)?],
)?;
let (psl_id,text): (String,String)=snapshot.query_row("SELECT f.source_id,f.value FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.predicate='psl' ORDER BY f.source_id LIMIT 1",[],|r| Ok((r.get(0)?,r.get(1)?))).context("import a complete PSL snapshot first")?;
let psl_count:u64=snapshot.query_row("SELECT count(*) FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.predicate='psl'",[],|r|store::unsigned(r,0))?;
ensure!(psl_count == 1, "exactly one active PSL snapshot required");
let psl_text: String = serde_json::from_str(&text)?;
let normalizer = Normalizer::new(psl_text.as_bytes(), psl_id.clone())?;
@ -93,8 +149,9 @@ pub fn build(db: &Connection, output: &Path) -> anyhow::Result<Receipt> {
.map(|s| Ok(serde_json::from_str(&s?)?))
.collect::<anyhow::Result<Vec<_>>>()?;
drop(statement);
let coverage_sha256 = crate::coverage::projection_digest(&snapshot)?;
let mut receipt = Receipt {
schema: "argand.site-registry/v1".into(),
schema: "argand.site-registry/v2".into(),
rules: store::RULE_VERSION.into(),
database_sha256: String::new(),
licenses_sha256: crate::digest(crate::release::LICENSES.as_bytes()),
@ -103,6 +160,11 @@ pub fn build(db: &Connection, output: &Path) -> anyhow::Result<Receipt> {
)?),
psl_source: psl_id,
sources,
review_policy: policy.clone(),
review_policy_sha256: policy_id,
reviewer_trust_sha256,
coverage_sha256,
decision_time_policy: "argand.site-decision-time/v1".into(),
entities: count(&snapshot, "entities")?,
properties: count(&snapshot, "properties")?,
edges: count(&snapshot, "edges")?,
@ -158,6 +220,8 @@ fn copy_canonical(source: &Connection, destination: &Connection) -> anyhow::Resu
"SELECT * FROM equivalences ORDER BY fingerprint",
5,
),
("votes", "SELECT * FROM votes ORDER BY sequence", 17),
("vote_auth", "SELECT * FROM vote_auth ORDER BY sequence", 7),
];
for (table, select, columns) in tables {
let placeholders = (1..=columns)
@ -175,6 +239,46 @@ fn copy_canonical(source: &Connection, destination: &Connection) -> anyhow::Resu
insert.execute(rusqlite::params_from_iter(values))?;
}
}
let mut batch_rows = source.prepare(
"SELECT id,source,retrieved_at,manifest_json,records FROM observation_batches WHERE complete=1 ORDER BY id",
)?;
let completed_batches = batch_rows
.query_map([], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, String>(3)?,
row.get::<_, i64>(4)?,
))
})?
.collect::<Result<Vec<_>, _>>()?;
for (id, provider, retrieved_at, manifest, _) in &completed_batches {
destination.execute(
"INSERT INTO observation_batches(id,source,retrieved_at,manifest_json) VALUES(?1,?2,?3,?4)",
params![id, provider, retrieved_at, manifest],
)?;
}
let mut observation_rows = source.prepare(
"SELECT o.* FROM observations o JOIN observation_batches b ON b.id=o.batch_id WHERE b.complete=1 ORDER BY o.fingerprint",
)?;
let mut rows = observation_rows.query([])?;
let mut insert = destination.prepare("INSERT INTO observations VALUES(?1,?2,?3,?4,?5)")?;
while let Some(row) = rows.next()? {
let values = (0..5)
.map(|index| row.get::<_, rusqlite::types::Value>(index))
.collect::<Result<Vec<_>, _>>()?;
insert.execute(rusqlite::params_from_iter(values))?;
}
drop(insert);
drop(rows);
drop(observation_rows);
for (id, _, _, _, records) in completed_batches {
destination.execute(
"UPDATE observation_batches SET records=?2,complete=1 WHERE id=?1",
params![id, records],
)?;
}
destination_transaction.commit()?;
source_transaction.commit()?;
Ok(())
@ -189,7 +293,7 @@ fn count(db: &Connection, table: &str) -> anyhow::Result<u64> {
}
fn project_names(db: &Connection) -> anyhow::Result<()> {
let mut stmt=db.prepare("SELECT f.id,f.subject,f.value FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.predicate='name' ORDER BY f.subject,f.id")?;
let mut stmt=db.prepare("SELECT f.id,f.subject,f.value,s.source,r.native_id,f.selector FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal JOIN sources s ON s.id=f.source_id JOIN records r ON r.source_id=f.source_id AND r.ordinal=f.ordinal WHERE f.predicate='name' ORDER BY f.subject,f.id")?;
let mut rows = stmt.query([])?;
while let Some(row) = rows.next()? {
let (id, subject, raw): (String, String, String) = (row.get(0)?, row.get(1)?, row.get(2)?);
@ -197,16 +301,22 @@ fn project_names(db: &Connection) -> anyhow::Result<()> {
let text = value["text"].as_str().context("name text missing")?;
match name_key(text) {
Ok(key) => {
let language = value["language"].as_str().unwrap_or("und");
let kind = value["kind"].as_str().unwrap_or("label");
let fingerprint = crate::digest(&serde_json::to_vec(&(
"argand.site-name/v1",
&subject,
&key,
text,
language,
kind,
row.get::<_, String>(3)?,
row.get::<_, String>(4)?,
row.get::<_, String>(5)?,
))?);
db.execute(
"INSERT INTO names VALUES(?1,?2,?3,?4,?5,?6)",
params![
subject,
key,
text,
value["language"].as_str().unwrap_or("und"),
value["kind"].as_str().unwrap_or("label"),
id
],
"INSERT INTO names VALUES(?1,?2,?3,?4,?5,?6,?7)",
params![subject, key, text, language, kind, id, fingerprint],
)?;
}
Err(error) => {
@ -218,7 +328,7 @@ fn project_names(db: &Connection) -> anyhow::Result<()> {
}
}
// All entities with website assertions exist even when names are absent.
let mut entities=db.prepare("SELECT DISTINCT f.subject FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.predicate IN('website','name') ORDER BY f.subject")?;
let mut entities=db.prepare("SELECT DISTINCT f.subject FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.predicate IN('website','name') ORDER BY f.subject")?;
for subject in entities.query_map([], |r| r.get::<_, String>(0))? {
let subject = subject?;
let mut names=db.prepare("SELECT DISTINCT text,language,kind FROM names WHERE entity=?1 ORDER BY CASE WHEN kind='label' THEN 0 ELSE 1 END,CASE WHEN language='en' THEN 0 ELSE 1 END,language,text")?;
@ -242,7 +352,7 @@ fn project_names(db: &Connection) -> anyhow::Result<()> {
}
fn project_facts(db: &Connection, normalizer: &Normalizer) -> anyhow::Result<()> {
let mut stmt=db.prepare("SELECT f.id,f.subject,f.predicate,f.value,s.source,f.selector,r.native_id FROM facts f JOIN sources s ON s.id=f.source_id JOIN selected_sources a ON a.id=s.id JOIN records r ON r.source_id=f.source_id AND r.ordinal=f.ordinal WHERE f.predicate IN('website','popularity') ORDER BY f.subject,f.id")?;
let mut stmt=db.prepare("SELECT f.id,f.subject,f.predicate,f.value,s.source,f.selector,r.native_id FROM facts f JOIN sources s ON s.id=f.source_id JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal JOIN records r ON r.source_id=f.source_id AND r.ordinal=f.ordinal WHERE f.predicate IN('website','popularity') ORDER BY f.subject,f.id")?;
let mut rows = stmt.query([])?;
while let Some(row) = rows.next()? {
let id: String = row.get(0)?;
@ -335,17 +445,12 @@ fn project_edge(
serde_json::to_string(&property)?
],
)?;
let names: String = db.query_row(
"SELECT names_fingerprint FROM entities WHERE id=?1",
[entity],
|r| r.get(0),
)?;
let relation = if source == "wikidata" {
"asserted_official"
} else {
"directory_listing"
};
let evidence = json!({"source":source,"native_id":native,"selector":selector,"assertion":value,"names_fingerprint":names,"normalization":store::RULE_VERSION});
let evidence = json!({"source":source,"native_id":native,"selector":selector,"assertion":value,"normalization":store::RULE_VERSION});
let mut identity_property = property.clone();
// Bind reviews to the normalization result. The complete PSL identity remains
// on the property, while comment-only or unrelated rule changes do not force
@ -353,9 +458,13 @@ fn project_edge(
identity_property.domain.psl_source.clear();
identity_property.domain.psl_sha256.clear();
let fingerprint = crate::digest(&serde_json::to_vec(&(
"argand.site-edge/v2",
entity,
&identity_property,
&evidence,
source,
native,
selector,
material_website_assertion(value),
))?);
// End-dated assertions stay as historical evidence, never current destinations.
// Future/partial starts require the operator to inspect the retained qualifiers.
@ -364,3 +473,14 @@ fn project_edge(
db.execute("INSERT INTO edges VALUES(?1,?2,?3,?4,?5,?6,?7) ON CONFLICT(fingerprint) DO UPDATE SET facts=json_insert(edges.facts,'$[#]',?8)",params![fingerprint,entity,property.id,relation,serde_json::to_string(&vec![id])?,serde_json::to_string(&evidence)?,eligible,id])?;
Ok(())
}
fn material_website_assertion(value: &Value) -> Value {
let mut material = value.clone();
if let Some(object) = material.as_object_mut() {
object.remove("revision");
if let Some(statement) = object.get_mut("statement").and_then(Value::as_object_mut) {
statement.remove("references");
}
}
material
}

248
crates/argand-site-registry/src/bundle.rs Normal file
View file

@ -0,0 +1,248 @@
// By Nic Weyand!
//! Deterministic, bounded evidence bundles referenced by reviewer votes.
use crate::{policy::SubjectKind, query::Registry};
use anyhow::{Context, ensure};
use rusqlite::OptionalExtension;
use serde::Serialize;
use serde_json::{Value, json};
const MAXIMUM_BUNDLE_BYTES: usize = 16 * 1024 * 1024;
/// Exact evidence presented for one granular review subject.
#[derive(Clone, Debug, Serialize)]
pub struct EvidenceBundle {
/// `argand.site-evidence-bundle/v1`.
pub schema: String,
/// Content digest over the remaining fields.
pub id: String,
/// Granular assertion type.
pub subject_kind: SubjectKind,
/// Stable material assertion fingerprint.
pub fingerprint: String,
/// Complete current source evidence needed for the decision.
pub evidence: Value,
/// Current crawler observations; empty until an observation batch is imported.
pub observations: Vec<Value>,
}
impl EvidenceBundle {
fn new(
subject_kind: SubjectKind,
fingerprint: &str,
evidence: Value,
observations: Vec<Value>,
) -> anyhow::Result<Self> {
ensure!(
crate::model::valid_digest(fingerprint),
"invalid evidence subject fingerprint"
);
let schema = "argand.site-evidence-bundle/v1".to_owned();
let id = crate::digest(&serde_json::to_vec(&(
&schema,
subject_kind,
fingerprint,
&evidence,
&observations,
))?);
let bundle = Self {
schema,
id,
subject_kind,
fingerprint: fingerprint.into(),
evidence,
observations,
};
ensure!(
serde_json::to_vec(&bundle)?.len() <= MAXIMUM_BUNDLE_BYTES,
"evidence bundle exceeds 16 MiB"
);
Ok(bundle)
}
}
/// Builds current evidence for a projected name or website edge.
///
/// # Errors
/// Rejects missing/mismatched subjects, oversized evidence, or corrupt registry data.
pub fn build(
registry: &Registry,
subject_kind: SubjectKind,
fingerprint: &str,
) -> anyhow::Result<EvidenceBundle> {
match subject_kind {
SubjectKind::Name => name(registry, fingerprint),
SubjectKind::Edge => edge(registry, fingerprint),
SubjectKind::Equivalence => {
anyhow::bail!("equivalence evidence needs the exact proposed entity pair")
}
}
}
fn name(registry: &Registry, fingerprint: &str) -> anyhow::Result<EvidenceBundle> {
let row: Option<(String, String, String, String, String, String)> = registry
.db
.query_row(
"SELECT entity,key,text,language,kind,fact FROM names WHERE fingerprint=?1",
[fingerprint],
|row| {
Ok((
row.get(0)?,
row.get(1)?,
row.get(2)?,
row.get(3)?,
row.get(4)?,
row.get(5)?,
))
},
)
.optional()?;
let (entity, key, text, language, kind, fact) = row.context("name fingerprint is absent")?;
let conflicts = name_conflicts(registry, &key, fingerprint)?;
EvidenceBundle::new(
SubjectKind::Name,
fingerprint,
json!({"entity":entity,"key":key,"text":text,"language":language,"kind":kind,"provenance":crate::evidence::fact(&registry.db,&fact)?,"conflicts":conflicts}),
observations(registry, fingerprint)?,
)
}
fn edge(registry: &Registry, fingerprint: &str) -> anyhow::Result<EvidenceBundle> {
let row: Option<(String, String, String, String, bool)> = registry
.db
.query_row(
"SELECT e.entity,p.derived_json,e.relation,e.evidence,e.eligible FROM edges e JOIN properties p ON p.id=e.property WHERE e.fingerprint=?1",
[fingerprint],
|row| {
Ok((
row.get(0)?,
row.get(1)?,
row.get(2)?,
row.get(3)?,
row.get(4)?,
))
},
)
.optional()?;
let (entity, property, relation, evidence, eligible) =
row.context("edge fingerprint is absent")?;
let facts: String = registry.db.query_row(
"SELECT facts FROM edges WHERE fingerprint=?1",
[fingerprint],
|row| row.get(0),
)?;
let mut provenance = Vec::new();
for fact in serde_json::from_str::<Vec<String>>(&facts)? {
provenance.push(crate::evidence::fact(&registry.db, &fact)?);
}
let property_value = serde_json::from_str::<Value>(&property)?;
let domain = property_value
.pointer("/domain/registrable_domain")
.and_then(Value::as_str)
.context("edge property has no registrable domain")?;
let conflicts = edge_conflicts(registry, domain, fingerprint)?;
let drift = crate::queue::drift(registry, fingerprint)?;
EvidenceBundle::new(
SubjectKind::Edge,
fingerprint,
json!({"entity":entity,"web_property":property_value,"relation":relation,"eligible":eligible,"assertion":serde_json::from_str::<Value>(&evidence)?,"provenance":provenance,"conflicts":conflicts,"drift":drift}),
observations(registry, fingerprint)?,
)
}
fn name_conflicts(registry: &Registry, key: &str, fingerprint: &str) -> anyhow::Result<Value> {
let total: u64 = registry.db.query_row(
"SELECT count(*) FROM names WHERE key=?1 AND fingerprint<>?2",
rusqlite::params![key, fingerprint],
|row| crate::store::unsigned(row, 0),
)?;
let mut statement = registry.db.prepare(
"SELECT fingerprint,entity,text,language,kind,fact FROM names WHERE key=?1 AND fingerprint<>?2 ORDER BY fingerprint LIMIT 256",
)?;
let conflicts = statement
.query_map(rusqlite::params![key, fingerprint], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, String>(3)?,
row.get::<_, String>(4)?,
row.get::<_, String>(5)?,
))
})?
.map(|row| {
let (fingerprint, entity, text, language, kind, fact) = row?;
Ok(json!({"fingerprint":fingerprint,"entity":entity,"text":text,"language":language,"kind":kind,"provenance":crate::evidence::fact(&registry.db,&fact)?}))
})
.collect::<anyhow::Result<Vec<_>>>()?;
Ok(json!({"total":total,"truncated":total>256,"items":conflicts}))
}
fn edge_conflicts(registry: &Registry, domain: &str, fingerprint: &str) -> anyhow::Result<Value> {
let from =
"FROM edges e JOIN properties p ON p.id=e.property WHERE p.domain=?1 AND e.fingerprint<>?2";
let total: u64 = registry.db.query_row(
&format!("SELECT count(*) {from}"),
rusqlite::params![domain, fingerprint],
|row| crate::store::unsigned(row, 0),
)?;
let mut statement = registry.db.prepare(&format!(
"SELECT e.fingerprint,e.entity,p.url,e.relation,e.facts,e.evidence {from} ORDER BY e.fingerprint LIMIT 256"
))?;
let conflicts = statement
.query_map(rusqlite::params![domain, fingerprint], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, String>(3)?,
row.get::<_, String>(4)?,
row.get::<_, String>(5)?,
))
})?
.map(|row| {
let (fingerprint, entity, url, relation, facts, assertion) = row?;
Ok(json!({"fingerprint":fingerprint,"entity":entity,"url":url,"relation":relation,"facts":serde_json::from_str::<Value>(&facts)?,"assertion":serde_json::from_str::<Value>(&assertion)?}))
})
.collect::<anyhow::Result<Vec<_>>>()?;
Ok(json!({"total":total,"truncated":total>256,"items":conflicts}))
}
/// Builds current evidence for an exact proposed equivalence.
///
/// # Errors
/// Rejects a stale proposal or oversized/corrupt evidence.
pub fn equivalence(
registry: &Registry,
pair: &crate::identity::Equivalence,
) -> anyhow::Result<EvidenceBundle> {
let current = crate::identity::propose(registry, &pair.entities[0], &pair.entities[1])?;
ensure!(
current.fingerprint == pair.fingerprint,
"equivalence evidence is stale"
);
EvidenceBundle::new(
SubjectKind::Equivalence,
&pair.fingerprint,
serde_json::to_value(current)?,
observations(registry, &pair.fingerprint)?,
)
}
fn observations(registry: &Registry, fingerprint: &str) -> anyhow::Result<Vec<Value>> {
let exists: bool = registry.db.query_row(
"SELECT EXISTS(SELECT 1 FROM sqlite_schema WHERE type='table' AND name='observations')",
[],
|row| row.get(0),
)?;
if !exists {
return Ok(Vec::new());
}
let mut statement = registry.db.prepare(
"SELECT document_json FROM observations WHERE subject_fingerprint=?1 ORDER BY fingerprint",
)?;
statement
.query_map([fingerprint], |row| row.get::<_, String>(0))?
.map(|row| Ok(serde_json::from_str(&row?)?))
.collect()
}

65
crates/argand-site-registry/src/catalog.rs
View file

@ -24,6 +24,14 @@ pub struct RegistryStats {
pub source_snapshots: u64,
/// Active source selections used for projections.
pub selected_sources: u64,
/// Complete snapshots retained for audit but excluded from active projections.
pub superseded_source_snapshots: u64,
/// Incomplete snapshots; immutable generations always report zero.
pub incomplete_source_snapshots: u64,
/// Conflicting snapshots; successful immutable generations always report zero.
pub conflicting_source_snapshots: u64,
/// Active source records after delta masking.
pub active_records: u64,
/// Retained source records.
pub records: u64,
/// Retained source facts.
@ -32,10 +40,26 @@ pub struct RegistryStats {
pub reviews: u64,
/// Decisions carrying verified reviewer authentication.
pub authenticated_reviews: u64,
/// Immutable v0.4 votes.
pub votes: u64,
/// Votes carrying retained authentication.
pub authenticated_votes: u64,
/// Sticky revocation votes retained in the generation.
pub vote_revocations: u64,
/// Explicit cross-source identity proposals.
pub equivalences: u64,
/// Current approvals expiring during the next seven days.
pub approvals_expiring_within_seven_days: u64,
/// v0.4 approval votes requesting expiry during the next seven days.
pub vote_approvals_expiring_within_seven_days: u64,
/// Complete immutable observation batches.
pub observation_batches: u64,
/// Subject-bound observations.
pub observations: u64,
/// Receipt-authenticated review policy digest.
pub review_policy_sha256: String,
/// Receipt-authenticated source coverage selection digest.
pub coverage_sha256: String,
/// Receipt-level entity count.
pub entities: u64,
/// Receipt-level strict URL count.
@ -163,18 +187,45 @@ impl Registry {
params![now.to_rfc3339(), deadline.to_rfc3339()],
|row| crate::store::unsigned(row, 0),
)?;
let vote_approvals_expiring_within_seven_days = self.db.query_row(
"SELECT count(*) FROM votes WHERE decision='approve' AND expires_at>?1 AND expires_at<=?2",
params![now.to_rfc3339(), deadline.to_rfc3339()],
|row| crate::store::unsigned(row, 0),
)?;
let source_snapshots = count("sources")?;
let selected_sources = count("selected_sources")?;
Ok(RegistryStats {
registry: self.identity.clone(),
rules: self.receipt.rules.clone(),
database_bytes: page_count.saturating_mul(page_size),
source_snapshots: count("sources")?,
selected_sources: count("selected_sources")?,
source_snapshots,
selected_sources,
superseded_source_snapshots: source_snapshots.saturating_sub(selected_sources),
incomplete_source_snapshots: 0,
conflicting_source_snapshots: 0,
active_records: count("active_records")?,
records: count("records")?,
facts: count("facts")?,
reviews: count("reviews")?,
authenticated_reviews: count("review_auth")?,
votes: count("votes")?,
authenticated_votes: count("vote_auth")?,
vote_revocations: self.db.query_row(
"SELECT count(*) FROM votes WHERE decision='revoke'",
[],
|row| crate::store::unsigned(row, 0),
)?,
equivalences: count("equivalences")?,
approvals_expiring_within_seven_days,
vote_approvals_expiring_within_seven_days,
observation_batches: self.db.query_row(
"SELECT count(*) FROM observation_batches WHERE complete=1",
[],
|row| crate::store::unsigned(row, 0),
)?,
observations: count("observations")?,
review_policy_sha256: self.receipt.review_policy_sha256.clone(),
coverage_sha256: self.receipt.coverage_sha256.clone(),
entities: self.receipt.entities,
properties: self.receipt.properties,
edges: self.receipt.edges,
@ -350,7 +401,7 @@ impl Registry {
&& category_id.bytes().all(|byte| byte.is_ascii_digit()),
"invalid category ID"
);
let mut statement = self.db.prepare("SELECT f.id FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.predicate='category' AND json_extract(f.value,'$.category_id')=?1 ORDER BY f.id")?;
let mut statement = self.db.prepare("SELECT f.id FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.predicate='category' AND json_extract(f.value,'$.category_id')=?1 ORDER BY f.id")?;
let mut metadata = Vec::new();
let mut output_bytes = 0;
for id in statement.query_map([category_id], |row| row.get::<_, String>(0))? {
@ -362,8 +413,8 @@ impl Registry {
crate::query::account_output(&mut output_bytes, &fact)?;
metadata.push(fact);
}
let total_members = self.db.query_row("SELECT count(*) FROM edges e WHERE e.entity IN(SELECT f.subject FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.predicate='category_membership' AND json_extract(f.value,'$.category_id')=?1)",[category_id],|row|crate::store::unsigned(row,0))?;
let members = self.candidates_for("SELECT e.fingerprint FROM edges e WHERE e.entity IN(SELECT f.subject FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.predicate='category_membership' AND json_extract(f.value,'$.category_id')=?1) ORDER BY e.entity,e.fingerprint LIMIT ?2",params![category_id,limit])?;
let total_members = self.db.query_row("SELECT count(*) FROM edges e WHERE e.entity IN(SELECT f.subject FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.predicate='category_membership' AND json_extract(f.value,'$.category_id')=?1)",[category_id],|row|crate::store::unsigned(row,0))?;
let members = self.candidates_for("SELECT e.fingerprint FROM edges e WHERE e.entity IN(SELECT f.subject FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.predicate='category_membership' AND json_extract(f.value,'$.category_id')=?1) ORDER BY e.entity,e.fingerprint LIMIT ?2",params![category_id,limit])?;
Ok(CategoryLookup {
category_id: category_id.into(),
metadata,
@ -374,8 +425,8 @@ impl Registry {
})
}
fn normalizer(&self) -> anyhow::Result<Normalizer> {
let (source, raw): (String, String) = self.db.query_row("SELECT f.source_id,f.value FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.predicate='psl' ORDER BY f.source_id LIMIT 1",[],|row|Ok((row.get(0)?,row.get(1)?))).context("generation has no PSL")?;
pub(crate) fn normalizer(&self) -> anyhow::Result<Normalizer> {
let (source, raw): (String, String) = self.db.query_row("SELECT f.source_id,f.value FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.predicate='psl' ORDER BY f.source_id LIMIT 1",[],|row|Ok((row.get(0)?,row.get(1)?))).context("generation has no PSL")?;
let text: String = serde_json::from_str(&raw)?;
Normalizer::new(text.as_bytes(), source)
}

884
crates/argand-site-registry/src/cli.rs
View file

File diff suppressed because it is too large Load diff

576
crates/argand-site-registry/src/coverage.rs Normal file
View file

@ -0,0 +1,576 @@
// By Nic Weyand!
//! Typed source coverage selection and active-record masking.
use crate::model::{CoverageKind, SourceManifest};
use anyhow::{Context, ensure};
use rusqlite::{Connection, params};
use std::collections::{BTreeMap, BTreeSet};
#[derive(Clone, Debug)]
struct Snapshot {
id: String,
source: String,
scope: String,
retrieved_at: String,
manifest: SourceManifest,
}
#[derive(Clone, Debug, Eq, PartialEq)]
struct Selected {
id: String,
precedence: u64,
coordinate: String,
}
/// Selects one coherent source coverage graph and materializes active records.
///
/// # Errors
/// Rejects missing/cross-source supersession, coverage forks, mixed legacy and
/// typed coverage, delta gaps, duplicate native records, and SQLite failures.
pub(crate) fn project(db: &Connection) -> anyhow::Result<()> {
let snapshots = snapshots(db)?;
let selected = select(&snapshots)?;
db.execute_batch("BEGIN IMMEDIATE")?;
let result = project_inner(db, &selected);
match result {
Ok(()) => db.execute_batch("COMMIT")?,
Err(error) => {
db.execute_batch("ROLLBACK")?;
return Err(error);
}
}
Ok(())
}
/// Stable digest of the complete selected coverage graph.
pub(crate) fn projection_digest(db: &Connection) -> anyhow::Result<String> {
let mut statement =
db.prepare("SELECT id,precedence,partition FROM selected_sources ORDER BY id")?;
let rows = statement
.query_map([], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, i64>(1)?,
row.get::<_, String>(2)?,
))
})?
.collect::<Result<Vec<_>, _>>()?;
Ok(crate::digest(&serde_json::to_vec(&(
"argand.site-coverage-selection/v1",
rows,
))?))
}
fn snapshots(db: &Connection) -> anyhow::Result<Vec<Snapshot>> {
let mut statement = db.prepare(
"SELECT id,source,scope,retrieved_at,manifest FROM sources WHERE complete=1 ORDER BY id",
)?;
statement
.query_map([], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, String>(3)?,
row.get::<_, String>(4)?,
))
})?
.map(|row| {
let (id, source, scope, retrieved_at, raw) = row?;
let manifest: SourceManifest = serde_json::from_str(&raw)?;
manifest.validate()?;
ensure!(
manifest.id()? == id && manifest.source.key() == source && manifest.scope == scope,
"stored source declaration differs from its identity"
);
Ok(Snapshot {
id,
source,
scope,
retrieved_at,
manifest,
})
})
.collect()
}
/// Returns the current frontier for one typed source coverage coordinate.
///
/// Used only by explicitly configured scheduled replacement. Invalid existing
/// coverage still fails closed through the same selector as generation builds.
pub(crate) fn frontier(
db: &Connection,
source: &str,
collection: &str,
coordinate: &str,
) -> anyhow::Result<Option<String>> {
let snapshots = snapshots(db)?;
if snapshots.is_empty() {
return Ok(None);
}
let selected = select(&snapshots)?;
Ok(selected
.into_iter()
.filter_map(|item| {
let snapshot = snapshots.iter().find(|snapshot| snapshot.id == item.id)?;
let coverage = snapshot.manifest.coverage.as_ref()?;
(snapshot.source == source
&& coverage.collection == collection
&& coverage.coordinate() == coordinate)
.then_some((item.precedence, item.id))
})
.max_by(|left, right| left.0.cmp(&right.0).then_with(|| left.1.cmp(&right.1)))
.map(|(_, id)| id))
}
#[allow(clippy::too_many_lines)] // One pass validates and selects the complete coverage graph.
fn select(snapshots: &[Snapshot]) -> anyhow::Result<Vec<Selected>> {
ensure!(
!snapshots.is_empty(),
"registry has no complete source snapshots"
);
let by_id = snapshots
.iter()
.map(|snapshot| (snapshot.id.as_str(), snapshot))
.collect::<BTreeMap<_, _>>();
let mut superseded = BTreeSet::new();
for snapshot in snapshots {
let Some(coverage) = &snapshot.manifest.coverage else {
continue;
};
for replaced in &coverage.supersedes {
ensure!(replaced != &snapshot.id, "source cannot supersede itself");
let prior = by_id
.get(replaced.as_str())
.context("coverage supersedes an absent or incomplete source")?;
ensure!(
prior.source == snapshot.source,
"coverage cannot supersede another provider"
);
if let Some(prior_coverage) = &prior.manifest.coverage {
ensure!(
prior_coverage.collection == coverage.collection,
"coverage cannot supersede another collection"
);
}
superseded.insert(replaced.clone());
}
}
reject_cycles(snapshots, &by_id)?;
let mut selected = legacy_selection(snapshots, &superseded);
let typed_sources = snapshots
.iter()
.filter(|snapshot| {
snapshot.manifest.coverage.is_some() && !superseded.contains(&snapshot.id)
})
.map(|snapshot| snapshot.source.as_str())
.collect::<BTreeSet<_>>();
for source in typed_sources {
ensure!(
!selected.iter().any(|item| {
by_id
.get(item.id.as_str())
.is_some_and(|snapshot| snapshot.source == source)
}),
"legacy and typed coverage cannot remain active for one source"
);
}
let mut collections: BTreeMap<(&str, &str), Vec<&Snapshot>> = BTreeMap::new();
for snapshot in snapshots {
if let Some(coverage) = &snapshot.manifest.coverage {
collections
.entry((&snapshot.source, &coverage.collection))
.or_default()
.push(snapshot);
}
}
for ((source, collection), members) in collections {
let frontiers = members
.iter()
.copied()
.filter(|snapshot| !superseded.contains(&snapshot.id))
.collect::<Vec<_>>();
ensure!(!frontiers.is_empty(), "coverage collection has no frontier");
let mut chains = Vec::new();
for frontier in frontiers {
chains.push(delta_chain(frontier, &by_id)?);
}
let mut coordinates = BTreeSet::new();
let mut full = 0_u8;
for chain in &chains {
let root = chain.first().context("empty coverage chain")?;
let root_coverage = root
.manifest
.coverage
.as_ref()
.context("typed chain has a legacy root")?;
if root_coverage.kind == CoverageKind::Full {
full = full.saturating_add(1);
}
ensure!(
coordinates.insert(root_coverage.coordinate()),
"coverage collection has conflicting active branches"
);
}
ensure!(
full == 0 || (full == 1 && chains.len() == 1),
"full coverage cannot compose with another active branch"
);
for chain in chains {
let coordinate = chain
.first()
.and_then(|snapshot| snapshot.manifest.coverage.as_ref())
.map(|coverage| format!("{source}:{collection}:{}", coverage.coordinate()))
.context("missing coverage coordinate")?;
for (precedence, snapshot) in chain.into_iter().enumerate() {
selected.push(Selected {
id: snapshot.id.clone(),
precedence: u64::try_from(precedence)?,
coordinate: coordinate.clone(),
});
}
}
}
selected.sort_by(|left, right| left.id.cmp(&right.id));
selected.dedup_by(|left, right| {
if left.id != right.id {
return false;
}
left.precedence = left.precedence.max(right.precedence);
true
});
Ok(selected)
}
fn legacy_selection(snapshots: &[Snapshot], superseded: &BTreeSet<String>) -> Vec<Selected> {
let mut latest: BTreeMap<(&str, &str), &Snapshot> = BTreeMap::new();
for snapshot in snapshots.iter().filter(|snapshot| {
snapshot.manifest.coverage.is_none() && !superseded.contains(&snapshot.id)
}) {
let slot = latest.entry((&snapshot.source, &snapshot.scope));
slot.and_modify(|current| {
if (&snapshot.retrieved_at, &snapshot.id) > (&current.retrieved_at, &current.id) {
*current = snapshot;
}
})
.or_insert(snapshot);
}
latest
.into_values()
.map(|snapshot| Selected {
id: snapshot.id.clone(),
precedence: 0,
coordinate: format!("{}:legacy:{}", snapshot.source, snapshot.scope),
})
.collect()
}
fn delta_chain<'a>(
frontier: &'a Snapshot,
by_id: &BTreeMap<&str, &'a Snapshot>,
) -> anyhow::Result<Vec<&'a Snapshot>> {
let mut reverse = Vec::new();
let mut current = frontier;
let mut seen = BTreeSet::new();
loop {
ensure!(seen.insert(current.id.as_str()), "coverage base cycle");
reverse.push(current);
let coverage = current
.manifest
.coverage
.as_ref()
.context("delta chain reached legacy source")?;
if coverage.kind != CoverageKind::Delta {
break;
}
let base_id = coverage.base.as_deref().context("delta has no base")?;
let base = by_id
.get(base_id)
.context("delta base is absent or incomplete")?;
let base_coverage = base
.manifest
.coverage
.as_ref()
.context("delta base uses legacy coverage")?;
ensure!(
base.source == current.source
&& base_coverage.collection == coverage.collection
&& base_coverage.coordinate() == coverage.coordinate(),
"delta base has different source coverage"
);
let expected = if base_coverage.kind == CoverageKind::Delta {
base_coverage
.sequence
.context("delta base has no sequence")?
.checked_add(1)
.context("delta sequence overflow")?
} else {
1
};
ensure!(
coverage.sequence == Some(expected),
"delta sequence is not consecutive"
);
current = base;
}
reverse.reverse();
Ok(reverse)
}
fn reject_cycles(snapshots: &[Snapshot], by_id: &BTreeMap<&str, &Snapshot>) -> anyhow::Result<()> {
for snapshot in snapshots {
let mut pending = vec![snapshot.id.as_str()];
let mut seen = BTreeSet::new();
while let Some(id) = pending.pop() {
ensure!(seen.insert(id), "coverage supersession cycle");
let current = by_id.get(id).context("coverage source disappeared")?;
if let Some(coverage) = &current.manifest.coverage {
pending.extend(coverage.supersedes.iter().map(String::as_str));
}
}
}
Ok(())
}
fn project_inner(db: &Connection, selected: &[Selected]) -> anyhow::Result<()> {
{
let mut insert =
db.prepare("INSERT INTO selected_sources(id,precedence,partition) VALUES(?1,?2,?3)")?;
for item in selected {
insert.execute(params![
item.id,
i64::try_from(item.precedence)?,
item.coordinate
])?;
}
}
let mut query = db.prepare(
"SELECT s.source,r.native_id,a.precedence,r.source_id,r.ordinal \
FROM records r JOIN selected_sources a ON a.id=r.source_id \
JOIN sources s ON s.id=r.source_id \
ORDER BY s.source,r.native_id,a.precedence DESC,r.source_id,r.ordinal",
)?;
let mut rows = query.query([])?;
let mut insert = db.prepare("INSERT INTO active_records VALUES(?1,?2)")?;
let mut previous: Option<(String, String, u64)> = None;
while let Some(row) = rows.next()? {
let source: String = row.get(0)?;
let native: String = row.get(1)?;
let precedence = crate::store::unsigned(row, 2)?;
let key = (source, native);
if let Some((prior_source, prior_native, prior_precedence)) = &previous
&& (&key.0, &key.1) == (prior_source, prior_native)
{
ensure!(
precedence < *prior_precedence,
"active source partitions contain duplicate native records"
);
continue;
}
insert.execute(params![row.get::<_, String>(3)?, row.get::<_, i64>(4)?])?;
previous = Some((key.0, key.1, precedence));
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::model::{Compression, Format, Source, SourceCoverage};
use chrono::{TimeZone, Utc};
fn snapshot(
name: &str,
kind: CoverageKind,
partition: Option<&str>,
base: Option<&str>,
sequence: Option<u64>,
supersedes: Vec<String>,
) -> anyhow::Result<Snapshot> {
let id = crate::digest(name.as_bytes());
let coverage = SourceCoverage {
collection: "entities".into(),
kind,
partition: partition.map(str::to_owned),
base: base.map(str::to_owned),
sequence,
supersedes,
};
coverage.validate()?;
let manifest = SourceManifest {
schema: "argand.site-source/v2".into(),
source: Source::Wikidata,
format: Format::WikidataEntities,
compression: Compression::None,
snapshot: name.into(),
scope: name.into(),
coverage: Some(coverage),
source_url: "https://www.wikidata.org/wiki/Special:EntityData/Q355.json".into(),
license: Source::Wikidata.license().into(),
license_url: Source::Wikidata.license_url().into(),
retrieved_at: Utc
.with_ymd_and_hms(2026, 9, 13, 0, 0, 0)
.single()
.context("test timestamp")?,
sha256: crate::digest(b"input"),
bytes: 5,
};
Ok(Snapshot {
id,
source: "wikidata".into(),
scope: name.into(),
retrieved_at: manifest.retrieved_at.to_rfc3339(),
manifest,
})
}
#[test]
fn full_supersedes_partitions_and_delta_keeps_its_base() -> anyhow::Result<()> {
let left = snapshot(
"left",
CoverageKind::Partition,
Some("left"),
None,
None,
Vec::new(),
)?;
let right = snapshot(
"right",
CoverageKind::Partition,
Some("right"),
None,
None,
Vec::new(),
)?;
let full = snapshot(
"full",
CoverageKind::Full,
None,
None,
None,
vec![left.id.clone(), right.id.clone()],
)?;
let delta = snapshot(
"delta",
CoverageKind::Delta,
None,
Some(&full.id),
Some(1),
vec![full.id.clone()],
)?;
let selected = select(&[left, right, full.clone(), delta.clone()])?;
assert_eq!(selected.len(), 2);
assert!(selected.iter().any(|item| item.id == full.id));
assert!(selected.iter().any(|item| item.id == delta.id));
Ok(())
}
#[test]
fn conflicting_full_and_partition_fail_closed() -> anyhow::Result<()> {
let full = snapshot("full", CoverageKind::Full, None, None, None, Vec::new())?;
let partition = snapshot(
"partition",
CoverageKind::Partition,
Some("part"),
None,
None,
Vec::new(),
)?;
assert!(select(&[full, partition]).is_err());
Ok(())
}
#[test]
fn disjoint_partitions_compose_but_overlap_fails() -> anyhow::Result<()> {
let left = snapshot(
"left",
CoverageKind::Partition,
Some("left"),
None,
None,
Vec::new(),
)?;
let right = snapshot(
"right",
CoverageKind::Partition,
Some("right"),
None,
None,
Vec::new(),
)?;
assert_eq!(select(&[left.clone(), right])?.len(), 2);
let overlap = snapshot(
"overlap",
CoverageKind::Partition,
Some("left"),
None,
None,
Vec::new(),
)?;
assert!(select(&[left, overlap]).is_err());
Ok(())
}
#[test]
fn missing_skipped_and_forked_delta_history_fails() -> anyhow::Result<()> {
let missing_id = crate::digest(b"absent");
let missing = snapshot(
"missing",
CoverageKind::Delta,
None,
Some(&missing_id),
Some(1),
vec![missing_id.clone()],
)?;
assert!(select(&[missing]).is_err());
let full = snapshot("full", CoverageKind::Full, None, None, None, Vec::new())?;
let skipped = snapshot(
"skipped",
CoverageKind::Delta,
None,
Some(&full.id),
Some(2),
vec![full.id.clone()],
)?;
assert!(select(&[full.clone(), skipped]).is_err());
let first = snapshot(
"first",
CoverageKind::Delta,
None,
Some(&full.id),
Some(1),
vec![full.id.clone()],
)?;
let fork = snapshot(
"fork",
CoverageKind::Delta,
None,
Some(&full.id),
Some(1),
vec![full.id.clone()],
)?;
assert!(select(&[full, first, fork]).is_err());
Ok(())
}
#[test]
fn legacy_and_typed_source_cannot_mix_without_explicit_migration() -> anyhow::Result<()> {
let mut legacy = snapshot("legacy", CoverageKind::Full, None, None, None, Vec::new())?;
legacy.manifest.schema = "argand.site-source/v1".into();
legacy.manifest.coverage = None;
let typed = snapshot(
"typed",
CoverageKind::Partition,
Some("p0"),
None,
None,
Vec::new(),
)?;
assert!(select(&[legacy, typed]).is_err());
Ok(())
}
}

13
crates/argand-site-registry/src/crux.rs
View file

@ -30,6 +30,9 @@ pub struct CruxDownload {
pub maximum_bytes_billed: u64,
/// Maximum exported CSV bytes.
pub maximum_output_bytes: u64,
/// Explicit monthly partition/delta coverage declaration.
#[serde(default)]
pub coverage: Option<crate::model::SourceCoverage>,
}
/// Acquires a complete `CrUX` projection using `GOOGLE_OAUTH_ACCESS_TOKEN`.
@ -79,7 +82,12 @@ pub async fn download(cache: &Path, request: &CruxDownload) -> anyhow::Result<Ca
let sha256 = crate::file_digest(&part)?;
let input = root.join(&sha256);
let manifest = SourceManifest {
schema: "argand.site-source/v1".into(),
schema: if request.coverage.is_some() {
"argand.site-source/v2"
} else {
"argand.site-source/v1"
}
.into(),
source: Source::Crux,
format: Format::CruxCsv,
compression: Compression::None,
@ -93,6 +101,7 @@ pub async fn download(cache: &Path, request: &CruxDownload) -> anyhow::Result<Ca
request.month,
request.country.as_deref().unwrap_or("global")
),
coverage: request.coverage.clone(),
source_url: "https://developer.chrome.com/docs/crux/bigquery/".into(),
license: Source::Crux.license().into(),
license_url: Source::Crux.license_url().into(),
@ -287,6 +296,7 @@ fn job_key(request: &CruxDownload) -> anyhow::Result<String> {
&request.month,
&request.country,
request.maximum_bytes_billed,
&request.coverage,
))?))
}
@ -346,6 +356,7 @@ mod tests {
country: Some("GB".into()),
maximum_bytes_billed: 1_000_000,
maximum_output_bytes: 1_000_000,
coverage: None,
};
let sql = query(&request)?;
let original_job = job_key(&request)?;

38
crates/argand-site-registry/src/diff.rs
View file

@ -44,8 +44,8 @@ struct Table {
const TABLES: &[Table] = &[
Table {
subject: "source_selection",
scan: "SELECT s.id,json_object('source_id',s.id,'manifest',json(s.manifest)) FROM sources s JOIN selected_sources a USING(id) ORDER BY s.id",
find: "SELECT json_object('source_id',s.id,'manifest',json(s.manifest)) FROM sources s JOIN selected_sources a USING(id) WHERE s.id=?1",
scan: "SELECT s.id,json_object('source_id',s.id,'precedence',a.precedence,'partition',a.partition,'manifest',json(s.manifest)) FROM sources s JOIN selected_sources a USING(id) ORDER BY s.id",
find: "SELECT json_object('source_id',s.id,'precedence',a.precedence,'partition',a.partition,'manifest',json(s.manifest)) FROM sources s JOIN selected_sources a USING(id) WHERE s.id=?1",
},
Table {
subject: "entity",
@ -54,13 +54,13 @@ const TABLES: &[Table] = &[
},
Table {
subject: "name",
scan: "SELECT fact,json_object('entity',entity,'fact',fact,'key',key,'text',text,'language',language,'kind',kind) FROM names ORDER BY fact",
find: "SELECT json_object('entity',entity,'fact',fact,'key',key,'text',text,'language',language,'kind',kind) FROM names WHERE fact=?1",
scan: "SELECT fingerprint,json_object('fingerprint',fingerprint,'entity',entity,'fact',fact,'key',key,'text',text,'language',language,'kind',kind) FROM names ORDER BY fingerprint",
find: "SELECT json_object('fingerprint',fingerprint,'entity',entity,'fact',fact,'key',key,'text',text,'language',language,'kind',kind) FROM names WHERE fingerprint=?1",
},
Table {
subject: "fact",
scan: "SELECT f.id,json_object('id',f.id,'source_id',f.source_id,'subject',f.subject,'predicate',f.predicate,'value',json(f.value),'selector',f.selector,'confidence',f.confidence) FROM facts f JOIN selected_sources s ON s.id=f.source_id ORDER BY f.id",
find: "SELECT json_object('id',f.id,'source_id',f.source_id,'subject',f.subject,'predicate',f.predicate,'value',json(f.value),'selector',f.selector,'confidence',f.confidence) FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.id=?1",
scan: "SELECT f.id,json_object('id',f.id,'source_id',f.source_id,'subject',f.subject,'predicate',f.predicate,'value',json(f.value),'selector',f.selector,'confidence',f.confidence) FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal ORDER BY f.id",
find: "SELECT json_object('id',f.id,'source_id',f.source_id,'subject',f.subject,'predicate',f.predicate,'value',json(f.value),'selector',f.selector,'confidence',f.confidence) FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.id=?1",
},
Table {
subject: "property",
@ -92,6 +92,26 @@ const TABLES: &[Table] = &[
scan: "SELECT fingerprint,json_object('fingerprint',fingerprint,'left_entity',left_entity,'right_entity',right_entity,'left_signature',left_signature,'right_signature',right_signature) FROM equivalences ORDER BY fingerprint",
find: "SELECT json_object('fingerprint',fingerprint,'left_entity',left_entity,'right_entity',right_entity,'left_signature',left_signature,'right_signature',right_signature) FROM equivalences WHERE fingerprint=?1",
},
Table {
subject: "vote",
scan: "SELECT v.id,json_object('id',v.id,'fingerprint',v.fingerprint,'subject_kind',v.subject_kind,'decision',v.decision,'reviewer',v.reviewer,'reason',v.reason,'evidence_bundle',v.evidence_bundle,'policy',v.policy_sha256,'reviewed_at',v.reviewed_at,'expires_at',v.expires_at,'role',v.role,'locale',v.locale,'country',v.country,'supersedes',json(v.supersedes_json),'accepted_at',v.accepted_at,'authentication',json_object('signer',a.signer,'signature_sha256',a.signature_sha256,'namespace',a.namespace,'decision_sha256',a.decision_sha256,'key_sha256',a.key_sha256)) FROM votes v JOIN vote_auth a USING(sequence) ORDER BY v.id",
find: "SELECT json_object('id',v.id,'fingerprint',v.fingerprint,'subject_kind',v.subject_kind,'decision',v.decision,'reviewer',v.reviewer,'reason',v.reason,'evidence_bundle',v.evidence_bundle,'policy',v.policy_sha256,'reviewed_at',v.reviewed_at,'expires_at',v.expires_at,'role',v.role,'locale',v.locale,'country',v.country,'supersedes',json(v.supersedes_json),'accepted_at',v.accepted_at,'authentication',json_object('signer',a.signer,'signature_sha256',a.signature_sha256,'namespace',a.namespace,'decision_sha256',a.decision_sha256,'key_sha256',a.key_sha256)) FROM votes v JOIN vote_auth a USING(sequence) WHERE v.id=?1",
},
Table {
subject: "observation_batch",
scan: "SELECT id,json_object('id',id,'source',source,'retrieved_at',retrieved_at,'manifest',json(manifest_json),'records',records,'complete',json(complete)) FROM observation_batches WHERE complete=1 ORDER BY id",
find: "SELECT json_object('id',id,'source',source,'retrieved_at',retrieved_at,'manifest',json(manifest_json),'records',records,'complete',json(complete)) FROM observation_batches WHERE complete=1 AND id=?1",
},
Table {
subject: "observation",
scan: "SELECT fingerprint,json_object('fingerprint',fingerprint,'batch_id',batch_id,'subject_kind',subject_kind,'subject_fingerprint',subject_fingerprint,'document',json(document_json)) FROM observations ORDER BY fingerprint",
find: "SELECT json_object('fingerprint',fingerprint,'batch_id',batch_id,'subject_kind',subject_kind,'subject_fingerprint',subject_fingerprint,'document',json(document_json)) FROM observations WHERE fingerprint=?1",
},
Table {
subject: "review_policy",
scan: "SELECT CAST(singleton AS TEXT),json_object('id',id,'document',json(document)) FROM review_policy ORDER BY singleton",
find: "SELECT json_object('id',id,'document',json(document)) FROM review_policy WHERE singleton=CAST(?1 AS INTEGER)",
},
];
/// Streams every material change between two authenticated generations.
@ -103,13 +123,13 @@ pub fn write(old: &Registry, new: &Registry, output: &mut dyn Write) -> anyhow::
inner: output,
written: 0,
};
output.line(&json!({"schema":"argand.site-diff/v3","type":"header","old":old.identity,"new":new.identity,"attribution":crate::release::attribution(),"descriptions_included":false}))?;
output.line(&json!({"schema":"argand.site-diff/v4","type":"header","old":old.identity,"new":new.identity,"old_coverage":old.receipt.coverage_sha256,"new_coverage":new.receipt.coverage_sha256,"old_policy":old.receipt.review_policy_sha256,"new_policy":new.receipt.review_policy_sha256,"attribution":crate::release::attribution(),"descriptions_included":false}))?;
let mut changes = 0_u64;
for table in TABLES {
changes += removed_or_changed(table, &old.db, &new.db, &mut output)?;
changes += added(table, &new.db, &old.db, &mut output)?;
}
output.line(&json!({"schema":"argand.site-diff/v3","type":"summary","changes":changes}))?;
output.line(&json!({"schema":"argand.site-diff/v4","type":"summary","changes":changes}))?;
Ok(())
}
@ -193,7 +213,7 @@ fn event(
})
.transpose()
};
output.line(&json!({"schema":"argand.site-diff/v3","type":"change","subject":subject,"change":change,"key":key,"before":parse(before)?,"after":parse(after)?}))?;
output.line(&json!({"schema":"argand.site-diff/v4","type":"change","subject":subject,"change":change,"key":key,"before":parse(before)?,"after":parse(after)?}))?;
Ok(())
}

12
crates/argand-site-registry/src/download.rs
View file

@ -32,6 +32,9 @@ pub struct Download {
pub scope: String,
/// Maximum downloaded object bytes.
pub maximum_bytes: u64,
/// Explicit full/partition/delta coverage; absent preserves legacy scope semantics.
#[serde(default)]
pub coverage: Option<crate::model::SourceCoverage>,
}
/// Result points to immutable cached bytes and their manifest.
@ -137,6 +140,7 @@ pub async fn download(cache: &Path, request: &Download) -> anyhow::Result<Cached
&& manifest.source_url == request.url
&& manifest.snapshot == request.snapshot
&& manifest.scope == request.scope
&& manifest.coverage == request.coverage
&& manifest.bytes <= request.maximum_bytes,
"cached source declaration differs from request"
);
@ -179,12 +183,18 @@ pub async fn download(cache: &Path, request: &Download) -> anyhow::Result<Cached
let sha256 = crate::file_digest(&part)?;
let bytes = part.metadata()?.len();
let manifest = SourceManifest {
schema: "argand.site-source/v1".into(),
schema: if request.coverage.is_some() {
"argand.site-source/v2"
} else {
"argand.site-source/v1"
}
.into(),
source: request.source,
format: request.format,
compression: request.compression,
snapshot: request.snapshot.clone(),
scope: request.scope.clone(),
coverage: request.coverage.clone(),
source_url: request.url.clone(),
license: request.source.license().into(),
license_url: request.source.license_url().into(),

2
crates/argand-site-registry/src/evidence.rs
View file

@ -50,7 +50,7 @@ fn fields(db: &Connection, entity: &str, names: bool) -> anyhow::Result<(u64, Ve
"f.predicate IN('P17','P159','P407','P1001','P297','P218','P219','P220')"
};
let from = format!(
"FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.subject=?1 AND {predicate}"
"FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.subject=?1 AND {predicate}"
);
let total = db.query_row(&format!("SELECT count(*) {from}"), [entity], |r| {
crate::store::unsigned(r, 0)

24
crates/argand-site-registry/src/generation.rs
View file

@ -51,15 +51,27 @@ impl Registry {
&& crate::digest(&attribution) == receipt.attribution_sha256,
"registry license or attribution digest mismatch"
);
let rules_supported = crate::store::supported_rule_version(&receipt.rules)
|| (allow_legacy_rules && crate::store::legacy_rule_version(&receipt.rules));
ensure!(
receipt.schema == "argand.site-registry/v1" && rules_supported,
"unsupported registry contract"
);
let current = receipt.schema == "argand.site-registry/v2"
&& crate::store::supported_rule_version(&receipt.rules)
&& crate::model::valid_digest(&receipt.review_policy_sha256)
&& receipt.review_policy.id()? == receipt.review_policy_sha256
&& crate::model::valid_digest(&receipt.coverage_sha256)
&& receipt.decision_time_policy == "argand.site-decision-time/v1"
&& (receipt.review_policy.allow_legacy_reviews
|| crate::model::valid_digest(&receipt.reviewer_trust_sha256));
let legacy = allow_legacy_rules
&& receipt.schema == "argand.site-registry/v1"
&& crate::store::legacy_rule_version(&receipt.rules);
ensure!(current || legacy, "unsupported registry contract");
let database = path.join("registry.sqlite");
let (db, database) = open_authenticated_database(&database, &receipt.database_sha256)?;
crate::store::configure(&db)?;
if current {
ensure!(
crate::coverage::projection_digest(&db)? == receipt.coverage_sha256,
"selected coverage graph differs from receipt"
);
}
Ok(Self {
db,
_database: database,

182
crates/argand-site-registry/src/identity.rs
View file

@ -49,10 +49,8 @@ pub fn propose(registry: &Registry, left: &str, right: &str) -> anyhow::Result<E
}
Ok(Equivalence {
fingerprint: crate::digest(&serde_json::to_vec(&(
"argand.site-equivalence/v2",
&entities,
&signatures,
&names,
crate::store::RULE_VERSION,
))?),
entities,
signatures,
@ -90,6 +88,39 @@ pub fn record_authenticated(
record_inner(db, registry, pair, review, Some(authentication))
}
/// Appends an authenticated v0.4 vote for an exact equivalence proposal.
///
/// # Errors
/// Rejects stale entity evidence, invalid scope, missing source identities, or
/// authentication that does not match the signed vote.
pub fn record_vote_authenticated(
db: &Connection,
registry: &Registry,
pair: &Equivalence,
vote: &crate::vote::Vote,
authentication: &crate::vote::Authentication,
) -> anyhow::Result<String> {
let expected = propose(registry, &pair.entities[0], &pair.entities[1])?;
ensure!(
pair.fingerprint == expected.fingerprint && vote.fingerprint == expected.fingerprint,
"identity vote fingerprint differs from current proposal"
);
for entity in &expected.entities {
let mut statement = registry.db.prepare("SELECT DISTINCT f.source_id FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.subject=?1")?;
for id in statement.query_map([entity], |row| row.get::<_, String>(0))? {
crate::store::source(db, &id?)?;
}
}
crate::vote::record_equivalence_authenticated_at(
db,
registry,
&expected,
vote,
authentication,
Utc::now(),
)
}
fn record_inner(
db: &Connection,
registry: &Registry,
@ -108,7 +139,7 @@ fn record_inner(
"identity reviews cannot assert a destination role"
);
for entity in &expected.entities {
let mut statement = registry.db.prepare("SELECT DISTINCT f.source_id FROM facts f JOIN selected_sources s ON s.id=f.source_id WHERE f.subject=?1")?;
let mut statement = registry.db.prepare("SELECT DISTINCT f.source_id FROM facts f JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal WHERE f.subject=?1")?;
for id in statement.query_map([entity], |r| r.get::<_, String>(0))? {
crate::store::source(db, &id?)?;
}
@ -152,6 +183,26 @@ pub(crate) fn expand(
registry: &Registry,
initial: &str,
now: DateTime<Utc>,
) -> anyhow::Result<Option<(BTreeSet<String>, Vec<Value>)>> {
expand_with_revocations(registry, initial, now, None)
}
pub(crate) fn expand_with_revocations(
registry: &Registry,
initial: &str,
now: DateTime<Utc>,
revocations: Option<&crate::revocation::VerifiedRevocations>,
) -> anyhow::Result<Option<(BTreeSet<String>, Vec<Value>)>> {
if !registry.receipt.review_policy.allow_legacy_reviews {
return expand_policy(registry, initial, now, revocations);
}
expand_legacy(registry, initial, now)
}
fn expand_legacy(
registry: &Registry,
initial: &str,
now: DateTime<Utc>,
) -> anyhow::Result<Option<(BTreeSet<String>, Vec<Value>)>> {
let mut entities = BTreeSet::from([initial.to_owned()]);
let mut pending = vec![initial.to_owned()];
@ -211,6 +262,68 @@ pub(crate) fn expand(
Ok(Some((entities, evidence.into_values().collect())))
}
fn expand_policy(
registry: &Registry,
initial: &str,
now: DateTime<Utc>,
revocations: Option<&crate::revocation::VerifiedRevocations>,
) -> anyhow::Result<Option<(BTreeSet<String>, Vec<Value>)>> {
let mut entities = BTreeSet::from([initial.to_owned()]);
let mut pending = vec![initial.to_owned()];
let mut evidence = BTreeMap::new();
while let Some(entity) = pending.pop() {
let mut statement = registry.db.prepare(
"SELECT fingerprint,left_entity,right_entity FROM equivalences WHERE left_entity=?1 OR right_entity=?1 ORDER BY fingerprint",
)?;
let pairs = statement
.query_map([&entity], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
))
})?
.collect::<Result<Vec<_>, _>>()?;
for (fingerprint, left, right) in pairs {
if revocations.is_some_and(|feed| {
feed.blocks(crate::policy::SubjectKind::Equivalence, &fingerprint)
}) {
continue;
}
let pair = propose(registry, &left, &right)?;
ensure!(
pair.fingerprint == fingerprint,
"stored equivalence fingerprint is invalid"
);
let bundle = crate::bundle::equivalence(registry, &pair)?;
let decision = crate::vote::decision_for_bundle(
registry,
crate::policy::SubjectKind::Equivalence,
&fingerprint,
&bundle.id,
now,
)?;
if decision.status != crate::vote::DecisionStatus::Approved {
continue;
}
let pair_ids = [left, right];
evidence.insert(
fingerprint.clone(),
json!({"fingerprint":fingerprint,"entities":pair_ids,"evidence_bundle":bundle.id,"policy_decision":decision,"source":"argand_reviewer_votes","source_identifier":fingerprint,"license":"CC0-1.0","confidence":9000}),
);
for id in pair_ids {
if entities.insert(id.clone()) {
pending.push(id);
}
}
if entities.len() > 64 || evidence.len() > 256 {
return Ok(None);
}
}
}
Ok(Some((entities, evidence.into_values().collect())))
}
pub(crate) enum CandidateSearch {
Ready {
matched_entities: u64,
@ -220,32 +333,73 @@ pub(crate) enum CandidateSearch {
AmbiguousIdentity {
matched_entities: u64,
},
NoActiveNameReview {
matched_entities: u64,
},
SafetyLimitExceeded {
matched_entities: u64,
},
}
pub(crate) fn candidate_search(
pub(crate) fn candidate_search_with_revocations(
registry: &Registry,
query: &str,
now: DateTime<Utc>,
revocations: Option<&crate::revocation::VerifiedRevocations>,
) -> anyhow::Result<CandidateSearch> {
let key = crate::normalize::name_key(query)?;
let mut statement = registry
.db
.prepare("SELECT DISTINCT entity FROM names WHERE key=?1 ORDER BY entity LIMIT 65")?;
let matched = statement
.query_map([key], |r| r.get::<_, String>(0))?
let mut statement = registry.db.prepare(
"SELECT entity,fingerprint FROM names WHERE key=?1 ORDER BY entity,fingerprint LIMIT 257",
)?;
let raw = statement
.query_map([key], |row| {
Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))
})?
.collect::<Result<Vec<_>, _>>()?;
if matched.is_empty() {
if raw.is_empty() {
return Ok(CandidateSearch::NoMatch);
}
if matched.len() > 64 {
let raw_entities = raw
.iter()
.map(|(entity, _)| entity.clone())
.collect::<BTreeSet<_>>();
if raw_entities.len() > 64 || raw.len() > 256 {
return Ok(CandidateSearch::SafetyLimitExceeded {
matched_entities: u64::try_from(matched.len())?,
matched_entities: u64::try_from(raw_entities.len())?,
});
}
let Some((entities, evidence)) = expand(registry, &matched[0], now)? else {
let matched = if registry.receipt.review_policy.require_name_votes {
let mut approved = BTreeSet::new();
for (entity, fingerprint) in raw {
if revocations
.is_some_and(|feed| feed.blocks(crate::policy::SubjectKind::Name, &fingerprint))
{
continue;
}
if crate::vote::decision(
registry,
crate::policy::SubjectKind::Name,
&fingerprint,
now,
)?
.status
== crate::vote::DecisionStatus::Approved
{
approved.insert(entity);
}
}
if approved.is_empty() {
return Ok(CandidateSearch::NoActiveNameReview {
matched_entities: u64::try_from(raw_entities.len())?,
});
}
approved.into_iter().collect::<Vec<_>>()
} else {
raw_entities.into_iter().collect::<Vec<_>>()
};
let Some((entities, evidence)) =
expand_with_revocations(registry, &matched[0], now, revocations)?
else {
return Ok(CandidateSearch::SafetyLimitExceeded {
matched_entities: u64::try_from(matched.len())?,
});

7
crates/argand-site-registry/src/lib.rs
View file

@ -3,7 +3,9 @@
pub mod adapters;
pub mod build;
pub mod bundle;
pub mod catalog;
mod coverage;
pub mod crux;
pub mod diff;
pub mod download;
@ -15,15 +17,20 @@ mod json;
pub mod model;
pub mod normalize;
pub mod observation;
pub mod observer;
pub mod policy;
pub mod query;
pub mod queue;
pub mod release;
mod resolution;
pub mod review;
pub mod revocation;
mod ssh;
pub mod store;
pub use resolution::{Resolution, ResolutionCounts, ResolutionStatus};
pub mod update;
pub mod vote;
use sha2::{Digest, Sha256};
use std::{fs::File, io::Read, path::Path};

112
crates/argand-site-registry/src/model.rs
View file

@ -88,11 +88,108 @@ pub enum Compression {
Bzip2,
}
/// How one source object participates in an explicitly declared coverage set.
#[derive(Clone, Copy, Debug, Deserialize, Serialize, Eq, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum CoverageKind {
/// One complete snapshot for the declared collection.
Full,
/// One publisher-declared disjoint partition of a collection.
Partition,
/// An ordered change set applied to an authenticated base object.
Delta,
}
/// Typed source replacement and composition semantics.
#[derive(Clone, Debug, Deserialize, Serialize, Eq, PartialEq)]
#[serde(deny_unknown_fields)]
pub struct SourceCoverage {
/// Stable provider collection, such as `wikidata-entities`.
pub collection: String,
/// Coverage behavior of this object.
pub kind: CoverageKind,
/// Stable disjoint partition name; absent for a collection-wide object.
#[serde(default)]
pub partition: Option<String>,
/// Exact prior source ID for a delta.
#[serde(default)]
pub base: Option<String>,
/// Consecutive sequence within a delta chain.
#[serde(default)]
pub sequence: Option<u64>,
/// Exact older source IDs replaced by this object.
#[serde(default)]
pub supersedes: Vec<String>,
}
impl SourceCoverage {
/// Validates local coverage syntax. Cross-snapshot relationships are checked at build time.
///
/// # Errors
/// Rejects ambiguous kinds, unsafe identifiers, and malformed source references.
pub fn validate(&self) -> anyhow::Result<()> {
fn identifier(value: &str) -> bool {
!value.is_empty()
&& value.len() <= 128
&& value.bytes().all(|byte| {
byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_' | b':' | b'.')
})
}
ensure!(identifier(&self.collection), "invalid coverage collection");
ensure!(
self.partition.as_deref().is_none_or(identifier),
"invalid coverage partition"
);
ensure!(
self.supersedes.len() <= 4096 && self.supersedes.iter().all(|id| valid_digest(id)),
"invalid coverage supersession"
);
let unique = self
.supersedes
.iter()
.collect::<std::collections::BTreeSet<_>>();
ensure!(
unique.len() == self.supersedes.len(),
"duplicate coverage supersession"
);
match self.kind {
CoverageKind::Full => ensure!(
self.partition.is_none() && self.base.is_none() && self.sequence.is_none(),
"full coverage cannot declare partition or delta coordinates"
),
CoverageKind::Partition => ensure!(
self.partition.is_some() && self.base.is_none() && self.sequence.is_none(),
"partition coverage needs only a partition"
),
CoverageKind::Delta => {
ensure!(
self.base.as_deref().is_some_and(valid_digest)
&& self.sequence.is_some_and(|sequence| sequence > 0),
"delta coverage needs a base digest and positive sequence"
);
ensure!(
self.base
.as_ref()
.is_some_and(|base| self.supersedes.contains(base)),
"delta must explicitly supersede its base"
);
}
}
Ok(())
}
/// Stable partition coordinate used to detect conflicting active branches.
#[must_use]
pub fn coordinate(&self) -> &str {
self.partition.as_deref().unwrap_or("__full__")
}
}
/// An immutable, externally auditable input declaration; local paths are separate.
#[derive(Clone, Debug, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct SourceManifest {
/// Must be `argand.site-source/v1`.
/// `argand.site-source/v1` for legacy scope semantics or v2 for typed coverage.
pub schema: String,
/// Approved provider.
pub source: Source,
@ -105,6 +202,9 @@ pub struct SourceManifest {
pub snapshot: String,
/// Replacement scope, e.g. `full` or `selection:facebook`.
pub scope: String,
/// Typed replacement semantics for schema v2; absent on legacy v1 declarations.
#[serde(default)]
pub coverage: Option<SourceCoverage>,
/// Original distribution URL, not an arbitrary mirror.
pub source_url: String,
/// Exact data license identifier.
@ -126,8 +226,11 @@ impl SourceManifest {
/// Returns a descriptive error for unsupported source declarations.
pub fn validate(&self) -> anyhow::Result<()> {
ensure!(
self.schema == "argand.site-source/v1",
"unsupported source schema"
matches!(
(self.schema.as_str(), self.coverage.as_ref()),
("argand.site-source/v1", None) | ("argand.site-source/v2", Some(_))
),
"unsupported source schema or coverage declaration"
);
ensure!(
self.license == self.source.license() && self.license_url == self.source.license_url(),
@ -155,6 +258,9 @@ impl SourceManifest {
| (Source::Psl, Format::PslText)
);
ensure!(valid, "source/format mismatch");
if let Some(coverage) = &self.coverage {
coverage.validate()?;
}
crate::download::validate_source_url(self.source, &self.source_url)?;
Ok(())
}

474
crates/argand-site-registry/src/observation.rs
View file

@ -1,11 +1,22 @@
// By Nic Weyand!
//! Extension contract for later crawler evidence. No network or ownership inference.
use anyhow::{Context, ensure};
use chrono::{DateTime, Utc};
use rusqlite::{Connection, OptionalExtension, params};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::{
io::{BufRead, BufReader, Read, Seek},
path::Path,
};
const MAXIMUM_BATCH_BYTES: u64 = 1024 * 1024 * 1024;
const MAXIMUM_RECORD_BYTES: usize = 1024 * 1024;
const MAXIMUM_RECORDS: u64 = 10_000_000;
/// Observed relationship, distinct from an entity-ownership claim.
#[derive(Clone, Debug, Deserialize, Serialize)]
#[derive(Clone, Debug, Deserialize, Serialize, Eq, PartialEq)]
#[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)]
pub enum ObservationKind {
/// An actual HTTP redirect with its status code.
@ -29,6 +40,40 @@ pub enum ObservationKind {
/// Country as declared by the site.
country: String,
},
/// HTTP response status observed without an asserted relationship.
HttpStatus {
/// Valid three-digit HTTP status.
status: u16,
},
/// Bounded observer failure retained instead of pretending no fetch occurred.
FetchFailure {
/// Stable allowlisted failure class.
class: String,
},
/// Hash of the complete public address set pinned for one request.
DnsResolution {
/// SHA-256 over sorted socket addresses.
addresses_sha256: String,
},
/// Hash of the verified leaf certificate returned for an HTTPS request.
TlsCertificate {
/// SHA-256 over the leaf certificate DER bytes.
certificate_sha256: String,
},
/// Rights-reviewed domain-registration state from an external adapter.
DomainRegistration {
/// `active`, `expiry_risk`, `expired`, `redemption`, or `unknown`.
state: String,
/// Registry-reported expiry when the source supplies one.
expires_at: Option<DateTime<Utc>>,
},
/// Result from an explicitly configured, rights-reviewed malware policy.
MalwarePolicy {
/// Stable publisher policy identifier, never an inferred vendor.
policy: String,
/// `clean`, `suspicious`, `malicious`, or `unknown`.
result: String,
},
}
/// Immutable evidence coordinates for a future crawler-source adapter.
@ -60,7 +105,7 @@ pub struct Observation {
}
/// Deterministic crawler evidence prepared for a later rights-reviewed adapter.
#[derive(Clone, Debug, Serialize)]
#[derive(Clone, Debug, Deserialize, Serialize, Eq, PartialEq)]
pub struct NormalizedObservation {
/// Versioned extension contract.
pub schema: String,
@ -99,7 +144,6 @@ impl Observation {
&self,
normalizer: &crate::normalize::Normalizer,
) -> anyhow::Result<NormalizedObservation> {
use anyhow::ensure;
ensure!(
!self.source.trim().is_empty()
&& self.source.len() <= 128
@ -163,7 +207,6 @@ impl Observation {
}
fn validate_relation(relation: &ObservationKind) -> anyhow::Result<()> {
use anyhow::ensure;
match relation {
ObservationKind::Redirect { status } => ensure!(
matches!(status, 301 | 302 | 303 | 307 | 308),
@ -181,7 +224,430 @@ fn validate_relation(relation: &ObservationKind) -> anyhow::Result<()> {
country.len() == 2 && country.bytes().all(|byte| byte.is_ascii_uppercase()),
"invalid country selector scope"
),
ObservationKind::HttpStatus { status } => {
ensure!((100..=599).contains(status), "invalid observed HTTP status");
}
ObservationKind::FetchFailure { class } => ensure!(
matches!(
class.as_str(),
"dns"
| "timeout"
| "tls"
| "connection"
| "http_status"
| "content_type"
| "content_encoding"
| "size_limit"
| "policy_block"
),
"invalid observer failure class"
),
ObservationKind::DnsResolution { addresses_sha256 } => ensure!(
crate::model::valid_digest(addresses_sha256),
"invalid DNS address-set digest"
),
ObservationKind::TlsCertificate { certificate_sha256 } => ensure!(
crate::model::valid_digest(certificate_sha256),
"invalid TLS certificate digest"
),
ObservationKind::DomainRegistration { state, expires_at } => {
ensure!(
matches!(
state.as_str(),
"active" | "expiry_risk" | "expired" | "redemption" | "unknown"
),
"invalid domain-registration state"
);
ensure!(
expires_at.is_none_or(|value| value.timestamp() >= 0),
"invalid domain expiry time"
);
}
ObservationKind::MalwarePolicy { policy, result } => ensure!(
!policy.trim().is_empty()
&& policy.len() <= 256
&& policy
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_' | b'.'))
&& matches!(
result.as_str(),
"clean" | "suspicious" | "malicious" | "unknown"
),
"invalid malware-policy observation"
),
ObservationKind::Canonical | ObservationKind::JsonLdSameAs | ObservationKind::Sitemap => {}
}
Ok(())
}
/// Immutable declaration for one JSONL observation batch.
#[derive(Clone, Debug, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct BatchManifest {
/// `argand.site-observation-source/v1`.
pub schema: String,
/// Producer or capture collection.
pub source: String,
/// HTTPS documentation for the observation source.
pub source_url: String,
/// Rights declaration for the emitted observation metadata.
pub license: String,
/// HTTPS evidence for the rights declaration.
pub license_url: String,
/// Retrieval time of the batch.
pub retrieved_at: DateTime<Utc>,
/// SHA-256 over exact JSONL bytes.
pub sha256: String,
/// Exact JSONL byte length.
pub bytes: u64,
}
impl BatchManifest {
/// Validates bounded source, rights, and object identity fields.
///
/// # Errors
/// Rejects unsupported, malformed, non-HTTPS, empty, or oversized declarations.
pub fn validate(&self) -> anyhow::Result<()> {
ensure!(
self.schema == "argand.site-observation-source/v1",
"unsupported observation batch schema"
);
ensure!(
!self.source.trim().is_empty() && self.source.len() <= 128,
"observation batch source is required and bounded"
);
for value in [&self.source_url, &self.license_url] {
let url = url::Url::parse(value)?;
ensure!(
url.scheme() == "https" && url.host_str().is_some(),
"observation documentation URLs must use HTTPS"
);
}
ensure!(
!self.license.trim().is_empty()
&& self.license.len() <= 128
&& self.bytes > 0
&& self.bytes <= MAXIMUM_BATCH_BYTES
&& crate::model::valid_digest(&self.sha256),
"invalid observation rights or object identity"
);
Ok(())
}
/// Stable identity over exact manifest fields.
///
/// # Errors
/// Returns validation or serialization errors.
pub fn id(&self) -> anyhow::Result<String> {
self.validate()?;
Ok(crate::digest(&serde_json::to_vec(self)?))
}
}
/// One JSONL record binding an observation to a granular review subject.
#[derive(Clone, Debug, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct Assertion {
/// Name, edge, or equivalence fingerprint being observed.
pub subject_kind: crate::policy::SubjectKind,
/// Exact material subject fingerprint.
pub subject_fingerprint: String,
/// Source-native observation evidence.
pub observation: Observation,
}
/// Imports a complete pinned JSONL batch against one verified candidate generation.
///
/// # Errors
/// Rejects altered/oversized batches, stale subjects, rights mismatches, malformed
/// records, duplicate conflicts, and SQLite/filesystem failures.
#[allow(clippy::too_many_lines)] // Streaming validation and append share one transaction boundary.
pub fn import(
db: &Connection,
registry: &crate::query::Registry,
manifest: &BatchManifest,
path: &Path,
) -> anyhow::Result<String> {
manifest.validate()?;
let id = manifest.id()?;
let mut file = crate::generation::open_no_follow(path)?;
ensure!(
file.metadata()?.is_file() && file.metadata()?.len() == manifest.bytes,
"observation batch length/type mismatch"
);
let mut hash = Sha256::new();
let observed = std::io::copy(&mut file, &mut HashWriter(&mut hash))?;
ensure!(
observed == manifest.bytes && format!("{:x}", hash.finalize()) == manifest.sha256,
"observation batch digest mismatch"
);
let existing: Option<bool> = db
.query_row(
"SELECT complete FROM observation_batches WHERE id=?1",
[&id],
|row| row.get(0),
)
.optional()?;
if existing == Some(true) {
return Ok(id);
}
ensure!(
existing.is_none(),
"incomplete observation batch is present"
);
file.rewind()?;
let normalizer = registry.normalizer()?;
db.execute_batch("BEGIN IMMEDIATE")?;
let result = (|| {
db.execute(
"INSERT INTO observation_batches(id,source,retrieved_at,manifest_json) VALUES(?1,?2,?3,?4)",
params![id, manifest.source, manifest.retrieved_at.to_rfc3339(), serde_json::to_string(manifest)?],
)?;
let mut reader = BufReader::new(file);
let mut line = Vec::new();
let mut records = 0_u64;
loop {
line.clear();
let read = reader
.by_ref()
.take(u64::try_from(MAXIMUM_RECORD_BYTES)? + 1)
.read_until(b'\n', &mut line)?;
if read == 0 {
break;
}
ensure!(
line.len() <= MAXIMUM_RECORD_BYTES,
"observation record exceeds 1 MiB"
);
if line.iter().all(u8::is_ascii_whitespace) {
continue;
}
records = records
.checked_add(1)
.context("observation record overflow")?;
ensure!(
records <= MAXIMUM_RECORDS,
"observation record cap exceeded"
);
let assertion: Assertion = serde_json::from_value(crate::json::parse(&line)?)?;
validate_subject(registry, &assertion)?;
ensure!(
assertion.observation.source == manifest.source
&& assertion.observation.license == manifest.license
&& assertion.observation.license_url == manifest.license_url,
"observation record disagrees with batch rights/source"
);
let normalized_observation = assertion.observation.normalize(&normalizer)?;
let document = serde_json::to_string(&normalized_observation)?;
db.execute(
"INSERT INTO observations VALUES(?1,?2,?3,?4,?5)",
params![
normalized_observation.fingerprint,
id,
assertion.subject_kind.key(),
assertion.subject_fingerprint,
document
],
)
.context("duplicate observation fingerprint")?;
}
ensure!(records > 0, "observation batch is empty");
db.execute(
"UPDATE observation_batches SET records=?2,complete=1 WHERE id=?1",
params![id, i64::try_from(records)?],
)?;
Ok(records)
})();
match result {
Ok(_) => db.execute_batch("COMMIT")?,
Err(error) => {
db.execute_batch("ROLLBACK")?;
return Err(error);
}
}
Ok(id)
}
fn validate_subject(
registry: &crate::query::Registry,
assertion: &Assertion,
) -> anyhow::Result<()> {
ensure!(
crate::model::valid_digest(&assertion.subject_fingerprint),
"invalid observation subject fingerprint"
);
let (table, column) = match assertion.subject_kind {
crate::policy::SubjectKind::Name => ("names", "fingerprint"),
crate::policy::SubjectKind::Edge => ("edges", "fingerprint"),
crate::policy::SubjectKind::Equivalence => ("equivalences", "fingerprint"),
};
let exists: bool = registry.db.query_row(
&format!("SELECT EXISTS(SELECT 1 FROM {table} WHERE {column}=?1)"),
[&assertion.subject_fingerprint],
|row| row.get(0),
)?;
ensure!(
exists,
"observation subject is absent from candidate generation"
);
Ok(())
}
struct HashWriter<'a>(&'a mut Sha256);
impl std::io::Write for HashWriter<'_> {
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
self.0.update(bytes);
Ok(bytes.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
/// Bounded observation evidence attached to one review subject.
#[derive(Debug, Serialize)]
pub struct Lookup {
/// Exact subject fingerprint.
pub subject_fingerprint: String,
/// Complete matching count before the output limit.
pub total: u64,
/// True when records were omitted by the output limit.
pub truncated: bool,
/// Normalized observations with their batch manifests.
pub observations: Vec<serde_json::Value>,
}
/// Reverse observation lookup for an exact URL, hostname, or registrable domain.
#[derive(Debug, Serialize)]
pub struct ReverseLookup {
/// Original lookup target.
pub input: String,
/// Strict normalized URL or domain.
pub normalized: serde_json::Value,
/// Complete matching observation count.
pub total: u64,
/// True when results were omitted by the output limit.
pub truncated: bool,
/// Subject-bound observations with batch declarations.
pub observations: Vec<serde_json::Value>,
}
/// Returns observations for one exact review subject.
///
/// # Errors
/// Rejects malformed fingerprints/limits or corrupt registry evidence.
pub fn lookup(
registry: &crate::query::Registry,
subject_fingerprint: &str,
limit: u32,
) -> anyhow::Result<Lookup> {
ensure!(
crate::model::valid_digest(subject_fingerprint),
"invalid observation subject fingerprint"
);
ensure!(
(1..=1000).contains(&limit),
"observation limit must be 1..1000"
);
let total = registry.db.query_row(
"SELECT count(*) FROM observations WHERE subject_fingerprint=?1",
[subject_fingerprint],
|row| crate::store::unsigned(row, 0),
)?;
let mut statement = registry.db.prepare(
"SELECT o.document_json,b.manifest_json FROM observations o JOIN observation_batches b ON b.id=o.batch_id WHERE o.subject_fingerprint=?1 ORDER BY o.fingerprint LIMIT ?2",
)?;
let rows = statement
.query_map(params![subject_fingerprint, limit], |row| {
Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))
})?
.map(|row| {
let (observation, manifest) = row?;
Ok(serde_json::json!({
"observation": serde_json::from_str::<serde_json::Value>(&observation)?,
"batch": serde_json::from_str::<serde_json::Value>(&manifest)?,
}))
})
.collect::<anyhow::Result<Vec<_>>>()?;
Ok(Lookup {
subject_fingerprint: subject_fingerprint.into(),
total,
truncated: total > u64::from(limit),
observations: rows,
})
}
/// Finds observations whose source or target matches an exact web target.
///
/// # Errors
/// Rejects malformed inputs/limits or corrupt observation evidence.
pub fn reverse_lookup(
registry: &crate::query::Registry,
input: &str,
limit: u32,
) -> anyhow::Result<ReverseLookup> {
ensure!(
(1..=1000).contains(&limit),
"observation limit must be 1..1000"
);
let parser = registry.normalizer()?;
let (normalized, predicate, arguments) = if input.contains("://") {
let property = parser.url(input)?;
let arguments = vec![property.url.clone(), String::new()];
(
serde_json::to_value(property)?,
"json_extract(o.document_json,'$.from.url')=?1 OR json_extract(o.document_json,'$.to.url')=?1",
arguments,
)
} else {
let domain = parser.domain(input)?;
let arguments = vec![domain.hostname.clone(), domain.registrable_domain.clone()];
(
serde_json::to_value(domain)?,
"json_extract(o.document_json,'$.from.domain.hostname')=?1 OR json_extract(o.document_json,'$.to.domain.hostname')=?1 OR json_extract(o.document_json,'$.from.domain.registrable_domain')=?2 OR json_extract(o.document_json,'$.to.domain.registrable_domain')=?2",
arguments,
)
};
let from = format!(
"FROM observations o JOIN observation_batches b ON b.id=o.batch_id WHERE {predicate}"
);
let total = registry.db.query_row(
&format!("SELECT count(*) {from}"),
params![arguments[0], arguments[1]],
|row| crate::store::unsigned(row, 0),
)?;
let mut statement = registry.db.prepare(&format!(
"SELECT o.subject_kind,o.subject_fingerprint,o.document_json,b.manifest_json {from} ORDER BY o.subject_kind,o.subject_fingerprint,o.fingerprint LIMIT ?3"
))?;
let rows = statement
.query_map(params![arguments[0], arguments[1], limit], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, String>(3)?,
))
})?
.map(|row| {
let (subject_kind, subject_fingerprint, observation, manifest) = row?;
Ok(serde_json::json!({
"subject_kind":subject_kind,
"subject_fingerprint":subject_fingerprint,
"observation":serde_json::from_str::<serde_json::Value>(&observation)?,
"batch":serde_json::from_str::<serde_json::Value>(&manifest)?,
}))
})
.collect::<anyhow::Result<Vec<_>>>()?;
let result = ReverseLookup {
input: input.into(),
normalized,
total,
truncated: total > u64::from(limit),
observations: rows,
};
let mut output = 0;
crate::query::account_output(&mut output, &result)?;
Ok(result)
}

1243
crates/argand-site-registry/src/observer.rs Normal file
View file

File diff suppressed because it is too large Load diff

287
crates/argand-site-registry/src/policy.rs Normal file
View file

@ -0,0 +1,287 @@
// By Nic Weyand!
//! Versioned reviewer thresholds compiled by every registry consumer.
use anyhow::ensure;
use serde::{Deserialize, Serialize};
use std::collections::{BTreeMap, BTreeSet};
/// Kind of immutable assertion receiving reviewer votes.
#[derive(
Clone, Copy, Debug, Deserialize, Serialize, clap::ValueEnum, Eq, Ord, PartialEq, PartialOrd,
)]
#[serde(rename_all = "snake_case")]
pub enum SubjectKind {
/// One normalized name or alias attached to one stable source entity.
Name,
/// One source-asserted entity-to-web-property relationship.
Edge,
/// One explicit equivalence between stable source entity IDs.
Equivalence,
}
impl SubjectKind {
/// Stable database and JSON spelling.
#[must_use]
pub const fn key(self) -> &'static str {
match self {
Self::Name => "name",
Self::Edge => "edge",
Self::Equivalence => "equivalence",
}
}
}
/// Independent reviewer and reviewer-group threshold for one subject kind.
#[derive(Clone, Copy, Debug, Deserialize, Serialize, Eq, PartialEq)]
#[serde(deny_unknown_fields)]
pub struct Threshold {
/// Distinct authenticated reviewer identities required.
pub approvals: u16,
/// Distinct configured groups required; unmapped identities form their own group.
pub groups: u16,
}
impl Threshold {
fn validate(self) -> anyhow::Result<()> {
ensure!(
(1..=32).contains(&self.approvals) && (1..=self.approvals).contains(&self.groups),
"invalid review threshold"
);
Ok(())
}
}
/// Exact publisher policy authenticated by the generation receipt.
#[derive(Clone, Debug, Deserialize, Serialize, Eq, PartialEq)]
#[serde(deny_unknown_fields)]
#[allow(clippy::struct_excessive_bools)] // Independent switches are authenticated public contract fields.
pub struct ReviewPolicy {
/// `argand.site-policy/v1`.
pub schema: String,
/// Human-readable bounded policy name.
pub name: String,
/// Name-to-entity decision threshold.
pub names: Threshold,
/// Entity-to-property decision threshold.
pub edges: Threshold,
/// Entity-equivalence decision threshold.
pub equivalences: Threshold,
/// A single authenticated revocation blocks use until explicitly superseded.
pub sticky_revocations: bool,
/// Release publisher identity may not supply a counted approval.
pub publisher_reviewer_separation: bool,
/// Whether name facts need votes before `resolve`; audit lookup is unaffected.
pub require_name_votes: bool,
/// Whether v0.3 legacy reviews may be used by the compatibility policy.
pub allow_legacy_reviews: bool,
/// Maximum effective approval age from trusted writer acceptance, at most 90 days.
pub maximum_approval_days: u16,
/// Whether an edge can qualify before any observation batch exists.
pub require_edge_observation: bool,
/// Optional maximum age of the latest edge observation.
pub maximum_observation_age_days: Option<u16>,
/// Whether a domain asserted for another entity places an approved edge on probation.
pub block_source_conflicts: bool,
/// Whether dangerous observer drift places an otherwise approved edge on probation.
pub block_dangerous_drift: bool,
/// Optional reviewer-to-independent-group mapping.
#[serde(default)]
pub reviewer_groups: BTreeMap<String, String>,
/// Optional stricter thresholds for documented material risk classes.
#[serde(default)]
pub risk_thresholds: BTreeMap<String, Threshold>,
}
impl ReviewPolicy {
/// Strict default used by the CLI and public [`crate::build::build`].
#[must_use]
pub fn reference() -> Self {
let threshold = Threshold {
approvals: 2,
groups: 2,
};
Self {
schema: "argand.site-policy/v1".into(),
name: "argand-reference-v1".into(),
names: threshold,
edges: threshold,
equivalences: threshold,
sticky_revocations: true,
publisher_reviewer_separation: true,
require_name_votes: true,
allow_legacy_reviews: false,
maximum_approval_days: 90,
require_edge_observation: false,
maximum_observation_age_days: Some(30),
block_source_conflicts: true,
block_dangerous_drift: true,
reviewer_groups: BTreeMap::new(),
risk_thresholds: BTreeMap::new(),
}
}
/// Explicit compatibility policy for migration/testing; never the CLI default.
#[must_use]
pub fn legacy_compatible() -> Self {
let threshold = Threshold {
approvals: 1,
groups: 1,
};
Self {
schema: "argand.site-policy/v1".into(),
name: "legacy-v0.3-compatibility".into(),
names: threshold,
edges: threshold,
equivalences: threshold,
sticky_revocations: true,
publisher_reviewer_separation: false,
require_name_votes: false,
allow_legacy_reviews: true,
maximum_approval_days: 90,
require_edge_observation: false,
maximum_observation_age_days: None,
block_source_conflicts: false,
block_dangerous_drift: false,
reviewer_groups: BTreeMap::new(),
risk_thresholds: BTreeMap::new(),
}
}
/// Validates bounded thresholds and identity/group declarations.
///
/// # Errors
/// Rejects unsupported schemas, unsafe identifiers, and inconsistent policy.
pub fn validate(&self) -> anyhow::Result<()> {
ensure!(
self.schema == "argand.site-policy/v1",
"unsupported review policy"
);
ensure!(
!self.name.trim().is_empty() && self.name.len() <= 256,
"review policy name is required and bounded"
);
self.names.validate()?;
self.edges.validate()?;
self.equivalences.validate()?;
ensure!(
(1..=90).contains(&self.maximum_approval_days)
&& self
.maximum_observation_age_days
.is_none_or(|days| (1..=365).contains(&days)),
"invalid review or observation age policy"
);
ensure!(
self.sticky_revocations || self.allow_legacy_reviews,
"new policies must preserve sticky revocations"
);
ensure!(
self.reviewer_groups.len() <= 4096
&& self.reviewer_groups.iter().all(|(reviewer, group)| {
!reviewer.trim().is_empty()
&& reviewer.len() <= 256
&& !group.trim().is_empty()
&& group.len() <= 256
}),
"invalid reviewer-group mapping"
);
ensure!(
self.risk_thresholds.len() <= 16
&& self.risk_thresholds.iter().all(|(risk, threshold)| {
matches!(risk.as_str(), "source_conflict" | "dangerous_drift")
&& threshold.validate().is_ok()
}),
"invalid risk-threshold mapping"
);
Ok(())
}
/// Content identity authenticated by a generation receipt.
///
/// # Errors
/// Returns validation or serialization errors.
pub fn id(&self) -> anyhow::Result<String> {
self.validate()?;
Ok(crate::digest(&serde_json::to_vec(self)?))
}
/// Threshold for an assertion kind.
#[must_use]
pub const fn threshold(&self, kind: SubjectKind) -> Threshold {
match kind {
SubjectKind::Name => self.names,
SubjectKind::Edge => self.edges,
SubjectKind::Equivalence => self.equivalences,
}
}
/// Raises the subject threshold for every current material risk class.
#[must_use]
pub fn threshold_for<'a>(
&self,
kind: SubjectKind,
risks: impl Iterator<Item = &'a str>,
) -> Threshold {
risks.fold(self.threshold(kind), |current, risk| {
self.risk_thresholds
.get(risk)
.map_or(current, |extra| Threshold {
approvals: current.approvals.max(extra.approvals),
groups: current.groups.max(extra.groups),
})
})
}
/// Counts distinct policy groups for reviewer identities.
#[must_use]
pub fn group_count<'a>(&self, reviewers: impl Iterator<Item = &'a str>) -> usize {
reviewers
.map(|reviewer| {
self.reviewer_groups
.get(reviewer)
.map_or(reviewer, String::as_str)
})
.collect::<BTreeSet<_>>()
.len()
}
}
impl Default for ReviewPolicy {
fn default() -> Self {
Self::legacy_compatible()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn reference_policy_requires_two_independent_identities() -> anyhow::Result<()> {
let policy = ReviewPolicy::reference();
policy.validate()?;
let mut custom = policy.clone();
custom.risk_thresholds.insert(
"dangerous_drift".into(),
Threshold {
approvals: 3,
groups: 2,
},
);
assert_eq!(
custom.threshold_for(SubjectKind::Edge, ["dangerous_drift"].into_iter()),
Threshold {
approvals: 3,
groups: 2
}
);
custom.validate()?;
assert_eq!(policy.group_count(["one", "two"].into_iter()), 2);
let mut same_group = policy;
same_group.reviewer_groups = BTreeMap::from([
("one".into(), "organization".into()),
("two".into(), "organization".into()),
]);
assert_eq!(same_group.group_count(["one", "two"].into_iter()), 1);
Ok(())
}
}

3
crates/argand-site-registry/src/query.rs
View file

@ -63,6 +63,8 @@ pub struct Candidate {
pub provenance: Vec<Value>,
/// Latest review, including expiry and its own provenance declaration.
pub review: Option<Value>,
/// Current policy result when this candidate was selected by `resolve`.
pub policy_decision: Option<crate::vote::PolicyDecision>,
/// Whether this assertion can be considered for approval.
pub eligible: bool,
}
@ -232,6 +234,7 @@ impl Registry {
evidence: serde_json::from_str(&evidence)?,
provenance,
review,
policy_decision: None,
eligible,
};
let mut output_bytes = 0;

639
crates/argand-site-registry/src/queue.rs Normal file
View file

@ -0,0 +1,639 @@
// By Nic Weyand!
//! Deterministic review queues and route-drift classification.
use crate::{
bundle::EvidenceBundle,
observation::{NormalizedObservation, ObservationKind},
policy::SubjectKind,
query::Registry,
vote::{DecisionStatus, PolicyDecision},
};
use anyhow::{Context, ensure};
use chrono::{DateTime, Utc};
use rusqlite::params;
use serde::Serialize;
use std::collections::{BTreeMap, BTreeSet};
const MAXIMUM_QUEUE_LIMIT: u32 = 1000;
const MAXIMUM_SCANNED_SUBJECTS: u32 = 100_000;
/// Why a subject needs operator attention.
#[derive(Clone, Debug, Serialize, Eq, Ord, PartialEq, PartialOrd)]
#[serde(rename_all = "snake_case")]
pub enum QueueReason {
/// No current approval votes exist.
NewSubject,
/// The configured independent-review quorum is not met.
InsufficientQuorum,
/// A sticky revocation blocks the subject.
Revoked,
/// Current approvals expired.
Expired,
/// Votes refer to an earlier evidence bundle.
StaleEvidence,
/// Votes were made under a different review-policy epoch.
StalePolicy,
/// Current approvals conflict across scopes.
Disputed,
/// Approval quorum exists but policy has placed the subject on probation.
Probationary,
/// Approval expires within seven days.
ExpiringSoon,
/// Current sources disagree about destination scope.
SourceConflict,
/// An eligible website has not been observed.
MissingObservation,
/// Current observation evidence indicates a material route change.
MaterialDrift,
}
/// One stable, risk-ordered unit of review work.
#[derive(Debug, Serialize)]
pub struct QueueItem {
/// Higher values appear first.
pub risk: u16,
/// Granular assertion type.
pub subject_kind: SubjectKind,
/// Material assertion identity.
pub fingerprint: String,
/// Stable owning entity where applicable.
pub entity_id: Option<String>,
/// Human-readable name or URL.
pub display: String,
/// Deterministic reasons for inclusion.
pub reasons: Vec<QueueReason>,
/// Current policy compilation.
pub policy_decision: PolicyDecision,
/// Exact current evidence a new vote must sign.
pub evidence_bundle: EvidenceBundle,
/// Website observation state for edge subjects.
pub drift: Option<DriftReport>,
}
/// Bounded deterministic queue result.
#[derive(Debug, Serialize)]
pub struct ReviewQueue {
/// Verified generation pin.
pub registry: String,
/// Exact evaluation time supplied by the caller.
pub at: DateTime<Utc>,
/// Complete number of queueable subjects within the scan bound.
pub total: u64,
/// Whether queue items were omitted by the output limit.
pub truncated: bool,
/// Risk-ordered review work.
pub items: Vec<QueueItem>,
}
/// A material drift category derived only from retained observations.
#[derive(Clone, Debug, Serialize, Eq, Ord, PartialEq, PartialOrd)]
#[serde(rename_all = "snake_case")]
pub enum DriftClass {
/// No observations are attached to the subject.
Unobserved,
/// Latest observation completed without a classified change.
Healthy,
/// Latest capture recorded a transport, policy, or server failure.
Unreachable,
/// Latest redirect crosses the candidate's registrable domain.
CrossDomainRedirect,
/// Latest canonical crosses the candidate's registrable domain.
CrossDomainCanonical,
/// Redirect destinations differ from the preceding capture.
RedirectTargetChanged,
/// Canonical destinations differ from the preceding capture.
CanonicalTargetChanged,
/// Public DNS address-set hashes differ from the preceding capture.
DnsChanged,
/// Verified leaf-certificate hashes differ from the preceding capture.
TlsCertificateChanged,
/// Retrieved content hashes differ from the preceding capture.
ContentChanged,
/// A rights-reviewed domain source reports expiry risk or inactive state.
DomainExpiryIndicator,
/// An explicitly configured malware policy reports suspicious or malicious.
MalwarePolicyBlocked,
}
/// Observation comparison for one exact website assertion.
#[derive(Clone, Debug, Serialize)]
pub struct DriftReport {
/// Exact edge fingerprint.
pub fingerprint: String,
/// Current candidate URL.
pub url: String,
/// Deterministic drift classes.
pub classes: Vec<DriftClass>,
/// Latest complete observation batch identity.
pub latest_batch: Option<String>,
/// Latest batch capture time.
pub latest_at: Option<DateTime<Utc>>,
/// Immediately preceding batch identity.
pub previous_batch: Option<String>,
/// Material change should receive fresh review.
pub review_required: bool,
/// Conservative candidate for a signed emergency revocation.
pub revocation_candidate: bool,
}
#[derive(Default)]
struct Snapshot {
id: String,
at: Option<DateTime<Utc>>,
failures: BTreeSet<String>,
redirect_targets: BTreeSet<String>,
canonical_targets: BTreeSet<String>,
dns: BTreeSet<String>,
certificates: BTreeSet<String>,
content: BTreeSet<String>,
domain_expiry_risk: bool,
malware_policy_blocked: bool,
}
/// Builds a read-only review queue under the generation's authenticated policy.
///
/// # Errors
/// Rejects unsafe limits and corrupt subject, vote, or observation evidence.
pub fn review_queue(
registry: &Registry,
at: DateTime<Utc>,
limit: u32,
maximum_subjects: u32,
) -> anyhow::Result<ReviewQueue> {
ensure!(
(1..=MAXIMUM_QUEUE_LIMIT).contains(&limit),
"review queue limit must be 1..1000"
);
ensure!(
(1..=MAXIMUM_SCANNED_SUBJECTS).contains(&maximum_subjects),
"review queue scan bound must be 1..100000"
);
let mut items = Vec::new();
let mut scanned = 0_u32;
scan_names(registry, at, maximum_subjects, &mut scanned, &mut items)?;
scan_edges(registry, at, maximum_subjects, &mut scanned, &mut items)?;
scan_equivalences(registry, at, maximum_subjects, &mut scanned, &mut items)?;
items.sort_by(|left, right| {
right
.risk
.cmp(&left.risk)
.then_with(|| left.subject_kind.cmp(&right.subject_kind))
.then_with(|| left.fingerprint.cmp(&right.fingerprint))
});
let total = u64::try_from(items.len())?;
items.truncate(usize::try_from(limit)?);
let queue = ReviewQueue {
registry: registry.identity.clone(),
at,
total,
truncated: total > u64::from(limit),
items,
};
let mut output = 0;
crate::query::account_output(&mut output, &queue)?;
Ok(queue)
}
fn scan_names(
registry: &Registry,
at: DateTime<Utc>,
maximum: u32,
scanned: &mut u32,
items: &mut Vec<QueueItem>,
) -> anyhow::Result<()> {
if !registry.receipt.review_policy.require_name_votes || *scanned >= maximum {
return Ok(());
}
let remaining = maximum - *scanned;
let mut statement = registry
.db
.prepare("SELECT fingerprint,entity,text,kind FROM names ORDER BY fingerprint LIMIT ?1")?;
let rows = statement
.query_map([remaining], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, String>(3)?,
))
})?
.collect::<Result<Vec<_>, _>>()?;
*scanned = scanned.saturating_add(u32::try_from(rows.len())?);
for (fingerprint, entity, text, kind) in rows {
let bundle = crate::bundle::build(registry, SubjectKind::Name, &fingerprint)?;
let decision = crate::vote::decision_for_bundle(
registry,
SubjectKind::Name,
&fingerprint,
&bundle.id,
at,
)?;
let (mut reasons, mut risk) = policy_reasons(&decision, at);
if reasons.is_empty() {
continue;
}
if kind == "alias" {
risk = risk.saturating_add(25);
}
reasons.sort();
items.push(QueueItem {
risk,
subject_kind: SubjectKind::Name,
fingerprint,
entity_id: Some(entity),
display: text,
reasons,
policy_decision: decision,
evidence_bundle: bundle,
drift: None,
});
}
Ok(())
}
fn scan_edges(
registry: &Registry,
at: DateTime<Utc>,
maximum: u32,
scanned: &mut u32,
items: &mut Vec<QueueItem>,
) -> anyhow::Result<()> {
if *scanned >= maximum {
return Ok(());
}
let remaining = maximum - *scanned;
let mut statement = registry
.db
.prepare("SELECT fingerprint FROM edges WHERE eligible=1 ORDER BY fingerprint LIMIT ?1")?;
let rows = statement
.query_map([remaining], |row| row.get::<_, String>(0))?
.collect::<Result<Vec<_>, _>>()?;
*scanned = scanned.saturating_add(u32::try_from(rows.len())?);
for fingerprint in rows {
let candidate = registry.candidate(&fingerprint)?;
let bundle = crate::bundle::build(registry, SubjectKind::Edge, &fingerprint)?;
let decision = crate::vote::decision_for_bundle(
registry,
SubjectKind::Edge,
&fingerprint,
&bundle.id,
at,
)?;
let drift = drift(registry, &fingerprint)?;
let (mut reasons, mut risk) = policy_reasons(&decision, at);
if domain_entity_conflict(registry, &candidate, at)? {
reasons.push(QueueReason::SourceConflict);
risk = risk.max(775);
}
if drift.classes == [DriftClass::Unobserved] {
reasons.push(QueueReason::MissingObservation);
risk = risk.max(300);
} else if drift.review_required {
reasons.push(QueueReason::MaterialDrift);
risk = risk.max(if drift.revocation_candidate { 975 } else { 850 });
}
if reasons.is_empty() {
continue;
}
reasons.sort();
reasons.dedup();
items.push(QueueItem {
risk,
subject_kind: SubjectKind::Edge,
fingerprint,
entity_id: Some(candidate.entity_id),
display: candidate.url,
reasons,
policy_decision: decision,
evidence_bundle: bundle,
drift: Some(drift),
});
}
Ok(())
}
pub(crate) fn domain_entity_conflict(
registry: &Registry,
candidate: &crate::query::Candidate,
at: DateTime<Utc>,
) -> anyhow::Result<bool> {
let property: crate::normalize::WebProperty =
serde_json::from_value(candidate.web_property.clone())?;
let equivalent = crate::identity::expand(registry, &candidate.entity_id, at)?.map_or_else(
|| std::collections::BTreeSet::from([candidate.entity_id.clone()]),
|(entities, _)| entities,
);
let mut statement = registry.db.prepare(
"SELECT DISTINCT other.entity FROM edges other JOIN properties p ON p.id=other.property WHERE p.domain=?1 ORDER BY other.entity",
)?;
for entity in statement.query_map([property.domain.registrable_domain], |row| {
row.get::<_, String>(0)
})? {
if !equivalent.contains(&entity?) {
return Ok(true);
}
}
Ok(false)
}
fn scan_equivalences(
registry: &Registry,
at: DateTime<Utc>,
maximum: u32,
scanned: &mut u32,
items: &mut Vec<QueueItem>,
) -> anyhow::Result<()> {
if *scanned >= maximum {
return Ok(());
}
let remaining = maximum - *scanned;
let mut statement = registry.db.prepare(
"SELECT fingerprint,left_entity,right_entity FROM equivalences ORDER BY fingerprint LIMIT ?1",
)?;
let rows = statement
.query_map([remaining], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
))
})?
.collect::<Result<Vec<_>, _>>()?;
*scanned = scanned.saturating_add(u32::try_from(rows.len())?);
for (fingerprint, left, right) in rows {
let pair = crate::identity::propose(registry, &left, &right)?;
ensure!(
pair.fingerprint == fingerprint,
"stored equivalence identity is stale"
);
let bundle = crate::bundle::equivalence(registry, &pair)?;
let decision = crate::vote::decision_for_bundle(
registry,
SubjectKind::Equivalence,
&fingerprint,
&bundle.id,
at,
)?;
let (mut reasons, risk) = policy_reasons(&decision, at);
if reasons.is_empty() {
continue;
}
reasons.sort();
items.push(QueueItem {
risk,
subject_kind: SubjectKind::Equivalence,
fingerprint,
entity_id: None,
display: format!("{left} = {right}"),
reasons,
policy_decision: decision,
evidence_bundle: bundle,
drift: None,
});
}
Ok(())
}
fn policy_reasons(decision: &PolicyDecision, at: DateTime<Utc>) -> (Vec<QueueReason>, u16) {
let mut reasons = Vec::new();
let mut risk = 0;
match decision.status {
DecisionStatus::Approved => {
if decision
.scopes
.iter()
.any(|scope| scope.expires_at <= at + chrono::Duration::days(7))
{
reasons.push(QueueReason::ExpiringSoon);
risk = 500;
}
}
DecisionStatus::Revoked => {
reasons.push(QueueReason::Revoked);
risk = 1000;
}
DecisionStatus::Expired => {
reasons.push(QueueReason::Expired);
risk = 825;
}
DecisionStatus::StaleEvidence => {
reasons.push(QueueReason::StaleEvidence);
risk = 900;
}
DecisionStatus::StalePolicy => {
reasons.push(QueueReason::StalePolicy);
risk = 925;
}
DecisionStatus::Disputed => {
reasons.push(QueueReason::Disputed);
risk = 950;
}
DecisionStatus::Probationary => {
reasons.push(QueueReason::Probationary);
risk = 925;
}
DecisionStatus::InsufficientReview => {
reasons.push(if decision.approvals == 0 {
QueueReason::NewSubject
} else {
QueueReason::InsufficientQuorum
});
risk = 700;
}
}
(reasons, risk)
}
/// Classifies observation changes for one exact edge without changing route state.
///
/// # Errors
/// Rejects absent/non-edge fingerprints and corrupt observation evidence.
#[allow(clippy::too_many_lines)] // Drift classes share one explicit transition precedence.
pub fn drift(registry: &Registry, fingerprint: &str) -> anyhow::Result<DriftReport> {
ensure!(
crate::model::valid_digest(fingerprint),
"invalid edge fingerprint"
);
let candidate = registry.candidate(fingerprint)?;
let property: crate::normalize::WebProperty =
serde_json::from_value(candidate.web_property.clone())?;
let mut statement = registry.db.prepare(
"SELECT o.batch_id,b.retrieved_at,o.document_json FROM observations o JOIN observation_batches b ON b.id=o.batch_id WHERE o.subject_kind='edge' AND o.subject_fingerprint=?1 AND b.complete=1 ORDER BY b.retrieved_at,o.batch_id,o.fingerprint",
)?;
let mut snapshots: BTreeMap<(DateTime<Utc>, String), Snapshot> = BTreeMap::new();
let mut rows = statement.query([fingerprint])?;
while let Some(row) = rows.next()? {
let id: String = row.get(0)?;
let at = DateTime::parse_from_rfc3339(&row.get::<_, String>(1)?)?.with_timezone(&Utc);
let observation: NormalizedObservation = serde_json::from_str(&row.get::<_, String>(2)?)?;
let snapshot = snapshots.entry((at, id.clone())).or_default();
snapshot.id = id;
snapshot.at = Some(at);
snapshot.content.insert(observation.content_sha256.clone());
match observation.relation {
ObservationKind::FetchFailure { class } => {
snapshot.failures.insert(class);
}
ObservationKind::HttpStatus { status } if status >= 500 => {
snapshot.failures.insert(format!("http_{status}"));
}
ObservationKind::Redirect { .. } => {
snapshot.redirect_targets.insert(observation.to.url.clone());
}
ObservationKind::Canonical => {
snapshot
.canonical_targets
.insert(observation.to.url.clone());
}
ObservationKind::DnsResolution { addresses_sha256 } => {
snapshot.dns.insert(addresses_sha256);
}
ObservationKind::TlsCertificate { certificate_sha256 } => {
snapshot.certificates.insert(certificate_sha256);
}
ObservationKind::DomainRegistration { state, expires_at } => {
snapshot.domain_expiry_risk |=
matches!(state.as_str(), "expiry_risk" | "expired" | "redemption")
|| expires_at
.is_some_and(|expiry| expiry <= at + chrono::Duration::days(30));
}
ObservationKind::MalwarePolicy { result, .. } => {
snapshot.malware_policy_blocked |=
matches!(result.as_str(), "suspicious" | "malicious");
}
_ => {}
}
}
let mut snapshots = snapshots.into_values().collect::<Vec<_>>();
let Some(latest) = snapshots.pop() else {
return Ok(DriftReport {
fingerprint: fingerprint.into(),
url: candidate.url,
classes: vec![DriftClass::Unobserved],
latest_batch: None,
latest_at: None,
previous_batch: None,
review_required: false,
revocation_candidate: false,
});
};
let previous = snapshots.last();
let mut classes = BTreeSet::new();
if !latest.failures.is_empty() {
classes.insert(DriftClass::Unreachable);
}
if contains_cross_domain(
registry,
&property.domain.registrable_domain,
&latest.redirect_targets,
)? {
classes.insert(DriftClass::CrossDomainRedirect);
}
if contains_cross_domain(
registry,
&property.domain.registrable_domain,
&latest.canonical_targets,
)? {
classes.insert(DriftClass::CrossDomainCanonical);
}
if let Some(previous) = previous {
if previous.redirect_targets != latest.redirect_targets {
classes.insert(DriftClass::RedirectTargetChanged);
}
if previous.canonical_targets != latest.canonical_targets {
classes.insert(DriftClass::CanonicalTargetChanged);
}
if !previous.dns.is_empty() && !latest.dns.is_empty() && previous.dns != latest.dns {
classes.insert(DriftClass::DnsChanged);
}
if !previous.certificates.is_empty()
&& !latest.certificates.is_empty()
&& previous.certificates != latest.certificates
{
classes.insert(DriftClass::TlsCertificateChanged);
}
if !previous.content.is_empty()
&& !latest.content.is_empty()
&& previous.content != latest.content
{
classes.insert(DriftClass::ContentChanged);
}
}
if latest.domain_expiry_risk {
classes.insert(DriftClass::DomainExpiryIndicator);
}
if latest.malware_policy_blocked {
classes.insert(DriftClass::MalwarePolicyBlocked);
}
if classes.is_empty() {
classes.insert(DriftClass::Healthy);
}
let revocation_candidate = classes.iter().any(|class| {
matches!(
class,
DriftClass::Unreachable
| DriftClass::CrossDomainRedirect
| DriftClass::CrossDomainCanonical
| DriftClass::DomainExpiryIndicator
| DriftClass::MalwarePolicyBlocked
)
});
let review_required = classes
.iter()
.any(|class| !matches!(class, DriftClass::Healthy | DriftClass::Unobserved));
Ok(DriftReport {
fingerprint: fingerprint.into(),
url: candidate.url,
classes: classes.into_iter().collect(),
latest_batch: Some(latest.id),
latest_at: latest.at,
previous_batch: previous.map(|snapshot| snapshot.id.clone()),
review_required,
revocation_candidate,
})
}
fn contains_cross_domain(
registry: &Registry,
expected: &str,
targets: &BTreeSet<String>,
) -> anyhow::Result<bool> {
let normalizer = registry.normalizer()?;
for target in targets {
let target = normalizer
.url(target)
.context("invalid stored observation URL")?;
if target.domain.registrable_domain != expected {
return Ok(true);
}
}
Ok(false)
}
/// Returns all material emergency-revocation candidates within a bounded scan.
///
/// # Errors
/// Rejects unsafe bounds and corrupt candidate/observation evidence.
pub fn revocation_candidates(
registry: &Registry,
maximum_subjects: u32,
) -> anyhow::Result<Vec<DriftReport>> {
ensure!(
(1..=MAXIMUM_SCANNED_SUBJECTS).contains(&maximum_subjects),
"revocation scan bound must be 1..100000"
);
let mut statement = registry
.db
.prepare("SELECT fingerprint FROM edges WHERE eligible=1 ORDER BY fingerprint LIMIT ?1")?;
let fingerprints = statement
.query_map(params![maximum_subjects], |row| row.get::<_, String>(0))?
.collect::<Result<Vec<_>, _>>()?;
let mut reports = Vec::new();
for fingerprint in fingerprints {
let report = drift(registry, &fingerprint)?;
if report.revocation_candidate {
reports.push(report);
}
}
Ok(reports)
}

202
crates/argand-site-registry/src/release.rs
View file

@ -21,6 +21,7 @@ pub fn attribution() -> Value {
"crux":{"license":"CC-BY-4.0","credit":"Chrome UX Report, Google","url":"https://developer.chrome.com/docs/crux/","license_url":"https://creativecommons.org/licenses/by/4.0/"},
"curlie":{"license":"CC-BY-3.0","credit":"With content from Curlie.org - the largest human-edited directory of the web. Contribute by submitting a website or becoming an editor.","url":"https://curlie.org/","license_url":"https://creativecommons.org/licenses/by/3.0/","public_display":"Use the prescribed HTML attribution on every page using Curlie content: https://curlie.org/docs/en/license.html"},
"psl":{"license":"MPL-2.0","url":"https://publicsuffix.org/list/","license_url":"https://mozilla.org/MPL/2.0/"},
"argand_candidate_observer":{"license":"CC0-1.0","url":"https://git.argand.org/nicweyand/argand-site-registry","license_url":"https://creativecommons.org/publicdomain/zero/1.0/","scope":"locally authored observation metadata; captured page content is not redistributed"},
"changes":"Argand normalizes and combines assertions; provider endorsement is not implied."})
}
@ -35,23 +36,55 @@ pub fn export(
include_descriptions: bool,
) -> anyhow::Result<()> {
argand_atomic::create_durable_with(output, |file| {
export_inner(registry, file, include_descriptions).map_err(std::io::Error::other)
export_inner(registry, file, include_descriptions, ExportMode::Active)
.map_err(std::io::Error::other)
})?;
Ok(())
}
/// Exports retained active and superseded assertions for audit, never admission.
///
/// # Errors
/// Returns existing-output, query, serialization, or filesystem errors.
pub fn export_audit(
registry: &Registry,
output: &Path,
include_descriptions: bool,
) -> anyhow::Result<()> {
argand_atomic::create_durable_with(output, |file| {
export_inner(registry, file, include_descriptions, ExportMode::Audit)
.map_err(std::io::Error::other)
})?;
Ok(())
}
#[derive(Clone, Copy)]
enum ExportMode {
Active,
Audit,
}
fn export_inner(
registry: &Registry,
file: &mut File,
include_descriptions: bool,
mode: ExportMode,
) -> anyhow::Result<()> {
let mut writer = BufWriter::new(file);
writeln!(
writer,
"{}",
json!({"schema":"argand.site-export/v1","registry":registry.identity,"attribution":attribution(),"descriptions_included":include_descriptions})
json!({"schema":"argand.site-export/v2","registry":registry.identity,"mode":match mode { ExportMode::Active => "active", ExportMode::Audit => "audit" },"rules":registry.receipt.rules,"coverage":{"schema":"argand.site-coverage-selection/v1","sha256":registry.receipt.coverage_sha256},"selected_sources":registry.receipt.sources.iter().map(crate::model::SourceManifest::id).collect::<anyhow::Result<Vec<_>>>()?,"attribution_sha256":registry.receipt.attribution_sha256,"attribution":attribution(),"descriptions_included":include_descriptions})
)?;
let mut stmt=registry.db.prepare("SELECT f.id,f.subject,f.predicate,f.value,f.selector,f.confidence,s.manifest,r.native_id,f.source_id FROM facts f JOIN sources s ON s.id=f.source_id JOIN records r ON r.source_id=f.source_id AND r.ordinal=f.ordinal WHERE s.complete=1 ORDER BY f.id")?;
let query = match mode {
ExportMode::Active => {
"SELECT f.id,f.subject,f.predicate,f.value,f.selector,f.confidence,s.manifest,r.native_id,f.source_id,'active',NULL FROM facts f JOIN sources s ON s.id=f.source_id JOIN records r ON r.source_id=f.source_id AND r.ordinal=f.ordinal JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal LEFT JOIN rejected j ON j.fact=f.id WHERE j.fact IS NULL ORDER BY f.id"
}
ExportMode::Audit => {
"SELECT f.id,f.subject,f.predicate,f.value,f.selector,f.confidence,s.manifest,r.native_id,f.source_id,CASE WHEN j.fact IS NOT NULL THEN 'rejected' WHEN a.source_id IS NULL THEN 'superseded' ELSE 'active' END,j.reason FROM facts f JOIN sources s ON s.id=f.source_id JOIN records r ON r.source_id=f.source_id AND r.ordinal=f.ordinal LEFT JOIN active_records a ON a.source_id=f.source_id AND a.ordinal=f.ordinal LEFT JOIN rejected j ON j.fact=f.id WHERE s.complete=1 ORDER BY f.id"
}
};
let mut stmt = registry.db.prepare(query)?;
let mut rows = stmt.query([])?;
while let Some(row) = rows.next()? {
let predicate: String = row.get(2)?;
@ -65,12 +98,61 @@ fn export_inner(
{
object.remove("description");
}
let selection_state: String = row.get(9)?;
let source_manifest: Value = serde_json::from_str(&row.get::<_, String>(6)?)?;
let source_name = source_manifest["source"]
.as_str()
.context("source manifest has no source")?;
let native_id: String = row.get(7)?;
let source_id: String = row.get(8)?;
let replacement_source_ids = if matches!(mode, ExportMode::Audit)
&& selection_state == "superseded"
{
let mut replacements = registry.db.prepare(
"SELECT DISTINCT active.source_id FROM active_records active JOIN records r ON r.source_id=active.source_id AND r.ordinal=active.ordinal JOIN sources s ON s.id=active.source_id WHERE s.source=?1 AND r.native_id=?2 AND active.source_id<>?3 ORDER BY active.source_id",
)?;
replacements
.query_map(
rusqlite::params![source_name, native_id, source_id],
|item| item.get::<_, String>(0),
)?
.collect::<Result<Vec<_>, _>>()?
} else {
Vec::new()
};
writeln!(
writer,
"{}",
json!({"type":"assertion","id":row.get::<_,String>(0)?,"subject":row.get::<_,String>(1)?,"predicate":predicate,"value":value,"selector":row.get::<_,String>(4)?,"confidence":row.get::<_,u16>(5)?,"source":serde_json::from_str::<Value>(&row.get::<_,String>(6)?)?,"source_identifier":row.get::<_,String>(7)?,"source_snapshot_id":row.get::<_,String>(8)?,"description_redacted":!include_descriptions && predicate=="category"})
json!({"type":"assertion","selection_state":selection_state,"rejection_reason":row.get::<_,Option<String>>(10)?,"replacement_source_ids":replacement_source_ids,"id":row.get::<_,String>(0)?,"subject":row.get::<_,String>(1)?,"predicate":predicate,"value":value,"selector":row.get::<_,String>(4)?,"confidence":row.get::<_,u16>(5)?,"source":source_manifest,"source_identifier":native_id,"source_snapshot_id":source_id,"description_redacted":!include_descriptions && predicate=="category"})
)?;
}
drop(rows);
drop(stmt);
if matches!(mode, ExportMode::Audit) {
let mut tombstones = registry.db.prepare(
"SELECT s.source,r.native_id,r.source_id,a.precedence,s.manifest FROM active_records current JOIN records r ON r.source_id=current.source_id AND r.ordinal=current.ordinal JOIN sources s ON s.id=r.source_id JOIN selected_sources a ON a.id=r.source_id WHERE NOT EXISTS(SELECT 1 FROM facts f WHERE f.source_id=r.source_id AND f.ordinal=r.ordinal) ORDER BY s.source,r.native_id,r.source_id",
)?;
let mut rows = tombstones.query([])?;
while let Some(row) = rows.next()? {
let source: String = row.get(0)?;
let native_id: String = row.get(1)?;
let source_id: String = row.get(2)?;
let precedence: i64 = row.get(3)?;
let mut replaced = registry.db.prepare(
"SELECT f.id FROM records prior JOIN sources s ON s.id=prior.source_id JOIN selected_sources a ON a.id=prior.source_id JOIN facts f ON f.source_id=prior.source_id AND f.ordinal=prior.ordinal WHERE s.source=?1 AND prior.native_id=?2 AND a.precedence<?3 ORDER BY f.id",
)?;
let replaced = replaced
.query_map(rusqlite::params![source, native_id, precedence], |prior| {
prior.get::<_, String>(0)
})?
.collect::<Result<Vec<_>, _>>()?;
writeln!(
writer,
"{}",
json!({"type":"tombstone","selection_state":"tombstoned","source":serde_json::from_str::<Value>(&row.get::<_,String>(4)?)?,"source_identifier":native_id,"source_snapshot_id":source_id,"replaces":replaced})
)?;
}
}
writer.flush()?;
Ok(())
}
@ -84,17 +166,66 @@ pub fn sign(
key: &Path,
pin: &str,
allowed_reviewers: &Path,
) -> anyhow::Result<()> {
sign_inner(generation, key, pin, allowed_reviewers, None)
}
/// Signs a generation while enforcing publisher-reviewer separation for its identity.
///
/// # Errors
/// Returns trust, policy, key, existing signature, or signing process failures.
pub fn sign_as(
generation: &Path,
key: &Path,
pin: &str,
allowed_reviewers: &Path,
publisher_identity: &str,
) -> anyhow::Result<()> {
ensure!(
!publisher_identity.trim().is_empty() && publisher_identity.len() <= 256,
"publisher identity is required and bounded"
);
sign_inner(
generation,
key,
pin,
allowed_reviewers,
Some(publisher_identity),
)
}
fn sign_inner(
generation: &Path,
key: &Path,
pin: &str,
allowed_reviewers: &Path,
publisher_identity: Option<&str>,
) -> anyhow::Result<()> {
let receipt = crate::ssh::sealed_input(&generation.join("COMPLETE.json"), 1024 * 1024)?;
ensure!(crate::digest(&receipt.bytes) == pin, "receipt pin mismatch");
let registry = Registry::open(generation, pin)?;
verify_reviewer_trust(&registry, allowed_reviewers)?;
crate::review::verify_all(&registry.db, allowed_reviewers)?;
crate::vote::verify_all(&registry.db, allowed_reviewers)?;
if registry.receipt.review_policy.publisher_reviewer_separation {
let identity = publisher_identity
.context("strict release policy requires the publisher identity before signing")?;
crate::vote::verify_publisher_separation(&registry, identity)?;
}
crate::ssh::sign(
"argand-site-registry",
&receipt.bytes,
key,
&generation.join("COMPLETE.json.sig"),
)?;
let signature_path = generation.join("COMPLETE.json.sig");
let separation = crate::ssh::sealed_input(&signature_path, 64 * 1024).and_then(|signature| {
crate::vote::verify_publisher_key_separation(&registry, &signature.bytes)
});
if let Err(error) = separation {
let _ = fs::remove_file(signature_path);
return Err(error);
}
File::open(generation)?.sync_all()?;
Ok(())
}
@ -121,10 +252,30 @@ pub fn verify_signed(
identity,
)?;
let registry = Registry::open(generation, &pin)?;
verify_reviewer_trust(&registry, allowed_reviewers)?;
crate::review::verify_all(&registry.db, allowed_reviewers)?;
crate::vote::verify_all(&registry.db, allowed_reviewers)?;
crate::vote::verify_publisher_separation(&registry, identity)?;
crate::vote::verify_publisher_key_separation(&registry, &signature.bytes)?;
Ok(registry)
}
fn verify_reviewer_trust(registry: &Registry, allowed_reviewers: &Path) -> anyhow::Result<()> {
if registry.receipt.reviewer_trust_sha256.is_empty() {
ensure!(
registry.receipt.review_policy.allow_legacy_reviews,
"strict release is missing a reviewer trust-root digest"
);
return Ok(());
}
let trust = crate::ssh::sealed_input(allowed_reviewers, 1024 * 1024)?;
ensure!(
crate::digest(&trust.bytes) == registry.receipt.reviewer_trust_sha256,
"reviewer trust root differs from generation receipt"
);
Ok(())
}
/// Activates a verified generation using one durable pointer. Refuses rollback
/// past distributed revocations; rebuild old inputs with the current review log.
///
@ -149,7 +300,7 @@ pub fn activate(
.truncate(false)
.open(parent.join("activation.lock"))?;
lock.try_lock().context("another activation is running")?;
let revocation: u64 = registry.db.query_row(
let legacy_revocation: u64 = registry.db.query_row(
"SELECT coalesce(max(sequence),0) FROM reviews WHERE decision='revoke'",
[],
|r| crate::store::unsigned(r, 0),
@ -157,7 +308,7 @@ pub fn activate(
if current.exists() {
let previous: Value = crate::read_json(current)?;
ensure!(
revocation
legacy_revocation
>= previous["revocation_sequence"]
.as_u64()
.context("invalid current pointer")?,
@ -178,7 +329,7 @@ pub fn activate(
argand_atomic::replace_durable(
current,
&serde_json::to_vec_pretty(
&json!({"schema":"argand.site-current/v1","generation":generation.canonicalize()?,"receipt_sha256":registry.identity,"signer":identity,"revocation_sequence":revocation}),
&json!({"schema":"argand.site-current/v2","generation":generation.canonicalize()?,"receipt_sha256":registry.identity,"signer":identity,"revocation_sequence":legacy_revocation,"vote_revocations_sha256":vote_revocations_sha256(&registry.db)?}),
)?,
)?;
Ok(())
@ -209,9 +360,46 @@ fn preserve_revocations(old: &Registry, new: &Registry) -> anyhow::Result<()> {
"rollback would replace revocation history"
);
}
let old_has_votes: bool = old.db.query_row(
"SELECT EXISTS(SELECT 1 FROM sqlite_schema WHERE type='table' AND name='votes')",
[],
|row| row.get(0),
)?;
if old_has_votes {
let mut statement = old.db.prepare(
"SELECT id,document_json,accepted_at FROM votes WHERE decision='revoke' ORDER BY id",
)?;
let mut rows = statement.query([])?;
while let Some(row) = rows.next()? {
let id: String = row.get(0)?;
let expected: (Vec<u8>, String) = (row.get(1)?, row.get(2)?);
let found = new
.db
.query_row(
"SELECT document_json,accepted_at FROM votes WHERE id=?1 AND decision='revoke'",
[&id],
|current| Ok((current.get(0)?, current.get(1)?)),
)
.context("rollback would discard an authenticated vote revocation")?;
ensure!(
found == expected,
"rollback would alter an authenticated vote revocation"
);
}
}
Ok(())
}
fn vote_revocations_sha256(db: &rusqlite::Connection) -> anyhow::Result<String> {
use sha2::{Digest, Sha256};
let mut hash = Sha256::new();
let mut statement = db.prepare("SELECT id FROM votes WHERE decision='revoke' ORDER BY id")?;
for id in statement.query_map([], |row| row.get::<_, String>(0))? {
hash.update(id?.as_bytes());
}
Ok(format!("{:x}", hash.finalize()))
}
/// Streams added/removed assertion fingerprints between two pinned generations.
///
/// # Errors

205
crates/argand-site-registry/src/resolution.rs
View file

@ -19,6 +19,8 @@ pub enum ResolutionStatus {
NoNameMatch,
/// Matching source entities lack an active reviewed equivalence chain.
AmbiguousIdentity,
/// Matching source names exist, but none has a current approval quorum.
NoActiveNameReview,
/// Identity or edge expansion exceeded defensive bounds.
SafetyLimitExceeded,
/// Matching entities have no currently eligible website assertion.
@ -36,12 +38,24 @@ pub enum ResolutionStatus {
pub struct ResolutionCounts {
/// Source entities matching the normalized name or alias.
pub matched_entities: u64,
/// Matching name assertions without a current approval quorum.
pub unapproved_names: u64,
/// Assertion candidates examined after identity review.
pub considered: u64,
/// Candidates eligible for destination review.
pub eligible: u64,
/// Eligible candidates without any review entry.
pub missing_review: u64,
/// Candidates whose votes do not satisfy the configured quorum.
pub insufficient_quorum: u64,
/// Candidates whose votes reference superseded evidence.
pub stale_evidence: u64,
/// Candidates whose votes were made under another policy epoch.
pub stale_policy: u64,
/// Candidates with conflicting current approval scopes.
pub disputed: u64,
/// Candidates held by observation or source-risk policy.
pub probationary: u64,
/// Candidates whose latest decision is a revocation.
pub revoked: u64,
/// Approvals whose validity interval has ended.
@ -97,10 +111,60 @@ impl Registry {
locale: Option<&str>,
country: Option<&str>,
now: DateTime<Utc>,
) -> anyhow::Result<Resolution> {
self.resolve_explained_inner(query, locale, country, now, None)
}
/// Resolves with a verified emergency revocation overlay before a full
/// replacement generation has reached this consumer.
///
/// # Errors
/// Returns incompatible feed, malformed request/data, or database failures.
pub fn resolve_explained_with_revocations(
&self,
query: &str,
locale: Option<&str>,
country: Option<&str>,
now: DateTime<Utc>,
revocations: &crate::revocation::VerifiedRevocations,
) -> anyhow::Result<Resolution> {
revocations.ensure_applicable(self, now)?;
self.resolve_explained_inner(query, locale, country, now, Some(revocations))
}
/// Returns only the destination after applying a verified emergency feed.
///
/// # Errors
/// Returns incompatible feed, malformed request/data, or database failures.
pub fn resolve_with_revocations(
&self,
query: &str,
locale: Option<&str>,
country: Option<&str>,
now: DateTime<Utc>,
revocations: &crate::revocation::VerifiedRevocations,
) -> anyhow::Result<Option<Candidate>> {
Ok(self
.resolve_explained_with_revocations(query, locale, country, now, revocations)?
.destination)
}
fn resolve_explained_inner(
&self,
query: &str,
locale: Option<&str>,
country: Option<&str>,
now: DateTime<Utc>,
revocations: Option<&crate::revocation::VerifiedRevocations>,
) -> anyhow::Result<Resolution> {
let key = name_key(query)?;
let mut counts = ResolutionCounts::default();
let candidates = match crate::identity::candidate_search(self, query, now)? {
let candidates = match crate::identity::candidate_search_with_revocations(
self,
query,
now,
revocations,
)? {
crate::identity::CandidateSearch::Ready {
matched_entities,
candidates,
@ -120,6 +184,16 @@ impl Registry {
counts,
));
}
crate::identity::CandidateSearch::NoActiveNameReview { matched_entities } => {
counts.matched_entities = matched_entities;
counts.unapproved_names = matched_entities;
return Ok(result(
key,
ResolutionStatus::NoActiveNameReview,
None,
counts,
));
}
crate::identity::CandidateSearch::SafetyLimitExceeded { matched_entities } => {
counts.matched_entities = matched_entities;
return Ok(result(
@ -130,12 +204,24 @@ impl Registry {
));
}
};
let (status, destination) = choose(candidates, locale, country, now, &mut counts)?;
let (status, destination) = if self.receipt.review_policy.allow_legacy_reviews {
choose_legacy(candidates, locale, country, now, &mut counts)?
} else {
choose_policy(
self,
candidates,
locale,
country,
now,
revocations,
&mut counts,
)?
};
Ok(result(key, status, destination, counts))
}
}
fn choose(
fn choose_legacy(
candidates: Vec<Candidate>,
locale: Option<&str>,
country: Option<&str>,
@ -222,6 +308,119 @@ fn choose(
})
}
#[allow(clippy::too_many_lines)] // Abstention counts and scope selection are one ordered decision pass.
fn choose_policy(
registry: &Registry,
candidates: Vec<Candidate>,
locale: Option<&str>,
country: Option<&str>,
now: DateTime<Utc>,
revocations: Option<&crate::revocation::VerifiedRevocations>,
counts: &mut ResolutionCounts,
) -> anyhow::Result<(ResolutionStatus, Option<Candidate>)> {
let mut best = None;
let mut score = 0;
let mut ambiguous = false;
for mut candidate in candidates {
counts.considered += 1;
if !candidate.eligible {
continue;
}
counts.eligible += 1;
if revocations.is_some_and(|feed| {
feed.blocks(crate::policy::SubjectKind::Edge, &candidate.fingerprint)
}) {
counts.revoked += 1;
continue;
}
let decision = crate::vote::decision(
registry,
crate::policy::SubjectKind::Edge,
&candidate.fingerprint,
now,
)?;
match decision.status {
crate::vote::DecisionStatus::Revoked => {
counts.revoked += 1;
continue;
}
crate::vote::DecisionStatus::Expired => {
counts.expired += 1;
continue;
}
crate::vote::DecisionStatus::StaleEvidence => {
counts.stale_evidence += 1;
continue;
}
crate::vote::DecisionStatus::StalePolicy => {
counts.stale_policy += 1;
continue;
}
crate::vote::DecisionStatus::Disputed => {
counts.disputed += 1;
continue;
}
crate::vote::DecisionStatus::Probationary => {
counts.probationary += 1;
continue;
}
crate::vote::DecisionStatus::InsufficientReview => {
counts.insufficient_quorum += 1;
continue;
}
crate::vote::DecisionStatus::Approved => {}
}
let mut selected_score = 0;
for scope in &decision.scopes {
let matches_country = !scope.country.is_empty()
&& country.is_some_and(|value| value.eq_ignore_ascii_case(&scope.country));
let matches_locale = !scope.locale.is_empty()
&& locale.is_some_and(|value| value.eq_ignore_ascii_case(&scope.locale));
let current = if scope.role == "regional" {
if (!scope.country.is_empty() && !matches_country)
|| (!scope.locale.is_empty() && !matches_locale)
{
continue;
}
2 + u8::from(matches_country) + u8::from(matches_locale)
} else if scope.role == "primary" {
1
} else {
continue;
};
selected_score = selected_score.max(current);
}
if selected_score == 0 {
counts.region_mismatch += 1;
continue;
}
counts.active_approvals += 1;
candidate.policy_decision = Some(decision);
if selected_score > score {
best = Some(candidate);
score = selected_score;
ambiguous = false;
} else if selected_score == score
&& best
.as_ref()
.is_some_and(|prior: &Candidate| prior.url != candidate.url)
{
ambiguous = true;
}
}
Ok(if ambiguous {
(ResolutionStatus::AmbiguousDestination, None)
} else if let Some(candidate) = best {
(ResolutionStatus::Resolved, Some(candidate))
} else if counts.eligible == 0 {
(ResolutionStatus::NoEligibleDestination, None)
} else if counts.region_mismatch > 0 {
(ResolutionStatus::RegionMismatch, None)
} else {
(ResolutionStatus::NoActiveReview, None)
})
}
fn result(
query: String,
status: ResolutionStatus,

459
crates/argand-site-registry/src/revocation.rs Normal file
View file

@ -0,0 +1,459 @@
// By Nic Weyand!
//! Small publisher-signed revocation overlays for pinned offline consumers.
use crate::{policy::SubjectKind, query::Registry, vote::DecisionStatus};
use anyhow::{Context, ensure};
use chrono::{DateTime, Utc};
use rusqlite::OptionalExtension;
use serde::{Deserialize, Serialize};
use std::{collections::BTreeMap, fs, path::Path};
/// SSH signature namespace for exact revocation-feed JSON bytes.
pub const SIGNATURE_NAMESPACE: &str = "argand-site-registry-revocations";
const MAXIMUM_FEED_LIFETIME_DAYS: i64 = 7;
/// One granular subject with retained revocation identities across policy epochs.
#[derive(Clone, Debug, Deserialize, Serialize, Eq, PartialEq)]
#[serde(deny_unknown_fields)]
pub struct RevocationEntry {
/// Name, website edge, or explicit entity equivalence.
pub subject_kind: SubjectKind,
/// Stable material assertion fingerprint blocked by this entry.
pub fingerprint: String,
/// Every accepted revocation vote ID across retained policy epochs, sorted.
pub revocations: Vec<String>,
/// Whether this subject remains blocked at the feed's effective time.
pub active: bool,
/// Fresh approval vote IDs that explicitly supersede every revocation.
pub superseding_votes: Vec<String>,
}
/// Complete cumulative emergency revocation state from one registry generation.
#[derive(Clone, Debug, Deserialize, Serialize, Eq, PartialEq)]
#[serde(deny_unknown_fields)]
pub struct RevocationFeed {
/// `argand.site-revocations/v1`.
pub schema: String,
/// Receipt digest of the generation that compiled this feed.
pub registry: String,
/// Registry derivation rules required by cached consumers.
pub rules: String,
/// Exact review-policy epoch under which entries were compiled.
pub policy_sha256: String,
/// Exact reviewer trust-root digest bound into the source generation.
pub reviewer_trust_sha256: String,
/// Explicit time used to evaluate acceptance and supersession.
pub effective_at: DateTime<Utc>,
/// Artifact refresh deadline; revocation votes themselves never expire.
pub expires_at: DateTime<Utc>,
/// Cumulative revocation subjects across policy epochs, including superseded history.
pub entries: Vec<RevocationEntry>,
}
impl RevocationFeed {
fn validate(&self) -> anyhow::Result<()> {
ensure!(
self.schema == "argand.site-revocations/v1",
"unsupported revocation feed"
);
ensure!(
crate::model::valid_digest(&self.registry)
&& crate::model::valid_digest(&self.policy_sha256)
&& crate::model::valid_digest(&self.reviewer_trust_sha256),
"revocation feed needs registry, policy, and trust-root digests"
);
ensure!(
self.rules == crate::store::RULE_VERSION,
"revocation feed rules are unsupported"
);
ensure!(
self.expires_at > self.effective_at
&& self.expires_at - self.effective_at
<= chrono::Duration::days(MAXIMUM_FEED_LIFETIME_DAYS),
"revocation feed lifetime must be at most seven days"
);
ensure!(
self.entries.len() <= 100_000,
"revocation feed exceeds 100000 subjects"
);
let mut previous = None;
let mut vote_ids = std::collections::BTreeSet::new();
for entry in &self.entries {
ensure!(
crate::model::valid_digest(&entry.fingerprint)
&& !entry.revocations.is_empty()
&& entry.revocations.len() <= 256
&& entry.superseding_votes.len() <= 256
&& entry
.revocations
.iter()
.chain(&entry.superseding_votes)
.all(|id| crate::model::valid_digest(id)),
"invalid revocation entry"
);
ensure!(
entry.revocations.windows(2).all(|pair| pair[0] < pair[1])
&& entry
.superseding_votes
.windows(2)
.all(|pair| pair[0] < pair[1]),
"revocation vote IDs must be sorted and unique"
);
ensure!(
entry.active || !entry.superseding_votes.is_empty(),
"inactive revocation needs explicit superseding votes"
);
let coordinate = (entry.subject_kind, entry.fingerprint.as_str());
ensure!(
previous.is_none_or(|prior| prior < coordinate),
"revocation entries must be sorted and unique"
);
previous = Some(coordinate);
for id in &entry.revocations {
ensure!(
vote_ids.insert(id),
"revocation vote ID reused across subjects"
);
}
}
Ok(())
}
}
/// A feed whose exact bytes have been authenticated by a trusted publisher.
#[derive(Clone, Debug)]
pub struct VerifiedRevocations {
feed: RevocationFeed,
/// SHA-256 of the exact signed feed bytes.
pub sha256: String,
/// Allowed-signers identity that authenticated the feed.
pub publisher: String,
}
impl VerifiedRevocations {
/// Returns the authenticated feed declaration.
#[must_use]
pub const fn feed(&self) -> &RevocationFeed {
&self.feed
}
/// Whether an exact material subject is currently blocked.
#[must_use]
pub fn blocks(&self, subject_kind: SubjectKind, fingerprint: &str) -> bool {
self.feed
.entries
.binary_search_by(|entry| {
(entry.subject_kind, entry.fingerprint.as_str()).cmp(&(subject_kind, fingerprint))
})
.is_ok_and(|index| self.feed.entries[index].active)
}
pub(crate) fn ensure_compatible(&self, registry: &Registry) -> anyhow::Result<()> {
ensure!(
self.feed.rules == registry.receipt.rules
&& self.feed.policy_sha256 == registry.receipt.review_policy_sha256
&& self.feed.reviewer_trust_sha256 == registry.receipt.reviewer_trust_sha256,
"revocation feed is incompatible with this cached registry"
);
Ok(())
}
pub(crate) fn ensure_applicable(
&self,
registry: &Registry,
now: DateTime<Utc>,
) -> anyhow::Result<()> {
self.ensure_compatible(registry)?;
ensure!(
self.feed.effective_at <= now + chrono::Duration::minutes(5),
"revocation feed effective time is in the future"
);
ensure!(now < self.feed.expires_at, "revocation feed has expired");
Ok(())
}
}
/// Creates a deterministic cumulative feed at an explicit evaluation time.
///
/// # Errors
/// Rejects malformed retained votes, corrupt subjects, or existing output paths.
pub fn export(
registry: &Registry,
output: &Path,
effective_at: DateTime<Utc>,
) -> anyhow::Result<()> {
let feed = compile(registry, effective_at)?;
argand_atomic::create_durable(output, &serde_json::to_vec_pretty(&feed)?)?;
Ok(())
}
/// Signs an exact feed after proving it matches the pinned source generation.
///
/// # Errors
/// Rejects altered feeds, unverified reviewer state, publisher/reviewer conflicts,
/// existing output paths, and signing failures.
pub fn sign(
registry: &Registry,
input: &Path,
output: &Path,
key: &Path,
allowed_reviewers: &Path,
publisher: &str,
) -> anyhow::Result<()> {
ensure!(
!publisher.trim().is_empty() && publisher.len() <= 256,
"publisher identity is required and bounded"
);
let bytes = crate::ssh::sealed_input(input, 16 * 1024 * 1024)?;
let feed: RevocationFeed = serde_json::from_value(crate::json::parse(&bytes.bytes)?)?;
feed.validate()?;
ensure!(
feed == compile(registry, feed.effective_at)?,
"revocation feed differs from its source generation"
);
verify_reviewer_trust(registry, allowed_reviewers)?;
crate::review::verify_all(&registry.db, allowed_reviewers)?;
crate::vote::verify_all(&registry.db, allowed_reviewers)?;
crate::vote::verify_publisher_separation(registry, publisher)?;
crate::ssh::sign(SIGNATURE_NAMESPACE, &bytes.bytes, key, output)?;
let signature = crate::ssh::sealed_input(output, 64 * 1024)?;
if let Err(error) = crate::vote::verify_publisher_key_separation(registry, &signature.bytes) {
let _ = fs::remove_file(output);
return Err(error);
}
Ok(())
}
/// Verifies a publisher-signed feed for application to a compatible cached registry.
///
/// # Errors
/// Rejects malformed bytes, untrusted signatures, incompatible registries, future
/// effective times, publisher/reviewer conflicts, or revocation rollback.
pub fn verify(
registry: &Registry,
input: &Path,
signature: &Path,
allowed_publishers: &Path,
publisher: &str,
now: DateTime<Utc>,
previous: Option<&VerifiedRevocations>,
) -> anyhow::Result<VerifiedRevocations> {
ensure!(
!publisher.trim().is_empty() && publisher.len() <= 256,
"publisher identity is required and bounded"
);
let bytes = crate::ssh::sealed_input(input, 16 * 1024 * 1024)?;
let signature = crate::ssh::sealed_input(signature, 64 * 1024)?;
let publishers = crate::ssh::sealed_input(allowed_publishers, 1024 * 1024)?;
crate::ssh::verify(
SIGNATURE_NAMESPACE,
&bytes.bytes,
&signature.bytes,
&publishers.bytes,
publisher,
)?;
let feed: RevocationFeed = serde_json::from_value(crate::json::parse(&bytes.bytes)?)?;
feed.validate()?;
ensure!(
feed.effective_at <= now + chrono::Duration::minutes(5),
"revocation feed effective time is in the future"
);
ensure!(now < feed.expires_at, "revocation feed has expired");
crate::vote::verify_publisher_separation(registry, publisher)?;
crate::vote::verify_publisher_key_separation(registry, &signature.bytes)?;
let verified = VerifiedRevocations {
feed,
sha256: crate::digest(&bytes.bytes),
publisher: publisher.into(),
};
verified.ensure_applicable(registry, now)?;
if verified.feed.registry == registry.identity {
ensure!(
verified.feed == compile(registry, verified.feed.effective_at)?,
"revocation feed differs from its exact source generation"
);
} else {
ensure!(
verified
.feed
.entries
.iter()
.all(|entry| entry.active && entry.superseding_votes.is_empty()),
"cross-generation revocation feeds may only add blocks"
);
}
if let Some(previous) = previous {
preserve(previous, &verified)?;
}
Ok(verified)
}
fn preserve(previous: &VerifiedRevocations, current: &VerifiedRevocations) -> anyhow::Result<()> {
ensure!(
current.feed.effective_at >= previous.feed.effective_at
&& current.feed.rules == previous.feed.rules
&& current.feed.policy_sha256 == previous.feed.policy_sha256
&& current.feed.reviewer_trust_sha256 == previous.feed.reviewer_trust_sha256,
"revocation feed would roll back its compatibility epoch"
);
let current_entries = current
.feed
.entries
.iter()
.map(|entry| ((entry.subject_kind, entry.fingerprint.as_str()), entry))
.collect::<BTreeMap<_, _>>();
for old in &previous.feed.entries {
let new = current_entries
.get(&(old.subject_kind, old.fingerprint.as_str()))
.context("revocation feed would discard a subject")?;
ensure!(
old.revocations
.iter()
.all(|id| new.revocations.binary_search(id).is_ok()),
"revocation feed would discard a vote"
);
ensure!(
!old.active || new.active || !new.superseding_votes.is_empty(),
"active revocation disappeared without explicit supersession"
);
}
Ok(())
}
fn compile(registry: &Registry, effective_at: DateTime<Utc>) -> anyhow::Result<RevocationFeed> {
ensure!(
!registry.receipt.review_policy.allow_legacy_reviews,
"v0.4 revocation feeds require authenticated vote policy"
);
let mut subjects: BTreeMap<(SubjectKind, String), Vec<String>> = BTreeMap::new();
let mut statement = registry.db.prepare(
"SELECT subject_kind,fingerprint,id,accepted_at FROM votes WHERE decision='revoke' ORDER BY subject_kind,fingerprint,id",
)?;
let mut rows = statement.query([])?;
while let Some(row) = rows.next()? {
let accepted_at =
DateTime::parse_from_rfc3339(&row.get::<_, String>(3)?)?.with_timezone(&Utc);
if accepted_at > effective_at {
continue;
}
let kind = match row.get::<_, String>(0)?.as_str() {
"name" => SubjectKind::Name,
"edge" => SubjectKind::Edge,
"equivalence" => SubjectKind::Equivalence,
_ => anyhow::bail!("stored vote has unknown subject kind"),
};
subjects
.entry((kind, row.get(1)?))
.or_default()
.push(row.get(2)?);
}
let mut entries = Vec::with_capacity(subjects.len());
for ((subject_kind, fingerprint), mut revocations) in subjects {
revocations.sort();
revocations.dedup();
let decision = current_decision(registry, subject_kind, &fingerprint, effective_at)?;
let active = decision
.as_ref()
.is_none_or(|decision| decision.status == DecisionStatus::Revoked);
let mut superseding_votes = if active {
Vec::new()
} else {
decision
.as_ref()
.into_iter()
.flat_map(|decision| &decision.scopes)
.flat_map(|scope| scope.votes.iter().cloned())
.collect::<Vec<_>>()
};
superseding_votes.sort();
superseding_votes.dedup();
ensure!(
active || !superseding_votes.is_empty(),
"revocation compilation lost its superseding quorum"
);
entries.push(RevocationEntry {
subject_kind,
fingerprint,
revocations,
active,
superseding_votes,
});
}
let feed = RevocationFeed {
schema: "argand.site-revocations/v1".into(),
registry: registry.identity.clone(),
rules: registry.receipt.rules.clone(),
policy_sha256: registry.receipt.review_policy_sha256.clone(),
reviewer_trust_sha256: registry.receipt.reviewer_trust_sha256.clone(),
effective_at,
expires_at: effective_at + chrono::Duration::days(MAXIMUM_FEED_LIFETIME_DAYS),
entries,
};
feed.validate()?;
Ok(feed)
}
fn current_decision(
registry: &Registry,
subject_kind: SubjectKind,
fingerprint: &str,
at: DateTime<Utc>,
) -> anyhow::Result<Option<crate::vote::PolicyDecision>> {
match subject_kind {
SubjectKind::Name => {
let exists: bool = registry.db.query_row(
"SELECT EXISTS(SELECT 1 FROM names WHERE fingerprint=?1)",
[fingerprint],
|row| row.get(0),
)?;
exists
.then(|| crate::vote::decision(registry, subject_kind, fingerprint, at))
.transpose()
}
SubjectKind::Edge => {
let exists: bool = registry.db.query_row(
"SELECT EXISTS(SELECT 1 FROM edges WHERE fingerprint=?1)",
[fingerprint],
|row| row.get(0),
)?;
exists
.then(|| crate::vote::decision(registry, subject_kind, fingerprint, at))
.transpose()
}
SubjectKind::Equivalence => {
let pair = registry
.db
.query_row(
"SELECT left_entity,right_entity FROM equivalences WHERE fingerprint=?1",
[fingerprint],
|row| Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?)),
)
.optional()?;
pair.map(|(left, right)| {
let pair = crate::identity::propose(registry, &left, &right)?;
ensure!(
pair.fingerprint == fingerprint,
"stored equivalence fingerprint is stale"
);
let bundle = crate::bundle::equivalence(registry, &pair)?;
crate::vote::decision_for_bundle(
registry,
subject_kind,
fingerprint,
&bundle.id,
at,
)
})
.transpose()
}
}
}
fn verify_reviewer_trust(registry: &Registry, allowed_reviewers: &Path) -> anyhow::Result<()> {
let trust = crate::ssh::sealed_input(allowed_reviewers, 1024 * 1024)?;
ensure!(
crate::digest(&trust.bytes) == registry.receipt.reviewer_trust_sha256,
"reviewer trust root differs from generation receipt"
);
Ok(())
}

26
crates/argand-site-registry/src/store.rs
View file

@ -17,7 +17,7 @@ use std::{
};
/// Adapter/normalization contract recorded in all generation identities.
pub const RULE_VERSION: &str = "argand.site-rules/v3";
pub const RULE_VERSION: &str = "argand.site-rules/v4";
/// Whether a signed immutable generation uses a reader-compatible rule contract.
#[must_use]
@ -26,7 +26,10 @@ pub fn supported_rule_version(version: &str) -> bool {
}
pub(crate) fn legacy_rule_version(version: &str) -> bool {
matches!(version, "argand.site-rules/v1" | "argand.site-rules/v2")
matches!(
version,
"argand.site-rules/v1" | "argand.site-rules/v2" | "argand.site-rules/v3"
)
}
/// Opens or migrates the local assertion store with bounded page cache.
@ -46,20 +49,37 @@ pub fn open(path: &Path) -> anyhow::Result<Connection> {
db.execute_batch(include_str!("../migrations/001.sql"))?;
db.execute_batch(include_str!("../migrations/002.sql"))?;
db.execute_batch(include_str!("../migrations/003.sql"))?;
db.execute_batch(include_str!("../migrations/004.sql"))?;
db.execute_batch(include_str!("../migrations/005.sql"))?;
db.execute_batch("COMMIT")?;
}
1 => {
db.execute_batch("BEGIN IMMEDIATE")?;
db.execute_batch(include_str!("../migrations/002.sql"))?;
db.execute_batch(include_str!("../migrations/003.sql"))?;
db.execute_batch(include_str!("../migrations/004.sql"))?;
db.execute_batch(include_str!("../migrations/005.sql"))?;
db.execute_batch("COMMIT")?;
}
2 => {
db.execute_batch("BEGIN IMMEDIATE")?;
db.execute_batch(include_str!("../migrations/003.sql"))?;
db.execute_batch(include_str!("../migrations/004.sql"))?;
db.execute_batch(include_str!("../migrations/005.sql"))?;
db.execute_batch("COMMIT")?;
}
3 => {}
3 => {
db.execute_batch("BEGIN IMMEDIATE")?;
db.execute_batch(include_str!("../migrations/004.sql"))?;
db.execute_batch(include_str!("../migrations/005.sql"))?;
db.execute_batch("COMMIT")?;
}
4 => {
db.execute_batch("BEGIN IMMEDIATE")?;
db.execute_batch(include_str!("../migrations/005.sql"))?;
db.execute_batch("COMMIT")?;
}
5 => {}
_ => anyhow::bail!("unsupported registry schema {version}"),
}
let rules: String = db.query_row(

175
crates/argand-site-registry/src/update.rs
View file

@ -32,6 +32,13 @@ pub struct Config {
/// Explicit billed `CrUX` jobs, empty by default.
#[serde(default)]
pub crux: Vec<crate::crux::CruxDownload>,
/// Optional explicit review policy; strict reference policy when absent.
pub review_policy: Option<PathBuf>,
/// Exact reviewer SSH trust root required for strict candidate builds.
pub reviewer_trust: Option<PathBuf>,
/// Replace the current typed full/partition frontier on each scheduled download.
#[serde(default)]
pub auto_supersede_typed_snapshots: bool,
}
/// Runs all declared imports, refusing candidate publication on any failure.
@ -39,7 +46,9 @@ pub struct Config {
///
/// # Errors
/// Returns configuration, source, lock, import, or build errors.
#[allow(clippy::too_many_lines)] // Scheduler order is explicit: acquire, import, build, then publish.
pub async fn run(config: &Config) -> anyhow::Result<PathBuf> {
validate_automatic_coordinates(config)?;
fs::create_dir_all(&config.generations)?;
let lock = OpenOptions::new() // atomic-writes: allow advisory lock inode must remain stable
.read(true)
@ -49,6 +58,7 @@ pub async fn run(config: &Config) -> anyhow::Result<PathBuf> {
.open(config.generations.join("update.lock"))?;
lock.try_lock().context("registry update already running")?;
let mut inputs = config.inputs.clone();
let mut db = crate::store::open(&config.database)?;
let now = Utc::now();
for request in &config.downloads {
let mut request = request.clone();
@ -56,6 +66,26 @@ pub async fn run(config: &Config) -> anyhow::Result<PathBuf> {
.snapshot
.replace("{date}", &now.format("%Y-%m-%d").to_string())
.replace("{month}", &now.format("%Y-%m").to_string());
if config.auto_supersede_typed_snapshots
&& let Some(coverage) = &mut request.coverage
&& matches!(
coverage.kind,
crate::model::CoverageKind::Full | crate::model::CoverageKind::Partition
)
{
anyhow::ensure!(
coverage.supersedes.is_empty(),
"automatic typed supersession cannot combine with explicit supersedes"
);
if let Some(frontier) = crate::coverage::frontier(
&db,
request.source.key(),
&coverage.collection,
coverage.coordinate(),
)? {
coverage.supersedes.push(frontier);
}
}
inputs.push(crate::download::download(&config.cache, &request).await?);
}
for request in &config.crux {
@ -67,10 +97,29 @@ pub async fn run(config: &Config) -> anyhow::Result<PathBuf> {
.format("%Y%m")
.to_string();
}
if config.auto_supersede_typed_snapshots
&& let Some(coverage) = &mut request.coverage
&& matches!(
coverage.kind,
crate::model::CoverageKind::Full | crate::model::CoverageKind::Partition
)
{
anyhow::ensure!(
coverage.supersedes.is_empty(),
"automatic typed supersession cannot combine with explicit supersedes"
);
if let Some(frontier) = crate::coverage::frontier(
&db,
crate::model::Source::Crux.key(),
&coverage.collection,
coverage.coordinate(),
)? {
coverage.supersedes.push(frontier);
}
}
inputs.push(crate::crux::download(&config.cache, &request).await?);
}
anyhow::ensure!(!inputs.is_empty(), "update config contains no sources");
let mut db = crate::store::open(&config.database)?;
for input in inputs {
let manifest: SourceManifest = crate::read_json(&input.manifest)?;
crate::store::import(&mut db, &manifest, &input.input)?;
@ -80,7 +129,18 @@ pub async fn run(config: &Config) -> anyhow::Result<PathBuf> {
let pending = config
.generations
.join(format!("pending-{}", now.format("%Y%m%dT%H%M%S%.9fZ")));
crate::build::build(&db, &pending)?;
let policy = config
.review_policy
.as_deref()
.map(crate::read_json)
.transpose()?
.unwrap_or_else(crate::policy::ReviewPolicy::reference);
crate::build::build_with_policy_and_trust(
&db,
&pending,
&policy,
config.reviewer_trust.as_deref(),
)?;
let pin = crate::file_digest(&pending.join("COMPLETE.json"))?;
let output = config.generations.join(format!("candidate-{pin}"));
if output.exists() {
@ -94,3 +154,114 @@ pub async fn run(config: &Config) -> anyhow::Result<PathBuf> {
}
Ok(output)
}
fn validate_automatic_coordinates(config: &Config) -> anyhow::Result<()> {
if !config.auto_supersede_typed_snapshots {
return Ok(());
}
let mut coordinates = std::collections::BTreeSet::new();
for request in &config.downloads {
if let Some(coverage) = &request.coverage
&& matches!(
coverage.kind,
crate::model::CoverageKind::Full | crate::model::CoverageKind::Partition
)
{
anyhow::ensure!(
coordinates.insert((
request.source.key(),
coverage.collection.as_str(),
coverage.coordinate()
)),
"automatic update repeats one source coverage coordinate"
);
}
}
for request in &config.crux {
if let Some(coverage) = &request.coverage
&& matches!(
coverage.kind,
crate::model::CoverageKind::Full | crate::model::CoverageKind::Partition
)
{
anyhow::ensure!(
coordinates.insert((
crate::model::Source::Crux.key(),
coverage.collection.as_str(),
coverage.coordinate()
)),
"automatic update repeats one source coverage coordinate"
);
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::model::{Compression, CoverageKind, Format, Source, SourceCoverage};
fn request(partition: &str) -> Download {
Download {
source: Source::Majestic,
format: Format::MajesticCsv,
compression: Compression::None,
url: "https://downloads.majestic.com/majestic_million.csv".into(),
snapshot: "{date}".into(),
scope: "full".into(),
maximum_bytes: 1,
coverage: Some(SourceCoverage {
collection: "default".into(),
kind: CoverageKind::Partition,
partition: Some(partition.into()),
base: None,
sequence: None,
supersedes: Vec::new(),
}),
}
}
#[test]
fn scheduled_supersession_rejects_duplicate_coordinates_before_acquisition()
-> anyhow::Result<()> {
let config = Config {
cache: "cache".into(),
database: "writer.sqlite".into(),
generations: "generations".into(),
downloads: vec![request("global"), request("global")],
inputs: Vec::new(),
crux: Vec::new(),
review_policy: None,
reviewer_trust: None,
auto_supersede_typed_snapshots: true,
};
let Some(error) = validate_automatic_coordinates(&config).err() else {
anyhow::bail!("duplicate coordinate was accepted");
};
assert!(
error
.to_string()
.contains("repeats one source coverage coordinate")
);
Ok(())
}
#[test]
fn separate_partition_coordinates_are_allowed() -> anyhow::Result<()> {
let config = Config {
cache: "cache".into(),
database: "writer.sqlite".into(),
generations: "generations".into(),
downloads: vec![request("GB"), request("US")],
inputs: Vec::new(),
crux: Vec::new(),
review_policy: None,
reviewer_trust: None,
auto_supersede_typed_snapshots: true,
};
validate_automatic_coordinates(&config)
}
}

881
crates/argand-site-registry/src/vote.rs Normal file
View file

@ -0,0 +1,881 @@
// By Nic Weyand!
//! Accepted-time reviewer votes and deterministic quorum compilation.
use crate::{bundle::EvidenceBundle, policy::SubjectKind, query::Registry};
use anyhow::{Context, ensure};
use base64::Engine;
use chrono::{DateTime, Utc};
use rusqlite::{Connection, params};
use serde::{Deserialize, Serialize};
use std::{collections::BTreeMap, path::Path};
/// SSH signature namespace for exact vote JSON bytes.
pub const SIGNATURE_NAMESPACE: &str = "argand-site-registry-vote";
/// Authenticated decision carried by one reviewer.
#[derive(Clone, Copy, Debug, Deserialize, Serialize, clap::ValueEnum, Eq, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum VoteDecision {
/// Support admitting this exact subject and evidence bundle.
Approve,
/// Block this exact subject until the revocation is explicitly superseded.
Revoke,
}
impl VoteDecision {
const fn key(self) -> &'static str {
match self {
Self::Approve => "approve",
Self::Revoke => "revoke",
}
}
}
/// One signed reviewer vote over a granular assertion and current evidence bundle.
#[derive(Clone, Debug, Deserialize, Serialize, Eq, PartialEq)]
#[serde(deny_unknown_fields)]
pub struct Vote {
/// `argand.site-vote/v1`.
pub schema: String,
/// Name, edge, or equivalence assertion.
pub subject_kind: SubjectKind,
/// Stable material assertion fingerprint.
pub fingerprint: String,
/// Approve or revoke.
pub decision: VoteDecision,
/// Identity that must match the SSH allowed-signers identity.
pub reviewer: String,
/// Human explanation of the decision.
pub reason: String,
/// Exact current evidence-bundle digest.
pub evidence_bundle: String,
/// Review-policy digest under which this decision was made.
pub policy: String,
/// Reviewer-asserted decision time, retained for audit.
pub reviewed_at: DateTime<Utc>,
/// Reviewer-requested expiry; required only for approvals.
pub expires_at: Option<DateTime<Utc>>,
/// Edge role; `unspecified` for names, equivalences, and revocations.
pub role: String,
/// Explicit reviewed locale or empty.
pub locale: String,
/// Explicit reviewed uppercase two-letter country or empty.
pub country: String,
/// Sticky revocation vote IDs explicitly superseded by this approval.
#[serde(default)]
pub supersedes: Vec<String>,
}
impl Vote {
/// Validates the signed decision independently of registry state.
///
/// # Errors
/// Rejects unsupported, oversized, malformed, or internally inconsistent votes.
pub fn validate(&self) -> anyhow::Result<()> {
ensure!(
self.schema == "argand.site-vote/v1",
"unsupported vote schema"
);
ensure!(
crate::model::valid_digest(&self.fingerprint)
&& crate::model::valid_digest(&self.evidence_bundle)
&& crate::model::valid_digest(&self.policy),
"vote needs full assertion, evidence, and policy digests"
);
ensure!(
!self.reviewer.trim().is_empty()
&& self.reviewer.len() <= 256
&& !self.reason.trim().is_empty()
&& self.reason.len() <= 8192,
"vote reviewer and reason are required and bounded"
);
ensure!(
self.locale.len() <= 64
&& self
.locale
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || byte == b'-'),
"invalid vote locale"
);
ensure!(
self.country.is_empty()
|| (self.country.len() == 2
&& self.country.bytes().all(|byte| byte.is_ascii_uppercase())),
"vote country must be uppercase two-letter code"
);
ensure!(
self.supersedes.len() <= 256
&& self
.supersedes
.iter()
.all(|id| crate::model::valid_digest(id)),
"invalid vote supersession list"
);
let unique = self
.supersedes
.iter()
.collect::<std::collections::BTreeSet<_>>();
ensure!(
unique.len() == self.supersedes.len(),
"duplicate superseded vote"
);
match self.decision {
VoteDecision::Approve => {
let expires = self.expires_at.context("approval needs an expiry")?;
ensure!(
expires > self.reviewed_at
&& expires - self.reviewed_at <= chrono::Duration::days(90),
"approval must expire within 90 days"
);
}
VoteDecision::Revoke => ensure!(
self.expires_at.is_none() && self.supersedes.is_empty(),
"revocations neither expire nor supersede another vote"
),
}
match (self.subject_kind, self.decision) {
(SubjectKind::Edge, VoteDecision::Approve) => {
ensure!(
matches!(self.role.as_str(), "primary" | "regional"),
"edge approval needs a primary or regional role"
);
ensure!(
self.role != "regional" || !self.locale.is_empty() || !self.country.is_empty(),
"regional approval needs locale or country evidence"
);
ensure!(
self.role != "primary" || (self.locale.is_empty() && self.country.is_empty()),
"primary is the unscoped global fallback"
);
}
_ => ensure!(
self.role == "unspecified" && self.locale.is_empty() && self.country.is_empty(),
"only edge approvals can assert destination scope"
),
}
Ok(())
}
}
/// Exact detached signature evidence retained with a vote.
#[derive(Clone, Debug)]
pub struct Authentication {
/// SSH allowed-signers identity.
pub signer: String,
/// SHA-256 of exact detached signature bytes.
pub signature_sha256: String,
/// Domain-separated SSH signature namespace.
pub namespace: String,
/// SHA-256 of exact signed vote JSON bytes.
pub decision_sha256: String,
/// SHA-256 of the SSH public-key blob embedded in the verified signature.
pub key_sha256: String,
decision_json: Vec<u8>,
signature: Vec<u8>,
}
/// Verifies exact vote bytes against an independently supplied reviewer trust file.
///
/// # Errors
/// Rejects malformed/oversized inputs, identity mismatch, and untrusted signatures.
pub fn authenticate(
decision: &Path,
signature: &Path,
allowed_reviewers: &Path,
identity: &str,
) -> anyhow::Result<(Vote, Authentication)> {
ensure!(
!identity.trim().is_empty() && identity.len() <= 256,
"reviewer identity is required and bounded"
);
let decision = crate::ssh::sealed_input(decision, 1024 * 1024)?;
let signature = crate::ssh::sealed_input(signature, 64 * 1024)?;
let allowed_reviewers = crate::ssh::sealed_input(allowed_reviewers, 1024 * 1024)?;
let vote: Vote = serde_json::from_value(crate::json::parse(&decision.bytes)?)?;
vote.validate()?;
ensure!(
vote.reviewer == identity,
"reviewer must equal authenticated identity"
);
crate::ssh::verify(
SIGNATURE_NAMESPACE,
&decision.bytes,
&signature.bytes,
&allowed_reviewers.bytes,
identity,
)?;
let key_sha256 = signature_key_sha256(&signature.bytes)?;
Ok((
vote,
Authentication {
signer: identity.into(),
signature_sha256: crate::digest(&signature.bytes),
namespace: SIGNATURE_NAMESPACE.into(),
decision_sha256: crate::digest(&decision.bytes),
key_sha256,
decision_json: decision.bytes,
signature: signature.bytes,
},
))
}
/// Records a verified name or edge vote using trusted writer acceptance time.
///
/// # Errors
/// Rejects stale evidence, invalid subjects, bad authentication, or SQLite failures.
pub fn record_authenticated(
db: &Connection,
registry: &Registry,
vote: &Vote,
authentication: &Authentication,
) -> anyhow::Result<String> {
record_authenticated_at(db, registry, vote, authentication, Utc::now())
}
/// Verifies an authenticated name or edge vote against current evidence without writing it.
///
/// # Errors
/// Rejects malformed authentication, stale evidence, invalid subjects, and unrelated
/// revocation supersession references.
pub fn verify_authenticated(
registry: &Registry,
vote: &Vote,
authentication: &Authentication,
) -> anyhow::Result<()> {
ensure!(
vote.subject_kind != SubjectKind::Equivalence,
"equivalence vote verification needs the exact proposed entity pair"
);
validate_authentication(vote, authentication)?;
let bundle = crate::bundle::build(registry, vote.subject_kind, &vote.fingerprint)?;
validate_bundle_and_subject(registry, vote, &bundle)?;
validate_supersedes(&registry.db, vote)
}
/// Verifies an authenticated equivalence vote against the exact current entity pair.
///
/// # Errors
/// Rejects malformed authentication, stale pair evidence, a policy mismatch, or
/// unrelated revocation supersession references.
pub fn verify_equivalence_authenticated(
registry: &Registry,
pair: &crate::identity::Equivalence,
vote: &Vote,
authentication: &Authentication,
) -> anyhow::Result<()> {
ensure!(
vote.subject_kind == SubjectKind::Equivalence && vote.fingerprint == pair.fingerprint,
"vote does not match equivalence proposal"
);
validate_authentication(vote, authentication)?;
let bundle = crate::bundle::equivalence(registry, pair)?;
ensure!(
bundle.id == vote.evidence_bundle,
"vote references stale equivalence evidence"
);
ensure!(
vote.policy == registry.receipt.review_policy_sha256,
"vote was made under a different review policy"
);
validate_supersedes(&registry.db, vote)
}
fn record_authenticated_at(
db: &Connection,
registry: &Registry,
vote: &Vote,
authentication: &Authentication,
accepted_at: DateTime<Utc>,
) -> anyhow::Result<String> {
ensure!(
vote.subject_kind != SubjectKind::Equivalence,
"equivalence vote needs the exact proposed entity pair"
);
validate_authentication(vote, authentication)?;
let bundle = crate::bundle::build(registry, vote.subject_kind, &vote.fingerprint)?;
validate_bundle_and_subject(registry, vote, &bundle)?;
validate_supersedes(db, vote)?;
let transaction = db.unchecked_transaction()?;
let id = append(db, vote, authentication, accepted_at)?;
transaction.commit()?;
Ok(id)
}
pub(crate) fn record_equivalence_authenticated_at(
db: &Connection,
registry: &Registry,
pair: &crate::identity::Equivalence,
vote: &Vote,
authentication: &Authentication,
accepted_at: DateTime<Utc>,
) -> anyhow::Result<String> {
verify_equivalence_authenticated(registry, pair, vote, authentication)?;
validate_supersedes(db, vote)?;
let transaction = db.unchecked_transaction()?;
db.execute(
"INSERT OR IGNORE INTO equivalences VALUES(?1,?2,?3,?4,?5)",
params![
pair.fingerprint,
pair.entities[0],
pair.entities[1],
pair.signatures[0],
pair.signatures[1]
],
)?;
let id = append(db, vote, authentication, accepted_at)?;
transaction.commit()?;
Ok(id)
}
fn validate_bundle_and_subject(
registry: &Registry,
vote: &Vote,
bundle: &EvidenceBundle,
) -> anyhow::Result<()> {
ensure!(
vote.policy == registry.receipt.review_policy_sha256,
"vote was made under a different review policy"
);
ensure!(
bundle.id == vote.evidence_bundle,
"vote references stale evidence"
);
if vote.subject_kind == SubjectKind::Edge && vote.decision == VoteDecision::Approve {
ensure!(
registry.candidate(&vote.fingerprint)?.eligible,
"ineligible edge cannot be approved"
);
}
Ok(())
}
fn validate_authentication(vote: &Vote, authentication: &Authentication) -> anyhow::Result<()> {
vote.validate()?;
ensure!(
authentication.signer == vote.reviewer
&& authentication.namespace == SIGNATURE_NAMESPACE
&& crate::model::valid_digest(&authentication.signature_sha256)
&& authentication.signature_sha256 == crate::digest(&authentication.signature)
&& authentication.decision_sha256 == crate::digest(&authentication.decision_json)
&& crate::model::valid_digest(&authentication.key_sha256)
&& authentication.key_sha256 == signature_key_sha256(&authentication.signature)?,
"vote authentication does not match decision"
);
let signed: Vote = serde_json::from_value(crate::json::parse(&authentication.decision_json)?)?;
ensure!(&signed == vote, "signed decision differs from vote");
Ok(())
}
pub(crate) fn signature_key_sha256(signature: &[u8]) -> anyhow::Result<String> {
let text = std::str::from_utf8(signature)?;
let mut encoded = String::new();
let mut inside = false;
let mut ended = false;
for line in text.lines() {
match line.trim() {
"-----BEGIN SSH SIGNATURE-----" if !inside && !ended => inside = true,
"-----END SSH SIGNATURE-----" if inside => {
inside = false;
ended = true;
}
value if inside => encoded.push_str(value),
value if !value.is_empty() => anyhow::bail!("malformed SSH signature armor"),
_ => {}
}
}
ensure!(!inside && ended, "incomplete SSH signature armor");
let decoded = base64::engine::general_purpose::STANDARD.decode(encoded)?;
ensure!(
decoded.len() >= 14 && &decoded[..6] == b"SSHSIG",
"invalid SSH signature envelope"
);
let version = u32::from_be_bytes(decoded[6..10].try_into()?);
ensure!(version == 1, "unsupported SSH signature version");
let key_len = usize::try_from(u32::from_be_bytes(decoded[10..14].try_into()?))?;
let end = 14_usize
.checked_add(key_len)
.context("SSH signature key length overflow")?;
ensure!(
key_len > 0 && key_len <= 16 * 1024 && end <= decoded.len(),
"invalid SSH signature public key"
);
Ok(crate::digest(&decoded[14..end]))
}
fn validate_supersedes(db: &Connection, vote: &Vote) -> anyhow::Result<()> {
for id in &vote.supersedes {
let valid: bool = db.query_row(
"SELECT EXISTS(SELECT 1 FROM votes WHERE id=?1 AND fingerprint=?2 AND subject_kind=?3 AND decision='revoke')",
params![id, vote.fingerprint, vote.subject_kind.key()],
|row| row.get(0),
)?;
ensure!(valid, "vote supersedes an absent or unrelated revocation");
}
Ok(())
}
fn append(
db: &Connection,
vote: &Vote,
authentication: &Authentication,
accepted_at: DateTime<Utc>,
) -> anyhow::Result<String> {
let id = authentication.decision_sha256.clone();
let changed = db.execute(
"INSERT OR IGNORE INTO votes(id,fingerprint,subject_kind,decision,reviewer,reason,evidence_bundle,policy_sha256,reviewed_at,expires_at,role,locale,country,supersedes_json,accepted_at,document_json) VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10,?11,?12,?13,?14,?15,?16)",
params![
id,
vote.fingerprint,
vote.subject_kind.key(),
vote.decision.key(),
vote.reviewer,
vote.reason,
vote.evidence_bundle,
vote.policy,
vote.reviewed_at.to_rfc3339(),
vote.expires_at.map(|time| time.to_rfc3339()),
vote.role,
vote.locale,
vote.country,
serde_json::to_string(&vote.supersedes)?,
accepted_at.to_rfc3339(),
authentication.decision_json,
],
)?;
if changed == 0 {
let matches: bool = db.query_row(
"SELECT EXISTS(SELECT 1 FROM votes v JOIN vote_auth a USING(sequence) WHERE v.id=?1 AND v.document_json=?2 AND a.signer=?3 AND a.signature_sha256=?4 AND a.namespace=?5 AND a.decision_sha256=?6 AND a.key_sha256=?7 AND a.signature=?8)",
params![
id,
authentication.decision_json,
authentication.signer,
authentication.signature_sha256,
authentication.namespace,
authentication.decision_sha256,
authentication.key_sha256,
authentication.signature
],
|row| row.get(0),
)?;
ensure!(matches, "vote ID collision or authentication mismatch");
return Ok(id);
}
let sequence = db.last_insert_rowid();
db.execute(
"INSERT INTO vote_auth VALUES(?1,?2,?3,?4,?5,?6,?7)",
params![
sequence,
authentication.signer,
authentication.signature_sha256,
authentication.namespace,
authentication.decision_sha256,
authentication.key_sha256,
authentication.signature,
],
)?;
Ok(id)
}
/// Result of compiling current authenticated votes under one generation policy.
#[derive(Clone, Debug, Serialize)]
pub struct PolicyDecision {
/// Compiled state.
pub status: DecisionStatus,
/// Exact current evidence bundle.
pub evidence_bundle: String,
/// Required distinct approvals.
pub approvals_required: u16,
/// Required distinct reviewer groups.
pub groups_required: u16,
/// Current evidence-matching approval votes examined.
pub approvals: u64,
/// Active sticky revocations.
pub revocations: Vec<String>,
/// Qualified edge scopes; one unscoped entry for name/equivalence approval.
pub scopes: Vec<ApprovedScope>,
/// Votes excluded because their evidence bundle is no longer current.
pub stale_evidence: u64,
/// Votes excluded because they were signed under another policy epoch.
pub stale_policy: u64,
/// Latest reviewer approvals that expired.
pub expired: u64,
/// Latest reviewer approvals whose effective start is in the future.
pub not_yet_valid: u64,
/// Signed policy rules currently holding an otherwise qualified subject.
pub policy_holds: Vec<String>,
}
/// Policy state for one granular subject.
#[derive(Clone, Copy, Debug, Serialize, Eq, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum DecisionStatus {
/// At least one exact scope satisfies the configured quorum.
Approved,
/// A sticky revocation has not been superseded by a complete new quorum.
Revoked,
/// Current evidence lacks a complete approval quorum.
InsufficientReview,
/// All current-evidence approvals have expired.
Expired,
/// Votes exist, but none references the current evidence bundle.
StaleEvidence,
/// Votes exist, but none was signed under the current policy epoch.
StalePolicy,
/// Current approvals conflict across destination scopes and no scope has quorum.
Disputed,
/// Approval quorum exists, but signed risk policy requires fresh review or evidence.
Probationary,
}
/// One exact role/scope that independently reached quorum.
#[derive(Clone, Debug, Serialize, Eq, Ord, PartialEq, PartialOrd)]
pub struct ApprovedScope {
/// `primary`, `regional`, or `unspecified`.
pub role: String,
/// Reviewed locale or empty.
pub locale: String,
/// Reviewed country or empty.
pub country: String,
/// Counted exact vote IDs.
pub votes: Vec<String>,
/// Earliest effective expiry among counted votes.
pub expires_at: DateTime<Utc>,
}
#[derive(Clone)]
struct StoredVote {
id: String,
vote: Vote,
accepted_at: DateTime<Utc>,
key_sha256: String,
}
/// Compiles a current name or edge decision under the authenticated generation policy.
///
/// # Errors
/// Rejects missing subjects and corrupt vote/policy rows.
pub fn decision(
registry: &Registry,
subject_kind: SubjectKind,
fingerprint: &str,
now: DateTime<Utc>,
) -> anyhow::Result<PolicyDecision> {
let bundle = crate::bundle::build(registry, subject_kind, fingerprint)?;
decision_for_bundle(registry, subject_kind, fingerprint, &bundle.id, now)
}
#[allow(clippy::too_many_lines)] // Quorum, expiry, supersession, and risk compile in one pass.
pub(crate) fn decision_for_bundle(
registry: &Registry,
subject_kind: SubjectKind,
fingerprint: &str,
bundle: &str,
now: DateTime<Utc>,
) -> anyhow::Result<PolicyDecision> {
let policy = &registry.receipt.review_policy;
policy.validate()?;
let mut risk_classes = Vec::new();
let mut source_conflict = false;
let mut dangerous_drift = false;
if subject_kind == SubjectKind::Edge {
let candidate = registry.candidate(fingerprint)?;
if policy.block_source_conflicts || policy.risk_thresholds.contains_key("source_conflict") {
source_conflict = crate::queue::domain_entity_conflict(registry, &candidate, now)?;
if source_conflict {
risk_classes.push("source_conflict");
}
}
if policy.block_dangerous_drift || policy.risk_thresholds.contains_key("dangerous_drift") {
dangerous_drift = crate::queue::drift(registry, fingerprint)?.revocation_candidate;
if dangerous_drift {
risk_classes.push("dangerous_drift");
}
}
}
let threshold = policy.threshold_for(subject_kind, risk_classes.iter().copied());
let votes = load(&registry.db, subject_kind, fingerprint)?;
let stale_policy = u64::try_from(
votes
.iter()
.filter(|vote| {
vote.vote.decision == VoteDecision::Approve
&& vote.vote.policy != registry.receipt.review_policy_sha256
})
.map(|vote| vote.vote.reviewer.as_str())
.collect::<std::collections::BTreeSet<_>>()
.len(),
)?;
let mut latest = BTreeMap::new();
for vote in votes
.iter()
.filter(|vote| vote.vote.policy == registry.receipt.review_policy_sha256)
{
latest.insert(vote.vote.reviewer.as_str(), vote);
}
let mut stale_evidence = 0;
let mut expired = 0;
let mut not_yet_valid = 0;
let mut active_approvals: BTreeMap<(String, String, String), Vec<&StoredVote>> =
BTreeMap::new();
for vote in latest.into_values() {
if vote.vote.decision != VoteDecision::Approve {
continue;
}
if vote.vote.evidence_bundle != bundle {
stale_evidence += 1;
continue;
}
let starts = vote.accepted_at.max(vote.vote.reviewed_at);
let requested = vote
.vote
.expires_at
.context("stored approval has no expiry")?;
let effective_expiry = requested.min(
vote.accepted_at + chrono::Duration::days(i64::from(policy.maximum_approval_days)),
);
if now < starts {
not_yet_valid += 1;
continue;
}
if now >= effective_expiry {
expired += 1;
continue;
}
active_approvals
.entry((
vote.vote.role.clone(),
vote.vote.locale.clone(),
vote.vote.country.clone(),
))
.or_default()
.push(vote);
}
let revocations = votes
.iter()
.filter(|vote| vote.vote.decision == VoteDecision::Revoke && vote.accepted_at <= now)
.map(|vote| vote.id.clone())
.collect::<Vec<_>>();
let active_scope_count = active_approvals.len();
let mut scopes = Vec::new();
let mut approval_count = 0_u64;
for ((role, locale, country), approvals) in active_approvals {
let mut unique_keys = BTreeMap::new();
for approval in approvals {
unique_keys.insert(approval.key_sha256.as_str(), approval);
}
let approvals = unique_keys.into_values().collect::<Vec<_>>();
approval_count = approval_count.saturating_add(u64::try_from(approvals.len())?);
let reviewers = approvals
.iter()
.map(|vote| vote.vote.reviewer.as_str())
.collect::<Vec<_>>();
let enough = approvals.len() >= usize::from(threshold.approvals)
&& policy.group_count(reviewers.into_iter()) >= usize::from(threshold.groups);
let supersedes_revocations = revocations.iter().all(|revocation| {
approvals
.iter()
.all(|approval| approval.vote.supersedes.contains(revocation))
});
if enough && (!policy.sticky_revocations || supersedes_revocations) {
let expires_at = approvals
.iter()
.filter_map(|vote| {
vote.vote.expires_at.map(|expires| {
expires.min(
vote.accepted_at
+ chrono::Duration::days(i64::from(policy.maximum_approval_days)),
)
})
})
.min()
.context("qualified approval scope has no expiry")?;
scopes.push(ApprovedScope {
role,
locale,
country,
votes: approvals.iter().map(|vote| vote.id.clone()).collect(),
expires_at,
});
}
}
let unresolved_revocation = !revocations.is_empty() && scopes.is_empty();
let mut policy_holds = Vec::new();
if subject_kind == SubjectKind::Edge && !scopes.is_empty() {
if policy.block_source_conflicts && source_conflict {
policy_holds.push("source_conflict".into());
}
let drift = crate::queue::drift(registry, fingerprint)?;
if policy.require_edge_observation
&& drift
.classes
.contains(&crate::queue::DriftClass::Unobserved)
{
policy_holds.push("missing_observation".into());
}
if let (Some(maximum_days), Some(latest_at)) =
(policy.maximum_observation_age_days, drift.latest_at)
&& (latest_at > now + chrono::Duration::minutes(5)
|| latest_at + chrono::Duration::days(i64::from(maximum_days)) < now)
{
policy_holds.push("stale_observation".into());
}
if policy.block_dangerous_drift && dangerous_drift {
policy_holds.push("dangerous_drift".into());
}
}
let status = if unresolved_revocation {
DecisionStatus::Revoked
} else if policy_holds.iter().any(|hold| hold == "source_conflict") {
DecisionStatus::Disputed
} else if !policy_holds.is_empty() {
DecisionStatus::Probationary
} else if !scopes.is_empty() {
DecisionStatus::Approved
} else if active_scope_count > 1 && approval_count > 0 {
DecisionStatus::Disputed
} else if expired > 0 && approval_count == 0 {
DecisionStatus::Expired
} else if stale_evidence > 0 && approval_count == 0 {
DecisionStatus::StaleEvidence
} else if stale_policy > 0 && approval_count == 0 {
DecisionStatus::StalePolicy
} else {
DecisionStatus::InsufficientReview
};
Ok(PolicyDecision {
status,
evidence_bundle: bundle.into(),
approvals_required: threshold.approvals,
groups_required: threshold.groups,
approvals: approval_count,
revocations,
scopes,
stale_evidence,
stale_policy,
expired,
not_yet_valid,
policy_holds,
})
}
fn load(
db: &Connection,
subject_kind: SubjectKind,
fingerprint: &str,
) -> anyhow::Result<Vec<StoredVote>> {
let mut statement = db.prepare(
"SELECT v.id,v.document_json,v.accepted_at,a.signer,a.signature_sha256,a.namespace,a.decision_sha256,a.key_sha256,a.signature FROM votes v JOIN vote_auth a USING(sequence) WHERE v.subject_kind=?1 AND v.fingerprint=?2 ORDER BY v.sequence",
)?;
let mut rows = statement.query(params![subject_kind.key(), fingerprint])?;
let mut result = Vec::new();
while let Some(row) = rows.next()? {
let document: Vec<u8> = row.get(1)?;
let vote: Vote = serde_json::from_value(crate::json::parse(&document)?)?;
let authentication = Authentication {
signer: row.get(3)?,
signature_sha256: row.get(4)?,
namespace: row.get(5)?,
decision_sha256: row.get(6)?,
key_sha256: row.get(7)?,
decision_json: document,
signature: row.get(8)?,
};
validate_authentication(&vote, &authentication)?;
ensure!(
row.get::<_, String>(0)? == authentication.decision_sha256,
"stored vote identity mismatch"
);
result.push(StoredVote {
id: authentication.decision_sha256,
vote,
accepted_at: DateTime::parse_from_rfc3339(&row.get::<_, String>(2)?)?
.with_timezone(&Utc),
key_sha256: authentication.key_sha256,
});
}
Ok(result)
}
/// Re-verifies every stored vote signature at trusted writer acceptance time.
///
/// # Errors
/// Rejects missing/altered proofs, invalid acceptance times, or untrusted signers.
pub fn verify_all(db: &Connection, allowed_reviewers: &Path) -> anyhow::Result<()> {
let allowed = crate::ssh::sealed_input(allowed_reviewers, 1024 * 1024)?;
let missing: u64 = db.query_row(
"SELECT count(*) FROM votes v LEFT JOIN vote_auth a USING(sequence) WHERE a.sequence IS NULL",
[],
|row| crate::store::unsigned(row, 0),
)?;
ensure!(missing == 0, "generation contains unauthenticated votes");
let mut statement = db.prepare(
"SELECT v.document_json,v.accepted_at,a.signer,a.signature_sha256,a.namespace,a.decision_sha256,a.key_sha256,a.signature FROM votes v JOIN vote_auth a USING(sequence) ORDER BY v.sequence",
)?;
let mut rows = statement.query([])?;
while let Some(row) = rows.next()? {
let decision: Vec<u8> = row.get(0)?;
let vote: Vote = serde_json::from_value(crate::json::parse(&decision)?)?;
let accepted_at =
DateTime::parse_from_rfc3339(&row.get::<_, String>(1)?)?.with_timezone(&Utc);
let authentication = Authentication {
signer: row.get(2)?,
signature_sha256: row.get(3)?,
namespace: row.get(4)?,
decision_sha256: row.get(5)?,
key_sha256: row.get(6)?,
decision_json: decision,
signature: row.get(7)?,
};
validate_authentication(&vote, &authentication)?;
crate::ssh::verify_at(
SIGNATURE_NAMESPACE,
&authentication.decision_json,
&authentication.signature,
&allowed.bytes,
&authentication.signer,
Some(accepted_at),
)?;
}
Ok(())
}
/// Rejects a publisher identity that supplied any vote when separation is enabled.
///
/// # Errors
/// Returns policy/SQLite failures or a publisher-reviewer conflict.
pub fn verify_publisher_separation(registry: &Registry, publisher: &str) -> anyhow::Result<()> {
if !registry.receipt.review_policy.publisher_reviewer_separation {
return Ok(());
}
let conflict: bool = registry.db.query_row(
"SELECT EXISTS(SELECT 1 FROM votes WHERE reviewer=?1)",
[publisher],
|row| row.get(0),
)?;
ensure!(!conflict, "release publisher also supplied a reviewer vote");
Ok(())
}
/// Rejects a publisher signature made by any physical key that supplied a vote.
///
/// # Errors
/// Returns malformed-signature, policy, SQLite, or key-separation failures.
pub(crate) fn verify_publisher_key_separation(
registry: &Registry,
publisher_signature: &[u8],
) -> anyhow::Result<()> {
if !registry.receipt.review_policy.publisher_reviewer_separation {
return Ok(());
}
let key = signature_key_sha256(publisher_signature)?;
let conflict: bool = registry.db.query_row(
"SELECT EXISTS(SELECT 1 FROM vote_auth WHERE key_sha256=?1)",
[key],
|row| row.get(0),
)?;
ensure!(
!conflict,
"release publisher key also supplied a reviewer vote"
);
Ok(())
}

42
crates/argand-site-registry/tests/cli.rs
View file

@ -34,16 +34,16 @@ fn all_source_import_review_resolve_revoke_and_signed_rollback() -> anyhow::Resu
fs::create_dir(root)?;
}
prepare_signer(root)?;
fs::write(
root.join("legacy-policy.json"),
serde_json::to_vec_pretty(
&argand_site_registry::policy::ReviewPolicy::legacy_compatible(),
)?,
)?;
let database = root.join("store.sqlite");
import_sources(root, &database)?;
let candidate = root.join("candidate");
let built = run(&[
"build",
"--database",
text(&database)?,
"--output",
text(&candidate)?,
])?;
let built = build_legacy(root, &database, &candidate)?;
let pin = built["pin"].as_str().context("missing pin")?;
let lookup = run(&[
"lookup",
@ -79,13 +79,7 @@ fn all_source_import_review_resolve_revoke_and_signed_rollback() -> anyhow::Resu
record_review(root, &database, &candidate, pin, &decision_path)?;
let approved = root.join("approved");
review_identity(root, &database, &candidate, pin, &decision)?;
let built = run(&[
"build",
"--database",
text(&database)?,
"--output",
text(&approved)?,
])?;
let built = build_legacy(root, &database, &approved)?;
let approved_pin = built["pin"].as_str().context("approved pin")?;
assert_eq!(
run(&[
@ -307,13 +301,7 @@ fn release_lifecycle(
fs::write(&revocation_path, serde_json::to_vec(&decision)?)?;
record_review(root, database, approved, approved_pin, &revocation_path)?;
let revoked = root.join("revoked");
let built = run(&[
"build",
"--database",
text(database)?,
"--output",
text(&revoked)?,
])?;
let built = build_legacy(root, database, &revoked)?;
let revoked_pin = built["pin"].as_str().context("revoked pin")?;
assert!(
run(&[
@ -513,6 +501,18 @@ fn prepare_signer(root: &Path) -> anyhow::Result<()> {
Ok(())
}
fn build_legacy(root: &Path, database: &Path, output: &Path) -> anyhow::Result<Value> {
run(&[
"build",
"--database",
text(database)?,
"--output",
text(output)?,
"--policy",
text(&root.join("legacy-policy.json"))?,
])
}
fn sign_review(root: &Path, decision: &Path) -> anyhow::Result<std::path::PathBuf> {
let status = Command::new("ssh-keygen")
.args([

7
crates/argand-site-registry/tests/common/mod.rs
View file

@ -76,6 +76,7 @@ pub fn manifest(source: Source, format: Format, bytes: &[u8]) -> anyhow::Result<
compression: Compression::None,
snapshot: "synthetic-fixture-v1".into(),
scope: "fixture".into(),
coverage: None,
source_url: source_url.into(),
license: source.license().into(),
license_url: source.license_url().into(),
@ -135,7 +136,11 @@ pub fn fixture(root: &Path) -> anyhow::Result<rusqlite::Connection> {
pub fn build(db: &rusqlite::Connection, root: &Path, name: &str) -> anyhow::Result<Registry> {
let generation = root.join(name);
argand_site_registry::build::build(db, &generation)?;
argand_site_registry::build::build_with_policy(
db,
&generation,
&argand_site_registry::policy::ReviewPolicy::legacy_compatible(),
)?;
Registry::open(
&generation,
&argand_site_registry::file_digest(&generation.join("COMPLETE.json"))?,

51
crates/argand-site-registry/tests/compatibility.rs Normal file
View file

@ -0,0 +1,51 @@
// By Nic Weyand!
//! Frozen v0.3 public-contract coordinates used by migration and rollback tests.
#[test]
fn v03_contract_golden_is_explicit_and_unchanged() -> anyhow::Result<()> {
let contract: serde_json::Value =
serde_json::from_str(include_str!("fixtures/v03-contract.json"))?;
assert_eq!(contract["crate_version"], "0.3.0");
assert_eq!(
contract["signed_commit"],
"ac8282093d8a815c6227cff86e1f40714d510bcd"
);
assert_eq!(contract["writer_schema"], 3);
assert_eq!(contract["rules"], "argand.site-rules/v3");
assert_eq!(contract["source_manifest_schema"], "argand.site-source/v1");
assert_eq!(
contract["generation_receipt_schema"],
"argand.site-registry/v1"
);
assert_eq!(contract["export_schema"], "argand.site-export/v1");
assert_eq!(contract["current_pointer_schema"], "argand.site-current/v1");
assert_eq!(contract["diff_schema"], "argand.site-diff/v3");
assert_eq!(contract["selection_schema"], "argand.site-selection/v1");
assert_eq!(
contract["review_signature_namespace"],
argand_site_registry::review::SIGNATURE_NAMESPACE
);
assert_eq!(
contract["release_signature_namespace"],
"argand-site-registry"
);
assert_eq!(
contract["receipt_fields"].as_array().map(Vec::len),
Some(11)
);
assert_eq!(contract["lookup_fields"].as_array().map(Vec::len), Some(6));
assert_eq!(
contract["candidate_fields"].as_array().map(Vec::len),
Some(14)
);
assert_eq!(
contract["resolution_fields"].as_array().map(Vec::len),
Some(5)
);
assert_eq!(
contract["resolution_statuses"].as_array().map(Vec::len),
Some(8)
);
assert_eq!(contract["review_fields"].as_array().map(Vec::len), Some(10));
Ok(())
}

133
crates/argand-site-registry/tests/coverage.rs Normal file
View file

@ -0,0 +1,133 @@
// By Nic Weyand!
//! Active coverage, delta masking, and audit-retention acceptance proof.
#[allow(dead_code)]
mod common;
use argand_site_registry::{
model::{CoverageKind, Format, Source, SourceCoverage},
query::Registry,
store,
};
use serde_json::json;
use std::fs;
#[test]
#[allow(clippy::too_many_lines)] // One scenario proves full, delta, tombstone, active, and audit behavior.
fn typed_full_and_delta_replace_active_records_but_preserve_audit_history() -> anyhow::Result<()> {
let root = tempfile::tempdir()?;
let mut db = common::fixture(root.path())?;
let original_bytes = serde_json::to_vec(&common::wikidata())?;
let original = common::manifest(Source::Wikidata, Format::WikidataEntities, &original_bytes)?;
let full_bytes = serde_json::to_vec(&common::wikidata())?;
let mut full = common::manifest(Source::Wikidata, Format::WikidataEntities, &full_bytes)?;
full.schema = "argand.site-source/v2".into();
full.snapshot = "synthetic-full-v2".into();
full.scope = "entities-full".into();
full.retrieved_at += chrono::Duration::hours(1);
full.coverage = Some(SourceCoverage {
collection: "entities".into(),
kind: CoverageKind::Full,
partition: None,
base: None,
sequence: None,
supersedes: vec![original.id()?],
});
let full_path = root.path().join("wikidata-full.json");
fs::write(&full_path, &full_bytes)?;
store::import(&mut db, &full, &full_path)?;
let delta_bytes = serde_json::to_vec(&json!({"entities":{"Q355":common::entity(
"Q355",
"Facebook",
&["Meta FB"],
&["https://facebook.com/"]
)}}))?;
let mut delta = common::manifest(Source::Wikidata, Format::WikidataEntities, &delta_bytes)?;
delta.schema = "argand.site-source/v2".into();
delta.snapshot = "synthetic-delta-v2".into();
delta.scope = "entities-delta-1".into();
delta.retrieved_at += chrono::Duration::hours(2);
delta.coverage = Some(SourceCoverage {
collection: "entities".into(),
kind: CoverageKind::Delta,
partition: None,
base: Some(full.id()?),
sequence: Some(1),
supersedes: vec![full.id()?],
});
let delta_path = root.path().join("wikidata-delta.json");
fs::write(&delta_path, &delta_bytes)?;
store::import(&mut db, &delta, &delta_path)?;
let generation = root.path().join("generation");
argand_site_registry::build::build_with_policy(
&db,
&generation,
&argand_site_registry::policy::ReviewPolicy::legacy_compatible(),
)?;
let pin = argand_site_registry::file_digest(&generation.join("COMPLETE.json"))?;
let registry = Registry::open(&generation, &pin)?;
assert_eq!(registry.lookup("FB", 20)?.total_entities, 0);
assert_eq!(registry.lookup("Meta FB", 20)?.total_entities, 1);
assert_eq!(registry.lookup("Atlas", 20)?.total_entities, 1);
let stats = registry.stats(common::timestamp()?)?;
assert_eq!(stats.selected_sources, 6);
assert_eq!(stats.superseded_source_snapshots, 1);
let active = root.path().join("active.jsonl");
let audit = root.path().join("audit.jsonl");
argand_site_registry::release::export(&registry, &active, false)?;
argand_site_registry::release::export_audit(&registry, &audit, false)?;
let active = fs::read_to_string(active)?;
let audit = fs::read_to_string(audit)?;
assert!(!active.contains("\"text\":\"FB\""));
assert!(active.contains("\"text\":\"Meta FB\""));
assert!(audit.contains("\"selection_state\":\"superseded\""));
assert!(audit.contains("\"text\":\"FB\""));
let tombstone_bytes = serde_json::to_vec(&json!({"entities":{"Q355":{
"id":"Q355",
"missing":""
}}}))?;
let mut tombstone =
common::manifest(Source::Wikidata, Format::WikidataEntities, &tombstone_bytes)?;
tombstone.schema = "argand.site-source/v2".into();
tombstone.snapshot = "synthetic-delta-v2-tombstone".into();
tombstone.scope = "entities-delta-2".into();
tombstone.retrieved_at += chrono::Duration::hours(3);
tombstone.coverage = Some(SourceCoverage {
collection: "entities".into(),
kind: CoverageKind::Delta,
partition: None,
base: Some(delta.id()?),
sequence: Some(2),
supersedes: vec![delta.id()?],
});
let tombstone_path = root.path().join("wikidata-tombstone.json");
fs::write(&tombstone_path, &tombstone_bytes)?;
store::import(&mut db, &tombstone, &tombstone_path)?;
let retired_path = root.path().join("retired");
argand_site_registry::build::build_with_policy(
&db,
&retired_path,
&argand_site_registry::policy::ReviewPolicy::legacy_compatible(),
)?;
let retired_pin = argand_site_registry::file_digest(&retired_path.join("COMPLETE.json"))?;
let retired = Registry::open(&retired_path, &retired_pin)?;
assert_eq!(retired.lookup("Facebook", 20)?.total_entities, 0);
assert_eq!(retired.lookup("Meta FB", 20)?.total_entities, 0);
let retired_active = root.path().join("retired-active.jsonl");
let retired_audit = root.path().join("retired-audit.jsonl");
argand_site_registry::release::export(&retired, &retired_active, false)?;
argand_site_registry::release::export_audit(&retired, &retired_audit, false)?;
assert!(
!fs::read_to_string(retired_active)?
.contains("\"subject\":\"argand:entity:wikidata:Q355\"")
);
let retired_audit = fs::read_to_string(retired_audit)?;
assert!(retired_audit.contains("\"type\":\"tombstone\""));
assert!(retired_audit.contains("\"selection_state\":\"tombstoned\""));
assert!(retired_audit.contains("\"source_identifier\":\"Q355\""));
Ok(())
}

45
crates/argand-site-registry/tests/failures.rs
View file

@ -23,7 +23,7 @@ fn version_one_store_migrates_without_losing_review_history() -> anyhow::Result<
let migrated = store::open(&path)?;
assert_eq!(
migrated.query_row("PRAGMA user_version", [], |row| row.get::<_, i64>(0))?,
3
5
);
assert_eq!(
migrated.query_row("SELECT rules FROM registry_metadata", [], |row| row
@ -43,6 +43,40 @@ fn version_one_store_migrates_without_losing_review_history() -> anyhow::Result<
Ok(())
}
#[test]
fn every_prior_writer_schema_migrates_to_v04() -> anyhow::Result<()> {
let root = tempfile::tempdir()?;
for version in 1..=4 {
let path = root.path().join(format!("v{version}.sqlite"));
let db = rusqlite::Connection::open(&path)?;
db.execute_batch(include_str!("../migrations/001.sql"))?;
if version >= 2 {
db.execute_batch(include_str!("../migrations/002.sql"))?;
}
if version >= 3 {
db.execute_batch(include_str!("../migrations/003.sql"))?;
}
if version >= 4 {
db.execute_batch(include_str!("../migrations/004.sql"))?;
}
drop(db);
let migrated = store::open(&path)?;
assert_eq!(
migrated.query_row("PRAGMA user_version", [], |row| row.get::<_, i64>(0))?,
5
);
for table in ["votes", "vote_auth", "observation_batches", "observations"] {
let exists: bool = migrated.query_row(
"SELECT EXISTS(SELECT 1 FROM sqlite_schema WHERE type='table' AND name=?1)",
[table],
|row| row.get(0),
)?;
assert!(exists, "{table} missing after schema {version} migration");
}
}
Ok(())
}
#[test]
fn externally_signed_unauthenticated_reviews_are_rejected() -> anyhow::Result<()> {
let root = tempfile::tempdir()?;
@ -236,6 +270,7 @@ fn activation_accepts_a_pinned_v1_previous_generation() -> anyhow::Result<()> {
drop(old);
let receipt_path = root.path().join("legacy-current/COMPLETE.json");
let mut receipt: serde_json::Value = argand_site_registry::read_json(&receipt_path)?;
receipt["schema"] = json!("argand.site-registry/v1");
receipt["rules"] = json!("argand.site-rules/v1");
fs::write(&receipt_path, serde_json::to_vec_pretty(&receipt)?)?;
let old_pin = argand_site_registry::file_digest(&receipt_path)?;
@ -597,6 +632,11 @@ async fn repeated_update_reuses_generation_and_failure_preserves_it() -> anyhow:
)?;
inputs.push(CachedSource { input, manifest });
}
let review_policy = root.path().join("policy.json");
fs::write(
&review_policy,
serde_json::to_vec(&argand_site_registry::policy::ReviewPolicy::legacy_compatible())?,
)?;
let config = argand_site_registry::update::Config {
cache: root.path().join("cache"),
database: root.path().join("data.sqlite"),
@ -604,6 +644,9 @@ async fn repeated_update_reuses_generation_and_failure_preserves_it() -> anyhow:
downloads: vec![],
inputs,
crux: vec![],
review_policy: Some(review_policy),
reviewer_trust: None,
auto_supersede_typed_snapshots: false,
};
let first = argand_site_registry::update::run(&config).await?;
let second = argand_site_registry::update::run(&config).await?;

21
crates/argand-site-registry/tests/fixtures/v03-contract.json vendored Normal file
View file

@ -0,0 +1,21 @@
{
"crate_version": "0.3.0",
"signed_commit": "ac8282093d8a815c6227cff86e1f40714d510bcd",
"writer_schema": 3,
"rules": "argand.site-rules/v3",
"source_manifest_schema": "argand.site-source/v1",
"generation_receipt_schema": "argand.site-registry/v1",
"export_schema": "argand.site-export/v1",
"current_pointer_schema": "argand.site-current/v1",
"diff_schema": "argand.site-diff/v3",
"selection_schema": "argand.site-selection/v1",
"review_signature_namespace": "argand-site-registry-review",
"release_signature_namespace": "argand-site-registry",
"generation_files": ["ATTRIBUTION.json", "COMPLETE.json", "LICENSE_SOURCES.md", "registry.sqlite"],
"receipt_fields": ["schema", "rules", "database_sha256", "licenses_sha256", "attribution_sha256", "psl_source", "sources", "entities", "properties", "edges", "rejected"],
"lookup_fields": ["query", "total_entities", "total_edges", "truncated", "candidates", "attribution"],
"candidate_fields": ["identity_provenance", "entity", "entity_id", "canonical_name", "url", "web_property", "property_scopes", "relation", "confidence", "fingerprint", "evidence", "provenance", "review", "eligible"],
"resolution_fields": ["query", "status", "destination", "counts", "attribution"],
"resolution_statuses": ["resolved", "no_name_match", "ambiguous_identity", "safety_limit_exceeded", "no_eligible_destination", "no_active_review", "region_mismatch", "ambiguous_destination"],
"review_fields": ["fingerprint", "decision", "reviewer", "reason", "evidence", "reviewed_at", "expires_at", "role", "locale", "country"]
}

7
crates/argand-site-registry/tests/identity.rs
View file

@ -173,7 +173,7 @@ fn name_collision_remains_ambiguous_until_review_and_changes_invalidate_link() -
}
#[test]
fn metadata_changes_invalidate_identity_fingerprint() -> anyhow::Result<()> {
fn metadata_changes_preserve_material_identity_and_refresh_evidence() -> anyhow::Result<()> {
let root = tempfile::tempdir()?;
let mut db = common::fixture(root.path())?;
let baseline = common::build(&db, root.path(), "metadata-baseline")?;
@ -182,6 +182,7 @@ fn metadata_changes_invalidate_identity_fingerprint() -> anyhow::Result<()> {
.entity_id
.clone();
let before = identity::propose(&baseline, &wiki, &curlie)?;
let before_bundle = argand_site_registry::bundle::equivalence(&baseline, &before)?;
let mut entity = common::entity("Q355", "Facebook", &["FB"], &["https://facebook.com/"]);
entity["claims"]["P17"] = json!([{"id":"Q355$country","rank":"normal","mainsnak":{"property":"P17","snaktype":"value","datavalue":{"type":"wikibase-entityid","value":{"id":"Q30"}}}}]);
@ -193,6 +194,8 @@ fn metadata_changes_invalidate_identity_fingerprint() -> anyhow::Result<()> {
store::import(&mut db, &source, &input)?;
let changed = common::build(&db, root.path(), "metadata-changed")?;
let after = identity::propose(&changed, &wiki, &curlie)?;
assert_ne!(before.fingerprint, after.fingerprint);
let after_bundle = argand_site_registry::bundle::equivalence(&changed, &after)?;
assert_eq!(before.fingerprint, after.fingerprint);
assert_ne!(before_bundle.id, after_bundle.id);
Ok(())
}

8
crates/argand-site-registry/tests/registry.rs
View file

@ -255,7 +255,7 @@ fn ambiguity_survives_limits_and_same_named_domains_do_not_merge() -> anyhow::Re
}
#[test]
fn changed_names_and_urls_invalidate_approval_but_history_survives() -> anyhow::Result<()> {
fn added_alias_preserves_edge_approval_and_history() -> anyhow::Result<()> {
let dir = tempfile::tempdir()?;
let mut db = fixture(dir.path())?;
let initial = build(&db, dir.path(), "initial")?;
@ -280,14 +280,16 @@ fn changed_names_and_urls_invalidate_approval_but_history_survives() -> anyhow::
std::fs::write(&path, bytes)?;
store::import(&mut db, &m, &path)?;
let r = build(&db, dir.path(), "changed")?;
assert!(
assert_eq!(
r.resolve(
"Facebook",
None,
None,
timestamp()? + chrono::Duration::days(1)
)?
.is_none()
.context("granular website approval survives an unrelated alias")?
.url,
"https://facebook.com/"
);
let sources: i64 = db.query_row(
"SELECT count(*) FROM sources WHERE source='wikidata' AND complete=1",

1109
crates/argand-site-registry/tests/v04.rs Normal file
View file

File diff suppressed because it is too large Load diff