argand-site-registry/crates/argand-site-registry

Argand Site Registry

A Rust library and CLI for an entity ↔ website/domain dataset. SQLite stores source assertions separately and builds indexed, immutable registry generations. The code follows the engine workspace's AGPL-3.0-or-later license; see the GNU AGPL. Data licenses and required credits are in LICENSE_SOURCES.md.

facebook → Facebook → facebook.com is a name-to-entity-to-registrable-domain lookup. The actual retained Wikidata destination is https://www.facebook.com/; normalization does not silently replace it with an apex URL. A mobile website is a separate property. The current retained Amazon entity, Q3884, has 13 P856 properties, including amazon.com, amazon.co.uk, and amazon.de.

Imported assertions enter the review queue. Only explicit, unexpired reviews can produce a resolve destination. Automatic updates build candidates; signing and activation are separate operator actions. This crate is a dataset component; existing Argand Navigate policy and collection admission still apply when a consumer integrates it into public search.

Install and run the offline acceptance example

Rust 1.97+ and OpenSSH (ssh-keygen) are required. From the standalone repository root:

cargo install --path crates/argand-site-registry --locked
argand-site-registry --help
cargo test -p argand-site-registry --all-targets --locked --offline

The native CLI tests import small source-shaped fixtures for all six sources, repeats the imports, resolves aliases, signs and activates an approved generation, revokes the destination, and rejects rollback past the revocation. Synthetic fixtures are authored in Rust test code; no provider datasets or signing keys are committed. To retain a local example for inspection, choose a new path:

ARGAND_REGISTRY_E2E_OUTPUT=/tmp/argand-site-registry-example \
  cargo test -p argand-site-registry --test cli --locked --offline -- --nocapture

The example contains candidate/, approved/, revoked/, JSON manifests, an attributed export and a disposable test key. Do not use that test key or those synthetic approvals for a real release.

Acquire and import sources

All paths are explicit. These commands use jq only to read CLI JSON output. They create data outside the checkout. Byte caps are upper bounds, not estimates of current source sizes. Increase a cap only after checking available storage.

export ARGAND_SITE_DATA="$HOME/.local/share/argand-site-registry"
mkdir -p "$ARGAND_SITE_DATA"

cat > "$ARGAND_SITE_DATA/full-coverage.json" <<'JSON'
{"collection":"default","kind":"full","partition":null,"base":null,"sequence":null,"supersedes":[]}
JSON
cat > "$ARGAND_SITE_DATA/wikidata-selection-coverage.json" <<'JSON'
{"collection":"entity-selections","kind":"partition","partition":"facebook-amazon","base":null,"sequence":null,"supersedes":[]}
JSON

argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
  --source psl --format psl-text \
  --url https://publicsuffix.org/list/public_suffix_list.dat \
  --snapshot "$(date -u +%F)" --scope full --maximum-bytes 1000000 \
  --coverage "$ARGAND_SITE_DATA/full-coverage.json" \
  > "$ARGAND_SITE_DATA/psl-download.json"

argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
  --source wikidata --format wikidata-entities \
  --url 'https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q355%7CQ3884&format=json&maxlag=5' \
  --snapshot "$(date -u +%F)" --scope selection:facebook-amazon \
  --coverage "$ARGAND_SITE_DATA/wikidata-selection-coverage.json" \
  --maximum-bytes 5000000 > "$ARGAND_SITE_DATA/wikidata-download.json"

argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
  --source majestic --format majestic-csv \
  --url https://downloads.majestic.com/majestic_million.csv \
  --snapshot "$(date -u +%F)" --scope full --maximum-bytes 250000000 \
  --coverage "$ARGAND_SITE_DATA/full-coverage.json" \
  > "$ARGAND_SITE_DATA/majestic-download.json"

argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
  --source curlie --format curlie-tar-gz \
  --url https://curlie.org/directory-dl \
  --snapshot "$(date -u +%F)" --scope full --maximum-bytes 1000000000 \
  --coverage "$ARGAND_SITE_DATA/full-coverage.json" \
  > "$ARGAND_SITE_DATA/curlie-download.json"

for source in psl wikidata majestic curlie; do
  argand-site-registry import --database "$ARGAND_SITE_DATA/import.sqlite" \
    --input "$(jq -r .input "$ARGAND_SITE_DATA/$source-download.json")" \
    --manifest "$(jq -r .manifest "$ARGAND_SITE_DATA/$source-download.json")"
done

The reusable default collection is safe because coverage graphs are separated by provider. For a replacement, copy the preceding manifest ID into the new coverage object's supersedes array. For a delta, also set kind: "delta", base to that ID, and a consecutive positive sequence. Use distinct stable partition names only for provider-declared disjoint subsets. The build rejects ambiguous or overlapping typed coverage.

For a full Wikidata dump, select a real dump URL from the official download index, then use --format wikidata-dump --compression gzip (or bzip2) and --scope full. The parser streams the documented top-level JSON array without assuming that an entity is physically one line, and handles concatenated compressed streams. It rejects a record above the authenticated 16 MiB default; after inspecting the exact object, a publisher can bind a larger limit with --maximum-record-bytes on download or manifest, up to the 256 MiB audit-safe ceiling. Do not use truthy RDF: it loses statement evidence. Full dumps need substantial disk space and a long sequential scan even though memory is bounded. A small entity selection is useful on limited hardware.

Wikidata's current Add/Change artifacts are XML history/stub streams. Wikidata's own download guidance warns that JSON embedded in XML dumps is unstable, so this release refuses that format. For bounded incremental refresh, acquire official Wikibase API wbgetentities JSON, declare typed delta coverage against the exact selected-entity base and apply consecutive sequences. A changed entity with no P856 statements emits an empty record, which becomes a website-evidence tombstone in the audit export. Reconcile API deltas against a later full JSON dump before replacing broad production coverage.

ROR

ROR publishes versioned CC0 dumps through Zenodo. The following dated example is the exact asset inspected for version 0.5; check the current ROR release page before acquiring a newer snapshot and use the checksum published by that Zenodo record.

argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
  --source ror --format ror-zip \
  --url https://zenodo.org/api/records/22099990/files/v2.12-2026-08-25-ror-data.zip/content \
  --snapshot v2.12-2026-08-25 --scope full --maximum-bytes 100000000 \
  --coverage "$ARGAND_SITE_DATA/full-coverage.json" \
  --provider-checksum md5:ce8807691455d4ada3216c31408e9e1a \
  --provider-checksum-url https://zenodo.org/api/records/22099990 \
  > "$ARGAND_SITE_DATA/ror-download.json"
argand-site-registry import --database "$ARGAND_SITE_DATA/import.sqlite" \
  --input "$(jq -r .input "$ARGAND_SITE_DATA/ror-download.json")" \
  --manifest "$(jq -r .manifest "$ARGAND_SITE_DATA/ror-download.json")"

The adapter accepts the official schema 2.1 JSON plus CSV ZIP, consumes only the documented fields, and fails on extra members or schema drift. ROR entities remain separate from Wikidata unless reviewed equivalence evidence joins exact stable IDs. Inactive and withdrawn organizations remain audit evidence but their website edges are ineligible for resolution. GeoNames is declared as upstream lineage for location metadata and is included in attribution output.

To reuse an already acquired file, retain its original retrieval time, source URL and snapshot/revision. First verify its acquisition receipt, then:

argand-site-registry manifest --input /data/Q355.json \
  --output /data/Q355.source.json --source wikidata --format wikidata-entities \
  --source-url https://www.wikidata.org/wiki/Special:EntityData/Q355.json \
  --snapshot retained-Q355-revision --scope selection:Q355 \
  --retrieved-at 2026-09-10T13:40:20.446514Z
argand-site-registry import --database "$ARGAND_SITE_DATA/import.sqlite" \
  --input /data/Q355.json --manifest /data/Q355.source.json

Replace paths, snapshot and time with the actual acquisition details. A manifest declares provenance; making one does not authenticate arbitrary file contents.

CrUX

The adapter queries the documented monthly BigQuery table and streams paginated results into this exact CSV projection:

SELECT DISTINCT origin, experimental.popularity.rank AS rank,
  '202608' AS yyyymm, '' AS country_code
FROM `chrome-ux-report.all.202608`
WHERE experimental.popularity.rank IS NOT NULL
ORDER BY origin, rank

The month above is an example of the documented table naming. Confirm that the desired month exists. For an audience-country dataset, use country: "GB" in the request; the adapter selects chrome-ux-report.country_gb.202608 and emits GB. The rank is a bucket; do not mix it numerically with Majestic's exact rank.

Create crux-request.json with your project and explicit limits:

{
  "project": "your-billing-project",
  "month": "202608",
  "country": null,
  "maximum_bytes_billed": 1000000000,
  "maximum_output_bytes": 500000000,
  "coverage": {
    "collection": "monthly-origins",
    "kind": "partition",
    "partition": "global",
    "base": null,
    "sequence": null,
    "supersedes": []
  }
}

Supply an authorized OAuth access token through GOOGLE_OAUTH_ACCESS_TOKEN using your credential manager, then run:

argand-site-registry crux-download --cache "$ARGAND_SITE_DATA/cache" \
  --request crux-request.json > "$ARGAND_SITE_DATA/crux-download.json"
argand-site-registry import --database "$ARGAND_SITE_DATA/import.sqlite" \
  --input "$(jq -r .input "$ARGAND_SITE_DATA/crux-download.json")" \
  --manifest "$(jq -r .manifest "$ARGAND_SITE_DATA/crux-download.json")"

No default billing project or unbounded query is provided. An interrupted job reuses its content-derived BigQuery job ID; result pages replay from the same query result. Keep job.json with the acquisition records. Expired server results require an operator to inspect the existing job. Pinned local exports of the exact CSV projection can instead use manifest --source crux --format crux-csv --source-url https://developer.chrome.com/docs/crux/bigquery/ with their actual retrieval time, query/snapshot identity and appropriate monthly:YYYYMM:country scope. The token is never written into a manifest.

Keep a partition coordinate stable across refreshes, such as global or GB. The month belongs in the source snapshot identity. With scheduled typed supersession enabled, the next month then replaces the same audience partition instead of accumulating stale popularity facts.

Common Crawl Web Graph

The adapter accepts the official domain-level rank object whose exact header is:

#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts

It reverses host_rev (com.facebook to facebook.com) and retains harmonic centrality, PageRank and n_hosts as source-separated popularity evidence. n_hosts means hosts belonging to the registered domain and is exposed as member_hosts; it is not an inbound-link count. The adapter creates no entity, name, official-site edge, review, vote or route.

Only exact domain-rank objects below the official data.commoncrawl.org/projects/hyperlinkgraph/<release>/domain/ hierarchy are allowlisted. Use compression: gzip, the exact release ID as the snapshot, a positive byte ceiling, and typed coverage. The downloader and importer hash and consume the complete object; a byte-range prefix must not be declared as the complete source. Common Crawl's Terms of Use are not an SPDX open-data license, so preserve the Terms link and re-review it on each acquisition.

Domain-rank files contain tens of millions of rows. Import the PSL and public identity sources first, then run web-graph-selection --database ... to obtain the exact candidate-domains:<sha256> scope. A scheduled update may instead use the {candidate_domains} token. The importer still parses and authenticates the complete stream but persists only matching registrable domains, with original row coordinates. This keeps the public updater reproducible and compact. Rank still cannot replace reviewer quorum or destination-safety checks.

Build, inspect and review

The strict default requires an independently maintained OpenSSH reviewer trust file and two independent approvals for each name, edge, and equivalence. Build a candidate and inspect its deterministic work queue:

argand-site-registry build --database "$ARGAND_SITE_DATA/import.sqlite" \
  --output "$ARGAND_SITE_DATA/candidate" \
  --audit-store "$ARGAND_SITE_DATA/audit-objects" \
  --reviewer-trust /secure/reviewer-allowed-signers \
  > "$ARGAND_SITE_DATA/candidate.json"
export ARGAND_SITE_PIN="$(jq -r .pin "$ARGAND_SITE_DATA/candidate.json")"

argand-site-registry lookup --generation "$ARGAND_SITE_DATA/candidate" \
  --pin "$ARGAND_SITE_PIN" --query facebook
argand-site-registry review-queue --generation "$ARGAND_SITE_DATA/candidate" \
  --pin "$ARGAND_SITE_PIN" --at 2026-09-13T00:00:00Z

A source-shaped Facebook result retains https://www.facebook.com/, its Wikidata Q355 identity, the facebook.com registrable domain, all relevant source facts, and source-separated popularity. An entity can carry example.com, example.co.uk, and example.de only when source evidence attaches each property to that exact entity. PSL parsing returns example.co.uk, not co.uk, as the registrable domain.

Prepare canonical vote JSON for the exact queued name or edge. The command fills the current evidence-bundle and policy digests. Never hand-copy an earlier digest.

argand-site-registry prepare-vote \
  --generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
  --subject-kind edge --fingerprint "$EDGE_FINGERPRINT" \
  --decision approve --reviewer reviewer-one \
  --reason "Verified entity, URL, and exact role from retained evidence" \
  --reviewed-at 2026-09-13T00:00:00Z \
  --expires-at 2026-10-13T00:00:00Z --role primary \
  --output /secure/edge-vote.json
ssh-keygen -Y sign -n argand-site-registry-vote \
  -f /secure/reviewer-one /secure/edge-vote.json
argand-site-registry verify-vote \
  --generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
  --decision /secure/edge-vote.json --signature /secure/edge-vote.json.sig \
  --allowed-reviewers /secure/reviewer-allowed-signers --identity reviewer-one
argand-site-registry vote --database "$ARGAND_SITE_DATA/import.sqlite" \
  --generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
  --decision /secure/edge-vote.json --signature /secure/edge-vote.json.sig \
  --allowed-reviewers /secure/reviewer-allowed-signers --identity reviewer-one

Repeat with another identity, group, and physical key. Prepare separate name votes for the exact label or alias used by the query. A regional edge approval uses --role regional plus --country GB, --locale en-GB, or both. A primary edge is an unscoped global fallback. If both country and locale are set, both must match. The resolver abstains on missing quorum, ambiguity, expiry, revocation, stale evidence, stale policy, or equal destinations.

The writer assigns accepted_at. Effective validity begins at the later of that time and signed reviewed_at, and ends no later than 90 days after acceptance. A revocation has no expiry. Every approval in a new quorum must pass each active revocation ID with --supersedes; ordinary later approvals remain blocked.

Observe an existing candidate

The observer is candidate-only and does not grant approval. It pins public DNS per hop, rejects private/link-local addresses, credentials, nondefault ports, HTTPS downgrade, compressed response bodies, oversized headers/bodies and more than five redirects. Store its cache outside Git, replay it without network access, import the immutable batch, and rebuild:

argand-site-registry observe \
  --generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
  --fingerprint "$EDGE_FINGERPRINT" \
  --capture "$ARGAND_SITE_DATA/captures/run-1" \
  --output "$ARGAND_SITE_DATA/observations/run-1.jsonl" \
  --manifest-output "$ARGAND_SITE_DATA/observations/run-1.source.json"
argand-site-registry observation-import \
  --database "$ARGAND_SITE_DATA/import.sqlite" \
  --generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
  --input "$ARGAND_SITE_DATA/observations/run-1.jsonl" \
  --manifest "$ARGAND_SITE_DATA/observations/run-1.source.json"

observe-replay reproduces JSONL from the cache without a request. observations looks up an exact subject, observation-lookup searches exact URLs or domains, and drift compares the latest two batches. A new observation changes the evidence bundle and requires fresh review; it never auto-renews a vote. For scheduled runs, {timestamp} in the three observe output paths expands once to a nanosecond UTC coordinate. The template service and timer in examples/ serialize enabled fingerprints with a runtime lock, cap response bandwidth, and run one candidate per process; operators still import and review the produced batch separately.

Release, export, update and recovery

Preview an entity pair, then use the dedicated equivalence vote commands. Two similar names or domains remain separate until the equivalence quorum passes:

argand-site-registry equivalence --generation "$ARGAND_SITE_DATA/candidate" \
  --pin "$ARGAND_SITE_PIN" --left SOURCE_ENTITY_ID --right OTHER_ENTITY_ID
argand-site-registry prepare-equivalence-vote \
  --generation "$ARGAND_SITE_DATA/candidate" --pin "$ARGAND_SITE_PIN" \
  --left SOURCE_ENTITY_ID --right OTHER_ENTITY_ID --decision approve \
  --reviewer reviewer-one --reason "Same entity under both source IDs" \
  --reviewed-at 2026-09-13T00:00:00Z \
  --expires-at 2026-10-13T00:00:00Z --output /secure/equivalence-vote.json

After appending complete quorums, rebuild and inspect stats, diff, review-queue, and evaluate. export writes selected, nonrejected facts. export-audit also writes superseded and rejected facts. Both preserve provenance and attribution, redact Curlie descriptions by default, and are evidence exports rather than admitted-route lists.

argand-site-registry export --generation "$ARGAND_SITE_DATA/reviewed" \
  --pin "$REVIEWED_PIN" --output "$ARGAND_SITE_DATA/active.jsonl"
argand-site-registry export-audit --generation "$ARGAND_SITE_DATA/reviewed" \
  --pin "$REVIEWED_PIN" --audit-store "$ARGAND_SITE_DATA/audit-objects" \
  --output "$ARGAND_SITE_DATA/audit.jsonl"
argand-site-registry verify-audit --generation "$ARGAND_SITE_DATA/reviewed" \
  --pin "$REVIEWED_PIN" --audit-store "$ARGAND_SITE_DATA/audit-objects"
argand-site-registry audit-checkpoint \
  --generation "$ARGAND_SITE_DATA/reviewed" --pin "$REVIEWED_PIN" \
  --audit-store "$ARGAND_SITE_DATA/audit-objects" \
  --recorded-at 2026-09-13T00:00:00Z \
  --output "$ARGAND_SITE_DATA/audit-retention.json"
argand-site-registry sign-audit-checkpoint \
  --input "$ARGAND_SITE_DATA/audit-retention.json" \
  --signature "$ARGAND_SITE_DATA/audit-retention.json.sig" \
  --key /secure/audit-retention-key
argand-site-registry verify-audit-checkpoint \
  --input "$ARGAND_SITE_DATA/audit-retention.json" \
  --signature "$ARGAND_SITE_DATA/audit-retention.json.sig" \
  --allowed-signers /secure/audit-retention-allowed-signers \
  --identity registry-retention
argand-site-registry sign --generation "$ARGAND_SITE_DATA/reviewed" \
  --pin "$REVIEWED_PIN" --key /secure/publisher-key \
  --allowed-reviewers /secure/reviewer-allowed-signers \
  --identity registry-publisher
argand-site-registry activate --generation "$ARGAND_SITE_DATA/reviewed" \
  --current "$ARGAND_SITE_DATA/current.json" \
  --allowed-signers /secure/publisher-allowed-signers \
  --allowed-reviewers /secure/reviewer-allowed-signers \
  --identity registry-publisher

Strict signing rejects a publisher identity or physical key used for any reviewer vote. Activation re-verifies the publisher, reviewer trust digest, every retained vote, and revocation continuity. Rollback is allowed only when the target retains every distributed legacy and vote revocation.

For emergency delivery, export-revocations creates a cumulative feed at an explicit time. sign-revocations recomputes it from the pinned generation before using the publisher's separate SSH key. verify-revocations checks its signature, compatibility, effective time, seven-day refresh deadline, and optional prior-feed continuity. Cross-generation feeds can add blocks but cannot restore a route. A consumer may pass --revocations, --revocation-signature, --allowed-publishers, and --publisher-identity to resolve so the signed block applies before a full replacement generation is installed. After bootstrap, also pass the last accepted feed through --previous-revocations and --previous-revocation-signature to reject a replacement that drops retained revocations.

The update configuration and systemd examples refresh candidates without a resident daemon. Its explicit auto_supersede_typed_snapshots = true setting replaces only the current frontier with the same provider, collection and full/partition coordinate; deltas and coverage-layout changes still require exact operator-supplied bases. Update jobs may download, import and build. Schedule observation commands independently according to risk. Use separate cache/capture/output paths and process concurrency limits; one observer invocation handles one candidate with explicit body, redirect and time bounds. No scheduled command approves, renews, signs or activates.

Storage and operating limits

Migrations 001 through 005 own writer schema 5. Source manifests v3 authenticate integrity checks, per-record bounds and declared lineage in addition to typed full, partition, or delta coverage. Deltas name an exact base, consecutive sequence and superseded object. Ambiguous coverage, overlap, cycles, gaps, cross-provider supersession and duplicate native records fail the build. V1 manifests remain isolated by provider and scope for compatibility.

Each generation contains registry.sqlite, LICENSE_SOURCES.md, ATTRIBUTION.json, and COMPLETE.json, plus an optional publisher signature. The receipt binds database bytes, source selection, policy, reviewer trust, licenses, attribution, and decision-time contract. A compact v3 receipt also binds every cold audit object by digest, length, counts, source, coverage and attribution. The runtime database omits superseded raw history after the bundle is durable; verify-audit must pass wherever that history is retained. The retention checkpoint authorizes no deletion and uses the separate argand-site-registry-audit-retention SSH namespace. Keep source objects, writer state, generations, pins, trust files and signatures for recovery.

Imports are streaming, transactional, resumable and idempotent. Defaults bound expanded bytes, record size/count, database growth, query output and diff output. Use command-line overrides only after checking the real object and local capacity. Raw datasets, credentials, signing keys and production review logs do not belong in this repository.

lookup, reverse lookup, popularity and categories are audit surfaces. Popularity, TLS, DNS, redirects, sameAs, ccTLD spelling and source confidence do not prove ownership or safety. Use resolve for navigation and keep a null destination as an intentional abstention. See TRUST.md, PUBLISHING.md, and LICENSE_SOURCES.md.