argand-site-registry/crates/argand-site-registry

Argand Site Registry

A Rust library and CLI for an entity ↔ website/domain dataset. SQLite stores source assertions separately and builds indexed, immutable registry generations. The code follows the engine workspace's AGPL-3.0-or-later license; see the GNU AGPL. Data licenses and required credits are in LICENSE_SOURCES.md.

facebook → Facebook → facebook.com is a name-to-entity-to-registrable-domain lookup. The actual retained Wikidata destination is https://www.facebook.com/; normalization does not silently replace it with an apex URL. A mobile website is a separate property. The current retained Amazon entity, Q3884, has 13 P856 properties, including amazon.com, amazon.co.uk, and amazon.de.

Imported assertions enter the review queue. Only explicit, unexpired reviews can produce a resolve destination. Automatic updates build candidates; signing and activation are separate operator actions. This crate is a dataset component; existing Argand Navigate policy and collection admission still apply when a consumer integrates it into public search.

Install and run the offline acceptance example

Rust 1.97+ and OpenSSH (ssh-keygen) are required. From the standalone repository root:

cargo install --path crates/argand-site-registry --locked
argand-site-registry --help
cargo test -p argand-site-registry --all-targets --locked --offline

The native CLI test imports small source-shaped fixtures for all five sources, repeats the imports, resolves aliases, signs and activates an approved generation, revokes the destination, and rejects rollback past the revocation. Synthetic fixtures are authored in Rust test code; no provider datasets or signing keys are committed. To retain a local example for inspection, choose a new path:

ARGAND_REGISTRY_E2E_OUTPUT=/tmp/argand-site-registry-example \
  cargo test -p argand-site-registry --test cli --locked --offline -- --nocapture

The example contains candidate/, approved/, revoked/, JSON manifests, an attributed export and a disposable test key. Do not use that test key or those synthetic approvals for a real release.

Acquire and import sources

All paths are explicit. These commands use jq only to read CLI JSON output. They create data outside the checkout. Byte caps are upper bounds, not estimates of current source sizes. Increase a cap only after checking available storage.

export ARGAND_SITE_DATA="$HOME/.local/share/argand-site-registry"
mkdir -p "$ARGAND_SITE_DATA"

argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
  --source psl --format psl-text \
  --url https://publicsuffix.org/list/public_suffix_list.dat \
  --snapshot "$(date -u +%F)" --scope full --maximum-bytes 1000000 \
  > "$ARGAND_SITE_DATA/psl-download.json"

argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
  --source wikidata --format wikidata-entities \
  --url 'https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q355%7CQ3884&format=json&maxlag=5' \
  --snapshot "$(date -u +%F)" --scope selection:facebook-amazon \
  --maximum-bytes 5000000 > "$ARGAND_SITE_DATA/wikidata-download.json"

argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
  --source majestic --format majestic-csv \
  --url https://downloads.majestic.com/majestic_million.csv \
  --snapshot "$(date -u +%F)" --scope full --maximum-bytes 250000000 \
  > "$ARGAND_SITE_DATA/majestic-download.json"

argand-site-registry download --cache "$ARGAND_SITE_DATA/cache" \
  --source curlie --format curlie-tar-gz \
  --url https://curlie.org/directory-dl \
  --snapshot "$(date -u +%F)" --scope full --maximum-bytes 1000000000 \
  > "$ARGAND_SITE_DATA/curlie-download.json"

for source in psl wikidata majestic curlie; do
  argand-site-registry import --database "$ARGAND_SITE_DATA/import.sqlite" \
    --input "$(jq -r .input "$ARGAND_SITE_DATA/$source-download.json")" \
    --manifest "$(jq -r .manifest "$ARGAND_SITE_DATA/$source-download.json")"
done

For a full Wikidata dump, select a real dump URL from the official download index, then use --format wikidata-dump --compression gzip (or bzip2) and --scope full. The parser handles the documented one-entity-per-line JSON array and concatenated compressed streams. Do not use truthy RDF: it loses statement evidence. Full dumps need substantial disk space and a long sequential scan even though memory is bounded. A small entity selection is useful on limited hardware.

To reuse an already acquired file, retain its original retrieval time, source URL and snapshot/revision. First verify its acquisition receipt, then:

argand-site-registry manifest --input /data/Q355.json \
  --output /data/Q355.source.json --source wikidata --format wikidata-entities \
  --source-url https://www.wikidata.org/wiki/Special:EntityData/Q355.json \
  --snapshot retained-Q355-revision --scope selection:Q355 \
  --retrieved-at 2026-09-10T13:40:20.446514Z
argand-site-registry import --database "$ARGAND_SITE_DATA/import.sqlite" \
  --input /data/Q355.json --manifest /data/Q355.source.json

Replace paths, snapshot and time with the actual acquisition details. A manifest declares provenance; making one does not authenticate arbitrary file contents.

CrUX

The adapter queries the documented monthly BigQuery table and streams paginated results into this exact CSV projection:

SELECT DISTINCT origin, experimental.popularity.rank AS rank,
  '202608' AS yyyymm, '' AS country_code
FROM `chrome-ux-report.all.202608`
WHERE experimental.popularity.rank IS NOT NULL
ORDER BY origin, rank

The month above is an example of the documented table naming. Confirm that the desired month exists. For an audience-country dataset, use country: "GB" in the request; the adapter selects chrome-ux-report.country_gb.202608 and emits GB. The rank is a bucket; do not mix it numerically with Majestic's exact rank.

Create crux-request.json with your project and explicit limits:

{
  "project": "your-billing-project",
  "month": "202608",
  "country": null,
  "maximum_bytes_billed": 1000000000,
  "maximum_output_bytes": 500000000
}

Supply an authorized OAuth access token through GOOGLE_OAUTH_ACCESS_TOKEN using your credential manager, then run:

argand-site-registry crux-download --cache "$ARGAND_SITE_DATA/cache" \
  --request crux-request.json > "$ARGAND_SITE_DATA/crux-download.json"
argand-site-registry import --database "$ARGAND_SITE_DATA/import.sqlite" \
  --input "$(jq -r .input "$ARGAND_SITE_DATA/crux-download.json")" \
  --manifest "$(jq -r .manifest "$ARGAND_SITE_DATA/crux-download.json")"

No default billing project or unbounded query is provided. An interrupted job reuses its content-derived BigQuery job ID; result pages replay from the same query result. Keep job.json with the acquisition records. Expired server results require an operator to inspect the existing job. Pinned local exports of the exact CSV projection can instead use manifest --source crux --format crux-csv --source-url https://developer.chrome.com/docs/crux/bigquery/ with their actual retrieval time, query/snapshot identity and appropriate monthly:YYYYMM:country scope. The token is never written into a manifest.

Build, look up and review

argand-site-registry build --database "$ARGAND_SITE_DATA/import.sqlite" \
  --output "$ARGAND_SITE_DATA/generation-1" > "$ARGAND_SITE_DATA/build-1.json"
export ARGAND_SITE_PIN="$(jq -r .pin "$ARGAND_SITE_DATA/build-1.json")"
argand-site-registry lookup --generation "$ARGAND_SITE_DATA/generation-1" \
  --pin "$ARGAND_SITE_PIN" --query facebook
argand-site-registry lookup --generation "$ARGAND_SITE_DATA/generation-1" \
  --pin "$ARGAND_SITE_PIN" --query amazon --limit 100

The real-source acceptance run produced:

Query	Entity	Example properties	Registrable domains
facebook	Facebook (Q355)	https://www.facebook.com/, https://m.facebook.com/	facebook.com
amazon	Amazon (Q3884)	https://www.amazon.com/, https://www.amazon.co.uk/, https://www.amazon.de/	amazon.com, amazon.co.uk, amazon.de

These properties were asserted on the same Wikidata entity. Hostname resemblance did not establish the relationship. Imported qualifiers remain in evidence and property_scopes; unknown locale/country remains null. Names, aliases and entity metadata carry their own fact-level source declarations. Regional locale/country and role are explicit reviewed assertions. They are separate from entity headquarters, ccTLD spelling and CrUX audience country.

Inspect the full statements, references, names/aliases, hostname spelling and independent current ownership/role evidence. A review JSON has this shape:

{
  "fingerprint": "COPY_THE_EXACT_64_CHARACTER_FINGERPRINT_FROM_LOOKUP",
  "decision": "approve",
  "reviewer": "operator identity",
  "reason": "How entity ownership and this exact destination role were verified",
  "evidence": "An immutable capture identifier or evidence digest",
  "reviewed_at": "2026-09-12T12:00:00Z",
  "expires_at": "2026-10-12T12:00:00Z",
  "role": "regional",
  "locale": "",
  "country": "GB"
}

Replace the example evidence and dates; approvals expire within 90 days. Use role: "primary" for the independently verified default and country: "DE" for a separately verified German regional property. A country-scoped review can leave locale empty. If both are specified, both must match the request.

argand-site-registry review --database "$ARGAND_SITE_DATA/import.sqlite" \
  --generation "$ARGAND_SITE_DATA/generation-1" --pin "$ARGAND_SITE_PIN" \
  --decision review.json
argand-site-registry build --database "$ARGAND_SITE_DATA/import.sqlite" \
  --output "$ARGAND_SITE_DATA/generation-2" > "$ARGAND_SITE_DATA/build-2.json"
export ARGAND_SITE_PIN="$(jq -r .pin "$ARGAND_SITE_DATA/build-2.json")"
argand-site-registry resolve --generation "$ARGAND_SITE_DATA/generation-2" \
  --pin "$ARGAND_SITE_PIN" --query amazon --country GB

After the corresponding real reviews, GB selects the reviewed UK property; DE selects the reviewed German property; otherwise an explicitly reviewed primary may be used. Unknown, expired, tied or entity-ambiguous requests return "destination": null. Result limits never hide ambiguity. Name/alias changes, changed statements/revisions, URLs or normalization evidence invalidate reviews. Deprecated, end-dated and non-value statements remain audit evidence and cannot be admitted. An unchanged PSL file with a new retrieval time preserves reviews.

Release, export, update and recovery

When two providers describe the same navigational entity, lookup deliberately shows both source IDs. Connect them only after reviewing their identities:

argand-site-registry equivalence --generation "$ARGAND_SITE_DATA/generation-2" \
  --pin "$ARGAND_SITE_PIN" --left SOURCE_ENTITY_ID --right OTHER_SOURCE_ENTITY_ID

Use the returned fingerprint in a review JSON with role: "unspecified", empty locale/country, a reason, immutable identity evidence and an expiry. Then repeat the command with --database "$ARGAND_SITE_DATA/import.sqlite" --decision identity-review.json and rebuild. resolve follows only active, explicitly reviewed equivalences and includes their provenance. Each destination still needs its own review. The original IDs, raw ambiguity counts and conflicting assertions remain visible. Changed names or website assertions invalidate the identity decision; identity revocations use the same append-only release log. Operator-authored decisions are published under CC0-1.0, separately from source data licenses.

Each generation contains registry.sqlite, LICENSE_SOURCES.md, ATTRIBUTION.json and a hash-binding COMPLETE.json. Distribute all four together. Keep the import database, review history and cached source bytes for recovery. The SQLite file includes raw relevant records and descriptions for audit; public consumers must obey the source attribution requirements. export streams source-bearing JSONL and omits descriptions by default:

argand-site-registry export --generation "$ARGAND_SITE_DATA/generation-2" \
  --pin "$ARGAND_SITE_PIN" --output "$ARGAND_SITE_DATA/assertions.jsonl"
argand-site-registry sign --generation "$ARGAND_SITE_DATA/generation-2" \
  --pin "$ARGAND_SITE_PIN" --key /secure/registry-signing-key
argand-site-registry activate --generation "$ARGAND_SITE_DATA/generation-2" \
  --current "$ARGAND_SITE_DATA/current.json" \
  --allowed-signers /secure/registry-allowed-signers --identity registry-publisher

Use an existing operator-controlled SSH signing key. The external allowed-signers file follows OpenSSH syntax: registry-publisher ssh-ed25519 PUBLIC_KEY. Neither keys nor the trust file should come from the downloaded dataset. Consumers can open Registry::open(path, trusted_receipt_sha256) once and reuse its indexed queries, or verify a publisher with release::verify_signed first. A hash proves integrity only relative to a trusted pin. Signature verification authenticates the publisher, not the truth of a source assertion.

diff --old PATH --old-pin HASH --new PATH --new-pin HASH streams added/removed edge fingerprints. verify --generation PATH --pin HASH checks every artifact bound by the receipt. To revoke, append a review with decision: "revoke", then rebuild, sign and activate. Re-activating an older signed generation supports rollback only if it retains every distributed revocation. Otherwise rebuild the older source selection with the current review log; never edit generations.

The update configuration and systemd service/timer provide weekly candidate refreshes without a resident daemon. Set absolute paths and an installed executable path. TOML paths do not expand environment variables. update --config /etc/argand-site-registry.toml downloads/imports all configured sources and builds only after they succeed. The same inputs and review log reuse the same generation. A nonzero exit is a failed refresh; the active pointer stays intact. Failed pending-* builds can be inspected before explicitly removing that incomplete directory. Acquisition, import and update operations take local locks; use one writer and keep old complete generations for rollback.

Downloads permit only the reviewed HTTPS source endpoints, validate each redirect, bound bytes and bind range resumes to strong ETags. Chunked/validatorless responses safely restart on interruption. PSL network attempts are limited to once per 24 hours per cache. CrUX is opt-in and may use {previous_month} in update configuration; monthly data may not yet be published on the first day. Wikidata source snapshot labels support {date} and {month} in scheduled downloads. No scheduled job signs, approves, renews approvals or activates links.

Storage and operating limits

Migration migrations/001.sql owns schema version 1. sources, records and facts preserve snapshot/native IDs, licenses, retrieval times and confidence; reviews is append-only. Complete source selection is latest retrieval time per provider/scope, with digest as the deterministic tie break. Use the same scope for a replacement snapshot, and separate scopes for deliberate independent selections. History and conflicts remain stored. A failed source cannot replace a complete one. Avoid overlapping full/partial scopes unless both evidences are intended to remain active.

Derived tables are selected_sources, entities, names, properties, edges, popularity and rejected. Entity IDs derive from source/native IDs; URL IDs derive from strict normalized URLs. Equal source entities merge across snapshots and equal URLs share a property. Explicit equivalences connect reviewed cross-source identities while preserving both IDs. Names are never an identity join. The canonical label rule prefers labels, then English, then language/text order. Original labels/aliases are kept. Popularity has its own source, target, observation period and audience scope and never creates an ownership edge. All derivations bind input fact IDs, PSL identity and the argand.site-rules/v1 contract through their generation receipt.

Imports use transactions of 256 relevant records with durable replay checkpoints. Restart replays the compressed stream and skips committed records. Large source records are capped at 16 MiB; SQLite has an 8 MiB page cache and disk-backed sorts. Builds stream a canonical sorted copy and never load the full registry into RAM. Names and entity metadata exposed by lookup are capped at 256 facts each, with uncapped totals; the complete assertions remain available in the database/export. Lookup returns at most 100 edges and reports all pre-limit ambiguity counts. The full store/history and each generation consume disk; there is no automatic pruning. The original compressed source is hashed before/after import, so expect extra sequential disk reads. These bounds are not a full-dump throughput claim.

The observation module defines future crawler evidence for redirects, canonical links, hreflang, JSON-LD sameAs, sitemaps and country selectors, with capture IDs, hashes, rights and confidence. It does not crawl or automatically infer ownership.

Source vandalism, compromised publishers and a domain changing ownership cannot be eliminated by hashes or popularity. Review expiration, exact evidence binding, signed releases, explicit revocations and conservative abstention contain those risks. Protect the writer database, signing key and consumer trust configuration. Do not feed raw lookup candidates straight into an automatic redirect consumer.

If an import fails, fix the input/format or reuse the matching original source manifest, then rerun the same import. Do not edit digests to make corrupted data pass. A source host/schema change needs an adapter review. HTTP 403/429 is a source-access failure; reuse an authorized retained snapshot or retry according to the provider's policy. A null resolution means evidence/review is missing, expired or ambiguous; lookup explains which assertions are involved.