From 2861337a453d0f624b18ce1f3e58cba21883449e Mon Sep 17 00:00:00 2001 From: nicweyand Date: Sun, 13 Sep 2026 09:14:10 -0400 Subject: [PATCH] docs: make registry README easier to follow --- README.md | 217 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 131 insertions(+), 86 deletions(-) diff --git a/README.md b/README.md index 180166c..dd0e4fb 100644 --- a/README.md +++ b/README.md @@ -1,128 +1,173 @@ # Argand Site Registry -Evidence-backed website identity and regional resolution, as a Rust library and CLI. +Argand Site Registry connects an organization, product or other named entity to +its official websites. It is built for navigational search and for choosing the +right regional site without hiding where each claim came from. -Look up an entity's names, aliases and websites; inspect each source assertion; -resolve an explicitly reviewed primary or regional destination. Sources, conflicting -claims, licenses and review history remain available for audit. Everything runs -locally with SQLite. No search engine, hosted account or GPU is required. +Given `facebook`, a registry can return: -`facebook → Facebook → facebook.com` is a name-to-entity-to-registrable-domain -lookup. The retained real Wikidata Facebook record has `www.facebook.com` and -`m.facebook.com` as distinct properties. Its Amazon record includes `amazon.com`, -`amazon.co.uk` and `amazon.de` on the same entity. Similar hostname spelling never -establishes common ownership. Imported links await review before `resolve` can -return a destination; source confidence is not a malware-safety guarantee. +```text +facebook -> Facebook (Wikidata Q355) -> https://www.facebook.com/ + registrable domain: facebook.com +``` -## Build and try it +An entity may have several legitimate properties. For example, one Amazon entity +can carry separately evidenced properties for `amazon.com`, `amazon.co.uk` and +`amazon.de`. The registry never joins entities merely because their names or +hostnames look alike. -Linux is the currently validated platform. Install Rust (tested with 1.98.1; the -inherited minimum is 1.97), a C compiler, CMake, Perl and OpenSSH (`ssh-keygen`). -Python 3.11+ is needed for release tooling and the Python example. SQLite is built -with the binary. Get the public source, or use a verified source release: +This repository provides the Rust library, command-line tools and synthetic test +fixtures. It does **not** publish a ready-to-use approved-link dataset. A registry +publisher imports source data, reviews destinations and distributes a signed +generation; a consumer verifies that generation before using it. + +## What it answers + +The two main query operations have different purposes: + +| Operation | Answer | Trust behavior | +| --- | --- | --- | +| `lookup` | What entities and websites do the sources associate with this name? | Returns evidence for inspection, including conflicts and unreviewed claims. | +| `resolve` | Which destination is approved for this name, locale and country? | Returns only an unambiguous, signed-review-backed, unexpired destination; otherwise it abstains. | + +The CLI can also inspect an entity by stable ID, reverse-lookup a URL or domain, +show source-separated popularity signals, inspect redacted Curlie categories, +compare generations and evaluate a pinned generation against JSONL judgments. + +Source confidence is evidence metadata, not a malware-safety score. An imported +official-site claim cannot become a `resolve` destination until an authorized +reviewer approves that exact claim. Applications should still apply their own +security and content policy. + +## How a registry is built + +```text +source downloads -> provenance-preserving imports -> immutable candidate + -> signed human reviews -> rebuilt generation -> signed activation + -> lookup / regional resolve +``` + +| Stage | Main commands | What changes | +| --- | --- | --- | +| Acquire | `download`, `crux-download`, `manifest` | Stores source bytes and a receipt describing their origin. | +| Import | `import` | Streams normalized facts into the local SQLite writer database. | +| Build | `build` | Produces a new immutable, content-pinned generation. | +| Review | `review`, `equivalence` | Records signed destination or entity-equivalence decisions against exact evidence fingerprints. | +| Publish | `sign`, `activate` | Verifies reviewer trust, signs a generation and atomically selects it. | +| Consume | `lookup`, `resolve`, `lookup-web`, `entity` | Reads a verified generation without changing it. | + +Automated updates stop after building a candidate. They cannot approve a link, +sign a release or activate it. + +## Try it locally + +Linux is the currently validated platform. You need Rust 1.97 or newer, a C +compiler, CMake, Perl and OpenSSH (`ssh-keygen`). Python 3.11 or newer is needed +for release checks and the Python example. SQLite is compiled into the binary. ```bash git clone https://git.argand.org/nicweyand/argand-site-registry.git cd argand-site-registry -``` - -From the root of the checkout or extracted release: - -```bash cargo fetch --locked cargo build --release --locked --offline -p argand-site-registry ./target/release/argand-site-registry --help ``` -If your Cargo configuration sets a different target directory, set -`CARGO_TARGET_DIR="$PWD/target"` before these commands. To install the CLI: - -```bash -cargo install --path crates/argand-site-registry --locked --offline -argand-site-registry --help -``` - -Run the native all-source example in a **new directory outside the checkout**: +To see the complete lifecycle without downloading provider data, run the native +fixture in a new directory outside the checkout: ```bash ARGAND_REGISTRY_E2E_OUTPUT=/tmp/site-registry-example \ cargo test -p argand-site-registry --test cli --locked --offline -- --nocapture ``` -It imports synthetic Wikidata, Majestic, CrUX, Curlie and PSL fixtures, checks -idempotency, joins explicitly reviewed identities, approves a destination, signs -and activates it, revokes it, and rejects rollback past that revocation. Fixture -signing keys and approvals are disposable test material. All production inputs -belong in a configurable data/cache directory outside the source checkout. +The fixture imports source-shaped Wikidata, Majestic Million, CrUX, Curlie and +Public Suffix List records. It proves repeatable imports, alias lookup, explicit +entity equivalence, destination review, signing, activation, revocation and +rollback protection. Its keys and approvals are disposable test material and +must never be used for a real registry. -Follow the [operator guide](crates/argand-site-registry/README.md) for actual -downloads, imports, regional reviews, signed datasets and weekly candidate updates. -CrUX acquisition requires explicit credentials and a billing cap. No source -downloads or scheduled jobs run during installation or tests. +Production inputs belong in a configurable data/cache directory outside the +source checkout. Installation and tests do not download datasets or start +scheduled jobs. -## Use it in another project +## Use a generation from Rust or Python -The [Rust example](crates/argand-site-registry/examples/lookup.rs) opens a generation -once with an externally trusted receipt hash and returns the full lookup envelope. -The [Python example](examples/lookup.py) calls the same native CLI; it does not -implement a second resolver. Both preserve attribution and fact provenance. +A consumer needs the generation directory and a trusted pin for its +`COMPLETE.json` receipt. Obtain that pin through a trusted publisher channel, or +verify the publisher signature with an independently configured key. A hash found +beside an untrusted download does not authenticate the download. + +The [Rust example](crates/argand-site-registry/examples/lookup.rs) reuses the +native reader. The [Python example](examples/lookup.py) calls the native CLI so it +keeps the same validation and response contract: ```bash cargo run --locked --offline -p argand-site-registry --example lookup -- \ - --generation /data/registry/generation --pin "$REGISTRY_TRUSTED_PIN" --query facebook + --generation /data/registry/generation \ + --pin "$REGISTRY_TRUSTED_PIN" \ + --query facebook -python3 examples/lookup.py --binary ./target/release/argand-site-registry \ - --generation /data/registry/generation --pin "$REGISTRY_TRUSTED_PIN" --query facebook +python3 examples/lookup.py \ + --binary ./target/release/argand-site-registry \ + --generation /data/registry/generation \ + --pin "$REGISTRY_TRUSTED_PIN" \ + --query facebook ``` -Obtain the pin from a trusted publisher channel or verify the release signature -against an independently configured key. Reading a hash from the same untrusted -download does not authenticate it. Never turn `lookup.candidates[0]` into an -automatic destination: use the native `resolve` result and your application's -own admission policy. A null destination is a meaningful abstention. +Do not turn `lookup.candidates[0]` into an automatic redirect. Use `resolve` and +preserve `destination: null` as a deliberate abstention. The +[consumer contract](docs/CONSUMERS.md) documents the Rust API, CLI JSON, signed +generation files, compatibility rules and Argand's pinned integration. -The [consumer contract](docs/CONSUMERS.md) covers Rust dependencies, CLI JSON, -SQLite/JSONL distribution, compatibility and Argand's pinned upstream integration. -Exact entity, URL/domain, source-separated popularity and redacted Curlie category -queries support audit tools. `resolve` also reports a stable abstention reason and -decision counts. The bounded `evaluate` command replays JSONL judgments and reports -accuracy plus native p50/p95 latency for one pinned generation. +## Operate or contribute -## Sources and trust +The [operator guide](crates/argand-site-registry/README.md) contains the complete +commands for acquiring each source, importing large files, reviewing regional +properties, signing releases, scheduling candidate updates and recovering state. +CrUX acquisition always requires explicit credentials and a billing cap. -Wikidata (CC0), Majestic Million (CC BY 3.0), CrUX (CC BY 4.0), Curlie (CC BY 3.0) -and the Public Suffix List (MPL 2.0) remain logically separate. Read the exact -[source licenses and attribution rules](LICENSE_SOURCES.md) before redistribution. -Curlie attribution applies to names and categories as well as descriptions. +The registry currently supports these source adapters: -The [trust policy](docs/TRUST.md) explains enforced checks, publisher responsibilities, -evidence standards, expiry and revocation. The [publisher runbook](docs/PUBLISHING.md) -covers authenticated reviewer decisions, candidate inspection and activation. -[CONTRIBUTING.md](CONTRIBUTING.md), -[GOVERNANCE.md](GOVERNANCE.md) and [SECURITY.md](SECURITY.md) cover contributions, -decisions, disputes and incidents. Pull requests cannot directly approve destinations. +| Source | Purpose | Data license | +| --- | --- | --- | +| Wikidata | Entity names, aliases, official websites and locale/country evidence | CC0 1.0 Universal | +| Majestic Million | Domain popularity rank | CC BY 3.0 Unported | +| Chrome UX Report (CrUX) | Popular-origin rank bucket | CC BY 4.0 International | +| Curlie | Human-curated names and categories | CC BY 3.0 Unported | +| Public Suffix List | Public-suffix and registrable-domain parsing | Mozilla Public License 2.0 | -## Development and releases +Read [LICENSE_SOURCES.md](LICENSE_SOURCES.md) before redistributing data. Curlie +attribution applies to names and categories as well as descriptions. The registry +keeps each source logically separate and retains provenance, licenses, retrieval +timestamps, source identifiers and conflicting evidence. + +For the security model and release process, see: + +- [Trust and evidence policy](docs/TRUST.md) +- [Publishing runbook](docs/PUBLISHING.md) +- [Security policy](SECURITY.md) +- [Governance](GOVERNANCE.md) +- [Full documentation index](docs/INDEX.md) + +Pull requests may add evidence or code, but cannot directly approve a production +destination. + +## Development ```bash cargo fetch --locked bash scripts/check.sh ``` -The check runs formatting, all-target compilation, strict Clippy, Rust tests and -documentation, Python release tests, and native Rust/Python consumer parity. -[RELEASING.md](docs/RELEASING.md) covers deterministic source archives, signature -verification and rebuilding outside the checkout. The Forgejo workflow requires -a dedicated isolated runner; it has no signing or dataset-promotion authority. +The check covers formatting, all-target compilation, strict Clippy, Rust tests, +API documentation, Python release tests and native Rust/Python consumer parity. +[RELEASING.md](docs/RELEASING.md) explains deterministic source archives and +independent signature verification; [VALIDATION.md](docs/VALIDATION.md) records +the current independent build and acceptance evidence. -The [validation record](docs/VALIDATION.md) reports independent builds and native -acceptance. Hosted checks use the repository's isolated Forgejo runner label and -have no dataset, review, signing or activation authority. - -The code remains **AGPL-3.0-or-later**; the complete license is in [LICENSE](LICENSE). -Original attribution is retained. [UPSTREAM.json](UPSTREAM.json) records the signed -Argand extraction revision and original file hashes. Version 0.3 advances the -standalone database to schema version 3; the migration and consumer rules are -documented in [CONSUMERS.md](docs/CONSUMERS.md). Argand pins this signed `v0.3.0` -release by full Git revision and no longer carries an embedded source copy. This -repository does not operate a public approved-link dataset. +The code is licensed under **AGPL-3.0-or-later**; see [LICENSE](LICENSE). Provider +data keeps its own license. [UPSTREAM.json](UPSTREAM.json) records the signed +Argand extraction revision and original file hashes. Argand pins the signed +`v0.3.0` release by full Git revision and no longer carries an embedded source +copy.