nicweyand/argand-site-registry
1
0
Fork 0
Code Issues Pull requests Projects Releases 7 Packages Wiki Activity Actions

release: implement site registry v0.4 trust pipeline

Browse source
This commit is contained in:
Nic Weyand 2026-09-13 12:22:05 -04:00
commit e26efc19fa
Signed by: nicweyand
SSH key fingerprint: SHA256:2te+ycJIQON/Wo/dH6+ZkFSQ4HnHWpetV2azx9E65dQ
67 changed files with 10698 additions and 640 deletions
Download patch file Download diff file Expand all files Collapse all files

278
README.md
View file

@ -1,69 +1,57 @@
# Argand Site Registry
Argand Site Registry connects an organization, product or other named entity to
its official websites. It is built for navigational search and for choosing the
right regional site without hiding where each claim came from.
Given `facebook`, a registry can return:
Argand Site Registry is a Rust toolkit for building a trustworthy map from names
to entities to official websites. It is useful for navigational search, regional
site selection, link directories, and any application that needs to explain why
it trusts a destination.
```text
facebook -> Facebook (Wikidata Q355) -> https://www.facebook.com/
registrable domain: facebook.com
facebook -> Facebook (Wikidata Q355) -> https://www.facebook.com/
hostname: www.facebook.com
registrable domain: facebook.com
public suffix: com
```
An entity may have several legitimate properties. For example, one Amazon entity
can carry separately evidenced properties for `amazon.com`, `amazon.co.uk` and
`amazon.de`. The registry never joins entities merely because their names or
hostnames look alike.
The repository contains the library, CLI, schemas, migrations, and synthetic
fixtures. It does not contain a preapproved production dataset. A publisher must
import source evidence, collect signed reviews, and distribute a signed registry
generation.
This repository provides the Rust library, command-line tools and synthetic test
fixtures. It does **not** publish a ready-to-use approved-link dataset. A registry
publisher imports source data, reviews destinations and distributes a signed
generation; a consumer verifies that generation before using it.
## The basic idea
## What it answers
The registry keeps four things separate:
The two main query operations have different purposes:
1. **Assertions:** Wikidata, Curlie, and other reviewed sources say that a name or
website is associated with an entity.
2. **Observations:** a bounded observer records redirects, canonical links,
hreflang, JSON-LD `sameAs`, sitemaps, DNS, TLS, and failures. These are evidence,
not proof of ownership.
3. **Votes:** authenticated reviewers approve or revoke one exact name, website
edge, or entity equivalence under a versioned policy.
4. **Releases:** a separate publisher signs an immutable generation. Consumers
verify it before resolving names.
| Operation | Answer | Trust behavior |
| --- | --- | --- |
| `lookup` | What entities and websites do the sources associate with this name? | Returns evidence for inspection, including conflicts and unreviewed claims. |
| `resolve` | Which destination is approved for this name, locale and country? | Returns only an unambiguous, signed-review-backed, unexpired destination; otherwise it abstains. |
`lookup` shows source evidence and conflicts. `resolve` returns a destination only
when both the name binding and website edge satisfy the configured review policy.
Otherwise it returns a typed abstention such as `no_active_name_review`,
`no_active_review`, or `ambiguous_destination`.
The CLI can also inspect an entity by stable ID, reverse-lookup a URL or domain,
show source-separated popularity signals, inspect redacted Curlie categories,
compare generations and evaluate a pinned generation against JSONL judgments.
Source confidence is evidence metadata, not a malware-safety score. An imported
official-site claim cannot become a `resolve` destination until an authorized
reviewer approves that exact claim. Applications should still apply their own
security and content policy.
## How a registry is built
One entity can have several independently reviewed regional properties:
```text
source downloads -> provenance-preserving imports -> immutable candidate
-> signed human reviews -> rebuilt generation -> signed activation
-> lookup / regional resolve
Example Store
primary -> https://example.com/
country=GB -> https://example.co.uk/
country=DE -> https://example.de/
```
| Stage | Main commands | What changes |
| --- | --- | --- |
| Acquire | `download`, `crux-download`, `manifest` | Stores source bytes and a receipt describing their origin. |
| Import | `import` | Streams normalized facts into the local SQLite writer database. |
| Build | `build` | Produces a new immutable, content-pinned generation. |
| Review | `review`, `equivalence` | Records signed destination or entity-equivalence decisions against exact evidence fingerprints. |
| Publish | `sign`, `activate` | Verifies reviewer trust, signs a generation and atomically selects it. |
| Consume | `lookup`, `resolve`, `lookup-web`, `entity` | Reads a verified generation without changing it. |
The relationship must come from evidence and review. Similar names, ccTLDs, and
hostnames never merge entities or create regional relationships by themselves.
Automated updates stop after building a candidate. They cannot approve a link,
sign a release or activate it.
## Install and try it
## Try it locally
Linux is the currently validated platform. You need Rust 1.97 or newer, a C
compiler, CMake, Perl and OpenSSH (`ssh-keygen`). Python 3.11 or newer is needed
for release checks and the Python example. SQLite is compiled into the binary.
Linux is the validated platform. You need Rust 1.97 or newer, a C toolchain,
CMake, Perl, Python 3.11 or newer, and OpenSSH `ssh-keygen`.
```bash
git clone https://git.argand.org/nicweyand/argand-site-registry.git
@ -73,85 +61,142 @@ cargo build --release --locked --offline -p argand-site-registry
./target/release/argand-site-registry --help
```
To see the complete lifecycle without downloading provider data, run the native
fixture in a new directory outside the checkout:
Run the complete source-shaped fixture without downloading provider data:
```bash
ARGAND_REGISTRY_E2E_OUTPUT=/tmp/site-registry-example \
cargo test -p argand-site-registry --test cli --locked --offline -- --nocapture
```
The fixture imports source-shaped Wikidata, Majestic Million, CrUX, Curlie and
Public Suffix List records. It proves repeatable imports, alias lookup, explicit
entity equivalence, destination review, signing, activation, revocation and
rollback protection. Its keys and approvals are disposable test material and
must never be used for a real registry.
Production inputs belong in a configurable data/cache directory outside the
source checkout. Installation and tests do not download datasets or start
scheduled jobs.
## Use a generation from Rust or Python
A consumer needs the generation directory and a trusted pin for its
`COMPLETE.json` receipt. Obtain that pin through a trusted publisher channel, or
verify the publisher signature with an independently configured key. A hash found
beside an untrusted download does not authenticate the download.
The [Rust example](crates/argand-site-registry/examples/lookup.rs) reuses the
native reader. The [Python example](examples/lookup.py) calls the native CLI so it
keeps the same validation and response contract:
Run the strict 0.4 trust and observer acceptance flow:
```bash
cargo run --locked --offline -p argand-site-registry --example lookup -- \
--generation /data/registry/generation \
--pin "$REGISTRY_TRUSTED_PIN" \
--query facebook
cargo test -p argand-site-registry --test v04 --locked --offline -- --nocapture
```
python3 examples/lookup.py \
--binary ./target/release/argand-site-registry \
--generation /data/registry/generation \
--pin "$REGISTRY_TRUSTED_PIN" \
The first fixture imports synthetic Wikidata, Majestic Million, CrUX, Curlie,
and Public Suffix List inputs. The second proves two-reviewer name and edge votes,
policy epochs, stale evidence, observation replay, drift, sticky revocation,
publisher separation, signed activation, and resistance to signature tampering.
All keys and approvals created by tests are disposable.
## Build and use a registry
Production data and keys belong outside the checkout. A typical lifecycle is:
```text
download/manifest -> import -> build candidate -> inspect/observe
-> prepare and sign votes -> rebuild -> evaluate -> sign -> activate
```
The strict default policy requires two independent reviewer identities and groups
for names, website edges, and entity equivalences. A build binds both the policy
and exact reviewer trust file into its receipt.
```bash
argand-site-registry build \
--database /data/site-registry/import.sqlite \
--output /data/site-registry/candidate \
--reviewer-trust /secure/site-registry/reviewer-allowed-signers
PIN=$(sha256sum /data/site-registry/candidate/COMPLETE.json | cut -d' ' -f1)
argand-site-registry lookup \
--generation /data/site-registry/candidate --pin "$PIN" --query facebook
argand-site-registry review-queue \
--generation /data/site-registry/candidate --pin "$PIN" \
--at 2026-09-13T00:00:00Z
```
For a queued name or edge, prepare canonical vote JSON and sign its exact bytes:
```bash
argand-site-registry prepare-vote \
--generation /data/site-registry/candidate --pin "$PIN" \
--subject-kind edge --fingerprint "$EDGE_FINGERPRINT" \
--decision approve --reviewer reviewer-one \
--reason "Verified exact entity, URL, and global role" \
--reviewed-at 2026-09-13T00:00:00Z \
--expires-at 2026-10-13T00:00:00Z --role primary \
--output /secure/site-registry/edge-vote.json
ssh-keygen -Y sign -n argand-site-registry-vote \
-f /secure/site-registry/reviewer-one /secure/site-registry/edge-vote.json
argand-site-registry vote \
--database /data/site-registry/import.sqlite \
--generation /data/site-registry/candidate --pin "$PIN" \
--decision /secure/site-registry/edge-vote.json \
--signature /secure/site-registry/edge-vote.json.sig \
--allowed-reviewers /secure/site-registry/reviewer-allowed-signers \
--identity reviewer-one
```
Repeat independently for the second reviewer and for the selected name binding,
then rebuild. `prepare-equivalence-vote`, `verify-equivalence-vote`, and
`equivalence-vote` provide the same flow for explicit cross-source entity links.
The [operator guide](crates/argand-site-registry/README.md) documents source
downloads, typed full/partition/delta coverage, observation commands, regional
roles, release signing, activation, and recovery.
Consumers should use `resolve`, not the first result from `lookup`:
```bash
argand-site-registry resolve \
--generation /data/site-registry/reviewed --pin "$REVIEWED_PIN" \
--query facebook
```
Do not turn `lookup.candidates[0]` into an automatic redirect. Use `resolve` and
preserve `destination: null` as a deliberate abstention. The
[consumer contract](docs/CONSUMERS.md) documents the Rust API, CLI JSON, signed
generation files, compatibility rules and Argand's pinned integration.
The [Rust example](crates/argand-site-registry/examples/lookup.rs) reuses a
verified reader. The [Python example](examples/lookup.py) invokes the same native
CLI and preserves its JSON contract.
## Operate or contribute
## Sources and licenses
The [operator guide](crates/argand-site-registry/README.md) contains the complete
commands for acquiring each source, importing large files, reviewing regional
properties, signing releases, scheduling candidate updates and recovering state.
CrUX acquisition always requires explicit credentials and a billing cap.
The registry currently supports these source adapters:
| Source | Purpose | Data license |
| Source | Consumed evidence | Data license |
| --- | --- | --- |
| Wikidata | Entity names, aliases, official websites and locale/country evidence | CC0 1.0 Universal |
| Majestic Million | Domain popularity rank | CC BY 3.0 Unported |
| Chrome UX Report (CrUX) | Popular-origin rank bucket | CC BY 4.0 International |
| Curlie | Human-curated names and categories | CC BY 3.0 Unported |
| Public Suffix List | Public-suffix and registrable-domain parsing | Mozilla Public License 2.0 |
| Wikidata | IDs, labels, aliases, P856 statements, selected locale/country metadata | CC0 1.0 Universal |
| Majestic Million | source-specific domain rank and supplied metrics | CC BY 3.0 Unported |
| Chrome UX Report | origin popularity bucket, month, optional audience country | CC BY 4.0 International |
| Curlie | site titles, categories, descriptions retained for audit | CC BY 3.0 Unported |
| Public Suffix List | ICANN and PRIVATE suffix rules | MPL 2.0 |
Read [LICENSE_SOURCES.md](LICENSE_SOURCES.md) before redistributing data. Curlie
attribution applies to names and categories as well as descriptions. The registry
keeps each source logically separate and retains provenance, licenses, retrieval
timestamps, source identifiers and conflicting evidence.
Popularity never proves identity or ownership. Curlie attribution applies to
names and categories as well as descriptions; copied descriptions are redacted
unless the caller explicitly exports them and satisfies the display obligations.
Read [LICENSE_SOURCES.md](LICENSE_SOURCES.md) before distributing provider data.
For the security model and release process, see:
Cloudflare Radar, default Tranco, Cisco Umbrella, arbitrary mirrors, and sources
without verified commercial reuse rights are unsupported. Adding a provider
requires an explicit adapter, current format inspection, provenance, rights
review, attribution rules, and tests.
- [Trust and evidence policy](docs/TRUST.md)
- [Publishing runbook](docs/PUBLISHING.md)
- [Security policy](SECURITY.md)
- [Governance](GOVERNANCE.md)
- [Full documentation index](docs/INDEX.md)
## Safety and reproducibility
Pull requests may add evidence or code, but cannot directly approve a production
destination.
- Raw sources remain logically separate and content-addressed outside Git.
- Every imported or derived fact keeps source, source identifier, license,
retrieval time, selector, and confidence.
- Typed coverage declares full snapshots, partitions, deltas, bases, and exact
supersession; conflicts fail closed.
- Active export contains selected, nonrejected facts. Audit export also retains
superseded and rejected evidence.
- Votes bind assertion, evidence-bundle, policy, reviewer, scope, and expiry.
- Revocations remain sticky until every member of a fresh quorum explicitly
supersedes them.
- Small publisher-signed cumulative revocation feeds can block a compatible
pinned offline registry before its replacement generation arrives. They expire
after seven days and cannot restore a route across generations.
- The observer follows at most five redirects, uses public DNS pinning, blocks
private and link-local networks, rejects HTTPS downgrade and nondefault ports,
and caps headers, bodies, time, and extracted links.
- Automated updates may download, import, observe, and build candidates. They do
not approve, renew, sign, activate, or silently choose a destination.
The code is **AGPL-3.0-or-later**. Provider data retains its own license. See the
[documentation index](docs/INDEX.md), [architecture](docs/ARCHITECTURE.md),
[trust model](docs/TRUST.md), [publishing runbook](docs/PUBLISHING.md), and
[consumer contract](docs/CONSUMERS.md).
## Development
@ -160,14 +205,7 @@ cargo fetch --locked
bash scripts/check.sh
```
The check covers formatting, all-target compilation, strict Clippy, Rust tests,
API documentation, Python release tests and native Rust/Python consumer parity.
[RELEASING.md](docs/RELEASING.md) explains deterministic source archives and
independent signature verification; [VALIDATION.md](docs/VALIDATION.md) records
the current independent build and acceptance evidence.
The code is licensed under **AGPL-3.0-or-later**; see [LICENSE](LICENSE). Provider
data keeps its own license. [UPSTREAM.json](UPSTREAM.json) records the signed
Argand extraction revision and original file hashes. Argand pins the signed
`v0.3.0` release by full Git revision and no longer carries an embedded source
copy.
The gate covers formatting, all-target compilation, strict Clippy, Rust tests,
API documentation, Python release checks, deterministic packaging, and native
Rust/Python consumer parity. Security validation is recorded separately in
[docs/VALIDATION.md](docs/VALIDATION.md).