feat: establish standalone Argand Site Registry

This commit is contained in:
Nic Weyand 2026-09-12 21:38:59 -04:00
commit 2a0fe1714b
Signed by: nicweyand
SSH key fingerprint: SHA256:2te+ycJIQON/Wo/dH6+ZkFSQ4HnHWpetV2azx9E65dQ
60 changed files with 10494 additions and 0 deletions

57
docs/CONSUMERS.md Normal file
View file

@ -0,0 +1,57 @@
# Consumer and compatibility contract
## Rust library
Use `argand_site_registry::query::Registry::open(generation, trusted_pin)` once per
immutable generation and reuse the reader. `lookup(query, limit)` returns evidence
and complete ambiguity counts; `resolve(query, locale, country, now)` returns an
optional reviewed candidate. Check the compiled example and API docs for exact
types. `selection_context` binds the full alternative set for downstream query
review. Preserve the returned provenance, scopes, counts and attribution.
For local integration, point a Cargo dependency at
`crates/argand-site-registry` inside an extracted standalone source tree. Once an
upstream repository is published, use its actual Git URL and a full reviewed `rev`
pin. Do not invent a crates.io version or track a mutable branch in production.
Both crates remain in this workspace; the atomic helper is a relative dependency.
## CLI and other languages
`lookup`, `resolve`, `verify` and the other commands emit JSON. The Python example
passes arguments directly to the native executable, preserving query text and the
entire response. A nonzero exit is an error. `destination: null` is a successful
abstention, not a request to pick the first lookup candidate. Render source names
as untrusted text and satisfy their source-specific attribution requirements.
Use a bounded process or service wrapper appropriate to your workload. The Python
example has a 60-second timeout and invokes a fresh reader per request; for repeated
low-latency queries, use the reusable Rust reader. No hosted API or Python package
registry publication is claimed by this repository.
## SQLite, JSONL and license scope
Distribute `registry.sqlite`, `COMPLETE.json`, `LICENSE_SOURCES.md` and
`ATTRIBUTION.json` together, plus the publisher signature when applicable. The
database contains audit records and source descriptions. Default JSONL export
omits descriptions and includes fact provenance plus an attribution envelope.
It is an assertion export, not a self-contained signed list of admitted routes.
Raw SQL inspection is useful for audit; it does not implement resolution policy.
Code version 0.1.0 is an initial interface. Schema/rule contracts are versioned
independently in receipts. Unsupported contracts fail closed. Pin source releases,
compile consumers and replay fixed fixtures before upgrades. Preserve import and
review history; never mutate complete generations to migrate them.
## Argand transition
UPSTREAM.json pins the exact Argand source baseline, including the beta agent's
selection-context API. The first standalone extraction preserves runtime Rust and
migration bytes. Argand currently consumes its embedded workspace crate; this
package does not silently redirect that dependency.
At cutover, coordinate with the Argand source/build owner, compare both trees with
the recorded baseline, carry any subsequent fixes forward, replace the embedded
dependency with a reviewed standalone revision, and run Argand's complete engine
and navigation-compiler gates. Preserve existing registry receipts and public
navigation admission. After cutover, develop the library upstream and update
Argand through explicit pinned dependency changes, avoiding permanent dual copies.

13
docs/INDEX.md Normal file
View file

@ -0,0 +1,13 @@
# Documentation
- [README](../README.md): build, examples and scope.
- [Operator guide](../crates/argand-site-registry/README.md): all source commands,
schema, reviews, regional resolution, releases and update configuration.
- [Source licenses](../crates/argand-site-registry/LICENSE_SOURCES.md): exact terms and attribution.
- [Consumers](CONSUMERS.md): Rust, Python/CLI, data distribution and Argand transition.
- [Trust](TRUST.md): enforced checks and publisher/consumer responsibilities.
- [Releasing](RELEASING.md): CI, source signing and archive verification.
- [Contributing](../CONTRIBUTING.md), [governance](../GOVERNANCE.md),
[security](../SECURITY.md): proposals, decisions and incidents.
- [Extraction design](superpowers/specs/2026-09-12-standalone-design.md) and
[implementation plan](superpowers/plans/2026-09-12-standalone.md).

92
docs/RELEASING.md Normal file
View file

@ -0,0 +1,92 @@
# Source and dataset releases
## Build environment and CI
The two-crate workspace requires the normal Rust/C build toolchain, OpenSSH and
Python 3.11+ for checks and packaging. The initial extraction declares Rust 1.97
and was tested on Linux with Rust 1.98.1. Install rustfmt and Clippy alongside the
compiler. TLS dependencies may require CMake and Perl. Keep Cargo.lock tracked.
Run `cargo fetch --locked` once, then `bash scripts/check.sh` offline. Cargo's
[workspace inheritance](https://doc.rust-lang.org/cargo/reference/workspaces.html)
and [locked/offline options](https://doc.rust-lang.org/cargo/commands/cargo-test.html)
define the build behavior. A lockfile pins dependencies, not the host compiler.
The Forgejo workflow uses the documented [workflow and context syntax](https://forgejo.org/docs/latest/user/actions/reference/).
Register `site-registry-isolated` only on a disposable, repository-scoped runner
with the above tools, two build jobs and at least 4 GiB memory. Use a pinned,
reviewed runner image. Do not mount production directories, share signing keys,
or use Argand's host runners. Follow Forgejo's [runner security guidance](https://forgejo.org/docs/latest/user/actions/security/).
The label is a deployment requirement, not a provisioned runner supplied by this
repository. The workflow runs on trusted main pushes or manual dispatch, fetches
the exact public commit without credentials, runs offline gates, compares two
source archives and reruns acceptance on extracted source. It neither publishes
artifacts nor signs them. Review outside contributions before allowing them to
execute on infrastructure. No hosted CI pass is claimed by a local test run.
## Deterministic source archive
From a reviewed, clean committed tree, choose a new output directory outside the
checkout (its parent must exist):
```bash
python3 scripts/source_release.py create --output /data/releases/site-registry-source
```
The command emits the full receipt `pin` and creates `source.tar.gz` plus
`RELEASE.json`. The archive contains only allowed tracked source files from the
exact commit, including both crates, lockfile and license texts. Tar metadata and
gzip timestamps are fixed; identical source commits and Python/zlib packaging
versions produce identical bytes. The receipt records those packaging versions.
Links, submodules, datasets, private-key extensions and unsafe paths are refused.
The complete source tree is limited to 2,000 files and 32 MiB; review these limits
before expanding them. The allowlist is not a secret scanner: review source contents
for embedded credentials and unrelated material before signing or publication.
The receipt records the Git commit/tree and each path's hash, size and mode.
It is written last and fsynced with its directory. Existing output is never
overwritten. An interrupted directory without a valid receipt is incomplete;
inspect it and choose a fresh output path. The verifier bounds decompression,
rejects unexpected/missing/duplicate files, and checks every member before any
extraction. It accepts an external receipt pin; a hash found in the same download
is insufficient authentication.
## Signing and consuming source
After all checks and source review, sign with an operator-controlled SSH key:
```bash
ssh-keygen -Y sign -n argand-site-registry-source -f /secure/source-signing-key \
/data/releases/site-registry-source/RELEASE.json
```
Distribute the archive, receipt and `RELEASE.json.sig` together. Consumers obtain
the accepted identity and allowed-signers file through an independent trusted
channel, verify the signature, then compute the now-authenticated receipt pin:
```bash
ssh-keygen -Y verify -n argand-site-registry-source \
-f /secure/source-allowed-signers -I registry-source-publisher \
-s /data/releases/site-registry-source/RELEASE.json.sig \
< /data/releases/site-registry-source/RELEASE.json
sha256sum /data/releases/site-registry-source/RELEASE.json
python3 scripts/source_release.py verify \
--release /data/releases/site-registry-source --pin "$REGISTRY_TRUSTED_PIN"
```
Set `REGISTRY_TRUSTED_PIN` to that authenticated receipt hash. Use a verifier you
already trust, not one extracted from an unchecked archive. After verification,
extract into an empty directory and run `cargo fetch --locked` and
`bash scripts/check.sh`. Preserve the receipt with the installation. A source
signature authenticates reviewed source bytes, not an arbitrary executable built
elsewhere. Binary distribution needs its own artifact hash and build-environment
receipt; the initial release tool packages source only.
## Dataset publication
Source releases contain no provider datasets or real approvals. Dataset publishers
follow the operator guide: import, inspect, review, build, diff, sign and activate.
The dataset namespace `argand-site-registry` is distinct from the source namespace
above. Keep `LICENSE_SOURCES.md` and `ATTRIBUTION.json` with the database and receipt.
The weekly update example creates candidates. It never approves, renews, signs or
activates them. Record actual acquisition cost and dataset size separately from
small-fixture test results. Follow SECURITY.md for incidents and revocation delivery.

63
docs/TRUST.md Normal file
View file

@ -0,0 +1,63 @@
# Trust and evidence policy
## What the implementation enforces
Inputs use reviewed source adapters and explicit manifests. URL normalization,
PSL parsing and stable identities are deterministic. Fact provenance, conflicting
evidence and source-specific popularity remain separate. Names and similar
hostnames never silently join entities. Invalid URLs and malformed imports fail
validation; a partial import does not replace a complete source selection.
Destination and identity decisions bind exact evidence fingerprints, including
names, assertions and normalization context. Reviews expire within 90 days.
Changed evidence invalidates earlier approvals. Resolution abstains on ambiguity,
ties, missing approval or ineligible claims; regional scopes must explicitly
match. The local writer owns the append-only review log.
Generations bind the database, license document and attribution to a completion
receipt. Consumers provide a trusted hash or verify an external publisher key.
Activation checks signatures and refuses rollback that loses distributed
revocations. Updates build candidates and cannot approve, sign or activate them.
## What a publisher must establish
The reviewer name and evidence locator in a decision are operator assertions.
The CLI validates their structure and evidence binding; it does not independently
authenticate the reviewer, retrieve their evidence or prove website ownership.
Protect the writer database and signing key with separate operating permissions.
Restrict who can author decisions and require human review before release signing.
Publish dated evidence supporting the exact entity, URL, relationship, role and
country/locale. Prefer independently corroborated primary evidence with immutable
capture identifiers. Record contrary evidence and uncertainty. TLS, DNS control,
registrable-domain spelling, redirects, `sameAs` or popularity alone cannot
establish every identity or role claim. Future crawler observations remain inputs
to review. Confidence values are assertion scores, not calibrated probabilities.
Choose expiry based on volatility, within the enforced maximum. Do not renew
blindly on a timer. Expired approval should lead to abstention until evidence is
reviewed. Disclose editorial conflicts and use an independent reviewer for a
disputed claim when possible. The initial implementation is a local single-writer
tool; it does not provide authenticated reviewer accounts or an enforced quorum.
## What consumers must preserve
Authenticate a release before opening it. Keep the full receipt pin and required
attribution with caches and exports. Use `resolve` for reviewed destinations, keep
null as abstention, and enforce application-specific malware/content/navigation
policy separately. A verified signature authenticates the publisher, not the truth
of every assertion. An official website may later be compromised.
Deliver revocations to every active consumer and derived catalogue, and bound cache
lifetimes. The resolver checks review expiry at query time; a detached cached URL
does not recheck itself. Preserve the current review history during rollback.
Copying SQLite rows or the JSONL export into a second resolver can bypass these
checks; use the native API or CLI for admission decisions.
## Community changes
Treat public submissions as untrusted evidence. Do not execute submitted content
or copy review decisions into production automatically. Source allowlist and
attribution changes need both implementation tests and documented rights review.
Reject unknown sources until those checks are complete. Every publisher may apply
stricter admission rules, and must state its actual review and incident procedures.

View file

@ -0,0 +1,37 @@
# Standalone Site Registry Implementation Plan
> **For agentic workers:** Use superpowers:executing-plans inline. No subagents or Argand worktrees.
**Goal:** Produce a reusable standalone repository and verifiable source release.
**Architecture:** Retain the registry and atomic helper as two workspace crates.
Preserve runtime bytes and wire packaging, consumer examples and release gates
around their existing APIs. Use an isolated build directory.
**Tech Stack:** Rust, SQLite, Python 3.11 standard library tooling, OpenSSH, Forgejo Actions.
### Task 1: Extract the verified baseline
- [x] Verify signed Argand commit 47911062b00d87f215ba61c41965faf8a7f4b7f7.
- [x] Export only `engine/crates/argand-site-registry` and `engine/crates/argand-atomic`.
- [x] Write `UPSTREAM.json` with original path/blob/hash evidence and a two-crate `Cargo.toml`.
- [ ] Prune the inherited lock with `cargo metadata --offline --format-version 1`; retain exact dependency versions.
### Task 2: Make standalone use and trust policy concrete
- [ ] Add root README, LICENSE, source-license entrypoint, CONTRIBUTING, SECURITY and governance docs.
- [ ] Add Rust `examples/lookup.rs` and Python `examples/lookup.py` consumers of existing query contracts.
- [ ] Run both consumers on the native fixture and compare full lookup outputs, including attribution.
### Task 3: Package and verify source releases
- [ ] Add deterministic source archive and verification commands in `scripts/source_release.py`.
- [ ] Test determinism, dirty-tree refusal, no-clobber, tampering and unsafe archive members in `tests/test_source_release.py`.
- [ ] Add `scripts/check.sh` and `.forgejo/workflows/ci.yml` using an isolated runner without release secrets.
- [ ] Document explicit release signing, downstream pins and incident response in `docs/RELEASING.md` and `docs/TRUST.md`.
### Task 4: Validate and land the bounded task
- [ ] Run `cargo fmt --all -- --check`, `cargo check --workspace --all-targets --locked --offline`,
`cargo clippy --workspace --all-targets --locked --offline -- -D warnings`,
`cargo test --workspace --locked --offline` and strict `cargo doc`.
- [ ] Build the native executable and retain the all-five-source CLI fixture outside the repo.
- [ ] Rebuild/run tests from the release archive outside the original workspace.
- [ ] Review every initial tracked file and dependency change, sign the local commit and release receipt.
- [ ] Record exact validation, publication and Argand cutover state; release any coordination window.

View file

@ -0,0 +1,37 @@
# Standalone Site Registry design
The user approved continuing standalone packaging, release CI and public governance
on 2026-09-12. Preserve the existing AGPL-3.0-or-later license and all data terms.
Create a separate two-crate workspace containing the current registry and atomic
file helper. Preserve Rust module boundaries and the existing source adapters,
SQLite schema, receipt contracts, explicit reviews and regional resolution.
Record the exact signed Argand baseline and original file hashes in UPSTREAM.json.
Do not copy Argand history, deployment configuration, datasets or signing keys.
The existing Argand checkout and shared build caches remain owned by the beta agent.
A separate package is preferable to a permanent second implementation or a new
hosted service: it reuses the existing contracts and runs locally. Extracting only
the CLI would lose the reusable Rust API. Preserve both CLI and library, and prove
non-Rust consumption through the JSON CLI without creating a second resolver.
Ship a focused Cargo.lock, complete code license, beginner quickstart, native Rust
and Python examples, contribution requirements, trust policy and incident/revocation
procedure. Source releases come from an exact committed tree, with deterministic
archives and hash receipts; signing is an explicit local release action. CI checks
formatting, all targets, strict lints, tests, documentation and a native all-source
fixture, and exercises the archive outside its Git checkout. No CI signing key,
provider credentials, production runner or automatic dataset promotion is included.
Keep Argand's current consumer operational. Until an independently hosted release
is accepted into Argand, its embedded copy remains the active consumer; document
the cutover and verify baseline parity. The initial standalone package has no
runtime behavior changes. Each dataset publisher chooses its own external trust
roots; review policy is public and cannot be replaced by popularity or hostname
similarity. An authenticated release is not proof of source accuracy or malware safety.
Acceptance: preserve upstream Rust/schema bytes; build with only this workspace;
run the complete two-crate offline gates plus fresh CLI and Rust/Python examples;
produce the same source archive twice; reject modified releases, unsafe paths and
release overwrites; inspect the initial Git inventory for private/unrelated content.
Remote publication and runner activation are separate from local package readiness.