feat: harden reviewed registry releases
All checks were successful
Standalone registry checks / check (push) Successful in 3m43s

This commit is contained in:
Nic Weyand 2026-09-13 01:19:27 -04:00
commit e83f43d00f
Signed by: nicweyand
SSH key fingerprint: SHA256:2te+ycJIQON/Wo/dH6+ZkFSQ4HnHWpetV2azx9E65dQ
37 changed files with 2856 additions and 286 deletions

35
docs/EVALUATION.md Normal file
View file

@ -0,0 +1,35 @@
# Resolver evaluation
`evaluate` replays authored judgments through the same native resolver used by
consumers. It opens one externally pinned immutable generation, streams up to the
configured case limit and reports correctness plus native p50/p95 query latency.
Pass an explicit `--at` time when the report must be replayable across approval
expiry boundaries; the chosen clock is included in the report.
The input is bounded to 16 MiB, each line to 64 KiB and case IDs must be unique.
Each nonempty JSONL line has this form:
```json
{"id":"facebook-primary","query":"facebook","locale":null,"country":null,"expected_status":"resolved","expected_entity_id":"argand:entity:SOURCE:ID","expected_url":"https://www.facebook.com/"}
```
`locale`, `country`, `expected_entity_id` and `expected_url` are optional. Status
is required and is one of `resolved`, `no_name_match`, `ambiguous_identity`,
`safety_limit_exceeded`, `no_eligible_destination`, `no_active_review`,
`region_mismatch` or `ambiguous_destination`. Expected URLs must be copied from a
pinned registry, including normalization such as a trailing slash.
```bash
argand-site-registry evaluate --generation /data/registry/reviewed \
--pin "$REVIEWED_PIN" --cases /data/evaluation/navigation.jsonl \
--maximum-cases 10000 --at 2026-09-13T00:00:00Z \
> /data/evaluation/report.json
jq -e '.failed == 0 and .passed == .total' /data/evaluation/report.json
```
Keep the corpus version and digest with the report. Include canonical names,
aliases, Unicode normalization, multiple scripts, every served country/locale,
unknown names, deceptive lookalikes, ambiguous entities, expired/revoked reviews
and tied destinations. Use synthetic or authorized query material; do not commit
private user logs. Latency values are process and hardware measurements. Compare
them only with an equivalent environment and sufficient sample size.