feat: harden reviewed registry releases
All checks were successful
Standalone registry checks / check (push) Successful in 3m43s
All checks were successful
Standalone registry checks / check (push) Successful in 3m43s
This commit is contained in:
parent
0bcd4a2fc9
commit
e83f43d00f
37 changed files with 2856 additions and 286 deletions
35
docs/EVALUATION.md
Normal file
35
docs/EVALUATION.md
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
# Resolver evaluation
|
||||
|
||||
`evaluate` replays authored judgments through the same native resolver used by
|
||||
consumers. It opens one externally pinned immutable generation, streams up to the
|
||||
configured case limit and reports correctness plus native p50/p95 query latency.
|
||||
Pass an explicit `--at` time when the report must be replayable across approval
|
||||
expiry boundaries; the chosen clock is included in the report.
|
||||
The input is bounded to 16 MiB, each line to 64 KiB and case IDs must be unique.
|
||||
|
||||
Each nonempty JSONL line has this form:
|
||||
|
||||
```json
|
||||
{"id":"facebook-primary","query":"facebook","locale":null,"country":null,"expected_status":"resolved","expected_entity_id":"argand:entity:SOURCE:ID","expected_url":"https://www.facebook.com/"}
|
||||
```
|
||||
|
||||
`locale`, `country`, `expected_entity_id` and `expected_url` are optional. Status
|
||||
is required and is one of `resolved`, `no_name_match`, `ambiguous_identity`,
|
||||
`safety_limit_exceeded`, `no_eligible_destination`, `no_active_review`,
|
||||
`region_mismatch` or `ambiguous_destination`. Expected URLs must be copied from a
|
||||
pinned registry, including normalization such as a trailing slash.
|
||||
|
||||
```bash
|
||||
argand-site-registry evaluate --generation /data/registry/reviewed \
|
||||
--pin "$REVIEWED_PIN" --cases /data/evaluation/navigation.jsonl \
|
||||
--maximum-cases 10000 --at 2026-09-13T00:00:00Z \
|
||||
> /data/evaluation/report.json
|
||||
jq -e '.failed == 0 and .passed == .total' /data/evaluation/report.json
|
||||
```
|
||||
|
||||
Keep the corpus version and digest with the report. Include canonical names,
|
||||
aliases, Unicode normalization, multiple scripts, every served country/locale,
|
||||
unknown names, deceptive lookalikes, ambiguous entities, expired/revoked reviews
|
||||
and tied destinations. Use synthetic or authorized query material; do not commit
|
||||
private user logs. Latency values are process and hardware measurements. Compare
|
||||
them only with an equivalent environment and sufficient sample size.
|
||||
Loading…
Add table
Add a link
Reference in a new issue