nicweyand
e83f43d00f
All checks were successful
Standalone registry checks / check (push) Successful in 3m43s
35 lines
1.8 KiB
Markdown
35 lines
1.8 KiB
Markdown
# Resolver evaluation
|
|
|
|
`evaluate` replays authored judgments through the same native resolver used by
|
|
consumers. It opens one externally pinned immutable generation, streams up to the
|
|
configured case limit and reports correctness plus native p50/p95 query latency.
|
|
Pass an explicit `--at` time when the report must be replayable across approval
|
|
expiry boundaries; the chosen clock is included in the report.
|
|
The input is bounded to 16 MiB, each line to 64 KiB and case IDs must be unique.
|
|
|
|
Each nonempty JSONL line has this form:
|
|
|
|
```json
|
|
{"id":"facebook-primary","query":"facebook","locale":null,"country":null,"expected_status":"resolved","expected_entity_id":"argand:entity:SOURCE:ID","expected_url":"https://www.facebook.com/"}
|
|
```
|
|
|
|
`locale`, `country`, `expected_entity_id` and `expected_url` are optional. Status
|
|
is required and is one of `resolved`, `no_name_match`, `ambiguous_identity`,
|
|
`safety_limit_exceeded`, `no_eligible_destination`, `no_active_review`,
|
|
`region_mismatch` or `ambiguous_destination`. Expected URLs must be copied from a
|
|
pinned registry, including normalization such as a trailing slash.
|
|
|
|
```bash
|
|
argand-site-registry evaluate --generation /data/registry/reviewed \
|
|
--pin "$REVIEWED_PIN" --cases /data/evaluation/navigation.jsonl \
|
|
--maximum-cases 10000 --at 2026-09-13T00:00:00Z \
|
|
> /data/evaluation/report.json
|
|
jq -e '.failed == 0 and .passed == .total' /data/evaluation/report.json
|
|
```
|
|
|
|
Keep the corpus version and digest with the report. Include canonical names,
|
|
aliases, Unicode normalization, multiple scripts, every served country/locale,
|
|
unknown names, deceptive lookalikes, ambiguous entities, expired/revoked reviews
|
|
and tied destinations. Use synthetic or authorized query material; do not commit
|
|
private user logs. Latency values are process and hardware measurements. Compare
|
|
them only with an equivalent environment and sufficient sample size.
|