37 lines
1.9 KiB
Markdown
37 lines
1.9 KiB
Markdown
# Resolver evaluation
|
|
|
|
`evaluate` streams authored JSONL judgments through the same pinned native reader
|
|
used by consumers. It reports correctness and p50/p95 latency. Pass `--at` for a
|
|
reproducible policy clock; approval expiry and future votes otherwise depend on
|
|
current time. Inputs are bounded to 16 MiB, lines to 64 KiB, and case IDs must be
|
|
unique.
|
|
|
|
```json
|
|
{"id":"facebook-primary","query":"facebook","locale":null,"country":null,"expected_status":"resolved","expected_entity_id":"argand:entity:SOURCE:ID","expected_url":"https://www.facebook.com/"}
|
|
```
|
|
|
|
Optional fields are `locale`, `country`, `expected_entity_id`, and `expected_url`.
|
|
Current statuses are `resolved`, `no_name_match`, `ambiguous_identity`,
|
|
`no_active_name_review`, `safety_limit_exceeded`, `no_eligible_destination`,
|
|
`no_active_review`, `region_mismatch`, and `ambiguous_destination`. Copy expected
|
|
URLs from a pinned registry, including normalization such as trailing slash.
|
|
|
|
```bash
|
|
argand-site-registry evaluate --generation /data/registry/reviewed \
|
|
--pin "$REVIEWED_PIN" --cases /data/evaluation/navigation.jsonl \
|
|
--maximum-cases 10000 --at 2026-09-13T00:00:00Z \
|
|
> /data/evaluation/report.json
|
|
jq -e '.failed == 0 and .passed == .total' /data/evaluation/report.json
|
|
```
|
|
|
|
Keep corpus version and digest, registry pin, policy digest, reviewer-trust digest,
|
|
process build and hardware with the report. Cover canonical names, aliases,
|
|
Unicode normalization, scripts, regions, unknown names, lookalikes, ambiguity,
|
|
missing name votes, missing edge votes, correlated reviewers, stale policy,
|
|
stale observations, expiry, revocation, sticky supersession and tied routes.
|
|
Segment failures by safe abstention and wrong resolved route; any wrong resolved
|
|
route is a release blocker.
|
|
|
|
Use synthetic or authorized queries and do not commit private user logs. Latency is
|
|
process and hardware evidence. Compare only equivalent native configurations with
|
|
sufficient samples.
|