Compare commits

..

3 commits

Author SHA1 Message Date
62e7a67cba
docs: publish the v0.6.2 catalog
Some checks failed
Standalone registry checks / check (push) Has been cancelled
2026-09-22 09:29:15 -04:00
3c89a540d9
fix: retain PSL in compact catalogs
Some checks failed
Standalone registry checks / check (push) Has been cancelled
2026-09-22 09:09:41 -04:00
3b6966c81a
fix: accept non-DNS Common Crawl graph rows
Some checks failed
Standalone registry checks / check (push) Has been cancelled
2026-09-22 09:01:39 -04:00
11 changed files with 116 additions and 44 deletions

View file

@ -1,5 +1,18 @@
# Changelog # Changelog
## 0.6.2 - 2026-09-22
- Retain the PSL normalization fact in compact catalogs and exercise the review
queue against the compact runtime rather than only the full writer projection.
## 0.6.1 - 2026-09-22
- Accept complete Common Crawl domain-rank releases containing provider rows
that are not valid DNS hostnames. Such rows remain authenticated and counted
in source coordinates but cannot match or enter the official-domain catalog.
- Keep malformed graph schemas and numeric fields fail-closed, with a regression
derived from the real `com.your_domain` provider row.
## 0.6.0 - 2026-09-22 ## 0.6.0 - 2026-09-22
- Add a streaming Common Crawl domain Web Graph adapter for harmonic-centrality, - Add a streaming Common Crawl domain Web Graph adapter for harmonic-centrality,

4
Cargo.lock generated
View file

@ -78,14 +78,14 @@ checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
[[package]] [[package]]
name = "argand-atomic" name = "argand-atomic"
version = "0.6.0" version = "0.6.2"
dependencies = [ dependencies = [
"tempfile", "tempfile",
] ]
[[package]] [[package]]
name = "argand-site-registry" name = "argand-site-registry"
version = "0.6.0" version = "0.6.2"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"argand-atomic", "argand-atomic",

View file

@ -4,7 +4,7 @@ resolver = "3"
members = ["crates/argand-atomic", "crates/argand-site-registry"] members = ["crates/argand-atomic", "crates/argand-site-registry"]
[workspace.package] [workspace.package]
version = "0.6.0" version = "0.6.2"
authors = ["Nic Weyand"] authors = ["Nic Weyand"]
edition = "2024" edition = "2024"
license = "AGPL-3.0-or-later" license = "AGPL-3.0-or-later"

View file

@ -15,7 +15,7 @@ facebook -> Facebook (Wikidata Q355) -> https://www.facebook.com/
The repository contains the library, CLI, schemas, migrations, synthetic The repository contains the library, CLI, schemas, migrations, synthetic
fixtures, and independently authenticated public trust roots. It does not place a fixtures, and independently authenticated public trust roots. It does not place a
mutable production database in Git. Publishers import source evidence, collect mutable production database in Git. Publishers import source evidence, collect
signed reviews, and distribute immutable signed registry generations. The first signed reviews, and distribute immutable signed registry generations. The current
public catalog generation is available as a release asset; see public catalog generation is available as a release asset; see
[Public catalog](docs/PUBLIC_CATALOG.md). [Public catalog](docs/PUBLIC_CATALOG.md).

View file

@ -43,9 +43,15 @@ impl SourceAdapter for DomainRanks {
let harmonic_value = nonnegative_finite(fields[1], "harmonic value")?; let harmonic_value = nonnegative_finite(fields[1], "harmonic value")?;
let pagerank_rank = positive_integer(fields[2], "PageRank rank")?; let pagerank_rank = positive_integer(fields[2], "PageRank rank")?;
let pagerank_value = nonnegative_finite(fields[3], "PageRank value")?; let pagerank_value = nonnegative_finite(fields[3], "PageRank value")?;
let target = reverse_domain(fields[4])?;
let member_hosts = positive_integer(fields[5], "member host count")?; let member_hosts = positive_integer(fields[5], "member host count")?;
source_row += 1; source_row += 1;
let Some(target) = reverse_domain(fields[4]) else {
// The provider graph contains a small amount of underscore and
// otherwise non-DNS host material. It cannot match the registry's
// normalized public identity domains, but it remains part of the
// authenticated input stream and source-row coordinate space.
continue;
};
if !self.targets.contains(&target) { if !self.targets.contains(&target) {
continue; continue;
} }
@ -118,18 +124,13 @@ fn nonnegative_finite(value: &str, field: &str) -> anyhow::Result<f64> {
Ok(parsed) Ok(parsed)
} }
fn reverse_domain(value: &str) -> anyhow::Result<String> { fn reverse_domain(value: &str) -> Option<String> {
ensure!( if value.is_empty() || value.len() > 253 || value != value.trim() {
!value.is_empty() && value.len() <= 253 && value == value.trim(), return None;
"invalid reversed domain" }
);
let labels = value.split('.').collect::<Vec<_>>(); let labels = value.split('.').collect::<Vec<_>>();
ensure!( if labels.len() < 2
labels.len() >= 2, || !labels.iter().all(|label| {
"reversed domain needs at least two labels"
);
ensure!(
labels.iter().all(|label| {
!label.is_empty() !label.is_empty()
&& label.len() <= 63 && label.len() <= 63
&& label && label
@ -143,8 +144,9 @@ fn reverse_domain(value: &str) -> anyhow::Result<String> {
.as_bytes() .as_bytes()
.last() .last()
.is_some_and(u8::is_ascii_alphanumeric) .is_some_and(u8::is_ascii_alphanumeric)
}), })
"invalid reversed domain label" {
); return None;
Ok(labels.into_iter().rev().collect::<Vec<_>>().join(".")) }
Some(labels.into_iter().rev().collect::<Vec<_>>().join("."))
} }

View file

@ -616,7 +616,8 @@ pub(crate) fn compact_runtime(db: &Connection) -> anyhow::Result<()> {
INSERT OR IGNORE INTO runtime_facts SELECT fact FROM names; INSERT OR IGNORE INTO runtime_facts SELECT fact FROM names;
INSERT OR IGNORE INTO runtime_facts SELECT fact FROM popularity; INSERT OR IGNORE INTO runtime_facts SELECT fact FROM popularity;
INSERT OR IGNORE INTO runtime_facts SELECT fact FROM rejected; INSERT OR IGNORE INTO runtime_facts SELECT fact FROM rejected;
INSERT OR IGNORE INTO runtime_facts SELECT item.value FROM edges,json_each(edges.facts) item;", INSERT OR IGNORE INTO runtime_facts SELECT item.value FROM edges,json_each(edges.facts) item;
INSERT OR IGNORE INTO runtime_facts SELECT id FROM facts WHERE predicate='psl';",
)?; )?;
db.execute( db.execute(
"DELETE FROM facts WHERE NOT EXISTS(SELECT 1 FROM runtime_facts r WHERE r.id=facts.id)", "DELETE FROM facts WHERE NOT EXISTS(SELECT 1 FROM runtime_facts r WHERE r.id=facts.id)",

View file

@ -267,6 +267,9 @@ fn compact_generation_keeps_runtime_results_and_authenticates_cold_history() ->
serde_json::to_value(full.lookup("Facebook", 20)?.candidates)?, serde_json::to_value(full.lookup("Facebook", 20)?.candidates)?,
serde_json::to_value(compact.lookup("Facebook", 20)?.candidates)? serde_json::to_value(compact.lookup("Facebook", 20)?.candidates)?
); );
let compact_queue =
argand_site_registry::queue::review_queue(&compact, common::timestamp()?, 100, 1_000)?;
assert!(!compact_queue.items.is_empty());
assert!( assert!(
std::fs::metadata(compact_path.join("registry.sqlite"))?.len() std::fs::metadata(compact_path.join("registry.sqlite"))?.len()
< std::fs::metadata(root.path().join("full/registry.sqlite"))?.len() < std::fs::metadata(root.path().join("full/registry.sqlite"))?.len()

View file

@ -153,7 +153,6 @@ fn domain_rank_parser_rejects_schema_and_value_drift() -> anyhow::Result<()> {
"#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n\n", "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n\n",
"#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n0\t1\t1\t1\tcom.example\t1\n", "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n0\t1\t1\t1\tcom.example\t1\n",
"#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n1\tNaN\t1\t1\tcom.example\t1\n", "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n1\tNaN\t1\t1\tcom.example\t1\n",
"#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n1\t1\t1\t1\tcom..example\t1\n",
"#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n1\t1\t1\t1\tcom.example\t0\n", "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n1\t1\t1\t1\tcom.example\t0\n",
] { ] {
let root = tempfile::tempdir()?; let root = tempfile::tempdir()?;
@ -167,6 +166,30 @@ fn domain_rank_parser_rejects_schema_and_value_drift() -> anyhow::Result<()> {
Ok(()) Ok(())
} }
#[test]
fn non_dns_provider_rows_are_skipped_without_losing_source_coordinates() -> anyhow::Result<()> {
let root = tempfile::tempdir()?;
let mut db = store::open(&root.path().join("store.sqlite"))?;
import_identity_candidates(&mut db, root.path())?;
let input = "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n\
1\t1\t1\t1\tcom.your_domain\t15\n\
2\t1\t2\t1\tcom.facebook\t18795\n";
import_graph(&mut db, root.path(), input.as_bytes())?;
let retained: Vec<(String, String)> = {
let mut statement = db.prepare(
"SELECT r.native_id,f.value FROM records r JOIN facts f ON f.source_id=r.source_id AND f.ordinal=r.ordinal WHERE f.predicate='popularity' ORDER BY r.native_id",
)?;
statement
.query_map([], |row| Ok((row.get(0)?, row.get(1)?)))?
.collect::<Result<_, _>>()?
};
assert_eq!(retained.len(), 1);
assert_eq!(retained[0].0, "row:2");
assert!(retained[0].1.contains("facebook.com"));
Ok(())
}
#[test] #[test]
fn domain_rank_source_url_is_exactly_allowlisted() -> anyhow::Result<()> { fn domain_rank_source_url_is_exactly_allowlisted() -> anyhow::Result<()> {
let good = "https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/cc-main-2022-may-jun-aug-domain-ranks.txt.gz"; let good = "https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/cc-main-2022-may-jun-aug-domain-ranks.txt.gz";

View file

@ -34,7 +34,7 @@ source attribution and application-specific malware/content policy.
## Current contracts ## Current contracts
Code version 0.6.0 uses writer schema 5 and `argand.site-rules/v4`. Code version 0.6.2 uses writer schema 5 and `argand.site-rules/v4`.
New compact `COMPLETE.json` files use `argand.site-registry/v3` and bind: New compact `COMPLETE.json` files use `argand.site-registry/v3` and bind:
- authenticated `registry.sqlite` bytes; - authenticated `registry.sqlite` bytes;
@ -102,9 +102,11 @@ revocation history. Never mutate a complete generation to migrate it. See
`UPSTREAM.json` records the original Argand extraction baseline and file hashes. `UPSTREAM.json` records the original Argand extraction baseline and file hashes.
Argand's prior integration pinned signed v0.5.0 revision Argand's prior integration pinned signed v0.5.0 revision
`3d3e08cdfd303df9fbd347a9bab2ba52ad575759`; the v0.6 downstream pin is recorded `3d3e08cdfd303df9fbd347a9bab2ba52ad575759`. Argand now pins the public signed
by the Argand integration commit after this source release. The public beta uses Site Registry as v0.6.2 source release at `3c89a540d910cb298ba752880efeb1ed157dab43` in
Navigate's authoritative auto-route catalog. Its native `navigation-catalog/v2` downstream commit `224616eb9f6685d1a656b113b8460fb80c0c5a6b`.
The public beta uses Site Registry as Navigate's authoritative auto-route
catalog. Its native `navigation-catalog/v2`
file is only a collection- and content-policy-bound serving projection compiled file is only a collection- and content-policy-bound serving projection compiled
from one exact registry generation; it is not a second independently curated from one exact registry generation; it is not a second independently curated
destination catalog. destination catalog.

View file

@ -1,14 +1,14 @@
# Public signed catalog # Public signed catalog
The v0.5.0 Forgejo release publishes the first immutable data generation that any The v0.6.2 Forgejo release publishes the current immutable data generation that
Site Registry consumer can verify and resolve: Argand and any other Site Registry consumer can verify and resolve:
- release: <https://git.argand.org/nicweyand/argand-site-registry/releases/tag/v0.5.0> - release: <https://git.argand.org/nicweyand/argand-site-registry/releases/tag/v0.6.2>
- asset: `argand-site-registry-catalog-20260920-v1.tar.gz` - asset: `argand-site-registry-catalog-v0.6.2.tar.gz`
- asset SHA-256: - asset SHA-256:
`d878fa057397effa5dc729d2fa3a689c8edd1f4112ef1326dd6131b3fdeab63e` `48b0cdf453862d858c4bec6c564360e1309605e30af9aba1f54a9446b9bdbe41`
- generation pin: - generation pin:
`ede14746da8817aafdf705dd88cfeabbe8d23e1991e43a304acd8eca9249b18a` `5e5d8fd5dc1864dc3f4c53ec71cb5ac64f6db592cfbc8cc56f48a444378e2309`
The release also carries a checksum file and an OpenSSH signature under namespace The release also carries a checksum file and an OpenSSH signature under namespace
`argand-site-registry-release`. Verify it against `argand-site-registry-release`. Verify it against
@ -17,13 +17,13 @@ The signed Git history is the independent channel for the trust root; do not lea
the only trusted key from the archive it authenticates. the only trusted key from the archive it authenticates.
```bash ```bash
sha256sum --check argand-site-registry-catalog-20260920-v1.tar.gz.sha256 sha256sum --check argand-site-registry-catalog-v0.6.2.tar.gz.sha256
ssh-keygen -Y verify \ ssh-keygen -Y verify \
-f trust/public-catalog-20260920/publisher-allowed-signers \ -f trust/public-catalog-20260920/publisher-allowed-signers \
-I argand-site-registry-publisher-v1 \ -I argand-site-registry-publisher-v1 \
-n argand-site-registry-release \ -n argand-site-registry-release \
-s argand-site-registry-catalog-20260920-v1.tar.gz.sig \ -s argand-site-registry-catalog-v0.6.2.tar.gz.sig \
< argand-site-registry-catalog-20260920-v1.tar.gz < argand-site-registry-catalog-v0.6.2.tar.gz
``` ```
After extraction, verify every member with `SHA256SUMS`, then authenticate the After extraction, verify every member with `SHA256SUMS`, then authenticate the
@ -31,25 +31,36 @@ generation and exact reviewer trust root:
```bash ```bash
argand-site-registry activate \ argand-site-registry activate \
--generation public-release-v0.5.0/catalog \ --generation public-release-v0.6.2/catalog \
--current current.json \ --current current.json \
--allowed-signers trust/public-catalog-20260920/publisher-allowed-signers \ --allowed-signers trust/public-catalog-20260920/publisher-allowed-signers \
--allowed-reviewers trust/public-catalog-20260920/reviewer-allowed-signers \ --allowed-reviewers trust/public-catalog-20260920/reviewer-allowed-signers \
--identity argand-site-registry-publisher-v1 --identity argand-site-registry-publisher-v1
argand-site-registry resolve \ argand-site-registry resolve \
--generation public-release-v0.5.0/catalog \ --generation public-release-v0.6.2/catalog \
--pin ede14746da8817aafdf705dd88cfeabbe8d23e1991e43a304acd8eca9249b18a \ --pin 5e5d8fd5dc1864dc3f4c53ec71cb5ac64f6db592cfbc8cc56f48a444378e2309 \
--query "yahoo mail" --query "facebook"
``` ```
## Scope and trust ## Scope and trust
This first catalog is deliberately small. Its disclosed policy uses one automated The v0.6.2 catalog contains 976 entities, 1,062 official-site edges, 20,178
evidence-gate reviewer group rather than claiming human-review quorum. Fresh exact multilingual name facts, and Common Crawl Web Graph evidence for 840 domains that
endpoint observations are required, and source conflicts or dangerous drift need already had imported identity assertions. Graph authority can prioritize review
two groups, so the single automated reviewer must abstain on those risks. Sticky and disambiguation, but cannot create an identity, official-site assertion,
revocations and publisher/reviewer key separation remain enabled. review, vote, or redirect. The archive includes all 33 authenticated cold audit
objects referenced by the compact runtime generation.
The bounded Wikidata discovery input is broad but not a representative or
high-demand sample. Its query and selection metadata are included for audit; raw
discovery output is never approval.
The disclosed policy uses one automated evidence-gate reviewer group rather than
claiming human-review quorum. Fresh exact endpoint observations are required, and
source conflicts or dangerous drift need two groups, so the single automated
reviewer must abstain on those risks. Sticky revocations and
publisher/reviewer-key separation remain enabled.
Consumers decide whether this policy is appropriate for their use. Preserve typed Consumers decide whether this policy is appropriate for their use. Preserve typed
abstentions, retain attribution, and apply independent malware and content policy. abstentions, retain attribution, and apply independent malware and content policy.
@ -60,3 +71,12 @@ The generation's approvals expire. Installing an immutable archive is not a prom
that every decision stays valid forever: use the resolver's requested time, that every decision stays valid forever: use the resolver's requested time,
consume cumulative signed revocation feeds when published, and move to a newly consume cumulative signed revocation feeds when published, and move to a newly
signed full generation before relying on renewed decisions. signed full generation before relying on renewed decisions.
## Regular updates
The public source repository includes the same updater used to refresh candidate
generations. The example systemd timer runs weekly. It can download, authenticate,
import and build, but it holds no publisher key and cannot approve, sign or
activate a candidate. That separation lets any consumer automate evidence updates
without allowing a compromised downloader or changed upstream dataset to silently
change redirects.

View file

@ -2,6 +2,14 @@
# Version 0.6.0 Web Graph and updater validation, 2026-09-22 # Version 0.6.0 Web Graph and updater validation, 2026-09-22
Patch release 0.6.1 additionally replays the real provider-shaped
`com.your_domain` case: the row remains in authenticated input/coordinate
accounting but is not retained as a DNS target. The following valid
`com.facebook` row is retained at its original `row:2` coordinate. Schema and
numeric corruption continue to fail closed.
The compact-generation lifecycle now also executes the review queue, proving its
PSL normalization input remains in the runtime catalog.
The evaluation contract was written before implementation. The new Common Crawl The evaluation contract was written before implementation. The new Common Crawl
domain-rank integration passed five default integration tests; its sixth test is domain-rank integration passed five default integration tests; its sixth test is
an explicit resource benchmark. The benchmark parsed 100,000 valid provider-shaped an explicit resource benchmark. The benchmark parsed 100,000 valid provider-shaped