release: add Web Graph authority evidence for v0.6

crates/argand-site-registry/LICENSE_SOURCES.md

@@ -14,6 +14,7 @@ listing is evidence of an assertion, not a guarantee of ownership or safety.
 | Chrome UX Report (CrUX), Google | [CC BY 4.0 International](https://creativecommons.org/licenses/by/4.0/), [`CC-BY-4.0`](https://developer.chrome.com/docs/crux/methodology) | [Monthly BigQuery dataset](https://developer.chrome.com/docs/crux/bigquery/): `origin`, `experimental.popularity.rank`, observation month, optional audience-country dataset code. The adapter produces `origin,rank,yyyymm,country_code` CSV. Rank is a coarse bucket, not a precise visit count. Audience country is not website jurisdiction. No API key or OAuth token is retained. |
 | Curlie | [CC BY 3.0 Unported](https://creativecommons.org/licenses/by/3.0/), [`CC-BY-3.0`](https://curlie.org/docs/en/license.html), including the attribution placement prescribed on that page | [Format documentation](https://curlie.org/docs/en/rdf.html), [official download redirect](https://curlie.org/directory-dl), currently [Passau-hosted archive](https://share.innkube.fim.uni-passau.de/curlie-rdf/curlie-rdf-all.tar.gz). Despite its RDF name, the current archive contains **literal TSV**. Content: URL, title, description, category ID. Structure: category ID, full category path, entry count, description, latitude, longitude. Archive notices are retained. |
 | Public Suffix List contributors | [Mozilla Public License 2.0](https://mozilla.org/MPL/2.0/), [`MPL-2.0`](https://publicsuffix.org/list/public_suffix_list.dat) | [Official list](https://publicsuffix.org/list/public_suffix_list.dat). All ICANN and PRIVATE rules, wildcard/exception rules, version/commit comments and notices. Used for hostname, registrable-domain and public-suffix derivations. Download at most once per day. |
+| Common Crawl Web Graph | [Common Crawl Terms of Use](https://commoncrawl.org/terms-of-use), `LicenseRef-Common-Crawl-Terms-of-Use` | [Official Web Graph releases](https://index.commoncrawl.org/web-graphs-index.html). The domain-rank adapter consumes the exact six-column rank file: harmonic-centrality rank/value, PageRank rank/value, reversed registered domain, and provider `n_hosts`. `n_hosts` is retained as `member_hosts`; it is not represented as inbound-linking hosts. This is authority/popularity evidence only and never establishes entity ownership, query popularity, safety, or route approval. |
 | Argand candidate observer | [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/), `CC0-1.0` | Local host-side observations authored by the registry publisher: HTTP status and redirect targets, canonical/hreflang/JSON-LD/sitemap/country-selector targets, public DNS-set hash, TLS leaf-certificate hash, bounded failure class, content hash and selectors. These records describe a capture; they do not incorporate page prose or prove ownership. |
 
 ## Attribution and distribution
@@ -56,6 +57,12 @@ listing is evidence of an assertion, not a guarantee of ownership or safety.
   and `facts`; exports include the PSL fact and original download locator.
   Changes to covered PSL source files must remain available under MPL 2.0.
   The Rust `publicsuffix` parser is MIT/Apache-2.0; that is separate from the list.
+* **Common Crawl Web Graph:** retain the exact release and object URL, retrieval
+  time, content digest, Terms of Use link, and identify Argand's reversed-domain
+  projection. Common Crawl's Terms are not an SPDX open-data license and may
+  change; re-review them for every new acquisition. The rank file describes a
+  crawl-derived graph and does not transfer rights in crawled pages. Do not use
+  rank alone to assert ownership, safety, trust, or an official destination.
 * **Argand observer:** locally produced observation metadata is dedicated under
   CC0. The fetched page remains subject to its own rights. The default observer
   stores only a bounded body in the local replay cache and emits normalized link,

crates/argand-site-registry/README.md

@@ -230,6 +230,36 @@ The month belongs in the source snapshot identity. With scheduled typed
 supersession enabled, the next month then replaces the same audience partition
 instead of accumulating stale popularity facts.
 
+### Common Crawl Web Graph
+
+The adapter accepts the official domain-level rank object whose exact header is:
+
+```text
+#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts
+```
+
+It reverses `host_rev` (`com.facebook` to `facebook.com`) and retains harmonic
+centrality, PageRank and `n_hosts` as source-separated popularity evidence.
+`n_hosts` means hosts belonging to the registered domain and is exposed as
+`member_hosts`; it is not an inbound-link count. The adapter creates no entity,
+name, official-site edge, review, vote or route.
+
+Only exact domain-rank objects below the official
+`data.commoncrawl.org/projects/hyperlinkgraph/<release>/domain/` hierarchy are
+allowlisted. Use `compression: gzip`, the exact release ID as the snapshot, a
+positive byte ceiling, and typed coverage. The downloader and importer hash and
+consume the complete object; a byte-range prefix must not be declared as the
+complete source. Common Crawl's Terms of Use are not an SPDX open-data license,
+so preserve the Terms link and re-review it on each acquisition.
+
+Domain-rank files contain tens of millions of rows. Import the PSL and public
+identity sources first, then run `web-graph-selection --database ...` to obtain
+the exact `candidate-domains:<sha256>` scope. A scheduled update may instead use
+the `{candidate_domains}` token. The importer still parses and authenticates the
+complete stream but persists only matching registrable domains, with original
+row coordinates. This keeps the public updater reproducible and compact. Rank
+still cannot replace reviewer quorum or destination-safety checks.
+
 ## Build, inspect and review
 
 The strict default requires an independently maintained OpenSSH reviewer trust

crates/argand-site-registry/examples/update.toml

@@ -56,6 +56,19 @@ collection = "default"
 kind = "full"
 supersedes = []
 
+# Optional Common Crawl Web Graph authority evidence. The updater imports
+# identity sources first, replaces this token with their exact sorted-domain
+# digest, authenticates the complete graph stream, and retains only matching
+# registrable domains. Popularity never authorizes a redirect.
+# [[downloads]]
+# source = "common_crawl_web_graph"
+# format = "common_crawl_domain_ranks_tsv"
+# compression = "gzip"
+# url = "https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/cc-main-2022-may-jun-aug-domain-ranks.txt.gz"
+# snapshot = "cc-main-2022-may-jun-aug"
+# scope = "{candidate_domains}"
+# maximum_bytes = 3000000000
+
 # Optional pinned acquisitions; repeat [[inputs]] for each source.
 # [[inputs]]
 # input = "/data/source-object.gz"

crates/argand-site-registry/src/adapters/common_crawl_web_graph.rs

@@ -0,0 +1,150 @@
+// By Nic Weyand!
+//! Streaming Common Crawl domain-rank projection; graph rank never creates identity.
+
+use super::{RecordSink, SourceAdapter, bounded_line};
+use crate::model::{Fact, Record};
+use anyhow::ensure;
+use serde_json::json;
+use std::{collections::BTreeSet, io::BufRead};
+
+const HEADER: [&str; 6] = [
+    "#harmonicc_pos",
+    "#harmonicc_val",
+    "#pr_pos",
+    "#pr_val",
+    "#host_rev",
+    "#n_hosts",
+];
+
+pub(super) struct DomainRanks {
+    pub(super) targets: BTreeSet<String>,
+}
+
+impl SourceAdapter for DomainRanks {
+    fn ingest(&self, input: &mut dyn BufRead, sink: &mut dyn RecordSink) -> anyhow::Result<()> {
+        let mut line = String::new();
+        ensure!(bounded_line(input, &mut line)? > 0, "empty domain-rank TSV");
+        ensure!(columns(&line)? == HEADER, "domain-rank TSV schema changed");
+
+        ensure!(
+            !self.targets.is_empty(),
+            "Web Graph candidate-domain selection is empty"
+        );
+        let mut source_row = 0_u64;
+        let mut emitted = 0_u64;
+        while bounded_line(input, &mut line)? > 0 {
+            ensure!(!line.trim().is_empty(), "blank domain-rank row");
+            let fields = columns(&line)?;
+            ensure!(
+                fields.len() == HEADER.len(),
+                "domain-rank column count changed"
+            );
+            let harmonic_rank = positive_integer(fields[0], "harmonic rank")?;
+            let harmonic_value = nonnegative_finite(fields[1], "harmonic value")?;
+            let pagerank_rank = positive_integer(fields[2], "PageRank rank")?;
+            let pagerank_value = nonnegative_finite(fields[3], "PageRank value")?;
+            let target = reverse_domain(fields[4])?;
+            let member_hosts = positive_integer(fields[5], "member host count")?;
+            source_row += 1;
+            if !self.targets.contains(&target) {
+                continue;
+            }
+            emitted += 1;
+            let raw = json!({
+                "harmonicc_pos": fields[0],
+                "harmonicc_val": fields[1],
+                "pr_pos": fields[2],
+                "pr_val": fields[3],
+                "host_rev": fields[4],
+                "n_hosts": fields[5],
+            });
+            sink.emit(Record {
+                native_id: format!("row:{source_row}"),
+                raw,
+                facts: vec![Fact {
+                    subject: target.clone(),
+                    predicate: "popularity".into(),
+                    value: json!({
+                        "target": target,
+                        "target_kind": "hostname",
+                        "harmonic_rank": harmonic_rank,
+                        "harmonic_value": harmonic_value,
+                        "pagerank_rank": pagerank_rank,
+                        "pagerank_value": pagerank_value,
+                        // Provider n_hosts counts hosts belonging to this domain. It is
+                        // not a count of distinct domains or hosts linking to the target.
+                        "member_hosts": member_hosts,
+                        "country_code": null,
+                        "period": null,
+                    }),
+                    selector: format!("row:{source_row}"),
+                    confidence: 10_000,
+                }],
+            })?;
+        }
+        ensure!(source_row > 0, "empty domain-rank dataset");
+        ensure!(
+            emitted > 0,
+            "Web Graph contains none of the selected candidate domains"
+        );
+        Ok(())
+    }
+}
+
+fn columns(line: &str) -> anyhow::Result<Vec<&str>> {
+    let line = line.strip_suffix('\n').unwrap_or(line);
+    let line = line.strip_suffix('\r').unwrap_or(line);
+    ensure!(
+        !line
+            .chars()
+            .any(|character| character.is_control() && character != '\t'),
+        "control in domain-rank row"
+    );
+    Ok(line.split('\t').collect())
+}
+
+fn positive_integer(value: &str, field: &str) -> anyhow::Result<u64> {
+    let parsed: u64 = value.parse()?;
+    ensure!(parsed > 0, "{field} must be positive");
+    Ok(parsed)
+}
+
+fn nonnegative_finite(value: &str, field: &str) -> anyhow::Result<f64> {
+    let parsed: f64 = value.parse()?;
+    ensure!(
+        parsed.is_finite() && parsed >= 0.0,
+        "{field} must be finite and nonnegative"
+    );
+    Ok(parsed)
+}
+
+fn reverse_domain(value: &str) -> anyhow::Result<String> {
+    ensure!(
+        !value.is_empty() && value.len() <= 253 && value == value.trim(),
+        "invalid reversed domain"
+    );
+    let labels = value.split('.').collect::<Vec<_>>();
+    ensure!(
+        labels.len() >= 2,
+        "reversed domain needs at least two labels"
+    );
+    ensure!(
+        labels.iter().all(|label| {
+            !label.is_empty()
+                && label.len() <= 63
+                && label
+                    .bytes()
+                    .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-')
+                && label
+                    .as_bytes()
+                    .first()
+                    .is_some_and(u8::is_ascii_alphanumeric)
+                && label
+                    .as_bytes()
+                    .last()
+                    .is_some_and(u8::is_ascii_alphanumeric)
+        }),
+        "invalid reversed domain label"
+    );
+    Ok(labels.into_iter().rev().collect::<Vec<_>>().join("."))
+}

crates/argand-site-registry/src/adapters/mod.rs

@@ -4,7 +4,11 @@
 use crate::model::{Fact, Format, Record};
 use anyhow::ensure;
 use serde_json::json;
-use std::io::{BufRead, Read};
+use std::{
+    collections::BTreeSet,
+    io::{BufRead, Read},
+};
+mod common_crawl_web_graph;
 pub(crate) mod csv_sources;
 mod curlie;
 mod ror;
@@ -39,6 +43,7 @@ pub fn adapter(
     format: Format,
     maximum_record_bytes: usize,
     coverage_delta: bool,
+    web_graph_targets: Option<BTreeSet<String>>,
 ) -> Box<dyn SourceAdapter> {
     match format {
         Format::WikidataDump => Box::new(wikidata::Wikidata {
@@ -58,6 +63,9 @@ pub fn adapter(
         Format::RorZip => Box::new(ror::Ror {
             maximum_record_bytes,
         }),
+        Format::CommonCrawlDomainRanksTsv => Box::new(common_crawl_web_graph::DomainRanks {
+            targets: web_graph_targets.unwrap_or_default(),
+        }),
     }
 }
 

crates/argand-site-registry/src/cli.rs

@@ -203,6 +203,11 @@ enum Command {
         #[arg(long)]
         maximum_database_growth_bytes: Option<u64>,
     },
+    /// Compute the exact candidate-domain scope for a compact Web Graph import.
+    WebGraphSelection {
+        #[arg(long)]
+        database: PathBuf,
+    },
     /// Build a new immutable generation; output must not exist.
     Build {
         #[arg(long)]
@@ -739,6 +744,9 @@ pub(super) async fn run() -> anyhow::Result<()> {
             maximum_records,
             maximum_database_growth_bytes,
         )?,
+        Command::WebGraphSelection { database } => serde_json::to_value(
+            registry::store::web_graph_selection(&registry::store::open(&database)?)?,
+        )?,
         Command::ObservationImport {
             database,
             generation,

crates/argand-site-registry/src/download.rs

@@ -102,6 +102,7 @@ pub fn validate_source_url(source: Source, input: &str) -> anyhow::Result<()> {
         }
         Source::Psl => host == "publicsuffix.org" && path == "/list/public_suffix_list.dat",
         Source::Ror => ror_url(host, path),
+        Source::CommonCrawlWebGraph => common_crawl_domain_ranks_url(host, path),
     };
     ensure!(allowed, "unreviewed source endpoint: {host}{path}");
     Ok(())
@@ -130,6 +131,10 @@ pub async fn download(cache: &Path, request: &Download) -> anyhow::Result<Cached
     );
     validate_source_url(request.source, &request.url)?;
     ensure!(request.maximum_bytes > 0, "maximum bytes must be positive");
+    ensure!(
+        request.scope != "{candidate_domains}",
+        "candidate-domain scope token is resolved only by the ordered update command"
+    );
     let requested_proof = requested_checksum_proof(request)?;
     let key = crate::digest(&serde_json::to_vec(request)?);
     let dir = cache.join(request.source.key()).join(key);
@@ -267,6 +272,25 @@ fn ror_url(host: &str, path: &str) -> bool {
         && components[5] == "content"
 }
 
+fn common_crawl_domain_ranks_url(host: &str, path: &str) -> bool {
+    let components = path.trim_start_matches('/').split('/').collect::<Vec<_>>();
+    if host != "data.commoncrawl.org"
+        || components.len() != 5
+        || components[0] != "projects"
+        || components[1] != "hyperlinkgraph"
+        || components[3] != "domain"
+    {
+        return false;
+    }
+    let release = components[2];
+    release.starts_with("cc-main-")
+        && release.len() <= 128
+        && release
+            .bytes()
+            .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-')
+        && components[4] == format!("{release}-domain-ranks.txt.gz")
+}
+
 fn requested_checksum_proof(request: &Download) -> anyhow::Result<Option<IntegrityProof>> {
     match (&request.provider_checksum, &request.provider_checksum_url) {
         (None, None) => Ok(None),
@@ -327,7 +351,11 @@ pub(crate) fn validate_integrity_evidence(
                 && evidence.path().rsplit_once('/').map(|item| item.0) == source_parent
                 && evidence.path().ends_with("/sha256sums.txt")
         }
-        Source::Majestic | Source::Crux | Source::Curlie | Source::Psl => false,
+        Source::Majestic
+        | Source::Crux
+        | Source::Curlie
+        | Source::Psl
+        | Source::CommonCrawlWebGraph => false,
     };
     ensure!(allowed, "unreviewed provider checksum evidence endpoint");
     Ok(())
@@ -536,6 +564,29 @@ fn reserve_psl_refresh(cache: &Path) -> anyhow::Result<()> {
 mod tests {
     use super::*;
 
+    #[tokio::test]
+    async fn web_graph_scope_token_requires_ordered_update() -> anyhow::Result<()> {
+        let root = tempfile::tempdir()?;
+        let request = Download {
+            source: Source::CommonCrawlWebGraph,
+            format: Format::CommonCrawlDomainRanksTsv,
+            compression: Compression::Gzip,
+            url: "https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/cc-main-2022-may-jun-aug-domain-ranks.txt.gz".into(),
+            snapshot: "cc-main-2022-may-jun-aug".into(),
+            scope: "{candidate_domains}".into(),
+            maximum_bytes: 3_000_000_000,
+            maximum_record_bytes: None,
+            coverage: None,
+            provider_checksum: None,
+            provider_checksum_url: None,
+        };
+        let Some(error) = download(root.path(), &request).await.err() else {
+            anyhow::bail!("scope token accepted outside update");
+        };
+        assert!(error.to_string().contains("ordered update command"));
+        Ok(())
+    }
+
     #[test]
     fn validator_bound_ranges_and_source_allowlist() -> anyhow::Result<()> {
         let url = "https://downloads.majestic.com/majestic_million.csv";

crates/argand-site-registry/src/model.rs

@@ -22,6 +22,8 @@ pub enum Source {
     Psl,
     /// Research Organization Registry organization records.
     Ror,
+    /// Common Crawl domain-level Web Graph ranks.
+    CommonCrawlWebGraph,
 }
 
 impl Source {
@@ -35,6 +37,7 @@ impl Source {
             Self::Curlie => "curlie",
             Self::Psl => "psl",
             Self::Ror => "ror",
+            Self::CommonCrawlWebGraph => "common_crawl_web_graph",
         }
     }
     /// Exact SPDX data license.
@@ -45,6 +48,7 @@ impl Source {
             Self::Majestic | Self::Curlie => "CC-BY-3.0",
             Self::Crux => "CC-BY-4.0",
             Self::Psl => "MPL-2.0",
+            Self::CommonCrawlWebGraph => "LicenseRef-Common-Crawl-Terms-of-Use",
         }
     }
     /// Authoritative license evidence page.
@@ -57,6 +61,7 @@ impl Source {
             Self::Curlie => "https://curlie.org/docs/en/license.html",
             Self::Psl => "https://publicsuffix.org/list/public_suffix_list.dat",
             Self::Ror => "https://ror.readme.io/docs/data-dump",
+            Self::CommonCrawlWebGraph => "https://commoncrawl.org/terms-of-use",
         }
     }
 }
@@ -79,6 +84,8 @@ pub enum Format {
     PslText,
     /// Official ROR release ZIP containing schema 2.1 JSON and CSV.
     RorZip,
+    /// Common Crawl's six-column domain-rank TSV.
+    CommonCrawlDomainRanksTsv,
 }
 
 /// How an immutable source object was checked before import.
@@ -408,8 +415,22 @@ impl SourceManifest {
                 | (Source::Curlie, Format::CurlieTarGz)
                 | (Source::Psl, Format::PslText)
                 | (Source::Ror, Format::RorZip)
+                | (
+                    Source::CommonCrawlWebGraph,
+                    Format::CommonCrawlDomainRanksTsv
+                )
         );
         ensure!(valid, "source/format mismatch");
+        if self.source == Source::CommonCrawlWebGraph {
+            let digest = self
+                .scope
+                .strip_prefix("candidate-domains:")
+                .context("Web Graph scope must bind the candidate-domain digest")?;
+            ensure!(
+                valid_digest(digest),
+                "invalid Web Graph candidate-domain digest"
+            );
+        }
         if let Some(coverage) = &self.coverage {
             coverage.validate()?;
         }

crates/argand-site-registry/src/release.rs

@@ -22,6 +22,7 @@ pub fn attribution() -> Value {
         "curlie":{"license":"CC-BY-3.0","credit":"With content from Curlie.org - the largest human-edited directory of the web. Contribute by submitting a website or becoming an editor.","url":"https://curlie.org/","license_url":"https://creativecommons.org/licenses/by/3.0/","public_display":"Use the prescribed HTML attribution on every page using Curlie content: https://curlie.org/docs/en/license.html"},
         "psl":{"license":"MPL-2.0","url":"https://publicsuffix.org/list/","license_url":"https://mozilla.org/MPL/2.0/"},
         "ror":{"license":"CC0-1.0","url":"https://ror.org/","license_url":"https://ror.readme.io/docs/data-dump","lineage_note":"ROR location metadata identifies GeoNames as an upstream CC BY 3.0 source","upstream_attribution":{"credit":"GeoNames","url":"https://www.geonames.org/","license_url":"https://creativecommons.org/licenses/by/3.0/"}},
+        "common_crawl_web_graph":{"license":"LicenseRef-Common-Crawl-Terms-of-Use","credit":"Common Crawl Foundation Web Graph","url":"https://commoncrawl.org/web-graphs","license_url":"https://commoncrawl.org/terms-of-use","scope":"domain-level harmonic centrality, PageRank, and member-host count; graph rank is not ownership or query popularity"},
         "argand_candidate_observer":{"license":"CC0-1.0","url":"https://git.argand.org/nicweyand/argand-site-registry","license_url":"https://creativecommons.org/publicdomain/zero/1.0/","scope":"locally authored observation metadata; captured page content is not redistributed"},
         "changes":"Argand normalizes and combines assertions; provider endorsement is not implied."})
 }

crates/argand-site-registry/src/store.rs

@@ -3,13 +3,16 @@
 
 use crate::{
     adapters::{self, RecordSink},
-    model::{Compression, Record, SourceManifest},
+    model::{Compression, Format, Record, SourceManifest},
+    normalize::Normalizer,
 };
 use anyhow::{Context, ensure};
 use rusqlite::{Connection, OptionalExtension, params};
+use serde::Serialize;
 use sha2::{Digest, Sha256};
 use std::{
     cell::RefCell,
+    collections::BTreeSet,
     io::{BufReader, Read},
     path::Path,
     rc::Rc,
@@ -19,6 +22,59 @@ use std::{
 /// Adapter/normalization contract recorded in all generation identities.
 pub const RULE_VERSION: &str = "argand.site-rules/v4";
 
+/// Deterministic Web Graph projection selected from already imported website evidence.
+#[derive(Clone, Debug, Eq, PartialEq, Serialize)]
+pub struct WebGraphSelection {
+    /// Replacement scope that must be bound into the Web Graph source manifest.
+    pub scope: String,
+    /// Number of distinct registrable domains retained from the graph.
+    pub domains: u64,
+}
+
+/// Computes the exact public-identity domain set used by a compact Web Graph import.
+///
+/// # Errors
+/// Requires one completed PSL source and at least one valid website assertion.
+pub fn web_graph_selection(db: &Connection) -> anyhow::Result<WebGraphSelection> {
+    let (selection, _) = web_graph_targets(db)?;
+    Ok(selection)
+}
+
+fn web_graph_targets(db: &Connection) -> anyhow::Result<(WebGraphSelection, BTreeSet<String>)> {
+    let (psl_source, encoded): (String, String) = db
+        .query_row(
+            "SELECT f.source_id,f.value FROM facts f JOIN sources s ON s.id=f.source_id WHERE s.complete=1 AND f.predicate='psl' ORDER BY f.source_id LIMIT 1",
+            [],
+            |row| Ok((row.get(0)?, row.get(1)?)),
+        )
+        .context("import a complete PSL snapshot before Common Crawl Web Graph")?;
+    let psl: String = serde_json::from_str(&encoded)?;
+    let normalizer = Normalizer::new(psl.as_bytes(), psl_source)?;
+    let mut statement = db.prepare(
+        "SELECT f.value FROM facts f JOIN sources s ON s.id=f.source_id WHERE s.complete=1 AND f.predicate='website' ORDER BY f.id",
+    )?;
+    let values = statement.query_map([], |row| row.get::<_, String>(0))?;
+    let mut domains = BTreeSet::new();
+    for encoded in values {
+        let value: serde_json::Value = serde_json::from_str(&encoded?)?;
+        if let Some(url) = value.get("url").and_then(serde_json::Value::as_str)
+            && let Ok(property) = normalizer.url(url)
+        {
+            domains.insert(property.domain.registrable_domain);
+        }
+    }
+    ensure!(
+        !domains.is_empty(),
+        "import website assertions before Common Crawl Web Graph"
+    );
+    let digest = crate::digest(&serde_json::to_vec(&domains)?);
+    let selection = WebGraphSelection {
+        scope: format!("candidate-domains:{digest}"),
+        domains: u64::try_from(domains.len())?,
+    };
+    Ok((selection, domains))
+}
+
 /// Whether a signed immutable generation uses a reader-compatible rule contract.
 #[must_use]
 pub fn supported_rule_version(version: &str) -> bool {
@@ -154,6 +210,17 @@ pub fn import_with_limits(
     limits: ImportLimits,
 ) -> anyhow::Result<String> {
     manifest.validate()?;
+    let graph_targets = if manifest.format == Format::CommonCrawlDomainRanksTsv {
+        let (selection, targets) = web_graph_targets(db)?;
+        ensure!(
+            manifest.scope == selection.scope,
+            "Web Graph manifest scope does not match current candidate domains; expected {}",
+            selection.scope
+        );
+        Some(targets)
+    } else {
+        None
+    };
     ensure!(
         limits.maximum_expanded_bytes > 0
             && limits.maximum_records > 0
@@ -225,6 +292,7 @@ pub fn import_with_limits(
                 .coverage
                 .as_ref()
                 .is_some_and(|coverage| coverage.kind == crate::model::CoverageKind::Delta),
+            graph_targets,
         )
         .ingest(&mut reader, &mut sink);
         parsed.and_then(|()| {

crates/argand-site-registry/src/update.rs

@@ -58,6 +58,7 @@ pub async fn run(config: &Config) -> anyhow::Result<PathBuf> {
         .open(config.generations.join("update.lock"))?;
     lock.try_lock().context("registry update already running")?;
     let mut inputs = config.inputs.clone();
+    let mut graph_downloads = Vec::new();
     let mut db = crate::store::open(&config.database)?;
     let now = Utc::now();
     for request in &config.downloads {
@@ -66,6 +67,10 @@ pub async fn run(config: &Config) -> anyhow::Result<PathBuf> {
             .snapshot
             .replace("{date}", &now.format("%Y-%m-%d").to_string())
             .replace("{month}", &now.format("%Y-%m").to_string());
+        if request.source == crate::model::Source::CommonCrawlWebGraph {
+            graph_downloads.push(request);
+            continue;
+        }
         if config.auto_supersede_typed_snapshots
             && let Some(coverage) = &mut request.coverage
             && matches!(
@@ -119,8 +124,31 @@ pub async fn run(config: &Config) -> anyhow::Result<PathBuf> {
         }
         inputs.push(crate::crux::download(&config.cache, &request).await?);
     }
-    anyhow::ensure!(!inputs.is_empty(), "update config contains no sources");
+    let mut graph_inputs = Vec::new();
+    let mut identity_inputs = Vec::new();
     for input in inputs {
+        let manifest: SourceManifest = crate::read_json(&input.manifest)?;
+        if manifest.source == crate::model::Source::CommonCrawlWebGraph {
+            graph_inputs.push(input);
+        } else {
+            identity_inputs.push(input);
+        }
+    }
+    anyhow::ensure!(
+        !identity_inputs.is_empty() || !graph_inputs.is_empty() || !graph_downloads.is_empty(),
+        "update config contains no sources"
+    );
+    for input in identity_inputs {
+        let manifest: SourceManifest = crate::read_json(&input.manifest)?;
+        crate::store::import(&mut db, &manifest, &input.input)?;
+    }
+    for mut request in graph_downloads {
+        if request.scope == "{candidate_domains}" {
+            request.scope = crate::store::web_graph_selection(&db)?.scope;
+        }
+        graph_inputs.push(crate::download::download(&config.cache, &request).await?);
+    }
+    for input in graph_inputs {
         let manifest: SourceManifest = crate::read_json(&input.manifest)?;
         crate::store::import(&mut db, &manifest, &input.input)?;
     }

crates/argand-site-registry/tests/common/mod.rs

@@ -71,6 +71,9 @@ pub fn manifest(source: Source, format: Format, bytes: &[u8]) -> anyhow::Result<
         Source::Ror => {
             "https://zenodo.org/api/records/22099990/files/v2.12-2026-08-25-ror-data.zip/content"
         }
+        Source::CommonCrawlWebGraph => {
+            "https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/cc-main-2022-may-jun-aug-domain-ranks.txt.gz"
+        }
     };
     Ok(SourceManifest {
         schema: "argand.site-source/v1".into(),

crates/argand-site-registry/tests/webgraph.rs

@@ -0,0 +1,239 @@
+// By Nic Weyand!
+//! Common Crawl Web Graph evidence stays source separated and cannot authorize routes.
+
+#![allow(dead_code)] // Shared integration helpers intentionally cover a wider fixture surface.
+
+mod common;
+
+use argand_site_registry::{
+    download::validate_source_url,
+    model::{Compression, Format, Source, SourceManifest},
+    query::ResolutionStatus,
+    store,
+};
+use std::{fmt::Write as _, time::Instant};
+
+const WEBGRAPH: &str = "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n\
+1\t3.2914686E7\t1\t0.018076941061056315\tcom.googleapis\t4482\n\
+2\t3.2131562E7\t3\t0.012273178013351222\tcom.facebook\t18795\n";
+
+fn import_identity_candidates(
+    db: &mut rusqlite::Connection,
+    root: &std::path::Path,
+) -> anyhow::Result<()> {
+    common::import(
+        db,
+        root,
+        Source::Psl,
+        Format::PslText,
+        common::PSL.as_bytes(),
+    )?;
+    common::import(
+        db,
+        root,
+        Source::Wikidata,
+        Format::WikidataEntities,
+        &serde_json::to_vec(&common::wikidata())?,
+    )?;
+    Ok(())
+}
+
+fn graph_manifest(db: &rusqlite::Connection, bytes: &[u8]) -> anyhow::Result<SourceManifest> {
+    let mut manifest = common::manifest(
+        Source::CommonCrawlWebGraph,
+        Format::CommonCrawlDomainRanksTsv,
+        bytes,
+    )?;
+    manifest.scope = store::web_graph_selection(db)?.scope;
+    Ok(manifest)
+}
+
+fn import_graph(
+    db: &mut rusqlite::Connection,
+    root: &std::path::Path,
+    bytes: &[u8],
+) -> anyhow::Result<()> {
+    let manifest = graph_manifest(db, bytes)?;
+    let input = root.join(format!("{}.graph", manifest.sha256));
+    std::fs::write(&input, bytes)?;
+    store::import(db, &manifest, &input)?;
+    Ok(())
+}
+
+#[test]
+fn domain_ranks_are_popularity_only_and_preserve_provider_fields() -> anyhow::Result<()> {
+    let root = tempfile::tempdir()?;
+    let mut db = store::open(&root.path().join("store.sqlite"))?;
+    import_identity_candidates(&mut db, root.path())?;
+    let identity_before: i64 = db.query_row(
+        "SELECT count(*) FROM facts WHERE predicate NOT IN ('psl','popularity')",
+        [],
+        |row| row.get(0),
+    )?;
+    import_graph(&mut db, root.path(), WEBGRAPH.as_bytes())?;
+
+    let identity_facts: i64 = db.query_row(
+        "SELECT count(*) FROM facts WHERE predicate NOT IN ('psl','popularity')",
+        [],
+        |row| row.get(0),
+    )?;
+    assert_eq!(identity_facts, identity_before);
+    for table in ["reviews", "votes"] {
+        let count: i64 = db.query_row(&format!("SELECT count(*) FROM {table}"), [], |row| {
+            row.get(0)
+        })?;
+        assert_eq!(count, 0, "Web Graph unexpectedly populated {table}");
+    }
+
+    let first = common::build(&db, root.path(), "first")?;
+    let second = common::build(&db, root.path(), "second")?;
+    assert_eq!(first.identity, second.identity);
+    assert_eq!(first.lookup("Facebook", 10)?.total_entities, 1);
+    assert_ne!(
+        first
+            .resolve_explained("Facebook", None, None, common::timestamp()?)?
+            .status,
+        ResolutionStatus::Resolved
+    );
+
+    assert_eq!(first.popularity("googleapis.com", 10)?.total, 0);
+
+    let lookup = first.popularity("facebook.com", 10)?;
+    assert_eq!(lookup.total, 1);
+    let observation = &lookup.observations[0];
+    assert_eq!(observation.source, "common_crawl_web_graph");
+    assert_eq!(observation.target, "facebook.com");
+    assert_eq!(observation.value["harmonic_rank"], 2);
+    assert_eq!(observation.value["harmonic_value"], 3.213_156_2E7);
+    assert_eq!(observation.value["pagerank_rank"], 3);
+    assert_eq!(
+        observation.value["pagerank_value"],
+        0.012_273_178_013_351_222_f64
+    );
+    assert_eq!(observation.value["member_hosts"], 18_795);
+    assert!(observation.value.get("source_hosts").is_none());
+    assert_eq!(
+        observation.provenance["source"]["license"],
+        "LicenseRef-Common-Crawl-Terms-of-Use"
+    );
+    let native_id: String = db.query_row(
+        "SELECT r.native_id FROM records r JOIN sources s ON s.id=r.source_id WHERE s.source='common_crawl_web_graph'",
+        [],
+        |row| row.get(0),
+    )?;
+    assert_eq!(native_id, "row:2");
+    Ok(())
+}
+
+#[test]
+fn graph_import_requires_identity_candidates_and_exact_bound_scope() -> anyhow::Result<()> {
+    let root = tempfile::tempdir()?;
+    let mut db = store::open(&root.path().join("store.sqlite"))?;
+    let mut manifest = common::manifest(
+        Source::CommonCrawlWebGraph,
+        Format::CommonCrawlDomainRanksTsv,
+        WEBGRAPH.as_bytes(),
+    )?;
+    manifest.scope = format!("candidate-domains:{}", "0".repeat(64));
+    let input = root.path().join("graph.tsv");
+    std::fs::write(&input, WEBGRAPH)?;
+    assert!(store::import(&mut db, &manifest, &input).is_err());
+
+    import_identity_candidates(&mut db, root.path())?;
+    assert!(store::import(&mut db, &manifest, &input).is_err());
+    manifest.scope = store::web_graph_selection(&db)?.scope;
+    store::import(&mut db, &manifest, &input)?;
+    Ok(())
+}
+
+#[test]
+fn domain_rank_parser_rejects_schema_and_value_drift() -> anyhow::Result<()> {
+    for bad in [
+        "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\n",
+        "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n\n",
+        "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n0\t1\t1\t1\tcom.example\t1\n",
+        "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n1\tNaN\t1\t1\tcom.example\t1\n",
+        "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n1\t1\t1\t1\tcom..example\t1\n",
+        "#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n1\t1\t1\t1\tcom.example\t0\n",
+    ] {
+        let root = tempfile::tempdir()?;
+        let mut db = store::open(&root.path().join("store.sqlite"))?;
+        import_identity_candidates(&mut db, root.path())?;
+        assert!(
+            import_graph(&mut db, root.path(), bad.as_bytes()).is_err(),
+            "accepted malformed Web Graph input {bad:?}"
+        );
+    }
+    Ok(())
+}
+
+#[test]
+fn domain_rank_source_url_is_exactly_allowlisted() -> anyhow::Result<()> {
+    let good = "https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/cc-main-2022-may-jun-aug-domain-ranks.txt.gz";
+    validate_source_url(Source::CommonCrawlWebGraph, good)?;
+    for bad in [
+        "http://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/cc-main-2022-may-jun-aug-domain-ranks.txt.gz",
+        "https://data.commoncrawl.org.evil.example/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/cc-main-2022-may-jun-aug-domain-ranks.txt.gz",
+        "https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/host/cc-main-2022-may-jun-aug-host-ranks.txt.gz",
+        "https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/other-domain-ranks.txt.gz",
+        "https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/cc-main-2022-may-jun-aug-domain-ranks.txt.gz?x=1",
+    ] {
+        assert!(
+            validate_source_url(Source::CommonCrawlWebGraph, bad).is_err(),
+            "accepted unreviewed Web Graph URL {bad}"
+        );
+    }
+    Ok(())
+}
+
+#[test]
+fn gzip_source_is_consumed_and_authenticated_end_to_end() -> anyhow::Result<()> {
+    let root = tempfile::tempdir()?;
+    let compressed = common::gzip(WEBGRAPH.as_bytes())?;
+    let input = root.path().join("domain-ranks.txt.gz");
+    std::fs::write(&input, &compressed)?;
+    let mut db = store::open(&root.path().join("store.sqlite"))?;
+    import_identity_candidates(&mut db, root.path())?;
+    let mut manifest = graph_manifest(&db, &compressed)?;
+    manifest.compression = Compression::Gzip;
+    store::import(&mut db, &manifest, &input)?;
+    let registry = common::build(&db, root.path(), "gzip")?;
+    assert_eq!(registry.popularity("facebook.com", 10)?.total, 1);
+    Ok(())
+}
+
+#[test]
+#[ignore = "explicit resource benchmark"]
+fn hundred_thousand_rows_import_within_engineering_budget() -> anyhow::Result<()> {
+    let mut input =
+        String::from("#harmonicc_pos\t#harmonicc_val\t#pr_pos\t#pr_val\t#host_rev\t#n_hosts\n");
+    for rank in 1..=100_000_u32 {
+        let reversed_domain = if rank == 50_000 {
+            "com.facebook".to_owned()
+        } else {
+            format!("com.example-{rank}")
+        };
+        writeln!(
+            input,
+            "{rank}\t{}\t{rank}\t{}\t{reversed_domain}\t1",
+            100_001 - rank,
+            0.1
+        )?;
+    }
+    let root = tempfile::tempdir()?;
+    let mut db = store::open(&root.path().join("store.sqlite"))?;
+    import_identity_candidates(&mut db, root.path())?;
+    let started = Instant::now();
+    import_graph(&mut db, root.path(), input.as_bytes())?;
+    assert!(
+        started.elapsed().as_secs_f64() < 10.0,
+        "100k Web Graph rows exceeded the 10-second engineering budget"
+    );
+    let retained: i64 = db.query_row(
+        "SELECT count(*) FROM facts WHERE predicate='popularity'",
+        [],
+        |row| row.get(0),
+    )?;
+    assert_eq!(retained, 1);
+    Ok(())
+}