feat: establish standalone Argand Site Registry

This commit is contained in:
Nic Weyand 2026-09-12 21:38:59 -04:00
commit 2a0fe1714b
Signed by: nicweyand
SSH key fingerprint: SHA256:2te+ycJIQON/Wo/dH6+ZkFSQ4HnHWpetV2azx9E65dQ
60 changed files with 10494 additions and 0 deletions

View file

@ -0,0 +1,60 @@
# Site Registry source licenses
Reviewed against the primary distribution and licensing pages on 2026-09-12.
The Rust code uses the workspace's **AGPL-3.0-or-later** license. Imported data keeps
its own licenses; neither the code license nor a merged export relicenses it.
Commercial reuse is supported subject to the following obligations. A provider's
listing is evidence of an assertion, not a guarantee of ownership or safety.
| Source | Exact data license and evidence | Distribution and consumed fields |
| --- | --- | --- |
| Wikidata | [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/), [`CC0-1.0`](https://www.wikidata.org/wiki/Wikidata:Licensing) | [JSON dumps](https://www.wikidata.org/wiki/Wikidata:Database_download), [entity JSON](https://www.wikidata.org/wiki/Special:EntityData/Q355.json). Entity ID, revision, all labels/aliases, full P856 statements including ranks, qualifiers and references; P17, P159, P407, P1001 and country/language code mappings P297/P218/P219/P220. Relevant raw entity records remain available for audit. |
| Majestic Million | [CC BY 3.0 Unported](https://creativecommons.org/licenses/by/3.0/), [`CC-BY-3.0`](https://majestic.com/reports/majestic-million) | [Official CSV](https://downloads.majestic.com/majestic_million.csv). Domain/IDN/TLD, global/TLD ranks, referring subnet/IP counts and their previous values. Ranks remain source-specific signals; no entity ownership is inferred. |
| Chrome UX Report (CrUX), Google | [CC BY 4.0 International](https://creativecommons.org/licenses/by/4.0/), [`CC-BY-4.0`](https://developer.chrome.com/docs/crux/methodology) | [Monthly BigQuery dataset](https://developer.chrome.com/docs/crux/bigquery/): `origin`, `experimental.popularity.rank`, observation month, optional audience-country dataset code. The adapter produces `origin,rank,yyyymm,country_code` CSV. Rank is a coarse bucket, not a precise visit count. Audience country is not website jurisdiction. No API key or OAuth token is retained. |
| Curlie | [CC BY 3.0 Unported](https://creativecommons.org/licenses/by/3.0/), [`CC-BY-3.0`](https://curlie.org/docs/en/license.html), including the attribution placement prescribed on that page | [Format documentation](https://curlie.org/docs/en/rdf.html), [official download redirect](https://curlie.org/directory-dl), currently [Passau-hosted archive](https://share.innkube.fim.uni-passau.de/curlie-rdf/curlie-rdf-all.tar.gz). Despite its RDF name, the current archive contains **literal TSV**. Content: URL, title, description, category ID. Structure: category ID, full category path, entry count, description, latitude, longitude. Archive notices are retained. |
| Public Suffix List contributors | [Mozilla Public License 2.0](https://mozilla.org/MPL/2.0/), [`MPL-2.0`](https://publicsuffix.org/list/public_suffix_list.dat) | [Official list](https://publicsuffix.org/list/public_suffix_list.dat). All ICANN and PRIVATE rules, wildcard/exception rules, version/commit comments and notices. Used for hostname, registrable-domain and public-suffix derivations. Download at most once per day. |
## Attribution and distribution
* **Wikidata:** CC0 imposes no attribution condition. Keep Wikidata IDs, source
links and revision evidence for traceability. CC0 structured data does not
extend to unrelated Wikipedia prose, images or linked websites.
* **Majestic:** credit “Majestic Million, Majestic”, link the source and CC BY 3.0,
retain supplied notices, and identify Argand's changes. The current distribution
page governs this import; older blog posts describe different historic terms.
* **CrUX:** credit “Chrome UX Report, Google”, link the source and CC BY 4.0,
retain notices and indicate the projection/normalization. BigQuery access and
billing are separate from the data license. This adapter requires an explicit
project and positive maximum-bytes-billed limit.
* **Curlie:** the requirement applies to **names, categories and descriptions**.
Every public page using Curlie content must include its prescribed HTML credit:
```html
<div title="Curlie Directory Attribution">
With content from <a style="text-decoration:none; color:#cd4932;" href="https://curlie.org/" title="Curlie - the largest human-edited directory of the web">Curlie.org</a> - the largest human-edited directory of the web. Contribute by submitting a website or becoming an editor.
</div>
```
Also retain the source and license links and indicate modifications. Imports
retain descriptions as audit data. Lookup/resolve never expose descriptions;
JSONL export omits them by default. `--include-descriptions` is an explicit
distribution choice: receiving applications must satisfy these obligations
before displaying the text. Treat directory descriptions as untrusted text,
never as executable HTML. A JSON attribution object alone does not satisfy
Curlie's public-page placement requirement.
* **PSL:** retain the list's notices, MPL license and access to its source form
when redistributing it. Generations retain the unmodified list in `records`
and `facts`; exports include the PSL fact and original download locator.
Changes to covered PSL source files must remain available under MPL 2.0.
The Rust `publicsuffix` parser is MIT/Apache-2.0; that is separate from the list.
Every generation contains this document and `ATTRIBUTION.json`, both hash-bound
by its signed completion receipt. Exports carry source manifests, licenses,
retrieval times, assertion selectors and confidence, plus a required attribution
envelope. Consumers must preserve these when extracting subsets or redistributing
derived facts. Source credits do not imply endorsement by any provider.
Cloudflare Radar, default Tranco, Cisco Umbrella, arbitrary mirrors and any other
unreviewed source are not supported. Adding a source requires verified commercial
reuse rights, its own adapter, provenance and attribution policy. A source format
or distribution-host change fails validation until the adapter is reviewed.