CHANGELOG.md

@@ -49,6 +49,10 @@ replacement, and each removed path, column or flag.
   `number` and its `name`, selectable by either; `misc.port` renders a TCP port number,
   with the IANA `service` name beside it, keeping the first service the registry
   describes for a port so a number selects one row.
+- `misc.tld` is the IANA root zone as a table: every delegated TLD keyed with its
+  leading dot, `.se` and `.xn--p1ai`, the register's `type` beside it, and a `unicode`
+  column holding the form the register displays, so `misc.tld[.рф]` selects the row
+  `.xn--p1ai` renders. A delegation the register marks "Not assigned" does not ship.
 - `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`,
   `locality`, `postal-code` and `street`, weighted by population and address counts
   and built from SCB, GeoNames, Trafikverket NVDB and the US Census Bureau, and an

DATA-LICENSES.md

@@ -24,6 +24,7 @@ Every shipped dataset, its source, its licence and the attribution it asks for.
 | `misc/protocol.tsv` | [IANA Protocol Numbers](https://www.iana.org/assignments/protocol-numbers/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/protocol.py` |
 | `misc/territory.tsv` | [datasets/country-codes](https://github.com/datasets/country-codes) | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/territory.py` |
 | `misc/timezone.tsv` | [IANA tzdb](https://www.iana.org/time-zones) 2026d `zone.tab` and the standard offset of each zone; weights from [GeoNames](https://www.geonames.org/) `cities15000` populations | [public domain](https://data.iana.org/time-zones/tzdb/LICENSE); [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | "Populations from GeoNames, www.geonames.org" | `data-import/timezone.py` |
+| `misc/tld.tsv` | [IANA Root Zone Database](https://www.iana.org/domains/root/db), checked against the root zone's [TLD list](https://data.iana.org/TLD/tlds-alpha-by-domain.txt) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/tld.py` |
 | `misc/useragent.tsv` | [top-user-agents](https://github.com/microlinkhq/top-user-agents) desktop and mobile lists | [MIT](https://github.com/microlinkhq/top-user-agents/blob/master/LICENSE.md) | "Copyright © 2020 Kiko Beats" | `data-import/useragent.py` |
 
 Every other category is hand-written under [`data/`](data), MIT like the code.

README.md

@@ -164,13 +164,13 @@ Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`
 carries `car`, `coordinate`, `creditcard` (Luhn-valid), `currency` (ISO 4217),
 `datetime` (RFC 3339), `emoji`, `httpmethod`, `httpstatus`, `language` (ISO 639-1,
 with its 639-2/T code), `mac`, `mimetype`, `objectid`, `port`, `protocol`,
-`territory` (ISO 3166-1), `timezone` (IANA), `useragent` and `uuid` (v4). Many carry
+`territory` (ISO 3166-1), `timezone` (IANA), `tld` (IANA root zone), `useragent` and
-sub-fields — `misc.currency.symbol`, `misc.territory.alpha2`, `misc.httpstatus.code` —
+`uuid` (v4). Many carry sub-fields — `misc.currency.symbol`, `misc.territory.alpha2`,
-which `--list` shows. `car`, `currency`, `httpmethod`, `httpstatus`, `language`,
+`misc.httpstatus.code` — which `--list` shows. `car`, `currency`, `httpmethod`,
-`mimetype`, `port`, `protocol`, `territory`, `timezone` and `useragent` are
+`httpstatus`, `language`, `mimetype`, `port`, `protocol`, `territory`, `timezone`,
-[tables](#table), so `misc.territory[SE].capital` and `misc.currency[Euro].symbol`
+`tld` and `useragent` are [tables](#table), so `misc.territory[SE].capital` and
-select a row; `car` and `useragent` carry no key or name, so they are drawn from rather
+`misc.currency[Euro].symbol` select a row; `car` and `useragent` carry no key or name,
-than selected in.
+so they are drawn from rather than selected in.
 
 `misc.httpmethod`, `misc.protocol` and `misc.port` are IANA's registries.
 `misc.httpmethod` is the eight methods RFC 9110 defines and PATCH; the register's
@@ -184,6 +184,13 @@ IANA `service` beside it, and selecting by number alone. A draw spans the whole
 register, so pin `misc.port[443]` where a fixture needs a port a reader recognises;
 it stops at 49150, IANA assigning nothing above, so an ephemeral source port is
 `{int(49152,65535)}`.
+`misc.tld` is the root zone's delegated TLDs, keyed with the leading dot
+`misc.territory.tld` already carries, `.se`. Its `type` is the register's —
+`country-code`, `generic`, `generic-restricted`, `infrastructure` or `sponsored` — and
+its `unicode` the form the register displays, `.рф` for `.xn--p1ai` and the key itself
+for an ASCII one, which selects the row too, so `misc.tld[.рф]` renders `.xn--p1ai`. A
+draw spans the whole zone, `.arpa` and `.zuerich` alike, so pin `misc.tld[.com]` where
+a fixture needs one a reader recognises.
 [`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
 
 `misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for most
@@ -1199,11 +1206,11 @@ the Development section below, and who ships a register the four above then draw
   stays a column. Layer your own `misc.territory` over the shipped one and you
   must layer `misc.timezone` too, or the link fails at load naming the row.
 - **A table whose register publishes no frequency draws evenly.** `misc.httpmethod`,
-  `misc.port`, `misc.httpstatus` and `misc.mimetype` weigh every row alike, so GET is
+  `misc.port`, `misc.httpstatus`, `misc.mimetype` and `misc.tld` weigh every row alike,
-  a ninth of the methods drawn. Goal 10 keeps an authored fact out of a sourced table,
+  so GET is a ninth of the methods drawn. Goal 10 keeps an authored fact out of a
-  and no register publishes how often a method or a port is used, so a weight here
+  sourced table, and no register publishes how often a method, a port or a TLD is used,
-  would be invented. Where one exists it is read, as `misc.timezone` reads GeoNames
+  so a weight here would be invented. Where one exists it is read, as `misc.timezone`
-  populations and `sv_SE.first-name` SCB bearers.
+  reads GeoNames populations and `sv_SE.first-name` SCB bearers.
 - **`misc.port` selects by number, and carries no `name`.** 29 of its services sit on
   more than one port, `http-alt` on three, so `misc.port[http-alt]` could name no one
   row. The number is what a port field holds anyway, and `misc.port[443].service`
@@ -1223,6 +1230,15 @@ the Development section below, and who ships a register the four above then draw
   every value names a row instead. A territory the register records no state for
   stands alone, which is `EH` alone, and naming one for it would be a claim
   fejkdata has no business making.
+- **`misc.tld` keys carry the leading dot, where other tables key on a bare code.**
+  The register spells a TLD `.se` and `misc.territory.tld` already ships it so, which a
+  bare key would make two spellings of one fact; `{/misc.tld}` also composes onto a
+  host with no separator. `misc.tld[se]` misses for it, which `todo.md` carries.
+- **`misc.tld` is a table of its own, and `misc.territory.tld` stays a column.** A
+  `parent` demands a child for every parent row, so linking them would drop every root
+  zone row naming no territory, which is most of them, and goal 10 holds a sourced
+  table whole. The loader refuses the link outright anyway: `tld` is a column of
+  `misc.territory`, and a table may not be named like a column of its ancestor.
 - **`misc.territory` carries a currency code, it does not link to `misc.currency`.**
   A `parent` demands a child for every parent row, and ISO 4217 registers codes no
   country's row can name: the funds codes (Mvdol, WIR Euro, US Dollar (Next day)),
@@ -1330,6 +1346,7 @@ docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/port.
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/protocol.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/territory.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/timezone.py
+docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/tld.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/useragent.py
 ```
 

data-import/tld.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Rebuild data/misc/tld.tsv from the IANA Root Zone Database.
+
+    data-import/tld.py [--source URL_OR_FILE] [--zone URL_OR_FILE] [--cache DIR] [--out FILE]
+
+A TLD ships when the database names a manager for it; one it marks "Not assigned" is a
+delegation the root zone no longer holds, and the zone file is fetched to prove the two
+agree. The key is the A-label the root zone holds; `unicode` is the form the database
+displays, less the bidi marks it wraps a right-to-left label in, and proved to punycode
+back to the key.
+"""
+import argparse
+import html
+import re
+import sys
+from pathlib import Path
+
+import source
+import tsv
+
+SOURCE = "https://www.iana.org/domains/root/db"
+ZONE = "https://data.iana.org/TLD/tlds-alpha-by-domain.txt"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "tld.tsv"
+CACHE = Path(__file__).resolve().parent / "cache"
+COLUMNS = ["tld", "type", "unicode"]
+TYPES = ("country-code", "generic", "generic-restricted", "infrastructure", "sponsored")
+BIDI = {0x200E: None, 0x200F: None}
+ROW = re.compile(r'<a href="/domains/root/db/([^"]+)\.html">([^<]+)</a>.*?</td>\s*<td>([^<]*)</td>\s*<td>(.*?)</td>', re.S)
+
+
+def rows(page):
+    for a_label, shown, kind, manager in ROW.findall(page):
+        if "Not assigned" in manager:
+            continue
+        kind = kind.strip()
+        if kind not in TYPES:
+            sys.exit(f"{a_label}: the database types it {kind!r}, which is none of {TYPES}")
+        yield {"tld": "." + a_label, "type": kind, "unicode": html.unescape(shown).strip().translate(BIDI)}
+
+
+def prove_each_unicode_form_is_its_key(table):
+    """The displayed form is the key's own spelling, which punycode encodes back to."""
+    for r in table:
+        label = r["unicode"][1:]
+        encoded = label if label.isascii() else "xn--" + label.encode("punycode").decode("ascii")
+        if "." + encoded != r["tld"]:
+            sys.exit(f"{r['tld']}: the database displays {r['unicode']!r}, which is no spelling of it")
+
+
+def main():
+    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    p.add_argument("--cache", default=str(CACHE))
+    p.add_argument("--source", default=SOURCE)
+    p.add_argument("--zone", default=ZONE)
+    p.add_argument("--out", default=str(OUT))
+    a = p.parse_args()
+    table = sorted(rows(source.fetch(a.source, a.cache, "root-db.html").decode("utf-8")), key=lambda r: r["tld"])
+    prove_each_unicode_form_is_its_key(table)
+    listing = source.fetch(a.zone, a.cache, "tlds-alpha-by-domain.txt").decode("utf-8")
+    delegated = {"." + line.strip().lower() for line in listing.splitlines() if line.strip() and not line.startswith("#")}
+    shipped = {r["tld"] for r in table}
+    if len(table) != len(delegated) or shipped != delegated:
+        sys.exit(f"the database assigns {len(table)} rows where the root zone holds {len(delegated)} TLDs: {sorted(shipped ^ delegated)[:10]}")
+    tsv.write(a.out, COLUMNS, table)
+
+
+if __name__ == "__main__":
+    main()

data/misc/tld.json

@@ -0,0 +1 @@
+{ "format": "{tld}", "rows": "tld.tsv", "key": "tld", "name": "unicode" }

data_test.go

@@ -119,6 +119,9 @@ func TestShippedMiscReferenceData(t *testing.T) {
 		"protocol":              regexp.MustCompile(`^\S+( \S+)*$`),
 		"protocol.name":         regexp.MustCompile(`\S`),
 		"protocol.number":       regexp.MustCompile(`^\d{1,3}$`),
+		"tld":                   regexp.MustCompile(`^\.[a-z0-9]([a-z0-9-]*[a-z0-9])?$`),
+		"tld.type":              regexp.MustCompile(`^(country-code|generic|generic-restricted|infrastructure|sponsored)$`),
+		"tld.unicode":           regexp.MustCompile(`^\.\S+$`),
 		"httpstatus":            regexp.MustCompile(`^[1-5]\d{2} \S.*$`),
 		"httpstatus.code":       regexp.MustCompile(`^[1-5]\d{2}$`),
 		"useragent":             regexp.MustCompile(`^Mozilla/5\.0 .+`),

table_test.go

@@ -9,6 +9,7 @@ import (
 	"strconv"
 	"strings"
 	"testing"
+	"unicode"
 )
 
 // writeFiles writes a data directory from a map of relative file name, extension
@@ -716,6 +717,13 @@ func TestShippedTables(t *testing.T) {
 		"port[465].service":                     "urd",
 		"port[2049].service":                    "nfs",
 		"protocol[Transmission Control].number": "6",
+		"tld[.se]":                              ".se",
+		"tld[.se].type":                         "country-code",
+		"tld[.com].type":                        "generic",
+		"tld[.museum].type":                     "sponsored",
+		"tld[.arpa].type":                       "infrastructure",
+		"tld[.рф]":                              ".xn--p1ai",
+		"tld[.xn--p1ai].unicode":                ".рф",
 		"territory[NP].timezone.offset":         "+05:45",
 	} {
 		if got := fake(t, f, path); got != want {
@@ -830,6 +838,30 @@ func TestEveryTimezoneRowIsWellFormed(t *testing.T) {
 	}
 }
 
+// TestEveryTldRowSpellsItsKey scans the file rather than drawing: 100 draws leave
+// most of 1438 rows unrendered, and the register wraps a right-to-left label in bidi
+// marks that belong to its display, not to the label.
+func TestEveryTldRowSpellsItsKey(t *testing.T) {
+	head, rows := shippedRows(t, "tld.tsv", "tld", "type", "unicode")
+	types := map[string]bool{"country-code": true, "generic": true, "generic-restricted": true, "infrastructure": true, "sponsored": true}
+	for i, r := range rows {
+		key, kind, shown := r[head["tld"]], r[head["type"]], r[head["unicode"]]
+		if !types[kind] {
+			t.Errorf("tld.tsv line %d: %s is typed %q, which the register assigns no TLD", i+2, key, kind)
+		}
+		for _, c := range shown {
+			if unicode.Is(unicode.Cf, c) {
+				t.Errorf("tld.tsv line %d: %s displays %q, carrying the format character %U", i+2, key, shown, c)
+			}
+		}
+		if idn := strings.HasPrefix(key, ".xn--"); idn && shown == key {
+			t.Errorf("tld.tsv line %d: the IDN %s displays its A-label, not a Unicode form", i+2, key)
+		} else if !idn && shown != key {
+			t.Errorf("tld.tsv line %d: the ASCII TLD %s displays %q", i+2, key, shown)
+		}
+	}
+}
+
 // shippedRows reads a shipped misc TSV as the loader does, proving it names the
 // columns asked for and that every row fills them.
 func shippedRows(t *testing.T, file string, want ...string) (map[string]int, [][]string) {

testdata/shipped_shape.txt

@@ -274,6 +274,10 @@ misc.timezone.offset	string
 misc.timezone.territory	string
 misc.timezone.weight	string
 misc.timezone.zone	string
+misc.tld	format "{tld}"	key tld	name unicode
+misc.tld.tld	string
+misc.tld.type	string
+misc.tld.unicode	string
 misc.useragent	format "{ua}"
 misc.useragent.browser	string
 misc.useragent.device	string

todo.md

@@ -27,6 +27,9 @@ by a `data-import/` script, as README goal 10 asks.
 - Audit the rest of the shipped set against goal 12 and give each a never-reaching
   path where it lacks one: `phone` draws live PTS and NANP ranges, `bankgiro`,
   `plusgiro` and `routing` draw live prefixes.
+- Stop `misc.territory[EH].tld` rendering `.eh`, the one shipped TLD `misc.tld` does
+  not hold: ISO 3166 reserves it for Western Sahara and the root zone has never been
+  delegated it, so no resolver answers for it and goal 1 promises otherwise.
 - Accept a middle name, and draw a shipped `personnummer` inside a *selected* sex;
   both want a draw group sharing its family's pins. Stop the conflict error naming a
   rewrite that returns a different value where the read it conflicts with sits inside
@@ -46,7 +49,6 @@ by a `data-import/` script, as README goal 10 asks.
 
 | Category | Shape | Source | Licence |
 |---|---|---|---|
-| `tld` | T | IANA root zone | CC0 |
 | `loglevel` | c | RFC 5424 severities | facts |
 | `ip` v4 documentation ranges, private, CIDR; `ipv6` `2001:db8::/32`; `mac` locally administered | t | RFC 5737, 1918, 9637 | facts |
 | `creditcard` per network: IIN, length, CVV, expiry | T+t | network rules, `{luhn()}` | facts |
@@ -108,7 +110,9 @@ by a `data-import/` script, as README goal 10 asks.
 - Name a spelling that works when a row selector misses: `misc.protocol[tcp]`,
   `misc.httpmethod[get]` and `misc.territory[se]` all answer "no row … has key or name"
   and stop there, where a case-insensitive match could name the row that exists, and a
-  table with no `name` column could say it selects by key alone.
+  table with no `name` column could say it selects by key alone. `misc.tld[se]` misses
+  on the leading dot its keys carry rather than on case, so the near miss is worth
+  naming whatever shape it takes.
 - Let a table column carry a `datatype`, so `--format json` writes
   `"safe": true` and `--format sql` a boolean rather than the text `'true'`. Today only
   a JSON field takes one, so `misc.httpmethod`'s booleans are typed in Go and text