Read misc.httpmethod, misc.protocol and misc.port from their IANA registries

CHANGELOG.md

@@ -43,6 +43,11 @@ replacement, and each removed path, column or flag.
   in. `misc.useragent` is the top-user-agents desktop and mobile lists with `browser`,
   `device` and `os` columns. `misc.car` keeps its makes and models in a `make` and a
   `model` column.
+- `misc.httpmethod`, `misc.protocol` and `misc.port` are IANA's registries as tables.
+  `misc.httpmethod` carries `safe` and `idempotent` beside the method; `misc.protocol`
+  the keyword, its `number` and its `name`; `misc.port` renders a TCP port number,
+  with the service `name` beside it, keeping the first service the registry lists for
+  a port so a number selects one row.
 - `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`,
   `locality`, `postal-code` and `street`, weighted by population and address counts
   and built from SCB, GeoNames, Trafikverket NVDB and the US Census Bureau, and an

DATA-LICENSES.md

@@ -16,9 +16,12 @@ Every shipped dataset, its source, its licence and the attribution it asks for.
 | `en_US/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` |
 | `sv_SE/sex.tsv`, `sv_SE/birth-number.tsv`, `sv_SE/title.tsv`, `en_US/sex.tsv`, `en_US/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 10) | — | — | — |
 | `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` |
+| `misc/httpmethod.tsv` | [IANA HTTP Method Registry](https://www.iana.org/assignments/http-methods/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpmethod.py` |
 | `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` |
 | `misc/language.tsv` | [datasets/language-codes](https://github.com/datasets/language-codes), the [Library of Congress](https://www.loc.gov/standards/iso639-2/) ISO 639-2 register | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/language.py` |
 | `misc/mimetype.tsv` | [mime-db](https://github.com/jshttp/mime-db), the IANA media type registry with filename extensions | [MIT](https://github.com/jshttp/mime-db/blob/master/LICENSE) | "Copyright (c) 2014 Jonathan Ong, Copyright (c) 2015-2022 Douglas Christopher Wilson" | `data-import/mimetype.py` |
+| `misc/port.tsv` | [IANA Service Name and Transport Protocol Port Number Registry](https://www.iana.org/assignments/service-names-port-numbers/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/port.py` |
+| `misc/protocol.tsv` | [IANA Protocol Numbers](https://www.iana.org/assignments/protocol-numbers/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/protocol.py` |
 | `misc/territory.tsv` | [datasets/country-codes](https://github.com/datasets/country-codes) | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/territory.py` |
 | `misc/timezone.tsv` | [IANA tzdb](https://www.iana.org/time-zones) 2026d `zone.tab` and the standard offset of each zone; weights from [GeoNames](https://www.geonames.org/) `cities15000` populations | [public domain](https://data.iana.org/time-zones/tzdb/LICENSE); [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | "Populations from GeoNames, www.geonames.org" | `data-import/timezone.py` |
 | `misc/useragent.tsv` | [top-user-agents](https://github.com/microlinkhq/top-user-agents) desktop and mobile lists | [MIT](https://github.com/microlinkhq/top-user-agents/blob/master/LICENSE.md) | "Copyright © 2020 Kiko Beats" | `data-import/useragent.py` |

README.md

@@ -162,14 +162,18 @@ Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`
 `username`, `version` and `word`, formatted per locale; `sv_SE` adds
 `personnummer` and `samordningsnummer`, `en_US` adds `ssn` and `itin`. `misc`
 carries `car`, `coordinate`, `creditcard` (Luhn-valid), `currency` (ISO 4217),
-`datetime` (RFC 3339), `emoji`, `httpstatus`, `language` (ISO 639-1, with its
-639-2/T code), `mac`, `mimetype`, `objectid`, `territory` (ISO 3166-1), `timezone`
-(IANA), `useragent` and `uuid` (v4). Many carry sub-fields — `misc.currency.symbol`,
+`datetime` (RFC 3339), `emoji`, `httpmethod`, `httpstatus`, `language` (ISO 639-1,
+with its 639-2/T code), `mac`, `mimetype`, `objectid`, `port`, `protocol`,
+`territory` (ISO 3166-1), `timezone` (IANA), `useragent` and `uuid` (v4). Many carry sub-fields — `misc.currency.symbol`,
 `misc.territory.alpha2`, `misc.httpstatus.code` — which `--list` shows. `car`,
 `currency`, `httpstatus`, `language`, `mimetype`, `territory`, `timezone` and
-`useragent` are [tables](#table), so `misc.territory[SE].capital` and
+`httpmethod`, `port`, `protocol` and `useragent` are [tables](#table), so
+`misc.territory[SE].capital` and
 `misc.currency[Euro].symbol` select a row; `car` and `useragent` carry no key or
-name, so they are drawn from rather than selected in.
+name, so they are drawn from rather than selected in. `misc.httpmethod`,
+`misc.protocol` and `misc.port` are IANA's registries: a method carries whether it
+is `safe` and `idempotent`, a protocol its `number`, and `misc.port` renders the
+number a port field holds, with the service `name` beside it.
 [`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
 
 `misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for most
@@ -1296,11 +1300,14 @@ a copy of the surname file:
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py
 docker compose run --rm --user "$(id -u):$(id -g)" -e TRAFIKVERKET_API_KEY data-import data-import/geo-se.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py
+docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpmethod.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpstatus.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/language.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/mimetype.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-se.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-us.py
+docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/port.py
+docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/protocol.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/territory.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/timezone.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/useragent.py

data-import/httpmethod.py

@@ -0,0 +1,42 @@
+#!/usr/bin/env python3
+"""Rebuild data/misc/httpmethod.tsv from the IANA HTTP Method Registry.
+
+    data-import/httpmethod.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
+
+A method whose name is not letters and hyphens does not ship, which drops the `*` the
+registry holds to stop anyone registering it.
+"""
+import argparse
+import csv
+import io
+import re
+from pathlib import Path
+
+import source
+import tsv
+
+SOURCE = "https://www.iana.org/assignments/http-methods/methods.csv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpmethod.tsv"
+CACHE = Path(__file__).resolve().parent / "cache"
+COLUMNS = ["idempotent", "method", "safe"]
+
+
+def rows(text):
+    for r in csv.DictReader(io.StringIO(text)):
+        method, safe, idempotent = r["Method Name"].strip(), r["Safe"].strip(), r["Idempotent"].strip()
+        if re.match(r"^[A-Za-z][A-Za-z-]*$", method) and safe and idempotent:
+            yield {"idempotent": idempotent, "method": method, "safe": safe}
+
+
+def main():
+    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    p.add_argument("--cache", default=str(CACHE))
+    p.add_argument("--source", default=SOURCE)
+    p.add_argument("--out", default=str(OUT))
+    a = p.parse_args()
+    table = sorted(rows(source.fetch(a.source, a.cache, "http-methods.csv").decode("utf-8")), key=lambda r: r["method"])
+    tsv.write(a.out, COLUMNS, table)
+
+
+if __name__ == "__main__":
+    main()

data-import/port.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+"""Rebuild data/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry.
+
+    data-import/port.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
+
+What ships is the TCP assignments in use: a row needs a service name, a port number and
+a description the registry has filled in. A port the registry lists more than once keeps
+the first service, so a number selects one row.
+"""
+import argparse
+import csv
+import io
+from pathlib import Path
+
+import source
+import tsv
+
+SOURCE = "https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.csv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "port.tsv"
+CACHE = Path(__file__).resolve().parent / "cache"
+COLUMNS = ["name", "number"]
+UNUSED = ("Reserved", "Unassigned", "IANA assigned this well-formed service name to replace an unregistered or squatted port.")
+
+
+def rows(text):
+    seen = set()
+    for r in csv.DictReader(io.StringIO(text)):
+        name, number = r["Service Name"].strip(), r["Port Number"].strip()
+        described = (r["Description"] or "").strip()
+        if r["Transport Protocol"].strip() != "tcp" or not name or not number.isdigit():
+            continue
+        if not described or described in UNUSED or number in seen:
+            continue
+        seen.add(number)
+        yield {"name": name, "number": number}
+
+
+def main():
+    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    p.add_argument("--cache", default=str(CACHE))
+    p.add_argument("--source", default=SOURCE)
+    p.add_argument("--out", default=str(OUT))
+    a = p.parse_args()
+    table = sorted(rows(source.fetch(a.source, a.cache, "service-names-port-numbers.csv").decode("utf-8")), key=lambda r: int(r["number"]))
+    tsv.write(a.out, COLUMNS, table)
+
+
+if __name__ == "__main__":
+    main()

data-import/protocol.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+"""Rebuild data/misc/protocol.tsv from the IANA Protocol Numbers registry.
+
+    data-import/protocol.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
+
+A number the registry leaves unassigned, reserves or marks deprecated does not ship.
+Where it names no protocol beside the keyword, the keyword is the name.
+"""
+import argparse
+import csv
+import io
+from pathlib import Path
+
+import source
+import tsv
+
+SOURCE = "https://www.iana.org/assignments/protocol-numbers/protocol-numbers-1.csv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "protocol.tsv"
+CACHE = Path(__file__).resolve().parent / "cache"
+COLUMNS = ["keyword", "name", "number"]
+SKIP = ("Unassigned", "Reserved", "deprecated")
+
+
+def rows(text):
+    for r in csv.DictReader(io.StringIO(text)):
+        number, keyword = r["Decimal"].strip(), r["Keyword"].strip()
+        if not number.isdigit() or not keyword or any(s in keyword for s in SKIP):
+            continue
+        yield {"keyword": keyword, "name": " ".join(r["Protocol"].split()) or keyword, "number": number}
+
+
+def main():
+    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    p.add_argument("--cache", default=str(CACHE))
+    p.add_argument("--source", default=SOURCE)
+    p.add_argument("--out", default=str(OUT))
+    a = p.parse_args()
+    table = sorted(rows(source.fetch(a.source, a.cache, "protocol-numbers-1.csv").decode("utf-8")), key=lambda r: int(r["number"]))
+    tsv.write(a.out, COLUMNS, table)
+
+
+if __name__ == "__main__":
+    main()

data/misc/httpmethod.json

@@ -0,0 +1 @@
+{ "format": "{method}", "rows": "httpmethod.tsv", "key": "method" }

data/misc/httpmethod.tsv

@@ -0,0 +1,41 @@
+idempotent	method	safe
+yes	ACL	no
+yes	BASELINE-CONTROL	no
+yes	BIND	no
+yes	CHECKIN	no
+yes	CHECKOUT	no
+no	CONNECT	no
+yes	COPY	no
+yes	DELETE	no
+yes	GET	yes
+yes	HEAD	yes
+yes	LABEL	no
+yes	LINK	no
+no	LOCK	no
+yes	MERGE	no
+yes	MKACTIVITY	no
+yes	MKCALENDAR	no
+yes	MKCOL	no
+yes	MKREDIRECTREF	no
+yes	MKWORKSPACE	no
+yes	MOVE	no
+yes	OPTIONS	yes
+yes	ORDERPATCH	no
+no	PATCH	no
+no	POST	no
+yes	PRI	yes
+yes	PROPFIND	yes
+yes	PROPPATCH	no
+yes	PUT	no
+yes	QUERY	yes
+yes	REBIND	no
+yes	REPORT	yes
+yes	SEARCH	yes
+yes	TRACE	yes
+yes	UNBIND	no
+yes	UNCHECKOUT	no
+yes	UNLINK	no
+yes	UNLOCK	no
+yes	UPDATE	no
+yes	UPDATEREDIRECTREF	no
+yes	VERSION-CONTROL	no

data/misc/port.json

@@ -0,0 +1 @@
+{ "format": "{number}", "rows": "port.tsv", "key": "number" }

data/misc/protocol.json

@@ -0,0 +1 @@
+{ "format": "{keyword}", "rows": "protocol.tsv", "key": "keyword" }

data/misc/protocol.tsv

@@ -0,0 +1,140 @@
+keyword	name	number
+HOPOPT	IPv6 Hop-by-Hop Option	0
+ICMP	Internet Control Message	1
+IGMP	Internet Group Management	2
+GGP	Gateway-to-Gateway	3
+IPv4	IPv4 encapsulation	4
+ST	Stream	5
+TCP	Transmission Control	6
+CBT	CBT	7
+EGP	Exterior Gateway Protocol	8
+IGP	any private interior gateway (used by Cisco for their IGRP)	9
+BBN-RCC-MON	BBN RCC Monitoring	10
+NVP-II	Network Voice Protocol	11
+PUP	PUP	12
+EMCON	EMCON	14
+XNET	Cross Net Debugger	15
+CHAOS	Chaos	16
+UDP	User Datagram	17
+MUX	Multiplexing	18
+DCN-MEAS	DCN Measurement Subsystems	19
+HMP	Host Monitoring	20
+PRM	Packet Radio Measurement	21
+XNS-IDP	XEROX NS IDP	22
+TRUNK-1	Trunk-1	23
+TRUNK-2	Trunk-2	24
+LEAF-1	Leaf-1	25
+LEAF-2	Leaf-2	26
+RDP	Reliable Data Protocol	27
+IRTP	Internet Reliable Transaction	28
+ISO-TP4	ISO Transport Protocol Class 4	29
+NETBLT	Bulk Data Transfer Protocol	30
+MFE-NSP	MFE Network Services Protocol	31
+MERIT-INP	MERIT Internodal Protocol	32
+DCCP	Datagram Congestion Control Protocol	33
+3PC	Third Party Connect Protocol	34
+IDPR	Inter-Domain Policy Routing Protocol	35
+XTP	XTP	36
+DDP	Datagram Delivery Protocol	37
+IDPR-CMTP	IDPR Control Message Transport Proto	38
+TP++	TP++ Transport Protocol	39
+IL	IL Transport Protocol	40
+IPv6	IPv6 encapsulation	41
+SDRP	Source Demand Routing Protocol	42
+IPv6-Route	Routing Header for IPv6	43
+IPv6-Frag	Fragment Header for IPv6	44
+IDRP	Inter-Domain Routing Protocol	45
+RSVP	Reservation Protocol	46
+GRE	Generic Routing Encapsulation	47
+DSR	Dynamic Source Routing Protocol	48
+BNA	BNA	49
+ESP	Encap Security Payload	50
+AH	Authentication Header	51
+I-NLSP	Integrated Net Layer Security TUBA	52
+NARP	NBMA Address Resolution Protocol	54
+Min-IPv4	Minimal IPv4 Encapsulation	55
+TLSP	Transport Layer Security Protocol using Kryptonet key management	56
+SKIP	SKIP	57
+IPv6-ICMP	ICMP for IPv6	58
+IPv6-NoNxt	No Next Header for IPv6	59
+IPv6-Opts	Destination Options for IPv6	60
+CFTP	CFTP	62
+SAT-EXPAK	SATNET and Backroom EXPAK	64
+KRYPTOLAN	Kryptolan	65
+RVD	MIT Remote Virtual Disk Protocol	66
+IPPC	Internet Pluribus Packet Core	67
+SAT-MON	SATNET Monitoring	69
+VISA	VISA Protocol	70
+IPCV	Internet Packet Core Utility	71
+CPNX	Computer Protocol Network Executive	72
+CPHB	Computer Protocol Heart Beat	73
+WSN	Wang Span Network	74
+PVP	Packet Video Protocol	75
+BR-SAT-MON	Backroom SATNET Monitoring	76
+SUN-ND	SUN ND PROTOCOL-Temporary	77
+WB-MON	WIDEBAND Monitoring	78
+WB-EXPAK	WIDEBAND EXPAK	79
+ISO-IP	ISO Internet Protocol	80
+VMTP	VMTP	81
+SECURE-VMTP	SECURE-VMTP	82
+VINES	VINES	83
+IPTM	Internet Protocol Traffic Manager	84
+NSFNET-IGP	NSFNET-IGP	85
+DGP	Dissimilar Gateway Protocol	86
+TCF	TCF	87
+EIGRP	EIGRP	88
+OSPFIGP	OSPFIGP	89
+Sprite-RPC	Sprite RPC Protocol	90
+LARP	Locus Address Resolution Protocol	91
+MTP	Multicast Transport Protocol	92
+AX.25	AX.25 Frames	93
+IPIP	IP-within-IP Encapsulation Protocol	94
+SCC-SP	Semaphore Communications Sec. Pro.	96
+ETHERIP	Ethernet-within-IP Encapsulation	97
+ENCAP	Encapsulation Header	98
+GMTP	GMTP	100
+IFMP	Ipsilon Flow Management Protocol	101
+PNNI	PNNI over IP	102
+PIM	Protocol Independent Multicast	103
+ARIS	ARIS	104
+SCPS	SCPS	105
+QNX	QNX	106
+A/N	Active Networks	107
+IPComp	IP Payload Compression Protocol	108
+SNP	Sitara Networks Protocol	109
+Compaq-Peer	Compaq Peer Protocol	110
+IPX-in-IP	IPX in IP	111
+VRRP	Virtual Router Redundancy Protocol	112
+PGM	PGM Reliable Transport Protocol	113
+L2TP	Layer Two Tunneling Protocol	115
+DDX	D-II Data Exchange (DDX)	116
+IATP	Interactive Agent Transfer Protocol	117
+STP	Schedule Transfer Protocol	118
+SRP	SpectraLink Radio Protocol	119
+UTI	UTI	120
+SMP	Simple Message Protocol	121
+PTP	Performance Transparency Protocol	123
+ISIS over IPv4	ISIS over IPv4	124
+FIRE	FIRE	125
+CRTP	Combat Radio Transport Protocol	126
+CRUDP	Combat Radio User Datagram	127
+SSCOPMCE	SSCOPMCE	128
+IPLT	IPLT	129
+SPS	Secure Packet Shield	130
+PIPE	Private IP Encapsulation within IP	131
+SCTP	Stream Control Transmission Protocol	132
+FC	Fibre Channel	133
+RSVP-E2E-IGNORE	RSVP-E2E-IGNORE	134
+Mobility Header	Mobility Header	135
+UDPLite	UDPLite	136
+MPLS-in-IP	MPLS-in-IP	137
+manet	MANET Protocols	138
+HIP	Host Identity Protocol	139
+Shim6	Shim6 Protocol	140
+WESP	Wrapped Encapsulating Security Payload	141
+ROHC	Robust Header Compression	142
+Ethernet	Ethernet	143
+AGGFRAG	AGGFRAG encapsulation payload for ESP	144
+NSH	Network Service Header	145
+Homa	Homa	146
+BIT-EMU	Bit-stream Emulation	147

todo.md

@@ -9,7 +9,7 @@ Ordered as the releases that ship it.
 Shape: T = table, t = template, c = choice. Read every factual list from a register
 by a `data-import/` script, as README goal 10 asks.
 
-- Add the remaining `misc` tables and templates from the table below.
+- Add the remaining `misc` tables and templates, one row of the table below per chunk.
 - Read `misc.emoji` from the Unicode `emoji-test.txt` register, and `misc.car` from an
   international make and model source; vPIC and Mobility Sweden are national, so they
   build `en_US.car` and `sv_SE.car`.
@@ -46,7 +46,8 @@ by a `data-import/` script, as README goal 10 asks.
 
 | Category | Shape | Source | Licence |
 |---|---|---|---|
-| `httpmethod`, `port`, `tld`, `protocol`, `loglevel` | T/c | IANA | CC0 |
+| `tld` | T | IANA root zone | CC0 |
+| `loglevel` | c | RFC 5424 severities | facts |
 | `ip` v4 documentation ranges, private, CIDR; `ipv6` `2001:db8::/32`; `mac` locally administered | t | RFC 5737, 1918, 9637 | facts |
 | `creditcard` per network: IIN, length, CVV, expiry | T+t | network rules, `{luhn()}` | facts |
 | `bic`, `isin`, `cusip` | t | structure rules | facts |