From ecc0b475ae8c673e9038fcb4ed8a812dc38c3696 Mon Sep 17 00:00:00 2001 From: Lilleman auf Larv Date: Fri, 18 Sep 2026 21:55:54 +0200 Subject: [PATCH] Name the everyday media type extensions, refuse an emptied table, and cut the false register claim --- CHANGELOG.md | 9 +++++---- DATA-LICENSES.md | 2 +- README.md | 11 ++++------- data-import/mimetype.py | 6 ++++-- data-import/tsv.py | 4 +++- data/misc/mimetype.tsv | 8 ++++---- todo.md | 9 +++------ 7 files changed, 24 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 734b537..357feb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,10 +26,11 @@ replacement, and each removed path, column or flag. capital, a currency and a TLD, with the columns `calling-code`, `capital`, `currency`, `flag`, `languages`, `numeric` and `tld` added; `misc.currency` the current ISO 4217 currencies with a minor unit, with `decimals` and `numeric` - added, and its symbols from CLDR; `misc.language` every ISO 639-1 language, with - a `code3` column carrying its ISO 639-2 code; `misc.httpstatus` every HTTP status - code in use in the IANA registry; and `misc.mimetype` every IANA media type that - has a filename extension. `DATA-LICENSES.md` lists each source. + added, and its symbols from CLDR; `misc.language` every ISO 639-2 entry carrying a + 639-1 code, with a `code3` column holding its 639-2/T code; `misc.httpstatus` each + code the IANA registry lists with a plain reason phrase; and `misc.mimetype` each + type mime-db records as IANA-registered and gives a filename extension. + `DATA-LICENSES.md` lists each source. - `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`, `locality`, `postal-code` and `street`, weighted by population and address counts and built from SCB, GeoNames, Trafikverket NVDB and the US Census Bureau, and an diff --git a/DATA-LICENSES.md b/DATA-LICENSES.md index 8ad6c45..2ff85a6 100644 --- a/DATA-LICENSES.md +++ b/DATA-LICENSES.md @@ -19,6 +19,6 @@ Every shipped dataset, its source, its licence and the attribution it asks for. | `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` | | `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` | | `misc/language.tsv` | [datasets/language-codes](https://github.com/datasets/language-codes), the [Library of Congress](https://www.loc.gov/standards/iso639-2/) ISO 639-2 register | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/language.py` | -| `misc/mimetype.tsv` | [mime-db](https://github.com/jshttp/mime-db), the IANA media type registry with filename extensions | [MIT](https://github.com/jshttp/mime-db/blob/master/LICENSE) | none required | `data-import/mimetype.py` | +| `misc/mimetype.tsv` | [mime-db](https://github.com/jshttp/mime-db), the IANA media type registry with filename extensions | [MIT](https://github.com/jshttp/mime-db/blob/master/LICENSE) | "Copyright (c) 2014 Jonathan Ong, Copyright (c) 2015-2022 Douglas Christopher Wilson" | `data-import/mimetype.py` | Every other category is hand-written JSON under [`data/`](data), MIT like the code. diff --git a/README.md b/README.md index 0752685..94ed628 100644 --- a/README.md +++ b/README.md @@ -163,15 +163,12 @@ Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name` `personnummer` and `samordningsnummer`, `en_US` adds `ssn` and `itin`. `misc` carries `car`, `coordinate`, `country` (ISO 3166), `creditcard` (Luhn-valid), `currency` (ISO 4217), `datetime` (RFC 3339), `emoji`, `httpstatus`, `language` -(ISO 639-1 and -2), `mac`, `mimetype`, `objectid`, `timezone` (IANA), `useragent` and -`uuid` (v4). Many carry sub-fields — `misc.currency.symbol`, +(ISO 639-1, with its 639-2/T code), `mac`, `mimetype`, `objectid`, `timezone` (IANA), +`useragent` and `uuid` (v4). Many carry sub-fields — `misc.currency.symbol`, `misc.country.alpha2`, `misc.httpstatus.code` — which `--list` shows. `country`, `currency`, `httpstatus`, `language` and `mimetype` are [tables](#table), so -`misc.country[SE].capital` and `misc.currency[Euro].symbol` select a row. Each -holds its whole register: every ISO 3166 country, ISO 4217 currency and ISO 639-1 -language, every HTTP status code in use, and every IANA media type that has a -filename extension. [`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's -source and licence. +`misc.country[SE].capital` and `misc.currency[Euro].symbol` select a row; +[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence. `sex`, `first-name` and `last-name` are tables weighted by bearers, from SCB, the SSA and the Census Bureau. `first-name` links to `sex`, so `sv_SE.sex[f].first-name` diff --git a/data-import/mimetype.py b/data-import/mimetype.py index 8ff1d44..1ac4fc2 100644 --- a/data-import/mimetype.py +++ b/data-import/mimetype.py @@ -3,7 +3,7 @@ data-import/mimetype.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] -Only IANA-registered types with a filename extension ship; the first extension is the one listed. +Only types mime-db records as IANA-registered, with a filename extension, ship. """ import argparse import json @@ -16,12 +16,14 @@ SOURCE = "https://raw.githubusercontent.com/jshttp/mime-db/master/db.json" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "mimetype.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["ext", "type"] +# mime-db lists extensions in no particular order, so where the everyday one is not first, name it. +FIXUPS = {"application/mp4": "mp4s", "audio/mpeg": "mp3", "audio/ogg": "ogg", "video/quicktime": "mov"} def rows(db): for mimetype, entry in db.items(): if entry.get("source") == "iana" and entry.get("extensions"): - yield {"ext": "." + entry["extensions"][0], "type": mimetype} + yield {"ext": "." + FIXUPS.get(mimetype, entry["extensions"][0]), "type": mimetype} def main(): diff --git a/data-import/tsv.py b/data-import/tsv.py index 0816359..f46fcc9 100644 --- a/data-import/tsv.py +++ b/data-import/tsv.py @@ -5,12 +5,14 @@ from pathlib import Path def write(path, columns, rows): - """Write the rows as a TSV; every cell must be non-empty and free of tabs, newlines and braces.""" + """Write the rows as a TSV; the table needs a row, and every cell must be non-empty and free of tabs, newlines and braces.""" lines = ["\t".join(columns)] for row in rows: cells = [str(row[c]) for c in columns] if not all(cells) or any(re.search(r"[\t\n{}]", c) for c in cells): raise ValueError(f"{path}: a cell is empty or holds a tab, newline or brace: {row}") lines.append("\t".join(cells)) + if len(lines) < 2: + raise ValueError(f"{path}: has no rows below its header; a table is at least two rows") Path(path).write_text("\n".join(lines) + "\n", encoding="utf-8") print(f"{path}: {len(lines) - 1} rows", file=sys.stderr) diff --git a/data/misc/mimetype.tsv b/data/misc/mimetype.tsv index 545966f..4bbf13a 100644 --- a/data/misc/mimetype.tsv +++ b/data/misc/mimetype.tsv @@ -62,7 +62,7 @@ ext type .musd application/mmt-usd+xml .mods application/mods+xml .m21 application/mp21 -.mp4 application/mp4 +.mp4s application/mp4 .doc application/msword .mxf application/mxf .nq application/n-quads @@ -511,8 +511,8 @@ ext type .mka audio/matroska .mxmf audio/mobile-xmf .m4a audio/mp4 -.mpga audio/mpeg -.oga audio/ogg +.mp3 audio/mpeg +.ogg audio/ogg .uva audio/vnd.dece.audio .eol audio/vnd.digital-winds .dra audio/vnd.dra @@ -688,7 +688,7 @@ ext type .mp4 video/mp4 .mpeg video/mpeg .ogv video/ogg -.qt video/quicktime +.mov video/quicktime .uvh video/vnd.dece.hd .uvm video/vnd.dece.mobile .uvp video/vnd.dece.pd diff --git a/todo.md b/todo.md index 8a203f1..5b061ee 100644 --- a/todo.md +++ b/todo.md @@ -22,8 +22,7 @@ value is composed; TSV says which values exist. - In a data file, refuse a choice whose items are templates with one identical field set, naming the TSV to write; an inline template is exempt, having no file. A choice of strings stays a choice. -- Convert `misc.timezone`, `car` and `useragent` to tables; `country`, `currency`, - `language`, `httpstatus` and `mimetype` are done. +- Convert `misc.timezone`, `car` and `useragent` to tables. - Let a path select a row by key or by name, `geo.SE.municipality[0180]` and `geo.SE.municipality[Stockholm]`, and descend to a linked table by name, `geo.SE.region[Skåne län].street`; a key or a linked table name may not equal @@ -171,10 +170,8 @@ address, phone, national id, company and date names each. conflicts with sits inside another category. Give the national ids one record shape in step 6, and report a struct column's draw conflict with the path spelling a tag takes rather than the reference spelling. -4. `misc` conversions and the new `misc` tables. `language`, `httpstatus` and - `mimetype` read their registers now, and `misc.country` keeps a currency code - rather than a `parent` link to `misc.currency` (README Decisions). Convert - `timezone`, `car` and `useragent`, and add the new `misc` tables. +4. `misc` conversions and the new `misc` tables. Convert `timezone`, `car` and + `useragent`, and add the new `misc` tables. 5. The remaining locale categories: company, phone, finance, vehicle, words. 6. Records with building-block columns across the shipped set; shape re-pin.