Read misc.territory, timezone, useragent and car as tables, and state the data rules behind them #23

Merged
lilleman merged 14 commits from misc-territory into main 2026-09-19 01:40:34 +02:00
4 changed files with 12 additions and 9 deletions
Showing only changes of commit 85d4760cde - Show all commits
+3 -2
View File
@@ -40,8 +40,9 @@ replacement, and each removed path, column or flag.
`misc.territory` does not ship, Antarctica's among them, and the constant `UTC` `misc.territory` does not ship, Antarctica's among them, and the constant `UTC`
are gone. It draws from 401 zones where it drew from 27 common ones, and evenly, are gone. It draws from 401 zones where it drew from 27 common ones, and evenly,
so a bare `misc.timezone` now renders a sub-zone such as `America/Indiana/Knox` so a bare `misc.timezone` now renders a sub-zone such as `America/Indiana/Knox`
far more often than a capital's: select inside a territory, or pin the zone. `misc.useragent` is the top-user-agents desktop and mobile lists with far more often than a capital's: select inside a territory, or pin the zone.
`browser`, `device` and `os` columns. `misc.car` keeps its makes and models, with `misc.useragent` is the top-user-agents desktop and mobile lists with `browser`,
`device` and `os` columns. `misc.car` keeps its makes and models, with
`misc.car.maker` renamed `misc.car.make`. `misc.car.maker` renamed `misc.car.make`.
- `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`, - `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`,
`locality`, `postal-code` and `street`, weighted by population and address counts `locality`, `postal-code` and `street`, weighted by population and address counts
+3 -3
View File
@@ -172,9 +172,9 @@ carries `car`, `coordinate`, `creditcard` (Luhn-valid), `currency` (ISO 4217),
name, so they are drawn from rather than selected in. name, so they are drawn from rather than selected in.
[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence. [`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
`misc.timezone` is every zone tzdb gives a shipped territory — 401 of them, from one `misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for
apiece for most to 29 for the US — with the territory's code and the zone's standard most of them to dozens for the largest, with the territory's code and the zone's
offset, not the offset in force on any given date, which a zone name is what you standard offset, not the offset in force on any given date, which a zone name is what you
store precisely to avoid. It links to `misc.territory`, so `misc.territory[SE].timezone` store precisely to avoid. It links to `misc.territory`, so `misc.territory[SE].timezone`
is `Europe/Stockholm` and a drawn territory and zone agree. `misc.useragent` carries is `Europe/Stockholm` and a drawn territory and zone agree. `misc.useragent` carries
`browser`, `device` and `os` beside the string, and `misc.car` a `make` and a `model`. `browser`, `device` and `os` beside the string, and `misc.car` a `make` and a `model`.
+5 -2
View File
@@ -27,9 +27,12 @@ REGIONS = ["africa", "antarctica", "asia", "australasia", "backward", "etcetera"
def member(tar, name): def member(tar, name):
try: try:
return tar.extractfile(name).read().decode("utf-8") f = tar.extractfile(name)
except KeyError: except KeyError:
sys.exit(f"{name}: the tarball no longer holds it; the tzdb layout has moved") f = None
if f is None:
sys.exit(f"{name}: the tarball no longer holds it as a file; the tzdb layout has moved")
return f.read().decode("utf-8")
def offsets(tar): def offsets(tar):
+1 -2
View File
@@ -19,8 +19,7 @@ SOURCE = "https://raw.githubusercontent.com/microlinkhq/top-user-agents/master/s
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "useragent.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "useragent.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["browser", "device", "os", "ua"] COLUMNS = ["browser", "device", "os", "ua"]
# First match wins: every Chromium fork carries Chrome's token too, so one this list # First match wins: a Chromium fork this list does not name ships as Chrome.
# does not name would ship as Chrome rather than be dropped.
BROWSERS = [("Edge", r"Edg(A|iOS)?/"), ("Opera", r"OPR/"), ("Samsung Internet", r"SamsungBrowser/"), BROWSERS = [("Edge", r"Edg(A|iOS)?/"), ("Opera", r"OPR/"), ("Samsung Internet", r"SamsungBrowser/"),
("Chrome", r"(Chrome|CriOS)/"), ("Firefox", r"(Firefox|FxiOS)/"), ("Safari", r"Version/[\d.]+ .*Safari")] ("Chrome", r"(Chrome|CriOS)/"), ("Firefox", r"(Firefox|FxiOS)/"), ("Safari", r"Version/[\d.]+ .*Safari")]
SYSTEMS = [("iOS", r"iPhone|iPad|CPU OS "), ("Android", r"Android"), ("ChromeOS", r"CrOS"), SYSTEMS = [("iOS", r"iPhone|iPad|CPU OS "), ("Android", r"Android"), ("ChromeOS", r"CrOS"),