Read misc.territory, timezone, useragent and car as tables, and state the data rules behind them #23

Merged
lilleman merged 14 commits from misc-territory into main 2026-09-19 01:40:34 +02:00
4 changed files with 12 additions and 9 deletions
Showing only changes of commit 85d4760cde - Show all commits
+3 -2
View File
@@ -40,8 +40,9 @@ replacement, and each removed path, column or flag.
`misc.territory` does not ship, Antarctica's among them, and the constant `UTC`
are gone. It draws from 401 zones where it drew from 27 common ones, and evenly,
so a bare `misc.timezone` now renders a sub-zone such as `America/Indiana/Knox`
far more often than a capital's: select inside a territory, or pin the zone. `misc.useragent` is the top-user-agents desktop and mobile lists with
`browser`, `device` and `os` columns. `misc.car` keeps its makes and models, with
far more often than a capital's: select inside a territory, or pin the zone.
`misc.useragent` is the top-user-agents desktop and mobile lists with `browser`,
`device` and `os` columns. `misc.car` keeps its makes and models, with
`misc.car.maker` renamed `misc.car.make`.
- `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`,
`locality`, `postal-code` and `street`, weighted by population and address counts
+3 -3
View File
@@ -172,9 +172,9 @@ carries `car`, `coordinate`, `creditcard` (Luhn-valid), `currency` (ISO 4217),
name, so they are drawn from rather than selected in.
[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
`misc.timezone` is every zone tzdb gives a shipped territory — 401 of them, from one
apiece for most to 29 for the US — with the territory's code and the zone's standard
offset, not the offset in force on any given date, which a zone name is what you
`misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for
most of them to dozens for the largest, with the territory's code and the zone's
standard offset, not the offset in force on any given date, which a zone name is what you
store precisely to avoid. It links to `misc.territory`, so `misc.territory[SE].timezone`
is `Europe/Stockholm` and a drawn territory and zone agree. `misc.useragent` carries
`browser`, `device` and `os` beside the string, and `misc.car` a `make` and a `model`.
+5 -2
View File
@@ -27,9 +27,12 @@ REGIONS = ["africa", "antarctica", "asia", "australasia", "backward", "etcetera"
def member(tar, name):
try:
return tar.extractfile(name).read().decode("utf-8")
f = tar.extractfile(name)
except KeyError:
sys.exit(f"{name}: the tarball no longer holds it; the tzdb layout has moved")
f = None
if f is None:
sys.exit(f"{name}: the tarball no longer holds it as a file; the tzdb layout has moved")
return f.read().decode("utf-8")
def offsets(tar):
+1 -2
View File
@@ -19,8 +19,7 @@ SOURCE = "https://raw.githubusercontent.com/microlinkhq/top-user-agents/master/s
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "useragent.tsv"
CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["browser", "device", "os", "ua"]
# First match wins: every Chromium fork carries Chrome's token too, so one this list
# does not name would ship as Chrome rather than be dropped.
# First match wins: a Chromium fork this list does not name ships as Chrome.
BROWSERS = [("Edge", r"Edg(A|iOS)?/"), ("Opera", r"OPR/"), ("Samsung Internet", r"SamsungBrowser/"),
("Chrome", r"(Chrome|CriOS)/"), ("Firefox", r"(Firefox|FxiOS)/"), ("Safari", r"Version/[\d.]+ .*Safari")]
SYSTEMS = [("iOS", r"iPhone|iPad|CPU OS "), ("Android", r"Android"), ("ChromeOS", r"CrOS"),