Read misc.territory, timezone, useragent and car as tables, and state the data rules behind them #23
+3
-2
@@ -40,8 +40,9 @@ replacement, and each removed path, column or flag.
|
|||||||
`misc.territory` does not ship, Antarctica's among them, and the constant `UTC`
|
`misc.territory` does not ship, Antarctica's among them, and the constant `UTC`
|
||||||
are gone. It draws from 401 zones where it drew from 27 common ones, and evenly,
|
are gone. It draws from 401 zones where it drew from 27 common ones, and evenly,
|
||||||
so a bare `misc.timezone` now renders a sub-zone such as `America/Indiana/Knox`
|
so a bare `misc.timezone` now renders a sub-zone such as `America/Indiana/Knox`
|
||||||
far more often than a capital's: select inside a territory, or pin the zone. `misc.useragent` is the top-user-agents desktop and mobile lists with
|
far more often than a capital's: select inside a territory, or pin the zone.
|
||||||
`browser`, `device` and `os` columns. `misc.car` keeps its makes and models, with
|
`misc.useragent` is the top-user-agents desktop and mobile lists with `browser`,
|
||||||
|
`device` and `os` columns. `misc.car` keeps its makes and models, with
|
||||||
`misc.car.maker` renamed `misc.car.make`.
|
`misc.car.maker` renamed `misc.car.make`.
|
||||||
- `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`,
|
- `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`,
|
||||||
`locality`, `postal-code` and `street`, weighted by population and address counts
|
`locality`, `postal-code` and `street`, weighted by population and address counts
|
||||||
|
|||||||
@@ -172,9 +172,9 @@ carries `car`, `coordinate`, `creditcard` (Luhn-valid), `currency` (ISO 4217),
|
|||||||
name, so they are drawn from rather than selected in.
|
name, so they are drawn from rather than selected in.
|
||||||
[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
|
[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
|
||||||
|
|
||||||
`misc.timezone` is every zone tzdb gives a shipped territory — 401 of them, from one
|
`misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for
|
||||||
apiece for most to 29 for the US — with the territory's code and the zone's standard
|
most of them to dozens for the largest, with the territory's code and the zone's
|
||||||
offset, not the offset in force on any given date, which a zone name is what you
|
standard offset, not the offset in force on any given date, which a zone name is what you
|
||||||
store precisely to avoid. It links to `misc.territory`, so `misc.territory[SE].timezone`
|
store precisely to avoid. It links to `misc.territory`, so `misc.territory[SE].timezone`
|
||||||
is `Europe/Stockholm` and a drawn territory and zone agree. `misc.useragent` carries
|
is `Europe/Stockholm` and a drawn territory and zone agree. `misc.useragent` carries
|
||||||
`browser`, `device` and `os` beside the string, and `misc.car` a `make` and a `model`.
|
`browser`, `device` and `os` beside the string, and `misc.car` a `make` and a `model`.
|
||||||
|
|||||||
@@ -27,9 +27,12 @@ REGIONS = ["africa", "antarctica", "asia", "australasia", "backward", "etcetera"
|
|||||||
|
|
||||||
def member(tar, name):
|
def member(tar, name):
|
||||||
try:
|
try:
|
||||||
return tar.extractfile(name).read().decode("utf-8")
|
f = tar.extractfile(name)
|
||||||
except KeyError:
|
except KeyError:
|
||||||
sys.exit(f"{name}: the tarball no longer holds it; the tzdb layout has moved")
|
f = None
|
||||||
|
if f is None:
|
||||||
|
sys.exit(f"{name}: the tarball no longer holds it as a file; the tzdb layout has moved")
|
||||||
|
return f.read().decode("utf-8")
|
||||||
|
|
||||||
|
|
||||||
def offsets(tar):
|
def offsets(tar):
|
||||||
|
|||||||
@@ -19,8 +19,7 @@ SOURCE = "https://raw.githubusercontent.com/microlinkhq/top-user-agents/master/s
|
|||||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "useragent.tsv"
|
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "useragent.tsv"
|
||||||
CACHE = Path(__file__).resolve().parent / "cache"
|
CACHE = Path(__file__).resolve().parent / "cache"
|
||||||
COLUMNS = ["browser", "device", "os", "ua"]
|
COLUMNS = ["browser", "device", "os", "ua"]
|
||||||
# First match wins: every Chromium fork carries Chrome's token too, so one this list
|
# First match wins: a Chromium fork this list does not name ships as Chrome.
|
||||||
# does not name would ship as Chrome rather than be dropped.
|
|
||||||
BROWSERS = [("Edge", r"Edg(A|iOS)?/"), ("Opera", r"OPR/"), ("Samsung Internet", r"SamsungBrowser/"),
|
BROWSERS = [("Edge", r"Edg(A|iOS)?/"), ("Opera", r"OPR/"), ("Samsung Internet", r"SamsungBrowser/"),
|
||||||
("Chrome", r"(Chrome|CriOS)/"), ("Firefox", r"(Firefox|FxiOS)/"), ("Safari", r"Version/[\d.]+ .*Safari")]
|
("Chrome", r"(Chrome|CriOS)/"), ("Firefox", r"(Firefox|FxiOS)/"), ("Safari", r"Version/[\d.]+ .*Safari")]
|
||||||
SYSTEMS = [("iOS", r"iPhone|iPad|CPU OS "), ("Android", r"Android"), ("ChromeOS", r"CrOS"),
|
SYSTEMS = [("iOS", r"iPhone|iPad|CPU OS "), ("Android", r"Android"), ("ChromeOS", r"CrOS"),
|
||||||
|
|||||||
Reference in New Issue
Block a user