larvit/fejkdata
3
0
Fork 0
Code Pull Requests Actions Activity

Read misc.territory, timezone, useragent and car as tables, and state the data rules behind them #23

Merged
lilleman merged 14 commits from misc-territory into main 2026-09-19 01:40:34 +02:00
Conversation 0 Commits 14 Files Changed 24
+44 -41
3 changed files with 44 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit d366a80629 - Show all commits
README.md
+6 -3
View File
@@ -177,11 +177,14 @@ of them to dozens for the largest, weighted by the population GeoNames records i
so `misc.territory[US].timezone` draws `America/New_York` far more often than
`America/Nome`. It links to `misc.territory`, so `misc.territory[SE].timezone` is
`Europe/Stockholm` and a drawn territory and zone agree. Its `offset` is the zone's
*standard* offset, so do not pair it with a drawn `misc.datetime`: half the year it is
the wrong one, which is what storing a zone name avoids. There is no `UTC` row, tzdb
*standard* offset, so do not pair it with a drawn `misc.datetime`: in a zone that
observes DST it is the wrong one half the year, which is what storing a zone name
avoids. A zone tzdb named recently — `Europe/Kyiv`, `America/Ciudad_Juarez` — is
rejected outright by a consumer resolving it against older tzdata, so where the
consumer validates the zone, pin one rather than draw it. There is no `UTC` row, tzdb
giving that name no territory — spell it as the text `"UTC"`. `misc.useragent` carries
`browser`, `device` and `os` beside the string, and `misc.car` a `make` and a `model`,
32 pairs until an international source replaces them.
until an international source replaces them.
ISO 3166-1 codes territories, not sovereign states, so that is what the table is
called: Greenland and Åland have codes of their own, and `misc.territory.country`
data-import/timezone.py
+7 -7
View File
@@ -5,8 +5,8 @@
The offset is the zone's standard offset, the first field of its Zone rule's last
continuation line, with a Link resolved to its target. The weight is the population
GeoNames records in the zone's cities of 15,000 or more, and that threshold for a
zone holding none, so a drawn zone is one people live in.
GeoNames records in the zone's cities of 15,000 or more, never below that threshold,
so a drawn zone is one people live in.
"""
import argparse
import csv
@@ -87,10 +87,8 @@ def population(body):
text = z.read("cities15000.txt").decode("utf-8")
per = {}
for city in csv.reader(io.StringIO(text), delimiter="\t", quoting=csv.QUOTE_NONE):
if len(city) > 17 and city[17]:
per[city[17]] = per.get(city[17], 0) + int(city[14] or 0)
if not per:
sys.exit("cities15000.txt: no city carried a timezone; the column order has moved")
if len(city) > 17 and city[17] and city[14].isdigit():
per[city[17]] = per.get(city[17], 0) + int(city[14])
return per
@@ -111,7 +109,7 @@ def rows(tar, territories, pop):
offset = utc_offset(raw)
if offset is None:
sys.exit(f"{zone}: standard offset {raw!r} is not a ±HH:MM the table can spell")
yield {"offset": offset, "territory": territory, "weight": pop.get(zone) or FLOOR, "zone": zone}
yield {"offset": offset, "territory": territory, "weight": max(pop.get(zone, 0), FLOOR), "zone": zone}
def main():
@@ -131,6 +129,8 @@ def main():
linked = {r["territory"] for r in table}
if missing := shipped - linked:
sys.exit(f"no zone for {sorted(missing)}; a parent row without a child is a load error")
if (floored := sum(1 for r in table if r["weight"] == FLOOR)) > len(table) // 2:
sys.exit(f"{floored} of {len(table)} zones fell to the floor; GeoNames' population or timezone column has moved")
tsv.write(a.out, COLUMNS, table)
data/misc/timezone.tsv
+31 -31
View File
@@ -53,7 +53,7 @@ offset territory weight zone
+02:00 NA 983097 Africa/Windhoek
-10:00 US 15000 America/Adak
-09:00 US 366200 America/Anchorage
-04:00 AI 2035 America/Anguilla
-04:00 AI 15000 America/Anguilla
-04:00 AG 51737 America/Antigua
-03:00 BR 858988 America/Araguaina
-03:00 AR 11817784 America/Argentina/Buenos_Aires
@@ -108,7 +108,7 @@ offset territory weight zone
-04:00 CA 122883 America/Glace_Bay
-04:00 CA 15000 America/Goose_Bay
-05:00 TC 27489 America/Grand_Turk
-04:00 GD 7500 America/Grenada
-04:00 GD 15000 America/Grenada
-04:00 GP 250260 America/Guadeloupe
-06:00 GT 7302205 America/Guatemala
-05:00 EC 11564603 America/Guayaquil
@@ -133,11 +133,11 @@ offset territory weight zone
-04:00 BO 6958595 America/La_Paz
-05:00 PE 21406340 America/Lima
-08:00 US 46301242 America/Los_Angeles
-04:00 SX 1400 America/Lower_Princes
-04:00 SX 15000 America/Lower_Princes
-03:00 BR 4805317 America/Maceio
-06:00 NI 2621115 America/Managua
-04:00 BR 3918958 America/Manaus
-04:00 MF 5700 America/Marigot
-04:00 MF 15000 America/Marigot
-04:00 MQ 243860 America/Martinique
-06:00 MX 1899165 America/Matamoros
-07:00 MX 2951216 America/Mazatlan
@@ -145,11 +145,11 @@ offset territory weight zone
-06:00 MX 4154924 America/Merida
-09:00 US 15000 America/Metlakatla
-06:00 MX 61603325 America/Mexico_City
-03:00 PM 6200 America/Miquelon
-03:00 PM 15000 America/Miquelon
-04:00 CA 381536 America/Moncton
-06:00 MX 9714766 America/Monterrey
-03:00 UY 2457462 America/Montevideo
-04:00 MS 1000 America/Montserrat
-04:00 MS 15000 America/Montserrat
-05:00 BS 319054 America/Nassau
-05:00 US 88988896 America/New_York
-09:00 US 15000 America/Nome
@@ -157,7 +157,7 @@ offset territory weight zone
-06:00 US 15000 America/North_Dakota/Beulah
-06:00 US 15000 America/North_Dakota/Center
-06:00 US 21382 America/North_Dakota/New_Salem
-02:00 GL 14798 America/Nuuk
-02:00 GL 15000 America/Nuuk
-06:00 MX 50784 America/Ojinaga
-05:00 PA 1692726 America/Panama
-03:00 SR 454064 America/Paramaribo
@@ -178,9 +178,9 @@ offset territory weight zone
-03:00 BR 129186469 America/Sao_Paulo
-02:00 GL 15000 America/Scoresbysund
-09:00 US 15000 America/Sitka
-04:00 BL 5988 America/St_Barthelemy
-04:00 BL 15000 America/St_Barthelemy
-03:30 CA 202443 America/St_Johns
-04:00 KN 12920 America/St_Kitts
-04:00 KN 15000 America/St_Kitts
-04:00 LC 45210 America/St_Lucia
-04:00 VI 70601 America/St_Thomas
-04:00 VC 48723 America/St_Vincent
@@ -189,13 +189,13 @@ offset territory weight zone
-04:00 GL 15000 America/Thule
-08:00 MX 3797218 America/Tijuana
-05:00 CA 24763654 America/Toronto
-04:00 VG 8449 America/Tortola
-04:00 VG 15000 America/Tortola
-07:00 CA 6340886 America/Vancouver
-07:00 CA 28201 America/Whitehorse
-06:00 CA 922566 America/Winnipeg
-09:00 US 15000 America/Yakutat
+10:00 AU 15000 Antarctica/Macquarie
+01:00 SJ 2368 Arctic/Longyearbyen
+01:00 SJ 15000 Arctic/Longyearbyen
+03:00 YE 6904010 Asia/Aden
+05:00 KZ 9055499 Asia/Almaty
+03:00 JO 4950859 Asia/Amman
@@ -277,14 +277,14 @@ offset territory weight zone
+05:00 RU 15407217 Asia/Yekaterinburg
+04:00 AM 2880620 Asia/Yerevan
-01:00 PT 20056 Atlantic/Azores
-04:00 BM 902 Atlantic/Bermuda
-04:00 BM 15000 Atlantic/Bermuda
+00:00 ES 2054431 Atlantic/Canary
-01:00 CV 252678 Atlantic/Cape_Verde
+00:00 FO 13200 Atlantic/Faroe
+00:00 FO 15000 Atlantic/Faroe
+00:00 PT 213794 Atlantic/Madeira
+00:00 IS 245356 Atlantic/Reykjavik
+00:00 SH 637 Atlantic/St_Helena
-03:00 FK 2213 Atlantic/Stanley
+00:00 SH 15000 Atlantic/St_Helena
-03:00 FK 15000 Atlantic/Stanley
+09:30 AU 1706678 Australia/Adelaide
+10:00 AU 5807246 Australia/Brisbane
+09:30 AU 17456 Australia/Broken_Hill
@@ -325,7 +325,7 @@ offset territory weight zone
+01:00 LU 131322 Europe/Luxembourg
+01:00 ES 42918247 Europe/Madrid
+01:00 MT 141714 Europe/Malta
+02:00 AX 10682 Europe/Mariehamn
+02:00 AX 15000 Europe/Mariehamn
+03:00 BY 6105410 Europe/Minsk
+01:00 MC 48977 Europe/Monaco
+03:00 RU 69342062 Europe/Moscow
@@ -336,7 +336,7 @@ offset territory weight zone
+02:00 LV 1206422 Europe/Riga
+01:00 IT 34033666 Europe/Rome
+04:00 RU 3546813 Europe/Samara
+01:00 SM 4500 Europe/San_Marino
+01:00 SM 15000 Europe/San_Marino
+01:00 BA 1905798 Europe/Sarajevo
+04:00 RU 1705529 Europe/Saratov
+03:00 UA 1620906 Europe/Simferopol
@@ -346,8 +346,8 @@ offset territory weight zone
+02:00 EE 1106943 Europe/Tallinn
+01:00 AL 1629643 Europe/Tirane
+04:00 RU 814189 Europe/Ulyanovsk
+01:00 LI 5197 Europe/Vaduz
+01:00 VA 829 Europe/Vatican
+01:00 LI 15000 Europe/Vaduz
+01:00 VA 15000 Europe/Vatican
+01:00 AT 5274040 Europe/Vienna
+02:00 LT 2103133 Europe/Vilnius
+03:00 RU 1837053 Europe/Volgograd
@@ -356,10 +356,10 @@ offset territory weight zone
+01:00 CH 3461664 Europe/Zurich
+03:00 MG 6115943 Indian/Antananarivo
+06:00 IO 15000 Indian/Chagos
+07:00 CX 1355 Indian/Christmas
+06:30 CC 120 Indian/Cocos
+07:00 CX 15000 Indian/Christmas
+06:30 CC 15000 Indian/Cocos
+03:00 KM 118353 Indian/Comoro
+05:00 TF 45 Indian/Kerguelen
+05:00 TF 15000 Indian/Kerguelen
+04:00 SC 22881 Indian/Mahe
+05:00 MV 103693 Indian/Maldives
+04:00 MU 701493 Indian/Mauritius
@@ -373,7 +373,7 @@ offset territory weight zone
-06:00 CL 15000 Pacific/Easter
+11:00 VU 35901 Pacific/Efate
+12:00 FJ 334403 Pacific/Fiji
+12:00 TV 6320 Pacific/Funafuti
+12:00 TV 15000 Pacific/Funafuti
-06:00 EC 15000 Pacific/Galapagos
-09:00 PF 15000 Pacific/Gambier
+11:00 SB 194707 Pacific/Guadalcanal
@@ -385,18 +385,18 @@ offset territory weight zone
+12:00 MH 15000 Pacific/Kwajalein
+12:00 MH 45701 Pacific/Majuro
-09:30 PF 15000 Pacific/Marquesas
+12:00 NR 1100 Pacific/Nauru
-11:00 NU 624 Pacific/Niue
+11:00 NF 880 Pacific/Norfolk
+12:00 NR 15000 Pacific/Nauru
-11:00 NU 15000 Pacific/Niue
+11:00 NF 15000 Pacific/Norfolk
+11:00 NC 137086 Pacific/Noumea
-11:00 AS 11500 Pacific/Pago_Pago
-11:00 AS 15000 Pacific/Pago_Pago
+09:00 PW 15000 Pacific/Palau
-08:00 PN 46 Pacific/Pitcairn
+11:00 FM 6942 Pacific/Pohnpei
-08:00 PN 15000 Pacific/Pitcairn
+11:00 FM 15000 Pacific/Pohnpei
+10:00 PG 588589 Pacific/Port_Moresby
-10:00 CK 13373 Pacific/Rarotonga
-10:00 CK 15000 Pacific/Rarotonga
+10:00 MP 48220 Pacific/Saipan
-10:00 PF 81958 Pacific/Tahiti
+12:00 KI 40311 Pacific/Tarawa
+13:00 TO 22400 Pacific/Tongatapu
+12:00 WF 1200 Pacific/Wallis
+12:00 WF 15000 Pacific/Wallis
1 offset territory weight zone
53 +02:00 NA 983097 Africa/Windhoek
54 -10:00 US 15000 America/Adak
55 -09:00 US 366200 America/Anchorage
56 -04:00 AI 2035 15000 America/Anguilla
57 -04:00 AG 51737 America/Antigua
58 -03:00 BR 858988 America/Araguaina
59 -03:00 AR 11817784 America/Argentina/Buenos_Aires
108 -04:00 CA 122883 America/Glace_Bay
109 -04:00 CA 15000 America/Goose_Bay
110 -05:00 TC 27489 America/Grand_Turk
111 -04:00 GD 7500 15000 America/Grenada
112 -04:00 GP 250260 America/Guadeloupe
113 -06:00 GT 7302205 America/Guatemala
114 -05:00 EC 11564603 America/Guayaquil
133 -04:00 BO 6958595 America/La_Paz
134 -05:00 PE 21406340 America/Lima
135 -08:00 US 46301242 America/Los_Angeles
136 -04:00 SX 1400 15000 America/Lower_Princes
137 -03:00 BR 4805317 America/Maceio
138 -06:00 NI 2621115 America/Managua
139 -04:00 BR 3918958 America/Manaus
140 -04:00 MF 5700 15000 America/Marigot
141 -04:00 MQ 243860 America/Martinique
142 -06:00 MX 1899165 America/Matamoros
143 -07:00 MX 2951216 America/Mazatlan
145 -06:00 MX 4154924 America/Merida
146 -09:00 US 15000 America/Metlakatla
147 -06:00 MX 61603325 America/Mexico_City
148 -03:00 PM 6200 15000 America/Miquelon
149 -04:00 CA 381536 America/Moncton
150 -06:00 MX 9714766 America/Monterrey
151 -03:00 UY 2457462 America/Montevideo
152 -04:00 MS 1000 15000 America/Montserrat
153 -05:00 BS 319054 America/Nassau
154 -05:00 US 88988896 America/New_York
155 -09:00 US 15000 America/Nome
157 -06:00 US 15000 America/North_Dakota/Beulah
158 -06:00 US 15000 America/North_Dakota/Center
159 -06:00 US 21382 America/North_Dakota/New_Salem
160 -02:00 GL 14798 15000 America/Nuuk
161 -06:00 MX 50784 America/Ojinaga
162 -05:00 PA 1692726 America/Panama
163 -03:00 SR 454064 America/Paramaribo
178 -03:00 BR 129186469 America/Sao_Paulo
179 -02:00 GL 15000 America/Scoresbysund
180 -09:00 US 15000 America/Sitka
181 -04:00 BL 5988 15000 America/St_Barthelemy
182 -03:30 CA 202443 America/St_Johns
183 -04:00 KN 12920 15000 America/St_Kitts
184 -04:00 LC 45210 America/St_Lucia
185 -04:00 VI 70601 America/St_Thomas
186 -04:00 VC 48723 America/St_Vincent
189 -04:00 GL 15000 America/Thule
190 -08:00 MX 3797218 America/Tijuana
191 -05:00 CA 24763654 America/Toronto
192 -04:00 VG 8449 15000 America/Tortola
193 -07:00 CA 6340886 America/Vancouver
194 -07:00 CA 28201 America/Whitehorse
195 -06:00 CA 922566 America/Winnipeg
196 -09:00 US 15000 America/Yakutat
197 +10:00 AU 15000 Antarctica/Macquarie
198 +01:00 SJ 2368 15000 Arctic/Longyearbyen
199 +03:00 YE 6904010 Asia/Aden
200 +05:00 KZ 9055499 Asia/Almaty
201 +03:00 JO 4950859 Asia/Amman
277 +05:00 RU 15407217 Asia/Yekaterinburg
278 +04:00 AM 2880620 Asia/Yerevan
279 -01:00 PT 20056 Atlantic/Azores
280 -04:00 BM 902 15000 Atlantic/Bermuda
281 +00:00 ES 2054431 Atlantic/Canary
282 -01:00 CV 252678 Atlantic/Cape_Verde
283 +00:00 FO 13200 15000 Atlantic/Faroe
284 +00:00 PT 213794 Atlantic/Madeira
285 +00:00 IS 245356 Atlantic/Reykjavik
286 +00:00 SH 637 15000 Atlantic/St_Helena
287 -03:00 FK 2213 15000 Atlantic/Stanley
288 +09:30 AU 1706678 Australia/Adelaide
289 +10:00 AU 5807246 Australia/Brisbane
290 +09:30 AU 17456 Australia/Broken_Hill
325 +01:00 LU 131322 Europe/Luxembourg
326 +01:00 ES 42918247 Europe/Madrid
327 +01:00 MT 141714 Europe/Malta
328 +02:00 AX 10682 15000 Europe/Mariehamn
329 +03:00 BY 6105410 Europe/Minsk
330 +01:00 MC 48977 Europe/Monaco
331 +03:00 RU 69342062 Europe/Moscow
336 +02:00 LV 1206422 Europe/Riga
337 +01:00 IT 34033666 Europe/Rome
338 +04:00 RU 3546813 Europe/Samara
339 +01:00 SM 4500 15000 Europe/San_Marino
340 +01:00 BA 1905798 Europe/Sarajevo
341 +04:00 RU 1705529 Europe/Saratov
342 +03:00 UA 1620906 Europe/Simferopol
346 +02:00 EE 1106943 Europe/Tallinn
347 +01:00 AL 1629643 Europe/Tirane
348 +04:00 RU 814189 Europe/Ulyanovsk
349 +01:00 LI 5197 15000 Europe/Vaduz
350 +01:00 VA 829 15000 Europe/Vatican
351 +01:00 AT 5274040 Europe/Vienna
352 +02:00 LT 2103133 Europe/Vilnius
353 +03:00 RU 1837053 Europe/Volgograd
356 +01:00 CH 3461664 Europe/Zurich
357 +03:00 MG 6115943 Indian/Antananarivo
358 +06:00 IO 15000 Indian/Chagos
359 +07:00 CX 1355 15000 Indian/Christmas
360 +06:30 CC 120 15000 Indian/Cocos
361 +03:00 KM 118353 Indian/Comoro
362 +05:00 TF 45 15000 Indian/Kerguelen
363 +04:00 SC 22881 Indian/Mahe
364 +05:00 MV 103693 Indian/Maldives
365 +04:00 MU 701493 Indian/Mauritius
373 -06:00 CL 15000 Pacific/Easter
374 +11:00 VU 35901 Pacific/Efate
375 +12:00 FJ 334403 Pacific/Fiji
376 +12:00 TV 6320 15000 Pacific/Funafuti
377 -06:00 EC 15000 Pacific/Galapagos
378 -09:00 PF 15000 Pacific/Gambier
379 +11:00 SB 194707 Pacific/Guadalcanal
385 +12:00 MH 15000 Pacific/Kwajalein
386 +12:00 MH 45701 Pacific/Majuro
387 -09:30 PF 15000 Pacific/Marquesas
388 +12:00 NR 1100 15000 Pacific/Nauru
389 -11:00 NU 624 15000 Pacific/Niue
390 +11:00 NF 880 15000 Pacific/Norfolk
391 +11:00 NC 137086 Pacific/Noumea
392 -11:00 AS 11500 15000 Pacific/Pago_Pago
393 +09:00 PW 15000 Pacific/Palau
394 -08:00 PN 46 15000 Pacific/Pitcairn
395 +11:00 FM 6942 15000 Pacific/Pohnpei
396 +10:00 PG 588589 Pacific/Port_Moresby
397 -10:00 CK 13373 15000 Pacific/Rarotonga
398 +10:00 MP 48220 Pacific/Saipan
399 -10:00 PF 81958 Pacific/Tahiti
400 +12:00 KI 40311 Pacific/Tarawa
401 +13:00 TO 22400 Pacific/Tongatapu
402 +12:00 WF 1200 15000 Pacific/Wallis