Rebuild the geo trees without route designations, with ZCTAs at the place holding most of them

README.md

@@ -1014,8 +1014,9 @@ renamed or retyped line is a major.
   whole.** `data/sv_SE` alone no longer loads: a test loads `data` and prefixes
   the locale, and `--no-shipped-data -d` takes the whole `data` folder or a set of
   one's own.
-- **The default embed holds every Swedish postort and the US places of 25,000 or
-  more.** Sweden fits whole in 700 KB; every US place of 10,000 would pass a
+- **The default embed holds every Swedish postort the import can place and give a
+  street-delivery code and a street, and the US places of 25,000 or more.** Sweden
+  fits whole in 700 KB; every US place of 10,000 would pass a
   megabyte and fetch 1,200 counties of TIGER files, so the threshold sits where the
   two countries match in size, and `--min-population` and
   `--streets-per-locality` on the import scripts build a fuller set. The two trees
@@ -1030,6 +1031,13 @@ renamed or retyped line is a major.
   carries stale spellings; the nearest code across a border named the wrong kommun
   half the time it was tried, so a postort none of the three rules place is
   dropped, as is one not cased like a place name.
+- **A highway designation is not a street, and a US postal code belongs to the place
+  holding most of its land inside places.** `I- 55 Bus` and `US Hwy 1` carry the
+  most address ranges in many places and would head every address, so the import
+  drops names spelled as a route. A ZCTA goes to the place its largest in-place part
+  lies in, and ships only when that place does; counting the land outside every
+  place too would drop a quarter of the places, whose codes straddle unincorporated
+  land, for a postal city the USPS mostly names the same way.
 - **`List` advertises direct descents only.** `region.municipality.locality` is
   listed, and `region.locality` resolves too but is not: the set of every descent
   through a chain of five tables is every subsequence of it, and the direct chain is

data-import/geo-se.py

@@ -2,12 +2,6 @@
 """Rebuild data/geo/SE/*.tsv from SCB (CC0), GeoNames (CC BY 4.0) and Trafikverket NVDB (CC0).
 
     TRAFIKVERKET_API_KEY=… data-import/geo-se.py [--key-file FILE] [--cache DIR] [--streets-per-locality N] [--out DIR]
-
-A locality is a GeoNames postort, placed in the municipality it names, else of its
-tätort, else of most of its codes, and weighted by its tätort's population, else its
-municipality's, else 200. Box codes are dropped by the digit after the postort's own
-prefix. Each NVDB street segment goes to the nearest postal code centroid; a locality
-keeps the N names with most segments.
 """
 import argparse
 import collections
@@ -95,7 +89,7 @@ def geonames(cache):
 def nvdb_segments(cache, key):
     path = cache / "nvdb-gatunamn.tsv"
     if not path.exists():
-        with open(path, "w", encoding="utf-8") as out:
+        with open(path.with_suffix(".part"), "w", encoding="utf-8") as out:
             change = "0"
             while True:
                 query = (
@@ -116,6 +110,7 @@ def nvdb_segments(cache, key):
                 change = result["INFO"]["LASTCHANGEID"]
                 if len(rows) < NVDB_PAGE:
                     break
+        path.with_suffix(".part").rename(path)
     for line in path.read_text(encoding="utf-8").splitlines():
         name, lon, lat = line.split("\t")
         yield name, float(lat), float(lon)
@@ -208,7 +203,8 @@ def streets(segments, codes, localities, per_locality):
     nearest = Nearest((r["lat"], r["lon"], r["locality"]) for r in codes if r["lat"] is not None and r["locality"] in localities)
     count = collections.Counter()
     for name, lat, lon in segments:
-        count[(nearest.find(lat, lon), name)] += 1
+        if name[0].isalpha():
+            count[(nearest.find(lat, lon), name)] += 1
     of = collections.defaultdict(list)
     for (locality, name), n in count.items():
         of[locality].append((n, name))

data-import/geo-us.py

@@ -2,11 +2,6 @@
 """Rebuild data/geo/US/*.tsv from the Census Bureau's Gazetteer, population estimates, ZCTA relationships and TIGER/Line files (public domain).
 
     data-import/geo-us.py [--cache DIR] [--min-population N] [--streets-per-locality N] [--out DIR]
-
-A locality is an incorporated place, or a consolidated city's balance, of at least N
-people, in the county holding most of it. Its postal codes are the ZCTAs mostly inside
-it, weighted by their TIGER address ranges, and its streets the N names with most
-address ranges in those codes.
 """
 import argparse
 import collections
@@ -32,10 +27,10 @@ OUT = Path(__file__).resolve().parent.parent / "data" / "geo" / "US"
 CACHE = Path(__file__).resolve().parent / "cache"
 ESTIMATE = "POPESTIMATE2025"
 CDP = "57"
+HIGHWAY = re.compile(r"\b(I- |Hwy |Rte |Route |Rd )\d")
 SUFFIX = re.compile(r" (city and borough|city|town|village|borough|municipality|comunidad|zona urbana|metropolitan government|metro government|consolidated government|unified government|urban county|corporation|plantation)( \(balance\))?$")
 # Places whose Census name is a merged government's; the postal city is what an address carries.
 NAMES = {"1303440": "Athens", "1304204": "Augusta", "1349008": "Macon", "2148006": "Louisville", "3011397": "Butte", "4732742": "Hartsville", "4752006": "Nashville"}
-# The predominant zone of each state.
 TIMEZONES = {
     "AK": "America/Anchorage", "AL": "America/Chicago", "AR": "America/Chicago", "AZ": "America/Phoenix",
     "CA": "America/Los_Angeles", "CO": "America/Denver", "CT": "America/New_York", "DC": "America/New_York",
@@ -122,7 +117,7 @@ def localities(cache, min_population, counties):
     out = {}
     for r in gazetteer(cache, "place"):
         geoid, population = r["GEOID"], place_population.get(r["GEOID"], 0)
-        if r["FUNCSTAT"] not in "AFN" or r["LSAD"] == CDP or population < min_population or not county_part.get(geoid):
+        if r["FUNCSTAT"] not in ("A", "F", "N") or r["LSAD"] == CDP or population < min_population or not county_part.get(geoid):
             continue
         county = max(county_part[geoid])[1]
         if county not in counties:
@@ -133,12 +128,13 @@ def localities(cache, min_population, counties):
 
 
 def postal_codes(cache, localities):
-    """Each ZCTA and the shipped place holding most of its land."""
+    """Each ZCTA whose largest part inside any place lies in a shipped place."""
     parts = {}
     for r in csv.DictReader(io.StringIO(text(tsv.fetch(ZCTA_PLACE, cache, "zcta-place.txt"))), delimiter="|"):
-        if r["GEOID_ZCTA5_20"] and r["GEOID_PLACE_20"] in localities:
+        if r["GEOID_ZCTA5_20"] and r["GEOID_PLACE_20"]:
             parts.setdefault(r["GEOID_ZCTA5_20"], []).append((int(r["AREALAND_PART"]), r["GEOID_PLACE_20"]))
-    return {zcta: max(p)[1] for zcta, p in parts.items()}
+    largest = {zcta: max(p)[1] for zcta, p in parts.items()}
+    return {zcta: place for zcta, place in largest.items() if place in localities}
 
 
 def streets(cache, counties, locality_of_zcta, per_locality):
@@ -153,7 +149,7 @@ def streets(cache, counties, locality_of_zcta, per_locality):
                 zips[r["TLID"]].add(r["ZIP"])
                 addresses[r["ZIP"]] += 1
         for r in tiger(cache, "featnames", county, {"TLID", "FULLNAME", "PAFLAG"}):
-            if r["PAFLAG"] == "P" and r["FULLNAME"]:
+            if r["PAFLAG"] == "P" and r["FULLNAME"] and not HIGHWAY.search(r["FULLNAME"]):
                 for z in zips.get(r["TLID"], ()):
                     count[(locality_of_zcta[z], r["FULLNAME"])] += 1
     of = collections.defaultdict(list)

data-import/tsv.py

@@ -20,10 +20,10 @@ def fetch(source, cache, name, magic=b"", data=None, headers=None):
                 body = r.read()
         except OSError:
             body = b""
-        if body.startswith(magic) and b"Request Rejected" not in body[:512]:
+        if body and body.startswith(magic) and b"Request Rejected" not in body[:512]:
             path.parent.mkdir(parents=True, exist_ok=True)
             path.write_bytes(body)
-        else:
+        elif attempt < 5:
             time.sleep(10 * attempt)
     sys.exit(f"{source}: no valid download in 5 attempts")
 
@@ -33,7 +33,8 @@ def write(path, columns, rows):
     lines = ["\t".join(columns)]
     for row in rows:
         cells = [str(row[c]) for c in columns]
-        assert all(cells) and not any(re.search(r"[\t\n{}]", c) for c in cells), row
+        if not all(cells) or any(re.search(r"[\t\n{}]", c) for c in cells):
+            raise ValueError(f"{path}: a cell is empty or holds a tab, newline or brace: {row}")
         lines.append("\t".join(cells))
     Path(path).write_text("\n".join(lines) + "\n", encoding="utf-8")
     print(f"{path}: {len(lines) - 1} rows", file=sys.stderr)

data/geo/SE/street.tsv

@@ -6263,10 +6263,10 @@ Strandvägen	Källby	8
 Ringvägen	Källby	7
 Laxvägen	Källby	6
 Nästakvarnsvägen	Källby	6
-2:a Svanviksvägen	Källby	5
 Badvägen	Källby	5
 Mellomkvarnsvägen	Källby	5
 Braxenvägen	Källby	4
+Mastvägen	Källby	4
 Tranbärsvägen	Källö-Knippla	8
 Dalvägen	Källö-Knippla	7
 Hamnbergsvägen	Källö-Knippla	7

data/geo/US/locality.tsv

@@ -91,10 +91,8 @@ code	name	municipality	population	lat	lon
 0603162	Atwater	06047	32133	37.350672	-120.594258
 0603386	Azusa	06037	47447	34.138524	-117.912253
 0603526	Bakersfield	06029	422165	35.353174	-119.037878
-0603666	Baldwin Park	06037	68880	34.082824	-117.971286
 0603820	Banning	06065	31852	33.945903	-116.899044
 0604758	Beaumont	06065	60835	33.884946	-116.9538
-0604870	Bell	06037	31625	33.979704	-118.179249
 0604982	Bellflower	06037	74719	33.887821	-118.12725
 0604996	Bell Gardens	06037	37367	33.964379	-118.154437
 0605108	Belmont	06081	28752	37.514398	-122.294475
@@ -106,7 +104,6 @@ code	name	municipality	population	lat	lon
 0608142	Brentwood	06013	66607	37.935578	-121.71897
 0608786	Buena Park	06059	81741	33.857353	-118.007373
 0608954	Burbank	06037	102988	34.190079	-118.326405
-0609066	Burlingame	06081	31929	37.589952	-122.363342
 0609710	Calexico	06025	38623	32.684859	-115.494738
 0610046	Camarillo	06111	68652	34.221823	-119.032154
 0610345	Campbell	06085	42490	37.280244	-121.953296
@@ -227,7 +224,6 @@ code	name	municipality	population	lat	lon
 0648914	Monterey Park	06037	57457	34.04838	-118.133165
 0649138	Moorpark	06111	35023	34.285942	-118.87704
 0649270	Moreno Valley	06065	214263	33.923253	-117.205685
-0649278	Morgan Hill	06085	45625	37.132495	-121.641891
 0649582	Mountain House	06077	25315	37.779716	-121.537195
 0649670	Mountain View	06085	85438	37.399686	-122.079269
 0650076	Murrieta	06065	114124	33.573214	-117.191007
@@ -291,7 +287,6 @@ code	name	municipality	population	lat	lon
 0666070	San Dimas	06037	32978	34.107957	-117.808529
 0667000	San Francisco	06075	826079	37.727239	-123.032229
 0667042	San Gabriel	06037	38186	34.094636	-118.098469
-0667056	Sanger	06019	27228	36.700864	-119.55773
 0667112	San Jacinto	06065	56014	33.798864	-116.995914
 0668000	San Jose	06085	989814	37.296011	-121.814552
 0668028	San Juan Capistrano	06059	35119	33.500889	-117.654388
@@ -500,7 +495,6 @@ code	name	municipality	population	lat	lon
 1276500	West Melbourne	12009	31285	28.050398	-80.658327
 1276582	Weston	12011	69588	26.100076	-80.401266
 1276600	West Palm Beach	12099	127189	26.745114	-80.127038
-1278250	Winter Garden	12095	48063	28.541217	-81.597043
 1278275	Winter Haven	12105	62313	28.045836	-81.732743
 1278300	Winter Park	12095	30444	28.59663	-81.345748
 1278325	Winter Springs	12117	39585	28.695481	-81.272086
@@ -546,7 +540,6 @@ code	name	municipality	population	lat	lon
 1373256	Statesboro	13031	35366	32.439408	-81.775106
 1373704	Stockbridge	13151	36331	33.529646	-84.233422
 1373784	Stonecrest	13089	61337	33.680667	-84.14525
-1374180	Sugar Hill	13135	29280	34.088966	-84.034941
 1377652	Tucker	13089	37290	33.845169	-84.202085
 1378324	Union City	13121	29910	33.593837	-84.56263
 1378800	Valdosta	13185	56186	30.850126	-83.278833
@@ -638,7 +631,6 @@ code	name	municipality	population	lat	lon
 1765000	Rockford	17201	147384	42.2593	-89.063478
 1765078	Rock Island	17161	36225	41.469444	-90.585307
 1765442	Romeoville	17197	41281	41.627783	-88.101237
-1766040	Round Lake Beach	17097	26902	42.382452	-88.080682
 1766703	St. Charles	17089	33594	41.919447	-88.310743
 1768003	Schaumburg	17031	77249	42.028423	-88.084745
 1770122	Skokie	17031	66544	42.03597	-87.739994
@@ -765,7 +757,6 @@ code	name	municipality	population	lat	lon
 2338740	Lewiston	23001	38866	44.089325	-70.172051
 2360545	Portland	23005	69911	43.633157	-70.185305
 2371990	South Portland	23005	27007	43.631402	-70.285989
-2401600	Annapolis	24003	40634	38.97216	-76.505334
 2404000	Baltimore	24510	569997	39.300061	-76.610518
 2408775	Bowie	24033	58383	38.954703	-76.738834
 2418750	College Park	24033	35418	38.996692	-76.936407
@@ -867,7 +858,6 @@ code	name	municipality	population	lat	lon
 2684000	Warren	26099	137138	42.492904	-83.025001
 2686000	Westland	26163	82333	42.319173	-83.380778
 2688940	Wyoming	26081	78588	42.890317	-85.706733
-2701486	Andover	27003	33874	45.25242	-93.337402
 2701900	Apple Valley	27037	55248	44.745301	-93.199413
 2702908	Austin	27099	26689	43.672098	-92.977863
 2706382	Blaine	27003	76603	45.170515	-93.206089
@@ -1043,26 +1033,22 @@ code	name	municipality	population	lat	lon
 3633139	Hempstead	36059	59819	40.704374	-73.619445
 3638077	Ithaca	36109	33632	42.443937	-76.503055
 3638264	Jamestown	36013	27663	42.097603	-79.236135
-3639853	Kiryas Joel	36071	47147	41.33901	-74.166941
 3642554	Lindenhurst	36103	27258	40.685751	-73.371594
 3643335	Long Beach	36059	34918	40.585835	-73.665403
 3647042	Middletown	36071	30098	41.444922	-74.423573
 3649121	Mount Vernon	36119	72304	40.913105	-73.829317
-3650034	Newburgh	36071	28059	41.503249	-74.01964
 3650617	New Rochelle	36119	85864	40.923243	-73.779344
 3651000	New York	36047	8584629	40.662712	-73.938677
 3651055	Niagara Falls	36063	47255	43.092841	-79.014743
 3653682	North Tonawanda	36063	29806	43.049157	-78.871224
 3655530	Ossining	36119	27579	41.15349	-73.870587
 3656979	Peekskill	36119	25891	41.287396	-73.923597
-3659223	Port Chester	36119	32303	41.004701	-73.667175
 3659641	Poughkeepsie	36027	32499	41.694595	-73.919869
 3663000	Rochester	36055	206108	43.169927	-77.616891
 3663264	Rockville Centre	36059	25770	40.664024	-73.638048
 3663418	Rome	36065	31682	43.226002	-75.489523
 3665255	Saratoga Springs	36091	28768	43.067704	-73.778142
 3665508	Schenectady	36093	69471	42.802691	-73.92696
-3670420	Spring Valley	36087	34482	41.114567	-74.048712
 3673000	Syracuse	36067	144896	43.04126	-76.143212
 3675484	Troy	36083	50912	42.735812	-73.675321
 3676540	Utica	36065	62978	43.096382	-75.225983
@@ -1091,7 +1077,6 @@ code	name	municipality	population	lat	lon
 3731400	High Point	37081	120571	35.990803	-79.992713
 3732260	Holly Springs	37183	50288	35.651994	-78.843196
 3733120	Huntersville	37119	68535	35.40681	-80.86868
-3733560	Indian Trail	37179	44303	35.06898	-80.651758
 3734200	Jacksonville	37133	72732	34.729759	-77.394
 3735200	Kannapolis	37025	61708	35.479868	-80.647422
 3735600	Kernersville	37067	29128	36.106171	-80.081895
@@ -1339,7 +1324,6 @@ code	name	municipality	population	lat	lon
 4816468	Converse	48029	32481	29.512488	-98.304625
 4816612	Coppell	48113	41386	32.963318	-96.990082
 4816624	Copperas Cove	48099	40118	31.119436	-97.913909
-4816696	Corinth	48121	25297	33.143686	-97.068592
 4817000	Corpus Christi	48355	317247	27.754252	-97.173385
 4817060	Corsicana	48349	25487	32.082404	-96.4667
 4819000	Dallas	48113	1329491	32.793333	-96.766513
@@ -1437,7 +1421,6 @@ code	name	municipality	population	lat	lon
 4872392	Texas City	48167	59733	29.424942	-94.890026
 4872530	The Colony	48121	46196	33.089598	-96.903368
 4874144	Tyler	48423	113723	32.317334	-95.306399
-4874492	University Park	48113	25323	32.848624	-96.79525
 4875428	Victoria	48469	66500	28.828528	-96.986015
 4876000	Waco	48309	147788	31.557886	-97.190841
 4876816	Waxahachie	48139	50090	32.40248	-96.844493
@@ -1449,7 +1432,6 @@ code	name	municipality	population	lat	lon
 4907690	Bountiful	49011	44988	40.872483	-111.8612
 4911320	Cedar City	49021	41871	37.683877	-113.09579
 4913850	Clearfield	49011	35144	41.097012	-112.01792
-4916270	Cottonwood Heights	49035	32265	40.613728	-111.814137
 4920120	Draper	49035	50652	40.49715	-111.864636
 4920810	Eagle Mountain	49049	66557	40.314272	-112.011983
 4924740	Farmington	49011	26266	40.984669	-111.906306
@@ -1468,7 +1450,6 @@ code	name	municipality	population	lat	lon
 4957300	Orem	49049	95214	40.298067	-111.699463
 4960930	Pleasant Grove	49049	37773	40.37145	-111.741264
 4962470	Provo	49049	114527	40.245312	-111.645057
-4964340	Riverton	49035	47395	40.517634	-111.963515
 4965110	Roy	49057	38794	41.17146	-112.048476
 4965330	St. George	49053	108713	37.077006	-113.576534
 4967000	Salt Lake City	49035	218428	40.775366	-111.927266
@@ -1491,7 +1472,6 @@ code	name	municipality	population	lat	lon
 5114968	Charlottesville	51540	44388	38.037658	-78.485381
 5116000	Chesapeake	51550	255332	36.679376	-76.301788
 5121344	Danville	51590	41647	36.581754	-79.408258
-5126496	Fairfax	51600	26772	38.853183	-77.299025
 5129744	Fredericksburg	51630	30393	38.299272	-77.486658
 5135000	Hampton	51650	137315	37.047961	-76.297293
 5135624	Harrisonburg	51660	50839	38.436255	-78.873303
@@ -1530,7 +1510,6 @@ code	name	municipality	population	lat	lon
 5338038	Lakewood	53053	62755	47.164092	-122.528106
 5340245	Longview	53015	38362	46.145707	-122.964299
 5340840	Lynnwood	53061	42132	47.828979	-122.300672
-5343150	Maple Valley	53033	28482	47.367147	-122.034815
 5343955	Marysville	53061	77686	48.080214	-122.154312
 5345005	Mercer Island	53033	25422	47.564004	-122.231214
 5347245	Moses Lake	53025	27469	47.12785	-119.276384
@@ -1565,7 +1544,6 @@ code	name	municipality	population	lat	lon
 5506500	Beloit	55105	36649	42.523506	-89.01766
 5510025	Brookfield	55133	41448	43.06432	-88.117794
 5511950	Caledonia	55101	25229	42.797411	-87.877405
-5519775	De Pere	55009	25590	44.429327	-88.08037
 5522300	Eau Claire	55035	72465	44.818343	-91.494769
 5525950	Fitchburg	55025	34615	42.985782	-89.425334
 5526275	Fond du Lac	55039	44541	43.772182	-88.440264

data/geo/US/municipality.tsv

@@ -255,7 +255,6 @@ code	name	region	population
 23001	Androscoggin County	ME	116487
 23005	Cumberland County	ME	317222
 23019	Penobscot County	ME	157967
-24003	Anne Arundel County	MD	603380
 24021	Frederick County	MD	302883
 24031	Montgomery County	MD	1074582
 24033	Prince George's County	MD	970374
@@ -389,7 +388,6 @@ code	name	region	population
 36067	Onondaga County	NY	466584
 36071	Orange County	NY	417669
 36083	Rensselaer County	NY	160510
-36087	Rockland County	NY	357397
 36091	Saratoga County	NY	241343
 36093	Schenectady County	NY	162581
 36103	Suffolk County	NY	1546090
@@ -603,7 +601,6 @@ code	name	region	population
 51540	Charlottesville city	VA	44388
 51550	Chesapeake city	VA	255332
 51590	Danville city	VA	41647
-51600	Fairfax city	VA	26772
 51630	Fredericksburg city	VA	30393
 51650	Hampton city	VA	137315
 51660	Harrisonburg city	VA	50839

table_test.go

@@ -44,7 +44,6 @@ var (
 	regionOf       = map[string]string{"0180": "01", "0184": "01", "1280": "12", "1281": "12", "1480": "14"}
 )
 
-// siblings adds two child tables under locality: postal-code, keyed, and street, keyless.
 func siblings() map[string]string {
 	return with(geo(), map[string]string{
 		"postal-code.json": `{"format":"{code}","rows":"postal-code.tsv","key":"code","parent":"locality"}`,

todo.md

@@ -54,10 +54,10 @@ countries; the README maps each to the native term.
 | Table | SE | US | Weight |
 |---|---|---|---|
 | `region` | län (21) | state and DC (50; Hawaii has no incorporated place) | population |
-| `municipality` | kommun (290) | county with a shipped place (666) | population |
-| `locality` | postort (1,522), tätort population | place of 25,000+ (1,601) | population |
-| `postal-code` | postnummer with street delivery (13,712) | ZCTA of a shipped place (7,401) | one; address ranges |
-| `street` | gatunamn, top 10 per postort (14,764) | street name, top 10 per place (16,010) | segments; address ranges |
+| `municipality` | kommun (290) | county with a shipped place (663) | population |
+| `locality` | postort (1,522), tätort population | place of 25,000+ (1,579) | population |
+| `postal-code` | postnummer with street delivery (13,712) | ZCTA of a shipped place (5,946) | one; address ranges |
+| `street` | gatunamn, top 10 per postort (14,764) | street name, top 10 per place (15,790) | segments; address ranges |
 
 - Shipped in step 2, README Data. `geo.SE.address` is a record over one consistent
   draw. Each region row carries its timezone, each locality its centroid.