Document the geo trees, their sources and the choices behind them, and mark step 2 done

2026-09-18 00:52:08 +02:00
parent 1029b789b5
commit db271f82ee
3 changed files with 63 additions and 13 deletions
@@ -26,3 +26,9 @@ replacement, and each removed path, column or flag.
  `currency`, `flag`, `languages`, `numeric` and `tld` added; `misc.currency` the
  current ISO 4217 currencies with a minor unit, with `decimals` and `numeric`
  added, and its symbols from CLDR. `DATA-LICENSES.md` lists each source.
+- `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`,
+  `locality`, `postal-code` and `street`, weighted by population and address counts
+  and built from SCB, GeoNames, Trafikverket NVDB and the US Census Bureau, and an
+  `address` record over one consistent draw of them. `sv_SE.address` and
+  `en_US.address` read those records, so `en_US.address.street` no longer carries
+  `name` and `suffix`, and a locale folder loads only beside `geo`.
@@ -18,6 +18,7 @@ fejkdata --seed 42 sv_SE.address               # the same address every run
 fejkdata -n 3 --separator ', ' sv_SE.word      # nät, barn, sol
 fejkdata --list                                # every path the data offers
 fejkdata 'misc.country[SE].capital'            # Stockholm — a table's row, selected by key or name
+fejkdata 'geo.SE.locality[Lund].street'         # Fjelievägen — a linked table, drawn inside the row
 fejkdata --data-path ./mydata sv_SE.word       # layer a directory over the shipped data
 fejkdata --no-shipped-data -d ./mydata --list  # only your data
 fejkdata 'name: {/sv_SE.person.last}'          # name: <a surname> — an inline template
@@ -227,6 +228,24 @@ Each locale carries `address`, `color`, `company`, `date`, `email`, `ip`,
 `misc.country[SE].capital` and `misc.currency[Euro].symbol` select a row;
 [`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.

+A `geo` folder holds one tree per country under its alpha-2 code: five
+[linked tables](#linked-tables) named alike, so a template ports across countries,
+and an `address` record over one consistent draw of them, which the locale's
+`address` reads.
+
+| Table | `geo.SE` | `geo.US` | Weight |
+|-------|----------|----------|--------|
+| `region` | län, by code or name | state, by USPS abbreviation or name; `code` is the FIPS code | population |
+| `municipality` | kommun, by code or name | county, by FIPS code or name | population |
+| `locality` | postort, by name | incorporated place of 25,000 people or more, by GEOID or name; Hawaii has none | population |
+| `postal-code` | postnummer with street delivery, by code | ZCTA, by code | one; address ranges |
+| `street` | gatunamn, the ten with most road segments per postort | street name, the ten with most address ranges per place | segments; address ranges |
+
+`geo.SE.region[Skåne län].municipality` draws a kommun in Skåne,
+`geo.SE.locality[Lund].street` a street in Lund, and
+`geo.US.region[IL].locality[Springfield]` settles which Springfield. A region row
+carries its `timezone`, a locality its `lat` and `lon`.
+
 ## Data format

 Every value is a **node**, nestable without limit:
@@ -983,6 +1002,26 @@ renamed or retyped line is a major.
 - **A path is walked once without drawing before it is walked for real.** A path
  that fails below its first level then moves no seeded stream, at the cost of one
  draw-free walk per call, which allocates nothing.
+- **A country's postal codes and streets are siblings under its locality.** No open
+  source pairs a Swedish street with its postnummer, and pairing the US through its
+  ZIPs would shape the two trees differently, so both draw inside the pinned
+  locality and an address agrees at that level. A street's own code is the exact
+  pairing to add when a source carries it.
+- **A locale's `address` reads its country's `geo` tree, so the shipped set loads
+  whole.** `data/sv_SE` alone no longer loads: a test loads `data` and prefixes
+  the locale, and `--no-shipped-data -d` takes the whole `data` folder or a set of
+  one's own.
+- **The default embed holds every Swedish postort and the US places of 25,000 or
+  more.** Sweden fits whole in 700 KB; every US place of 10,000 would pass a
+  megabyte and fetch 1,200 counties of TIGER files, so the threshold sits where the
+  two countries match in size, and `--min-population` and
+  `--streets-per-locality` on the import scripts build a fuller set. The two trees
+  add about 20 ms to `New`, which loads the shipped set in about 45 ms.
+- **A postort's kommun comes from its name, its tätort or its codes, never from
+  distance.** GeoNames leaves a fifth of Sweden's codes without a kommun and
+  carries stale spellings; the nearest code across a border named the wrong kommun
+  half the time it was tried, so a postort none of the three rules place is
+  dropped, as is one not cased like a place name.
 - **`List` advertises direct descents only.** `region.municipality.locality` is
  listed, and `region.locality` resolves too but is not: the set of every descent
  through a chain of five tables is every subsequence of it, and the direct chain is
@@ -1037,6 +1076,8 @@ A shipped table built from a source is rebuilt by its script under
 ```sh
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/country.py
 docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py
+docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py
+docker compose run --rm --user "$(id -u):$(id -g)" -e TRAFIKVERKET_API_KEY data-import data-import/geo-se.py
 ```

 To release, head `CHANGELOG.md` with the version's section in place of `Unreleased`
@@ -53,19 +53,22 @@ countries; the README maps each to the native term.

 | Table | SE | US | Weight |
 |---|---|---|---|
-| `region` | län (21) | state (56) | population |
-| `municipality` | kommun (290) | county (3,234) | population |
-| `locality` | postort (~1,780), tätort population | place (~19,500) | population |
-| `postal-code` | postnummer (~10,500 deliverable) | ZCTA (33,791) | address count or 1 |
-| `street` | gatunamn, top N per locality | street name, top N per place | address count |
+| `region` | län (21) | state and DC (50; Hawaii has no incorporated place) | population |
+| `municipality` | kommun (290) | county with a shipped place (666) | population |
+| `locality` | postort (1,522), tätort population | place of 25,000+ (1,601) | population |
+| `postal-code` | postnummer with street delivery (13,712) | ZCTA of a shipped place (7,401) | one; address ranges |
+| `street` | gatunamn, top 10 per postort (14,764) | street name, top 10 per place (16,010) | segments; address ranges |

- `geo.SE.address` is a record over one consistent draw: street, number, postal
-  code, locality; `geo.SE.locality[Lund].address` stays inside Lund. Each region
-  row carries its timezone, each locality its centroid.
+- Shipped in step 2, README Data. `geo.SE.address` is a record over one consistent
+  draw. Each region row carries its timezone, each locality its centroid.
+- Let `geo.SE.locality[Lund].address` descend from a selected row into the
+  template beside the family: the path step must reach a sibling category and the
+  outer selector's pins seed every draw group of the render.
+- Ship the fuller sets, every US place of 10,000 and more streets per locality, as
+  packs; `--min-population` and `--streets-per-locality` on the scripts build them.
 - v0.1.0 ships SE and US; then NO, DK, FI, NL, FR, AU, CA, ES, GB, DE.
- SE streets come from NVDB per kommun and postnummer from GeoNames per postort,
-  box codes dropped by the third-digit rule; the pairing is approximate. Revisit
-  an application to Lantmäteriet for the exact pairing after v0.1.0.
+- Revisit an application to Lantmäteriet for the exact street to postnummer
+  pairing after v0.1.0; today a street goes to the nearest postal code centroid.

 #### Builtins the data cannot express

@@ -157,8 +160,8 @@ address, phone, national id, company and date names each.
 #### Order of work

 1. Table node, key and name selection, parent links, consistent draws, the
-   choice-of-rows fence, `DATA-LICENSES.md`, `data-import/`.
-2. `geo/SE` and `geo/US`, and `address` in both locales on top of them.
+   choice-of-rows fence, `DATA-LICENSES.md`, `data-import/` — done.
+2. `geo/SE` and `geo/US`, and `address` in both locales on top of them — done.
 3. Weighted person names and valid ids in both locales; `date()`.
 4. `misc` conversions and the new `misc` tables.
 5. The remaining locale categories: company, phone, finance, vehicle, words.