diff --git a/CHANGELOG.md b/CHANGELOG.md index 075ad90..b83d133 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,3 +26,9 @@ replacement, and each removed path, column or flag. `currency`, `flag`, `languages`, `numeric` and `tld` added; `misc.currency` the current ISO 4217 currencies with a minor unit, with `decimals` and `numeric` added, and its symbols from CLDR. `DATA-LICENSES.md` lists each source. +- `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`, + `locality`, `postal-code` and `street`, weighted by population and address counts + and built from SCB, GeoNames, Trafikverket NVDB and the US Census Bureau, and an + `address` record over one consistent draw of them. `sv_SE.address` and + `en_US.address` read those records, so `en_US.address.street` no longer carries + `name` and `suffix`, and a locale folder loads only beside `geo`. diff --git a/README.md b/README.md index 0fab5ea..d5db683 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,7 @@ fejkdata --seed 42 sv_SE.address # the same address every run fejkdata -n 3 --separator ', ' sv_SE.word # nät, barn, sol fejkdata --list # every path the data offers fejkdata 'misc.country[SE].capital' # Stockholm — a table's row, selected by key or name +fejkdata 'geo.SE.locality[Lund].street' # Fjelievägen — a linked table, drawn inside the row fejkdata --data-path ./mydata sv_SE.word # layer a directory over the shipped data fejkdata --no-shipped-data -d ./mydata --list # only your data fejkdata 'name: {/sv_SE.person.last}' # name: — an inline template @@ -227,6 +228,24 @@ Each locale carries `address`, `color`, `company`, `date`, `email`, `ip`, `misc.country[SE].capital` and `misc.currency[Euro].symbol` select a row; [`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence. +A `geo` folder holds one tree per country under its alpha-2 code: five +[linked tables](#linked-tables) named alike, so a template ports across countries, +and an `address` record over one consistent draw of them, which the locale's +`address` reads. + +| Table | `geo.SE` | `geo.US` | Weight | +|-------|----------|----------|--------| +| `region` | län, by code or name | state, by USPS abbreviation or name; `code` is the FIPS code | population | +| `municipality` | kommun, by code or name | county, by FIPS code or name | population | +| `locality` | postort, by name | incorporated place of 25,000 people or more, by GEOID or name; Hawaii has none | population | +| `postal-code` | postnummer with street delivery, by code | ZCTA, by code | one; address ranges | +| `street` | gatunamn, the ten with most road segments per postort | street name, the ten with most address ranges per place | segments; address ranges | + +`geo.SE.region[Skåne län].municipality` draws a kommun in Skåne, +`geo.SE.locality[Lund].street` a street in Lund, and +`geo.US.region[IL].locality[Springfield]` settles which Springfield. A region row +carries its `timezone`, a locality its `lat` and `lon`. + ## Data format Every value is a **node**, nestable without limit: @@ -983,6 +1002,26 @@ renamed or retyped line is a major. - **A path is walked once without drawing before it is walked for real.** A path that fails below its first level then moves no seeded stream, at the cost of one draw-free walk per call, which allocates nothing. +- **A country's postal codes and streets are siblings under its locality.** No open + source pairs a Swedish street with its postnummer, and pairing the US through its + ZIPs would shape the two trees differently, so both draw inside the pinned + locality and an address agrees at that level. A street's own code is the exact + pairing to add when a source carries it. +- **A locale's `address` reads its country's `geo` tree, so the shipped set loads + whole.** `data/sv_SE` alone no longer loads: a test loads `data` and prefixes + the locale, and `--no-shipped-data -d` takes the whole `data` folder or a set of + one's own. +- **The default embed holds every Swedish postort and the US places of 25,000 or + more.** Sweden fits whole in 700 KB; every US place of 10,000 would pass a + megabyte and fetch 1,200 counties of TIGER files, so the threshold sits where the + two countries match in size, and `--min-population` and + `--streets-per-locality` on the import scripts build a fuller set. The two trees + add about 20 ms to `New`, which loads the shipped set in about 45 ms. +- **A postort's kommun comes from its name, its tätort or its codes, never from + distance.** GeoNames leaves a fifth of Sweden's codes without a kommun and + carries stale spellings; the nearest code across a border named the wrong kommun + half the time it was tried, so a postort none of the three rules place is + dropped, as is one not cased like a place name. - **`List` advertises direct descents only.** `region.municipality.locality` is listed, and `region.locality` resolves too but is not: the set of every descent through a chain of five tables is every subsequence of it, and the direct chain is @@ -1037,6 +1076,8 @@ A shipped table built from a source is rebuilt by its script under ```sh docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/country.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py +docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py +docker compose run --rm --user "$(id -u):$(id -g)" -e TRAFIKVERKET_API_KEY data-import data-import/geo-se.py ``` To release, head `CHANGELOG.md` with the version's section in place of `Unreleased` diff --git a/todo.md b/todo.md index 85d8390..598b907 100644 --- a/todo.md +++ b/todo.md @@ -53,19 +53,22 @@ countries; the README maps each to the native term. | Table | SE | US | Weight | |---|---|---|---| -| `region` | län (21) | state (56) | population | -| `municipality` | kommun (290) | county (3,234) | population | -| `locality` | postort (~1,780), tätort population | place (~19,500) | population | -| `postal-code` | postnummer (~10,500 deliverable) | ZCTA (33,791) | address count or 1 | -| `street` | gatunamn, top N per locality | street name, top N per place | address count | +| `region` | län (21) | state and DC (50; Hawaii has no incorporated place) | population | +| `municipality` | kommun (290) | county with a shipped place (666) | population | +| `locality` | postort (1,522), tätort population | place of 25,000+ (1,601) | population | +| `postal-code` | postnummer with street delivery (13,712) | ZCTA of a shipped place (7,401) | one; address ranges | +| `street` | gatunamn, top 10 per postort (14,764) | street name, top 10 per place (16,010) | segments; address ranges | -- `geo.SE.address` is a record over one consistent draw: street, number, postal - code, locality; `geo.SE.locality[Lund].address` stays inside Lund. Each region - row carries its timezone, each locality its centroid. +- Shipped in step 2, README Data. `geo.SE.address` is a record over one consistent + draw. Each region row carries its timezone, each locality its centroid. +- Let `geo.SE.locality[Lund].address` descend from a selected row into the + template beside the family: the path step must reach a sibling category and the + outer selector's pins seed every draw group of the render. +- Ship the fuller sets, every US place of 10,000 and more streets per locality, as + packs; `--min-population` and `--streets-per-locality` on the scripts build them. - v0.1.0 ships SE and US; then NO, DK, FI, NL, FR, AU, CA, ES, GB, DE. -- SE streets come from NVDB per kommun and postnummer from GeoNames per postort, - box codes dropped by the third-digit rule; the pairing is approximate. Revisit - an application to Lantmäteriet for the exact pairing after v0.1.0. +- Revisit an application to Lantmäteriet for the exact street to postnummer + pairing after v0.1.0; today a street goes to the nearest postal code centroid. #### Builtins the data cannot express @@ -157,8 +160,8 @@ address, phone, national id, company and date names each. #### Order of work 1. Table node, key and name selection, parent links, consistent draws, the - choice-of-rows fence, `DATA-LICENSES.md`, `data-import/`. -2. `geo/SE` and `geo/US`, and `address` in both locales on top of them. + choice-of-rows fence, `DATA-LICENSES.md`, `data-import/` — done. +2. `geo/SE` and `geo/US`, and `address` in both locales on top of them — done. 3. Weighted person names and valid ids in both locales; `date()`. 4. `misc` conversions and the new `misc` tables. 5. The remaining locale categories: company, phone, finance, vehicle, words.