todo.md

@@ -9,22 +9,27 @@ Plan for the major data update. Research notes: `~/code/claude-handoffs/fejkdata
 #### Structure (proposal, pending review)
 
 `New` costs ~23 ms and 58k allocations for today's 200 KB, one allocation per
-node, so real-world tables cannot ship as nested JSON objects.
+node, so real-world tables cannot ship as nested JSON objects. JSON says how a
+value is composed; TSV says which values exist.
 
 - Add a table node: a category JSON naming a `rows` TSV beside it, whose header
   names the columns and whose rows are the draw. Options `key` (the column a path
-  selects a row by), `weight` (the column that skews the draw), `parent` (the
+  selects a row by), `name` (the column a path also selects by), `weight` (the
-  table a column links to). `{name}` tokens read columns; a cell is a string node.
+  column that skews the draw), `parent` (the table a column links to). `{name}`
-  Index rows on first draw; at load prove only the header, the options and any
+  tokens read columns; a cell is a string node, so it may carry tokens. Index
-  cell token.
+  rows on first draw; at load prove only the header, the options and any cell
-- Reject a choice of same-shaped templates naming the table spelling, and convert
+  token. A TSV nothing names is a load error.
-  `misc.country`, `currency`, `language`, `httpstatus`, `mimetype`, `timezone`, `car`.
+- In a data file, refuse a choice whose items are templates with one identical
+  field set, naming the TSV to write; an inline template is exempt, having no
+  file. A choice of strings stays a choice.
+- Convert `misc.country`, `currency`, `language`, `httpstatus`, `mimetype`,
+  `timezone`, `car` and `useragent` to tables.
 - Let a path select a row by key or by name, `geo.SE.municipality[0180]` and
   `geo.SE.municipality[Stockholm]`, and descend to a linked table by name,
   `geo.SE.region[Skåne län].street`; a key or a linked table name may not equal
   a column name (load error). A name that matches several rows outside a pinned
   parent is an error listing the keys; inside one, `region[IL].locality[Springfield]`
-  resolves.
+  resolves. Brackets need quoting in zsh; `:` would not.
 - Draw linked tables consistently within one render and draw group: the first
   table drawn pins its ancestors, and a descendant is drawn inside them.
 - Keep country data under `geo/<alpha2>/`; a locale's `address` reads its
@@ -33,13 +38,15 @@ node, so real-world tables cannot ship as nested JSON objects.
 - Ship full registers as packs, each a Go module with its own `embed.FS` and a
   zip for `--data-path`; the default embed stays under ~1 MB per country and
   `New` under ~50 ms.
-- Add `DATA-LICENSES.md` listing every dataset, its licence and its attribution.
+- Add `DATA-LICENSES.md` listing every dataset, its licence and its attribution,
+  and a `data-import/` directory of Python scripts that rebuild each TSV from its
+  source, so a refresh is one command per dataset.
 - Make every shipped category a record with its building blocks as columns
   (`sv_SE.person` → `femalefirst`, `malefirst`; `misc.uuid` → `variant`).
 - Share the handle lists between `email.local` and `username` only if that is a
   clean win — a reference between shipped categories is a major once tagged.
 
-#### Geo tree (proposal, pending review)
+#### Geo tree
 
 Five linked tables per country, generic names so a template ports across
 countries; the README maps each to the native term.
@@ -48,91 +55,114 @@ countries; the README maps each to the native term.
 |---|---|---|---|
 | `region` | län (21) | state (56) | population |
 | `municipality` | kommun (290) | county (3,234) | population |
-| `locality` | postort (~1,780) | place (~19,500) | population |
+| `locality` | postort (~1,780), tätort population | place (~19,500) | population |
 | `postal-code` | postnummer (~10,500 deliverable) | ZCTA (33,791) | address count or 1 |
 | `street` | gatunamn, top N per locality | street name, top N per place | address count |
 
 - `geo.SE.address` is a record over one consistent draw: street, number, postal
-  code, locality; `geo.SE.locality[stockholm].address` stays inside Stockholm.
+  code, locality; `geo.SE.locality[Lund].address` stays inside Lund. Each region
+  row carries its timezone, each locality its centroid.
 - v0.1.0 ships SE and US; then NO, DK, FI, NL, FR, AU, CA, ES, GB, DE.
 - SE streets come from NVDB per kommun and postnummer from GeoNames per postort,
   box codes dropped by the third-digit rule; the pairing is approximate. Revisit
   an application to Lantmäteriet for the exact pairing after v0.1.0.
 
-#### Sources (`research-geo-se.md`, `research-geo-world.md`)
+#### Builtins the data cannot express
 
-| Data | Source | Licence |
+- `{date(from,to,layout)}`: a date in `[from, to]` in a Go layout, and `{age(min,max)}`
-|---|---|---|
+  as the birthdate spelling; `{time(layout)}`. Personnummer, SSN, birthdate, card
-| SE län, kommun, population, tätorter | SCB | CC0 |
+  expiry, unix time and ISO datetime all build on it.
-| SE street names per kommun | Trafikverket NVDB Gatunamn | CC0 |
+- Derivations: `{isin()}` (Luhn over letters expanded to digits), `{cusip()}`,
-| SE postnummer → postort | GeoNames SE.zip (stale, retired codes included) | CC BY 4.0 |
+  `{aba()}` (3-7-1 weights), `{vin()}` (position 9 over the whole; a sample taking
-| SE full addresses | Lantmäteriet Belägenhetsadress | CC BY 4.0 + personal-data terms, application |
+  the WMI, since the check sits mid-string).
-| SE Göteborg, Malmö full addresses | OpenAddresses | CC0 |
+- Samples: `{base64url(n)}` for JWT shapes; `{btc()}`, `{eth()}` after v0.1.0
-| US states, counties, places, population, ZCTA | Census (Gazetteer, SUB-EST2025, ZCTA relationships) | public domain |
+  (Base58Check and EIP-55 need sha256 and keccak).
-| US addresses, streets | DOT National Address Database | public domain |
-| NO addresses | Kartverket Matrikkelen; Bring postnummer | CC BY 4.0; NLOD 2.0 |
-| DK addresses | DAWA / DAR | CC BY 4.0 |
-| FI addresses, postal codes | DVV/SYKE; Posti PCF/BAF | CC BY 4.0; Posti terms |
-| NL addresses | BAG | CC0 |
-| FR addresses | BAN, La Poste hexasmal | Licence Ouverte 2.0 |
-| AU addresses | G-NAF | CC BY 4.0 EULA |
-| CA addresses | StatCan National Address Register | StatCan Open Licence |
-| ES addresses | Catastro INSPIRE AD (no Basque, Navarre) | Catastro licence, verify |
-| GB streets, postcodes | OS Open Names, Code-Point Open (no house numbers) | OGL v3 |
-| DE Gemeinden, addresses | Destatis GV100, 15 Länder Hauskoordinaten (no Bavaria) | dl-de/by-2-0 |
-| Cities with population, any country | GeoNames cities500 | CC BY 4.0 |
-| Names by frequency | SCB, SSA, Census | CC0, public domain |
-| Never | OpenStreetMap, OpenPLZ, countries-states-cities-database | ODbL share-alike |
 
-An LLM is a source only for non-factual lists (product adjectives, catchphrases,
+#### Categories
-hacker phrases), never for names, places, codes or ids.
-
-#### Categories to add
 
 Union of @faker-js/faker, Python Faker, gofakeit, Datafaker, Bogus, Mimesis, Chance
 and Ruby faker (`research-libraries.md`). What sets fejkdata apart, and what every
 issue tracker asks for: real values, frequency weights, and facts that agree across
-a record. No pop-culture catalogues.
+a record. No pop-culture catalogues. An LLM writes only non-factual lists
+(catchphrases, hacker phrases, product adjectives), never names, places, codes or
+ids. Shape: T = table, t = template, c = choice.
 
-- Person: real first and last names weighted by official frequency tables (SCB, SSA,
+`sv_SE` (`research-sources-se.md`)
-  Census), middle name, generic (non-gendered) first names, gender, title, suffix,
+
-  birthdate and age, job title/area/level, blood type weighted by prevalence, marital
+| Category | Shape | Source | Licence |
-  status, nationality and demonym, valid national ids (personnummer, samordningsnummer,
+|---|---|---|---|
-  SSN, EIN, and one per new locale), passport number, bio.
+| `person` first (female, male, generic), middle, last, weighted | T | SCB 2022 whole-population xlsx, Skatteverket 2026 surnames | CC0, "Källa: SCB" |
-- Address: the geo tree, secondary address (`lgh 1201`, `Apt 4B`), PO box, building
+| `person` title, gender, birthdate, age, blood type weighted | t | geblod.nu distribution | facts |
-  number, latitude and longitude inside the locality, timezone by country and region,
+| `personnummer`, `samordningsnummer` | t | Skatteverket test series: date + 238/239, Luhn | CC0 |
-  cardinal direction, continent, full ISO 3166 country row (name per locale, alpha2,
+| `organisationsnummer` by form, `vat` | t | Bolagsverket group digits, Luhn, `SE…01` | facts |
-  alpha3, numeric, calling code, TLD, capital, currency, flag emoji, languages), ISO
+| `company` name patterns, legal form weighted | t | Bolagsverket registrations 2025 | CC BY 2.5 SE |
-  3166-2 subdivisions.
+| `sni` industry codes | T | SCB SNI 2025 | "Källa: SCB" |
-- Company: name patterns per locale, legal suffix, valid organisationsnummer and VAT
+| `ssyk` occupations | T | SCB SSYK 2012 | "Källa: SCB" |
-  number (SE), EIN (US), DUNS, industry (SNI/NAICS), department, profession,
+| `phone` mobile, fixed per area code, E.164 | T+t | PTS numbering plan 2024 | facts |
-  catchphrase, slogan, buzzwords, domain from the name.
+| `bankgiro`, `plusgiro`, `bankaccount` clearing + Luhn/mod-11 | T+t | Bankinfrastruktur CSV, Bankgirot rules | facts |
-- Internet: ipv4 public/private/CIDR, ipv6, hostname, TLD, port, protocol, http method
+| `licenseplate` `ABC 123`, `ABC 12A`, blocked combinations | t | Transportstyrelsen | facts |
-  and version, log level, slug, password, JWT shape, hash (md5/sha1/sha256), avatar
+| `car` make, model weighted by registrations | T | Mobility Sweden 2025 | cite |
-  and placeholder image URLs, user agent per browser, safe email domains, full IANA
+| `word` noun, verb, adjective, adverb; `sentence` | T+t | SALDO | CC BY 4.0 |
-  mime list with extensions correlated.
+| `date`, `time`, month and weekday names, `holiday` | t+T | CLDR sv, lag 1989:253 | Unicode |
-- Finance: BIC, BBAN, ABA routing with check digit, credit card per network with
+| `price` `1 234,56 kr`, `email` domains, `url` `.se`/`.nu` | t | curated | — |
-  valid prefix, length, CVV and expiry, bankgiro and plusgiro (SE), clearing number,
+| `country` localised names keyed by alpha2 | T | CLDR sv territory names | Unicode |
-  ISIN and CUSIP with check digit, crypto addresses, stock ticker, transaction type
+| `address` | t | `{/geo.SE…}` | — |
-  and description, amount by currency.
+
-- Commerce: product name, adjective, material, description, department, brand, SKU,
+`en_US` (`research-sources-world.md`, part1–4)
-  promo code, EAN/UPC/ISBN/GTIN/ASIN, IMEI, garment size, price per locale.
+
-- Date and time: `{date(from,to,layout)}` and `{time()}` builtins, birthdate by age
+| Category | Shape | Source | Licence |
-  range, weekday and month names per locale, ISO datetime, unix time, duration, cron.
+|---|---|---|---|
-- Phone: national, international and E.164 per locale, area code, extension.
+| `person` first by sex weighted, last weighted, middle, generic | T | SSA baby names, Census 2010 surnames | public domain |
-- Text: per-locale word classes (noun, verb, adjective, adverb, preposition), sentence,
+| `ssn`, `itin`, `ein` valid ranges | t | SSA POMS, IRS | facts |
-  paragraph, lorem ipsum, question, quote, hacker and hipster phrases.
+| `company` names, suffix, `naics` | t+T | SEC tickers for patterns, NAICS 2022 | public domain |
-- Vehicle: real make and model pairs, type, fuel, transmission, color, year, VIN with
+| `phone` NANP with valid NPA/NXX | t | NANPA rules | facts |
-  check digit, license plate per country, bicycle. Airline: airline, airport (IATA,
+| `routing` ABA with check, `bankaccount` | t | Fed prefix ranges | facts |
-  ICAO, tied to the geo tree), aircraft, flight number, seat, record locator.
+| `licenseplate` per state | T | hand-authored patterns | facts |
-- Science and nature: element (symbol, number), unit, scientist, planet, animal by
+| `car` make, model | T | NHTSA vPIC | public domain |
-  class, plant.
+| `word`, `sentence`, `paragraph` | T+t | Moby POS or WordNet | public domain / WordNet |
-- Food and drink, book (title, author, genre, publisher), music (genre, instrument),
+| `date`, `time`, names, `holiday` | t+T | CLDR en | Unicode |
-  movie genre, sport, programming language, OS, database (column, type, engine),
+| `price` `$1,234.56`, `email`, `url`, `address` | t | curated, `{/geo.US…}` | — |
-  version control (branch, commit message, sha), semver, file (name, extension, path).
+
-- Color: named per locale, hex, rgb, hsl, cmyk, css function.
+`misc` (locale-neutral)
-- Locales: nb_NO, da_DK, fi_FI, de_DE, en_GB, nl_NL, fr_FR, es_ES with person, address,
+
-  phone, national id, company and date names each.
+| Category | Shape | Source | Licence |
+|---|---|---|---|
+| `country` name, alpha2, alpha3, numeric, calling code, TLD, capital, currency, flag, languages | T | datasets/country-codes | PDDL |
+| `currency` code, numeric, name, symbol, decimals | T | ISO 4217 list-one, CLDR symbols | free, Unicode |
+| `language` ISO 639-1/2 | T | LoC | public domain |
+| `timezone` zone, country, offset | T | tzdb zone1970.tab | public domain |
+| `mimetype` type, extensions | T | IANA + mime-db | CC0, MIT |
+| `httpstatus`, `httpmethod`, `port`, `tld`, `protocol`, `loglevel` | T/c | IANA | CC0 |
+| `useragent` per browser | T | top-user-agents | MIT |
+| `ip` v4 documentation ranges, private, CIDR; `ipv6` `2001:db8::/32`; `mac` locally administered | t | RFC 5737, 1918, 9637 | facts |
+| `creditcard` per network: IIN, length, CVV, expiry | T+t | network rules, `{luhn()}` | facts |
+| `bic`, `iban` (exists), `isin`, `cusip` | t | structure rules | facts |
+| `ean`, `upc`, `gtin`, `isbn`, `issn`, `imei`, `asin`, `sku` | t | check-digit rules, GS1 demo prefix 952 | facts |
+| `product` name, adjective, material, department, `garmentsize` | T/c | LLM-written lists | — |
+| `airport` IATA, ICAO, country, municipality; `airline`; `aircraft`; `flight`, `seat`, `pnr` | T+t | OurAirports, Wikidata | public domain, CC0 |
+| `element`, `unit`, `planet` | T | PubChem, BIPM, NASA | public domain, CC BY 4.0 |
+| `animal` by class, `plant` | T | Wikidata | CC0 |
+| `bloodtype` weighted (global) | c | AABB | facts |
+| `color` named, hex, rgb, hsl, cmyk | T+t | CSS Color 4, XKCD | W3C, CC0 |
+| `programminglanguage`, `os`, `database` column/type/engine, `fileext`, `semver` (exists) | T/c | Linguist, curated | MIT |
+| `commit` message, branch, `sha` | t | LLM-written, `{hex(40)}` | — |
+| `password`, `hash`, `jwt`, `slug`, `hostname`, `avatar`, `placeholderimage` | t | shapes | — |
+| `book` title, author from Gutenberg; `genre`; `instrument`; `dish`, `ingredient` | T | Gutenberg catalog, MusicBrainz, USDA | CC0 |
+| `lorem`, `hacker`, `hipster`, `catchphrase`, `buzzword`, `quote` | T/c | lorem ipsum, LLM-written | — |
+| `coordinate` (exists), `direction`, `continent`, `emoji` (exists), `uuid`, `ulid`, `objectid` | c/t | — | — |
+
+Later locales: nb_NO, da_DK, fi_FI, de_DE, en_GB, nl_NL, fr_FR, es_ES with person,
+address, phone, national id, company and date names each.
+
+#### Order of work
+
+1. Table node, key and name selection, parent links, consistent draws, the
+   choice-of-rows fence, `DATA-LICENSES.md`, `data-import/`.
+2. `geo/SE` and `geo/US`, and `address` in both locales on top of them.
+3. Weighted person names and valid ids in both locales; `date()`.
+4. `misc` conversions and the new `misc` tables.
+5. The remaining locale categories: company, phone, finance, vehicle, words.
+6. Records with building-block columns across the shipped set; shape re-pin.
 
 ### Release