larvit/fejkdata
3
0
Fork 0
Code Pull Requests Actions Activity

Plan the major data update #18

Merged
lilleman merged 3 commits from worktree-data-update-plan into main 2026-09-17 11:20:05 +02:00
Conversation 0 Commits 3 Files Changed 1
+109 -79
1 changed files with 109 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 06a93bb96d - Show all commits
todo.md
+109 -79
View File
@@ -9,22 +9,27 @@ Plan for the major data update. Research notes: `~/code/claude-handoffs/fejkdata
#### Structure (proposal, pending review)
`New` costs ~23 ms and 58k allocations for today's 200 KB, one allocation per
node, so real-world tables cannot ship as nested JSON objects.
node, so real-world tables cannot ship as nested JSON objects. JSON says how a
value is composed; TSV says which values exist.
- Add a table node: a category JSON naming a `rows` TSV beside it, whose header
names the columns and whose rows are the draw. Options `key` (the column a path
selects a row by), `weight` (the column that skews the draw), `parent` (the
table a column links to). `{name}` tokens read columns; a cell is a string node.
Index rows on first draw; at load prove only the header, the options and any
cell token.
- Reject a choice of same-shaped templates naming the table spelling, and convert
`misc.country`, `currency`, `language`, `httpstatus`, `mimetype`, `timezone`, `car`.
selects a row by), `name` (the column a path also selects by), `weight` (the
column that skews the draw), `parent` (the table a column links to). `{name}`
tokens read columns; a cell is a string node, so it may carry tokens. Index
rows on first draw; at load prove only the header, the options and any cell
token. A TSV nothing names is a load error.
- In a data file, refuse a choice whose items are templates with one identical
field set, naming the TSV to write; an inline template is exempt, having no
file. A choice of strings stays a choice.
- Convert `misc.country`, `currency`, `language`, `httpstatus`, `mimetype`,
`timezone`, `car` and `useragent` to tables.
- Let a path select a row by key or by name, `geo.SE.municipality[0180]` and
`geo.SE.municipality[Stockholm]`, and descend to a linked table by name,
`geo.SE.region[Skåne län].street`; a key or a linked table name may not equal
a column name (load error). A name that matches several rows outside a pinned
parent is an error listing the keys; inside one, `region[IL].locality[Springfield]`
resolves.
resolves. Brackets need quoting in zsh; `:` would not.
- Draw linked tables consistently within one render and draw group: the first
table drawn pins its ancestors, and a descendant is drawn inside them.
- Keep country data under `geo/<alpha2>/`; a locale's `address` reads its
@@ -33,13 +38,15 @@ node, so real-world tables cannot ship as nested JSON objects.
- Ship full registers as packs, each a Go module with its own `embed.FS` and a
zip for `--data-path`; the default embed stays under ~1 MB per country and
`New` under ~50 ms.
- Add `DATA-LICENSES.md` listing every dataset, its licence and its attribution.
- Add `DATA-LICENSES.md` listing every dataset, its licence and its attribution,
and a `data-import/` directory of Python scripts that rebuild each TSV from its
source, so a refresh is one command per dataset.
- Make every shipped category a record with its building blocks as columns
(`sv_SE.person` → `femalefirst`, `malefirst`; `misc.uuid` → `variant`).
- Share the handle lists between `email.local` and `username` only if that is a
clean win — a reference between shipped categories is a major once tagged.
#### Geo tree (proposal, pending review)
#### Geo tree
Five linked tables per country, generic names so a template ports across
countries; the README maps each to the native term.
@@ -48,91 +55,114 @@ countries; the README maps each to the native term.
|---|---|---|---|
| `region` | län (21) | state (56) | population |
| `municipality` | kommun (290) | county (3,234) | population |
| `locality` | postort (~1,780) | place (~19,500) | population |
| `locality` | postort (~1,780), tätort population | place (~19,500) | population |
| `postal-code` | postnummer (~10,500 deliverable) | ZCTA (33,791) | address count or 1 |
| `street` | gatunamn, top N per locality | street name, top N per place | address count |
- `geo.SE.address` is a record over one consistent draw: street, number, postal
code, locality; `geo.SE.locality[stockholm].address` stays inside Stockholm.
code, locality; `geo.SE.locality[Lund].address` stays inside Lund. Each region
row carries its timezone, each locality its centroid.
- v0.1.0 ships SE and US; then NO, DK, FI, NL, FR, AU, CA, ES, GB, DE.
- SE streets come from NVDB per kommun and postnummer from GeoNames per postort,
box codes dropped by the third-digit rule; the pairing is approximate. Revisit
an application to Lantmäteriet for the exact pairing after v0.1.0.
#### Sources (`research-geo-se.md`, `research-geo-world.md`)
#### Builtins the data cannot express
| Data | Source | Licence |
|---|---|---|
| SE län, kommun, population, tätorter | SCB | CC0 |
| SE street names per kommun | Trafikverket NVDB Gatunamn | CC0 |
| SE postnummer → postort | GeoNames SE.zip (stale, retired codes included) | CC BY 4.0 |
| SE full addresses | Lantmäteriet Belägenhetsadress | CC BY 4.0 + personal-data terms, application |
| SE Göteborg, Malmö full addresses | OpenAddresses | CC0 |
| US states, counties, places, population, ZCTA | Census (Gazetteer, SUB-EST2025, ZCTA relationships) | public domain |
| US addresses, streets | DOT National Address Database | public domain |
| NO addresses | Kartverket Matrikkelen; Bring postnummer | CC BY 4.0; NLOD 2.0 |
| DK addresses | DAWA / DAR | CC BY 4.0 |
| FI addresses, postal codes | DVV/SYKE; Posti PCF/BAF | CC BY 4.0; Posti terms |
| NL addresses | BAG | CC0 |
| FR addresses | BAN, La Poste hexasmal | Licence Ouverte 2.0 |
| AU addresses | G-NAF | CC BY 4.0 EULA |
| CA addresses | StatCan National Address Register | StatCan Open Licence |
| ES addresses | Catastro INSPIRE AD (no Basque, Navarre) | Catastro licence, verify |
| GB streets, postcodes | OS Open Names, Code-Point Open (no house numbers) | OGL v3 |
| DE Gemeinden, addresses | Destatis GV100, 15 Länder Hauskoordinaten (no Bavaria) | dl-de/by-2-0 |
| Cities with population, any country | GeoNames cities500 | CC BY 4.0 |
| Names by frequency | SCB, SSA, Census | CC0, public domain |
| Never | OpenStreetMap, OpenPLZ, countries-states-cities-database | ODbL share-alike |
- `{date(from,to,layout)}`: a date in `[from, to]` in a Go layout, and `{age(min,max)}`
as the birthdate spelling; `{time(layout)}`. Personnummer, SSN, birthdate, card
expiry, unix time and ISO datetime all build on it.
- Derivations: `{isin()}` (Luhn over letters expanded to digits), `{cusip()}`,
`{aba()}` (3-7-1 weights), `{vin()}` (position 9 over the whole; a sample taking
the WMI, since the check sits mid-string).
- Samples: `{base64url(n)}` for JWT shapes; `{btc()}`, `{eth()}` after v0.1.0
(Base58Check and EIP-55 need sha256 and keccak).
An LLM is a source only for non-factual lists (product adjectives, catchphrases,
hacker phrases), never for names, places, codes or ids.
#### Categories to add
#### Categories
Union of @faker-js/faker, Python Faker, gofakeit, Datafaker, Bogus, Mimesis, Chance
and Ruby faker (`research-libraries.md`). What sets fejkdata apart, and what every
issue tracker asks for: real values, frequency weights, and facts that agree across
a record. No pop-culture catalogues.
a record. No pop-culture catalogues. An LLM writes only non-factual lists
(catchphrases, hacker phrases, product adjectives), never names, places, codes or
ids. Shape: T = table, t = template, c = choice.
- Person: real first and last names weighted by official frequency tables (SCB, SSA,
Census), middle name, generic (non-gendered) first names, gender, title, suffix,
birthdate and age, job title/area/level, blood type weighted by prevalence, marital
status, nationality and demonym, valid national ids (personnummer, samordningsnummer,
SSN, EIN, and one per new locale), passport number, bio.
- Address: the geo tree, secondary address (`lgh 1201`, `Apt 4B`), PO box, building
number, latitude and longitude inside the locality, timezone by country and region,
cardinal direction, continent, full ISO 3166 country row (name per locale, alpha2,
alpha3, numeric, calling code, TLD, capital, currency, flag emoji, languages), ISO
3166-2 subdivisions.
- Company: name patterns per locale, legal suffix, valid organisationsnummer and VAT
number (SE), EIN (US), DUNS, industry (SNI/NAICS), department, profession,
catchphrase, slogan, buzzwords, domain from the name.
- Internet: ipv4 public/private/CIDR, ipv6, hostname, TLD, port, protocol, http method
and version, log level, slug, password, JWT shape, hash (md5/sha1/sha256), avatar
and placeholder image URLs, user agent per browser, safe email domains, full IANA
mime list with extensions correlated.
- Finance: BIC, BBAN, ABA routing with check digit, credit card per network with
valid prefix, length, CVV and expiry, bankgiro and plusgiro (SE), clearing number,
ISIN and CUSIP with check digit, crypto addresses, stock ticker, transaction type
and description, amount by currency.
- Commerce: product name, adjective, material, description, department, brand, SKU,
promo code, EAN/UPC/ISBN/GTIN/ASIN, IMEI, garment size, price per locale.
- Date and time: `{date(from,to,layout)}` and `{time()}` builtins, birthdate by age
range, weekday and month names per locale, ISO datetime, unix time, duration, cron.
- Phone: national, international and E.164 per locale, area code, extension.
- Text: per-locale word classes (noun, verb, adjective, adverb, preposition), sentence,
paragraph, lorem ipsum, question, quote, hacker and hipster phrases.
- Vehicle: real make and model pairs, type, fuel, transmission, color, year, VIN with
check digit, license plate per country, bicycle. Airline: airline, airport (IATA,
ICAO, tied to the geo tree), aircraft, flight number, seat, record locator.
- Science and nature: element (symbol, number), unit, scientist, planet, animal by
class, plant.
- Food and drink, book (title, author, genre, publisher), music (genre, instrument),
movie genre, sport, programming language, OS, database (column, type, engine),
version control (branch, commit message, sha), semver, file (name, extension, path).
- Color: named per locale, hex, rgb, hsl, cmyk, css function.
- Locales: nb_NO, da_DK, fi_FI, de_DE, en_GB, nl_NL, fr_FR, es_ES with person, address,
phone, national id, company and date names each.
`sv_SE` (`research-sources-se.md`)
| Category | Shape | Source | Licence |
|---|---|---|---|
| `person` first (female, male, generic), middle, last, weighted | T | SCB 2022 whole-population xlsx, Skatteverket 2026 surnames | CC0, "Källa: SCB" |
| `person` title, gender, birthdate, age, blood type weighted | t | geblod.nu distribution | facts |
| `personnummer`, `samordningsnummer` | t | Skatteverket test series: date + 238/239, Luhn | CC0 |
| `organisationsnummer` by form, `vat` | t | Bolagsverket group digits, Luhn, `SE…01` | facts |
| `company` name patterns, legal form weighted | t | Bolagsverket registrations 2025 | CC BY 2.5 SE |
| `sni` industry codes | T | SCB SNI 2025 | "Källa: SCB" |
| `ssyk` occupations | T | SCB SSYK 2012 | "Källa: SCB" |
| `phone` mobile, fixed per area code, E.164 | T+t | PTS numbering plan 2024 | facts |
| `bankgiro`, `plusgiro`, `bankaccount` clearing + Luhn/mod-11 | T+t | Bankinfrastruktur CSV, Bankgirot rules | facts |
| `licenseplate` `ABC 123`, `ABC 12A`, blocked combinations | t | Transportstyrelsen | facts |
| `car` make, model weighted by registrations | T | Mobility Sweden 2025 | cite |
| `word` noun, verb, adjective, adverb; `sentence` | T+t | SALDO | CC BY 4.0 |
| `date`, `time`, month and weekday names, `holiday` | t+T | CLDR sv, lag 1989:253 | Unicode |
| `price` `1 234,56 kr`, `email` domains, `url` `.se`/`.nu` | t | curated | — |
| `country` localised names keyed by alpha2 | T | CLDR sv territory names | Unicode |
| `address` | t | `{/geo.SE…}` | — |
`en_US` (`research-sources-world.md`, part1–4)
| Category | Shape | Source | Licence |
|---|---|---|---|
| `person` first by sex weighted, last weighted, middle, generic | T | SSA baby names, Census 2010 surnames | public domain |
| `ssn`, `itin`, `ein` valid ranges | t | SSA POMS, IRS | facts |
| `company` names, suffix, `naics` | t+T | SEC tickers for patterns, NAICS 2022 | public domain |
| `phone` NANP with valid NPA/NXX | t | NANPA rules | facts |
| `routing` ABA with check, `bankaccount` | t | Fed prefix ranges | facts |
| `licenseplate` per state | T | hand-authored patterns | facts |
| `car` make, model | T | NHTSA vPIC | public domain |
| `word`, `sentence`, `paragraph` | T+t | Moby POS or WordNet | public domain / WordNet |
| `date`, `time`, names, `holiday` | t+T | CLDR en | Unicode |
| `price` `$1,234.56`, `email`, `url`, `address` | t | curated, `{/geo.US…}` | — |
`misc` (locale-neutral)
| Category | Shape | Source | Licence |
|---|---|---|---|
| `country` name, alpha2, alpha3, numeric, calling code, TLD, capital, currency, flag, languages | T | datasets/country-codes | PDDL |
| `currency` code, numeric, name, symbol, decimals | T | ISO 4217 list-one, CLDR symbols | free, Unicode |
| `language` ISO 639-1/2 | T | LoC | public domain |
| `timezone` zone, country, offset | T | tzdb zone1970.tab | public domain |
| `mimetype` type, extensions | T | IANA + mime-db | CC0, MIT |
| `httpstatus`, `httpmethod`, `port`, `tld`, `protocol`, `loglevel` | T/c | IANA | CC0 |
| `useragent` per browser | T | top-user-agents | MIT |
| `ip` v4 documentation ranges, private, CIDR; `ipv6` `2001:db8::/32`; `mac` locally administered | t | RFC 5737, 1918, 9637 | facts |
| `creditcard` per network: IIN, length, CVV, expiry | T+t | network rules, `{luhn()}` | facts |
| `bic`, `iban` (exists), `isin`, `cusip` | t | structure rules | facts |
| `ean`, `upc`, `gtin`, `isbn`, `issn`, `imei`, `asin`, `sku` | t | check-digit rules, GS1 demo prefix 952 | facts |
| `product` name, adjective, material, department, `garmentsize` | T/c | LLM-written lists | — |
| `airport` IATA, ICAO, country, municipality; `airline`; `aircraft`; `flight`, `seat`, `pnr` | T+t | OurAirports, Wikidata | public domain, CC0 |
| `element`, `unit`, `planet` | T | PubChem, BIPM, NASA | public domain, CC BY 4.0 |
| `animal` by class, `plant` | T | Wikidata | CC0 |
| `bloodtype` weighted (global) | c | AABB | facts |
| `color` named, hex, rgb, hsl, cmyk | T+t | CSS Color 4, XKCD | W3C, CC0 |
| `programminglanguage`, `os`, `database` column/type/engine, `fileext`, `semver` (exists) | T/c | Linguist, curated | MIT |
| `commit` message, branch, `sha` | t | LLM-written, `{hex(40)}` | — |
| `password`, `hash`, `jwt`, `slug`, `hostname`, `avatar`, `placeholderimage` | t | shapes | — |
| `book` title, author from Gutenberg; `genre`; `instrument`; `dish`, `ingredient` | T | Gutenberg catalog, MusicBrainz, USDA | CC0 |
| `lorem`, `hacker`, `hipster`, `catchphrase`, `buzzword`, `quote` | T/c | lorem ipsum, LLM-written | — |
| `coordinate` (exists), `direction`, `continent`, `emoji` (exists), `uuid`, `ulid`, `objectid` | c/t | — | — |
Later locales: nb_NO, da_DK, fi_FI, de_DE, en_GB, nl_NL, fr_FR, es_ES with person,
address, phone, national id, company and date names each.
#### Order of work
1. Table node, key and name selection, parent links, consistent draws, the
choice-of-rows fence, `DATA-LICENSES.md`, `data-import/`.
2. `geo/SE` and `geo/US`, and `address` in both locales on top of them.
3. Weighted person names and valid ids in both locales; `date()`.
4. `misc` conversions and the new `misc` tables.
5. The remaining locale categories: company, phone, finance, vehicle, words.
6. Records with building-block columns across the shipped set; shape re-pin.
### Release