Order the plan as the releases that ship it, and move the repo's own facts out of the handoff #24

Merged
lilleman merged 5 commits from plan-as-releases into main 2026-09-19 02:00:39 +02:00
3 changed files with 147 additions and 155 deletions
+4
View File
@@ -4,6 +4,10 @@
- Tests first, in their own commit; the implementation follows in the next. A re-pin of seeded output or of `testdata/shipped_shape.txt` is its own commit.
- A change under `data/` or to the shape pin adds its `CHANGELOG.md` entry under `Unreleased` in the same PR, and so does a change to a flag, an exit code, an exported name, a fence, a builtin or the lowest Go; what is major is the README's Versioning table.
- One-line commit messages: no ticket prefix, no repo name, no authorship trailers.
- Merges are fast-forward only, which `tea pr merge` cannot spell — every style it
offers fails with "is it still open?", as does a merge whose required checks are
still pending, so read the checks before believing the style is the problem. Use
`tea api /repos/{owner}/{repo}/pulls/<n>/merge -f Do=fast-forward-only`.
- Hard tabs. No comment by default; delete a restatement, a rationale, history, or a file preamble.
- One spelling per result: reject the other at `New`, and let the error name the spelling to use.
- A standing choice a reader would relitigate goes under Decisions in the README, not in a comment.
+11
View File
@@ -1151,6 +1151,8 @@ App developers writing tests and fixtures, in Go and at a shell:
tells the two apart, `sex[f].first-name[Kim]`, the ambiguity error spells each
row inside its parent, and a name repeating inside one parent row is refused at
load, since nothing could then select it.
- **No pop-culture catalogues.** Every other faker ships film, band and character
names; fejkdata ships none.
- **`misc` is what every locale shares.** A category whose facts differ by country
belongs in that country's locale, read from the register that country's own
records use; `misc` takes only sources that are international. NHTSA vPIC and
@@ -1238,6 +1240,15 @@ docker compose run --rm --user "$(id -u):$(id -g)" fmt # gofmt -w .
docker compose run --rm --user "$(id -u):$(id -g)" tidy # go mod tidy
```
A `--user` command answering `permission denied` on `/cache` has met root-owned
files in the `gocache` volume: Docker creates the volume root-owned, and every
command here that omits `--user` writes into it as root. Hand it back, and again
whenever it recurs:
```sh
docker compose run --rm --user root --entrypoint chown test -R "$(id -u):$(id -g)" /cache
```
Every pull request runs `docker build .` against both the latest and the lowest
supported Go, and must pass before it can be merged — unless it changes none of
the files the build and its tests read, nor the workflow itself, in which case
+132 -155
View File
@@ -1,103 +1,66 @@
# Release checklist
# Plan
## Before v0.1.0
Ordered as the releases that ship it.
## v0.1.0
### Data
Plan for the major data update. Research notes: `~/code/claude-handoffs/fejkdata/2026-09-16/`.
Shape: T = table, t = template, c = choice. Read every factual list from a register
by a `data-import/` script, as README goal 10 asks.
#### Structure (proposal, pending review)
`New` costs ~23 ms and 58k allocations for today's 200 KB, one allocation per
node, so real-world tables cannot ship as nested JSON objects. JSON says how a
value is composed; TSV says which values exist.
- Add a table node: a category JSON naming a `rows` TSV beside it, whose header
names the columns and whose rows are the draw. Options `key` (the column a path
selects a row by), `name` (the column a path also selects by), `weight` (the
column that skews the draw), `parent` (the table a column links to). `{name}`
tokens read columns; a cell is a string node, so it may carry tokens. Index
rows on first draw; at load prove only the header, the options and any cell
token. A TSV nothing names is a load error.
- In a data file, refuse a choice whose items are templates with one identical
field set, naming the TSV to write; an inline template is exempt, having no
file. A choice of strings stays a choice.
- Convert `misc.timezone`, `car` and `useragent` to tables — done in step 4.
- Let a path select a row by key or by name, `geo.SE.municipality[0180]` and
`geo.SE.municipality[Stockholm]`, and descend to a linked table by name,
`geo.SE.region[Skåne län].street`; a key or a linked table name may not equal
a column name (load error). A name that matches several rows outside a pinned
parent is an error listing the keys; inside one, `region[IL].locality[Springfield]`
resolves. Brackets need quoting in zsh; `:` would not.
- Draw linked tables consistently within one render and draw group: the first
table drawn pins its ancestors, and a descendant is drawn inside them.
- Keep country data under `geo/<alpha2>/`; a locale's `address` reads its
country's tree by reference, and a locale's `territory` table carries the
localised names keyed by alpha2.
- Ship full registers as packs, each a Go module with its own `embed.FS` and a
zip for `--data-path`; the default embed stays under ~1 MB per country and
`New` under ~50 ms.
- Add `DATA-LICENSES.md` listing every dataset, its licence and its attribution,
and a `data-import/` directory of Python scripts that rebuild each TSV from its
source, so a refresh is one command per dataset.
- Make every shipped category a record with its building blocks as columns
(`sv_SE.person` → `first`, `last`, `sex`, done in step 3; `misc.uuid` → `variant`).
- Share the handle lists between `email.local` and `username` only if that is a
clean win — a reference between shipped categories is a major once tagged.
#### Geo tree
Five linked tables per country, generic names so a template ports across
countries; the README maps each to the native term.
| Table | SE | US | Weight |
|---|---|---|---|
| `region` | län (21) | state and DC (50; Hawaii has no incorporated place) | population |
| `municipality` | kommun (290) | county with a shipped place | population |
| `locality` | postort, tätort population | place of 25,000+ | population |
| `postal-code` | postnummer with street delivery | ZCTA of a shipped place | one; address ranges |
| `street` | gatunamn, top 10 per postort | street name, top 10 per place | segments; address ranges |
- Shipped in step 2, README Data. `geo.SE.address` is a record over one consistent
draw. Each region row carries its timezone, each locality its centroid.
- Let `geo.SE.locality[Lund].address` descend from a selected row into the
template beside the family: the path step must reach a sibling category and the
outer selector's pins seed every draw group of the render.
- Ship the fuller sets, every US place of 10,000 and more streets per locality, as
packs; `--min-population` and `--streets-per-locality` on the scripts build them.
- Fill the 398 Swedish localities weighted 200 from SCB småorter before v0.1.0.
- Add the remaining `misc` tables and templates from the table below.
- Read `misc.emoji` from the Unicode `emoji-test.txt` register, and `misc.car` from an
international make and model source; vPIC and Mobility Sweden are national, so they
build `en_US.car` and `sv_SE.car`.
- Add the remaining locale categories: company, phone, finance, vehicle, words.
- Give every shipped category building-block columns, `misc.uuid` → `variant` among
them, and one record shape across the national ids; re-pin the shape.
- Give the address records one column set across countries: `region` and
`municipality` as building-block columns on `geo.SE.address` too, in step 6.
- v0.1.0 ships SE and US; then NO, DK, FI, NL, FR, AU, CA, ES, GB, DE.
- Revisit an application to Lantmäteriet for the exact street to postnummer
pairing after v0.1.0; today a street goes to the nearest postal code centroid.
`municipality` as columns on `geo.SE.address` too.
- Fill the 398 Swedish localities weighted 200 from SCB småorter.
- Accept a middle name, and draw a shipped `personnummer` inside a *selected* sex;
both want a draw group sharing its family's pins. Stop the conflict error naming a
rewrite that returns a different value where the read it conflicts with sits inside
another category, and report a struct column's draw conflict with the path spelling
a tag takes.
- Let `geo.SE.locality[Lund].address` descend from a selected row into the template
beside the family: require the path step to reach a sibling category, and seed every
draw group of the render from the outer selector's pins.
- Read `email.local`'s and `username`'s handles from the shipped name tables, which
goal 10 asks for and the hand-written list they use today does not meet. Keep the
handle shape: `username` shortens a surname to `ahl` or `sjo`, and the US one draws
unisex given names, neither of which falls out of the tables unaided.
- Decide whether `email.local` and `username` share one list: a reference between
shipped categories breaks a consumer once tagged, so it rides a 0.(x+1).0 after that.
#### Builtins the data cannot express
- `{date(from,to,'layout')}` and `{time('layout')}` shipped in step 3; `age()` is
rejected, README Decisions. Add a `unix` layout once something needs it.
- Derivations: `{isin()}` (Luhn over letters expanded to digits), `{cusip()}`,
`{aba()}` (3-7-1 weights), `{vin()}` (position 9 over the whole; a sample taking
the WMI, since the check sits mid-string).
- Samples: `{base64url(n)}` for JWT shapes; `{btc()}`, `{eth()}` after v0.1.0
(Base58Check and EIP-55 need sha256 and keccak).
#### Categories
Union of @faker-js/faker, Python Faker, gofakeit, Datafaker, Bogus, Mimesis, Chance
and Ruby faker (`research-libraries.md`). What sets fejkdata apart, and what every
issue tracker asks for: real values, frequency weights, and facts that agree across
a record. No pop-culture catalogues. An LLM writes only non-factual lists
(catchphrases, hacker phrases, product adjectives), never names, places, codes or
ids. Shape: T = table, t = template, c = choice.
`sv_SE` (`research-sources-se.md`)
`misc` (locale-neutral)
| Category | Shape | Source | Licence |
|---|---|---|---|
| `person` first (female, male, generic), middle, last, weighted | T | SCB 2022 whole-population xlsx, Skatteverket 2026 surnames | CC0, "Källa: SCB" |
| `person` title, sex, birthdate, age, blood type weighted | t | geblod.nu distribution | facts |
| `personnummer`, `samordningsnummer` | t | Skatteverket test series: date + 238/239, Luhn | CC0 |
| `httpmethod`, `port`, `tld`, `protocol`, `loglevel` | T/c | IANA | CC0 |
| `ip` v4 documentation ranges, private, CIDR; `ipv6` `2001:db8::/32`; `mac` locally administered | t | RFC 5737, 1918, 9637 | facts |
| `creditcard` per network: IIN, length, CVV, expiry | T+t | network rules, `{luhn()}` | facts |
| `bic`, `isin`, `cusip` | t | structure rules | facts |
| `ean`, `upc`, `gtin`, `isbn`, `issn`, `imei`, `asin`, `sku` | t | check-digit rules, GS1 demo prefix 952 | facts |
| `product` name, adjective, material, department, `garmentsize` | T/c | LLM-written lists | — |
| `airport` IATA, ICAO, territory, municipality; `airline`; `aircraft`; `flight`, `seat`, `pnr` | T+t | OurAirports, Wikidata | public domain, CC0 |
| `element`, `unit`, `planet` | T | PubChem, BIPM, NASA | public domain, CC BY 4.0 |
| `animal` by class, `plant` | T | Wikidata | CC0 |
| `bloodtype` weighted (global) | c | AABB | facts |
| `color` named, hex, rgb, hsl, cmyk | T+t | CSS Color 4, XKCD | W3C, CC0 |
| `programminglanguage`, `os`, `browser`, `database` column/type/engine, `fileext` | T/c | Linguist, curated | MIT |
| `commit` message, branch, `sha` | t | LLM-written, `{hex(40)}` | — |
| `password`, `hash`, `jwt`, `slug`, `hostname`, `avatar`, `placeholderimage` | t | shapes | — |
| `book` title, author; `genre`; `instrument`; `dish`, `ingredient` | T | Gutenberg catalog, MusicBrainz, USDA | CC0 |
| `lorem`, `hacker`, `hipster`, `catchphrase`, `buzzword`, `quote` | T/c | lorem ipsum, LLM-written | — |
| `direction`, `continent`, `ulid` | c/t | — | — |
`sv_SE` (source research: `research-sources-se.md`, in the handoff)
| Category | Shape | Source | Licence |
|---|---|---|---|
| `person` title, birthdate, age, blood type weighted | t | geblod.nu distribution | facts |
| `organisationsnummer` by form, `vat` | t | Bolagsverket group digits, Luhn, `SE…01` | facts |
| `company` name patterns, legal form weighted | t | Bolagsverket registrations 2025 | CC BY 2.5 SE |
| `sni` industry codes | T | SCB SNI 2025 | "Källa: SCB" |
@@ -107,87 +70,101 @@ ids. Shape: T = table, t = template, c = choice.
| `licenseplate` `ABC 123`, `ABC 12A`, blocked combinations | t | Transportstyrelsen | facts |
| `car` make, model weighted by registrations | T | Mobility Sweden 2025 | cite |
| `word` noun, verb, adjective, adverb; `sentence` | T+t | SALDO | CC BY 4.0 |
| `date`, `time`, month and weekday names, `holiday` | t+T | CLDR sv, lag 1989:253 | Unicode |
| `price` `1 234,56 kr`, `email` domains, `url` `.se`/`.nu` | t | curated | — |
| month and weekday names, `holiday` | t+T | CLDR sv, lag 1989:253 | Unicode |
| `territory` localised names keyed by alpha2 | T | CLDR sv territory names | Unicode |
| `address` | t | `{/geo.SE…}` | — |
`en_US` (`research-sources-world.md`, part1–4)
`en_US` (source research: `research-sources-world.md` part1–4, in the handoff)
| Category | Shape | Source | Licence |
|---|---|---|---|
| `person` first by sex weighted, last weighted, middle, generic | T | SSA baby names, Census 2010 surnames | public domain |
| `ssn`, `itin`, `ein` valid ranges | t | SSA POMS, IRS | facts |
| `ein` valid ranges | t | IRS | facts |
| `company` names, suffix, `naics` | t+T | SEC tickers for patterns, NAICS 2022 | public domain |
| `phone` NANP with valid NPA/NXX | t | NANPA rules | facts |
| `routing` ABA with check, `bankaccount` | t | Fed prefix ranges | facts |
| `licenseplate` per state | T | hand-authored patterns | facts |
| `car` make, model | T | NHTSA vPIC | public domain |
| `word`, `sentence`, `paragraph` | T+t | Moby POS or WordNet | public domain / WordNet |
| `date`, `time`, names, `holiday` | t+T | CLDR en | Unicode |
| `price` `$1,234.56`, `email`, `url`, `address` | t | curated, `{/geo.US…}` | — |
| month and weekday names, `holiday` | t+T | CLDR en | Unicode |
`misc` (locale-neutral)
### Builtins the data cannot express
| Category | Shape | Source | Licence |
|---|---|---|---|
| `territory` name, alpha2, alpha3, numeric, calling code, TLD, capital, country, currency, flag, languages | T | datasets/country-codes | PDDL |
| `currency` code, numeric, name, symbol, decimals | T | ISO 4217 list-one, CLDR symbols | free, Unicode |
| `language` ISO 639-1/2 | T | LoC | public domain |
| `timezone` zone, territory, offset | T | tzdb zone.tab | public domain |
| `mimetype` type, extensions | T | IANA + mime-db | CC0, MIT |
| `httpstatus`, `httpmethod`, `port`, `tld`, `protocol`, `loglevel` | T/c | IANA | CC0 |
| `useragent` browser, device, os; selection per browser once `browser` is a table | T | top-user-agents | MIT |
| `ip` v4 documentation ranges, private, CIDR; `ipv6` `2001:db8::/32`; `mac` locally administered | t | RFC 5737, 1918, 9637 | facts |
| `creditcard` per network: IIN, length, CVV, expiry | T+t | network rules, `{luhn()}` | facts |
| `bic`, `iban` (exists), `isin`, `cusip` | t | structure rules | facts |
| `ean`, `upc`, `gtin`, `isbn`, `issn`, `imei`, `asin`, `sku` | t | check-digit rules, GS1 demo prefix 952 | facts |
| `product` name, adjective, material, department, `garmentsize` | T/c | LLM-written lists | — |
| `airport` IATA, ICAO, country, municipality; `airline`; `aircraft`; `flight`, `seat`, `pnr` | T+t | OurAirports, Wikidata | public domain, CC0 |
| `element`, `unit`, `planet` | T | PubChem, BIPM, NASA | public domain, CC BY 4.0 |
| `animal` by class, `plant` | T | Wikidata | CC0 |
| `bloodtype` weighted (global) | c | AABB | facts |
| `color` named, hex, rgb, hsl, cmyk | T+t | CSS Color 4, XKCD | W3C, CC0 |
| `programminglanguage`, `os`, `database` column/type/engine, `fileext`, `semver` (exists) | T/c | Linguist, curated | MIT |
| `commit` message, branch, `sha` | t | LLM-written, `{hex(40)}` | — |
| `password`, `hash`, `jwt`, `slug`, `hostname`, `avatar`, `placeholderimage` | t | shapes | — |
| `book` title, author from Gutenberg; `genre`; `instrument`; `dish`, `ingredient` | T | Gutenberg catalog, MusicBrainz, USDA | CC0 |
| `lorem`, `hacker`, `hipster`, `catchphrase`, `buzzword`, `quote` | T/c | lorem ipsum, LLM-written | — |
| `coordinate` (exists), `direction`, `continent`, `emoji` (exists), `uuid`, `ulid`, `objectid` | c/t | — | — |
- Add `{isin()}` (Luhn over letters expanded to digits), `{cusip()}`, `{aba()}`
(3-7-1 weights) and `{vin()}` (position 9 over the whole; a sample taking the WMI,
since the check sits mid-string).
- Add `{base64url(n)}` for JWT shapes.
- Add a `unix` layout to `date()` once something needs it.
Later locales: nb_NO, da_DK, fi_FI, de_DE, en_GB, nl_NL, fr_FR, es_ES with person,
address, phone, national id, company and date names each.
### Open questions to settle
#### Order of work
1. Table node, key and name selection, parent links, consistent draws, the
choice-of-rows fence, `DATA-LICENSES.md`, `data-import/` — done.
2. `geo/SE` and `geo/US`, and `address` in both locales on top of them — done.
3. Weighted person names and valid ids in both locales; `date()` — done. Left for
later: middle names, and drawing a shipped `personnummer` inside a *selected*
sex, both of which want a draw group that shares its family's pins; stop the
conflict error naming a rewrite that returns a different value where the read it
conflicts with sits inside another category. Give the national ids one
record shape in step 6, and report a struct column's draw conflict with the path
spelling a tag takes rather than the reference spelling.
4. `misc` conversions and the new `misc` tables. Convert `timezone`, `car` and
`useragent`, and add the new `misc` tables.
5. Give the hand-written sets a source, as goal 10 asks: read `misc.emoji` from
the Unicode emoji-test.txt register, and `misc.car` from an international make
and model source, vPIC and Mobility Sweden being national and so `en_US.car`'s
and `sv_SE.car`'s.
6. The remaining locale categories: company, phone, finance, vehicle, words.
7. Records with building-block columns across the shipped set; shape re-pin.
- Decide whether `misc.browser` becomes a parent of `misc.useragent`, so
`misc.browser[Chrome].useragent` resolves. Adding a `parent` after v0.1.0 breaks a
consumer, so it rides a 0.(x+1).0.
- Decide whether `misc.timezone` keeps every zone tzdb names: one added recently,
`Europe/Kyiv` among them, is rejected by a consumer resolving it against older
tzdata, and the population weight makes that zone likelier, not rarer. Today the
README names the action a reader takes instead; narrowing what ships is the
alternative.
- Decide whether `misc.territory.country` is renamed `sovereign`: it holds a code, and
`country` collides with the `misc.country` path it replaced. A rename after v0.1.0
breaks a consumer, so it rides a 0.(x+1).0.
- Decide whether `parent: territory` stays, given a `--data-path` override of
`misc.territory` now fails `New` unless `misc.timezone` is overridden with it.
- Decide whether the shipped `url` and `email` domains are restricted to RFC 2606's
reserved names: 17 of the 40 distinct ones are not, sitting on `.se`, `.nu`, `.io`
and `.co`, which are live ccTLDs anyone can register. Restricting them costs the
locale flavour `example.se` and `.nu` were chosen for.
- Record why fejkdata ships no pop-culture catalogue. README Decisions states the
choice with no reason, because goal 10 does not supply one: it admits a hand-written
set where no register exists, and Wikidata, MusicBrainz and the Gutenberg catalog are
registers this plan already reads for `book`, `instrument` and `animal`.
### Release
- Rewrite `[Unreleased]` as what v0.1.0 holds, not as a diff against `main`: nothing
has shipped, so "no longer paths", "where it used to fail" and "where it used to be
an empty string" describe versions no reader can have installed.
- Move the README's Decisions section to `docs/decisions.md`, leaving a one-line index
of the titles in `AGENTS.md`, and drop the `AGENTS.md` line pointing decisions at the
README.
- Rewrite `CHANGELOG.md`'s `[Unreleased]` as what v0.1.0 holds: nothing has shipped, so
"no longer paths", "where it used to fail" and "where it used to be an empty string"
describe versions no reader can have installed.
- Ship prebuilt binaries so the CLI needs no Go: GoReleaser attaching them to the Gitea
release the tag workflow publishes, a container image, Homebrew and Scoop. A checkout
build prints `devel` for `--version`; the binaries carry the stamped tag.
- CLI without Go — investigate prebuilt binaries: GoReleaser attaching them to
the Gitea release the tag workflow publishes, a container image, Homebrew and
Scoop. A checkout build prints `devel` for `--version`; the binaries carry the stamped tag.
- Homepage — a simple page for fejkdata with an in-browser generator: the library
compiled to WebAssembly, so visitors generate as much data as they like in their
own browser.
## v0.2.0
- Ship the full registers as packs, each a Go module with its own `embed.FS` and a zip
for `--data-path`: every US place of 10,000 and more streets per locality, built by
`--min-population` and `--streets-per-locality`. The default embed stays under ~1 MB
per country and `New` under ~50 ms.
- Add the locales nb_NO, da_DK, fi_FI, de_DE, en_GB, nl_NL, fr_FR and es_ES, with
person, address, phone, national id, company and date names each, and the `geo/`
trees NO, DK, FI, NL, FR, AU, CA, ES, GB and DE their `address` reads by reference;
AU and CA get a tree with no locale of their own.
- Add `{btc()}` and `{eth()}`, which need sha256 and keccak for Base58Check and EIP-55.
- Pair a street with its exact postnummer, which needs an application to Lantmäteriet;
today a street goes to the nearest postal code centroid. It rewrites shipped rows, so
it is a major once 1.0 is cut and a minor before that.
## v0.3.0
Names no items yet, so planning it is a chunk of its own: it takes the additive work
that lands after v0.2.0, and it has to add something, since a minor only adds. Its job
is to be the release with no breaking change that v1.0.0 waits on — v0.2.0 cannot be,
because pairing a street with its exact postnummer rewrites shipped rows.
## v1.0.0
- Cut it once the shipped data is in its record shape and one full minor has shipped
with no breaking change, per the README's Versioning table. It plans no work of its
own: v0.1.0 settles the record shape and v0.3.0 is the quiet minor.
## Not release-bound
- Stop `/cache` collecting root-owned files, which today every command documented
without `--user` leaves for the next `--user` one to trip over, and which the README
answers with a chown a reader has to repeat. Put `--user` on all of them, or give the
`x-go` anchor a `user:` — that one needs a mechanism, since Compose interpolates from
the process environment and neither `UID` nor `GID` is exported there, so
`user: "${UID}:${GID}"` resolves to `":"`.
- Publish a homepage with an in-browser generator: the library compiled to WebAssembly,
so visitors generate as much data as they like in their own browser.