diff --git a/CHANGELOG.md b/CHANGELOG.md index 357feb2..82f2ffa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ replacement, and each removed path, column or flag. - First release: the CLI, the library and the shipped data set. - Table categories: a category JSON naming a `rows` TSV beside it, with the options `key`, `name`, `weight` and `parent`; a path selects a row by key or name, - `misc.country[SE]`, and descends to a linked table by name; linked tables draw + `misc.territory[SE]`, and descends to a linked table by name; linked tables draw consistently within one render and draw group. `rows` is an option, so no template may carry a field of that name. A `name` without a `key` resolves inside the table's `parent`. Refused at `New`: a `name` without a `key` or a @@ -21,16 +21,28 @@ replacement, and each removed path, column or flag. row of, or two paths pinning different rows of one table. - `New` refuses a root choice of templates sharing one format and one set of string fields, naming the rows TSV to write instead. -- `misc.country`, `misc.currency`, `misc.language`, `misc.httpstatus` and - `misc.mimetype` are tables. `misc.country` is every ISO 3166 country that has a - capital, a currency and a TLD, with the columns `calling-code`, `capital`, - `currency`, `flag`, `languages`, `numeric` and `tld` added; `misc.currency` the +- `misc.territory`, `misc.currency`, `misc.language`, `misc.httpstatus` and + `misc.mimetype` are tables. `misc.territory` is every ISO 3166-1 territory that has + a capital, a currency and a TLD, with the columns `calling-code`, `capital`, + `country`, `currency`, `flag`, `languages`, `numeric` and `tld` added, `country` + holding the alpha-2 code of the sovereign state it belongs to and its own where it + is one; `misc.currency` the current ISO 4217 currencies with a minor unit, with `decimals` and `numeric` added, and its symbols from CLDR; `misc.language` every ISO 639-2 entry carrying a 639-1 code, with a `code3` column holding its 639-2/T code; `misc.httpstatus` each code the IANA registry lists with a plain reason phrase; and `misc.mimetype` each type mime-db records as IANA-registered and gives a filename extension. `DATA-LICENSES.md` lists each source. +- `misc.timezone`, `misc.car` and `misc.useragent` are tables too. `misc.timezone` + is every zone tzdb `zone.tab` gives a shipped territory, with `offset` holding the + zone's standard UTC offset and `territory` linking to `misc.territory`, so + `misc.territory[SE].timezone` draws `Europe/Stockholm`; zones of a territory + `misc.territory` does not ship, Antarctica's among them, and the constant `UTC` + are gone; spell that one as the text `"UTC"`. A `weight` column carries the + population GeoNames records in each zone, so a draw favours the zones people live + in. `misc.useragent` is the top-user-agents desktop and mobile lists with `browser`, + `device` and `os` columns. `misc.car` keeps its makes and models in a `make` and a + `model` column. - `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`, `locality`, `postal-code` and `street`, weighted by population and address counts and built from SCB, GeoNames, Trafikverket NVDB and the US Census Bureau, and an diff --git a/DATA-LICENSES.md b/DATA-LICENSES.md index 2ff85a6..97f6c7f 100644 --- a/DATA-LICENSES.md +++ b/DATA-LICENSES.md @@ -14,11 +14,13 @@ Every shipped dataset, its source, its licence and the attribution it asks for. | `sv_SE/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` | | `en_US/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` | | `en_US/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` | -| `sv_SE/sex.tsv`, `sv_SE/birth-number.tsv`, `sv_SE/title.tsv`, `en_US/sex.tsv`, `en_US/title.tsv` | curated (Skatteverket's test birth numbers are facts) | — | — | — | -| `misc/country.tsv` | [datasets/country-codes](https://github.com/datasets/country-codes) | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/country.py` | +| `sv_SE/sex.tsv`, `sv_SE/birth-number.tsv`, `sv_SE/title.tsv`, `en_US/sex.tsv`, `en_US/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 10) | — | — | — | | `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` | | `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` | | `misc/language.tsv` | [datasets/language-codes](https://github.com/datasets/language-codes), the [Library of Congress](https://www.loc.gov/standards/iso639-2/) ISO 639-2 register | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/language.py` | | `misc/mimetype.tsv` | [mime-db](https://github.com/jshttp/mime-db), the IANA media type registry with filename extensions | [MIT](https://github.com/jshttp/mime-db/blob/master/LICENSE) | "Copyright (c) 2014 Jonathan Ong, Copyright (c) 2015-2022 Douglas Christopher Wilson" | `data-import/mimetype.py` | +| `misc/territory.tsv` | [datasets/country-codes](https://github.com/datasets/country-codes) | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/territory.py` | +| `misc/timezone.tsv` | [IANA tzdb](https://www.iana.org/time-zones) 2026d `zone.tab` and the standard offset of each zone; weights from [GeoNames](https://www.geonames.org/) `cities15000` populations | [public domain](https://data.iana.org/time-zones/tzdb/LICENSE); [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | "Populations from GeoNames, www.geonames.org" | `data-import/timezone.py` | +| `misc/useragent.tsv` | [top-user-agents](https://github.com/microlinkhq/top-user-agents) desktop and mobile lists | [MIT](https://github.com/microlinkhq/top-user-agents/blob/master/LICENSE.md) | "Copyright © 2020 Kiko Beats" | `data-import/useragent.py` | -Every other category is hand-written JSON under [`data/`](data), MIT like the code. +Every other category is hand-written under [`data/`](data), MIT like the code. diff --git a/README.md b/README.md index ad915dc..c1bf021 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,8 @@ fejkdata sv_SE.person.last # Eriksson fejkdata --seed 42 sv_SE.address # the same address every run fejkdata -n 3 --separator ', ' sv_SE.word # nät, barn, sol fejkdata --list # every path the data offers -fejkdata 'misc.country[SE].capital' # Stockholm — a table's row, selected by key or name -fejkdata 'geo.SE.locality[Lund].street' # Fjelievägen — a linked table, drawn inside the row +fejkdata 'misc.territory[SE].capital' # Stockholm — a table's row, selected by key or name +fejkdata 'geo.SE.locality[Lund].street' # Fjelievägen — a linked table, drawn inside the row fejkdata --data-path ./mydata sv_SE.word # layer a directory over the shipped data fejkdata --no-shipped-data -d ./mydata --list # only your data fejkdata 'name: {/sv_SE.person.last}' # name: — an inline template @@ -161,15 +161,36 @@ Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name` `ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`, `username`, `version` and `word`, formatted per locale; `sv_SE` adds `personnummer` and `samordningsnummer`, `en_US` adds `ssn` and `itin`. `misc` -carries `car`, `coordinate`, `country` (ISO 3166), `creditcard` (Luhn-valid), -`currency` (ISO 4217), `datetime` (RFC 3339), `emoji`, `httpstatus`, `language` -(ISO 639-1, with its 639-2/T code), `mac`, `mimetype`, `objectid`, `timezone` +carries `car`, `coordinate`, `creditcard` (Luhn-valid), `currency` (ISO 4217), +`datetime` (RFC 3339), `emoji`, `httpstatus`, `language` (ISO 639-1, with its +639-2/T code), `mac`, `mimetype`, `objectid`, `territory` (ISO 3166-1), `timezone` (IANA), `useragent` and `uuid` (v4). Many carry sub-fields — `misc.currency.symbol`, -`misc.country.alpha2`, `misc.httpstatus.code` — which `--list` shows. `country`, -`currency`, `httpstatus`, `language` and `mimetype` are [tables](#table), so -`misc.country[SE].capital` and `misc.currency[Euro].symbol` select a row; +`misc.territory.alpha2`, `misc.httpstatus.code` — which `--list` shows. `car`, +`currency`, `httpstatus`, `language`, `mimetype`, `territory`, `timezone` and +`useragent` are [tables](#table), so `misc.territory[SE].capital` and +`misc.currency[Euro].symbol` select a row; `car` and `useragent` carry no key or +name, so they are drawn from rather than selected in. [`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence. +`misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for most +of them to dozens for the largest, weighted by the population GeoNames records in each, +so `misc.territory[US].timezone` draws `America/New_York` far more often than +`America/Nome`. It links to `misc.territory`, so `misc.territory[SE].timezone` is +`Europe/Stockholm` and a drawn territory and zone agree. Its `offset` is the zone's +*standard* offset, so do not pair it with a drawn `misc.datetime`: in a zone that +observes DST it is the wrong one half the year, which is what storing a zone name +avoids. A zone tzdb named recently — `Europe/Kyiv`, `America/Ciudad_Juarez` — is +rejected outright by a consumer resolving it against older tzdata, so where the +consumer validates the zone, pin one rather than draw it. There is no `UTC` row, tzdb +giving that name no territory — spell it as the text `"UTC"`. `misc.useragent` carries +`browser`, `device` and `os` beside the string, and `misc.car` a `make` and a `model`, +until an international source replaces them. + +ISO 3166-1 codes territories, not sovereign states, so that is what the table is +called: Greenland and Åland have codes of their own, and `misc.territory.country` +names the state each belongs to — `DK` for Greenland, `FI` for Åland, and its own +code for a sovereign one, or for one the register names no state for. + `sex`, `first-name` and `last-name` are tables weighted by bearers, from SCB, the SSA and the Census Bureau. `first-name` links to `sex`, so `sv_SE.sex[f].first-name` draws a woman's name, and a name both sexes carry is a row under each, so @@ -447,11 +468,11 @@ inline template has no file beside it, so there it stays a choice. ### Row selection -`[key]` or `[name]` after a table's name selects one row: `misc.country[SE]` and -`misc.country[Sweden]` name one row, and `misc.country[SE].capital` reads its column. +`[key]` or `[name]` after a table's name selects one row: `misc.territory[SE]` and +`misc.territory[Sweden]` name one row, and `misc.territory[SE].capital` reads its column. A name naming several rows is an error listing their keys, unless a row selected before it settles which ([Linked tables](#linked-tables)). A selector is part of the path, so it works -wherever a path does: `Fake`, `FakeRecord`, a `{/misc.country[SE].capital}` reference +wherever a path does: `Fake`, `FakeRecord`, a `{/misc.territory[SE].capital}` reference and a struct tag. A dot inside the brackets belongs to the key or name, so `city[St. Louis]` selects it. A path starts with a name, and `[` still opens a JSON array at the start of a CLI argument, so `'[SE]'` alone names nothing. @@ -772,6 +793,13 @@ App developers writing tests and fixtures, in Go and at a shell: 9. **Fast enough to be free** — a value renders in about a microsecond and `New` parses and validates the whole set once upfront, so generating fixtures stays noise against a test's own runtime. +10. **Data is sourced, or on its way there** — a shipped fact, a name, place, + code, id or classification, is read from a register or open dataset by a + [`data-import/`](data-import) script wherever one exists to read; where none + does yet a small hand-written set ships and [`todo.md`](todo.md) carries the + step that replaces it. Only non-factual copy stays authored. A sourced table + holds the rows its source holds: none is added by hand, and one is dropped + only by a rule the script states. ## Decisions @@ -1025,8 +1053,7 @@ App developers writing tests and fixtures, in Go and at a shell: A table is a category with a TSV beside its file, so only a root choice has the spelling the fence names; a nested choice of same-shaped templates and an inline one keep loading. Fields must all be strings because a cell is a string node: a - choice whose items carry a nested choice, as `misc.car` does, is not one table but - two linked ones, which a later conversion writes. + choice whose items carry a nested choice is not one table but two linked ones. - **A table is a record of string columns.** Its columns are the CSV header and the `INSERT` column list, fixed by the TSV header, so a table is a record by construction; every column is a string until a typed column option earns its place. @@ -1124,11 +1151,46 @@ App developers writing tests and fixtures, in Go and at a shell: tells the two apart, `sex[f].first-name[Kim]`, the ambiguity error spells each row inside its parent, and a name repeating inside one parent row is refused at load, since nothing could then select it. -- **`misc.country` carries a currency code, it does not link to `misc.currency`.** +- **`misc` is what every locale shares.** A category whose facts differ by country + belongs in that country's locale, read from the register that country's own + records use; `misc` takes only sources that are international. NHTSA vPIC and + Mobility Sweden's registrations are national, so they build `en_US.car` and + `sv_SE.car`; `misc.car` waits for an international source rather than take one + of theirs. +- **A register's canonical spelling loses to the one its domain writes.** Where a + source offers several spellings of one fact, the shipped one is what records in + that domain carry. `misc.timezone` reads `zone.tab` and not the `zone1970.tab` + that supersedes it, because the latter keeps one zone per set of countries that + have agreed since 1970: it spells Sweden `Europe/Berlin`, and no Swedish system + writes that. For the same reason `misc.language` takes the ISO 639-2 register's + first synonym over CLDR, which says "Chinese, Mandarin" for `zh`. +- **`misc.territory` is the spine, and a `misc` table naming a territory links to + it.** Where every territory row has a child the column is a `parent`, and the + import drops the child rows whose territory the set does not ship — 17 of + `misc.timezone`'s, Antarctica's ten among them. Agreement across a record is + worth more than the last rows of a table. Where no such link can hold the fact + stays a column. Layer your own `misc.territory` over the shipped one and you + must layer `misc.timezone` too, or the link fails at load naming the row. +- **`misc.car` is one flat table, not a make linked to its models.** A row is a make + and a model drawn together, so no render pairs a Volvo with a RAV4. Two linked + tables would reach the same pairs and add a selector nothing asks for; a make alone + is `misc.car.make`. +- **`misc.territory` names its sovereign in a column, and there is no `misc.country` + table.** ISO 3166-1 codes territories, so `territory` is the honest name, and + `is_independent` in the register gives each one's state. A second table of the 195 + sovereigns would hold a `DK` row beside the territory `DK` row, both carrying + Denmark's capital, currency, flag and TLD — two owners for one fact, drifting at + the next import. Splitting the columns to avoid that is worse: put `capital` on + the territory alone and a country row can no longer name Copenhagen. The column + cannot be a `parent`, since a table never reaches its own family; a test proves + every value names a row instead. A territory the register records no state for + stands alone, which is `EH` alone, and naming one for it would be a claim + fejkdata has no business making. +- **`misc.territory` carries a currency code, it does not link to `misc.currency`.** A `parent` demands a child for every parent row, and ISO 4217 registers codes no country's row can name: the funds codes (Mvdol, WIR Euro, US Dollar (Next day)), and VED beside VES, both Venezuela's, of which a country row names one. Linking - would trade the register for the link, and `misc.country.currency` already pairs a + would trade the register for the link, and `misc.territory.currency` already pairs a country with its currency in one draw. - **An extension may name two media types.** `.xml`, `.rtf`, `.sub`, `.mpp` and `.ac` each name two rows of `misc.mimetype`. Separating them would mean dropping a @@ -1207,15 +1269,17 @@ and `names-us.py` rejects a client for a while after a burst, so `--surnames` ta a copy of the surname file: ```sh -docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/country.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py -docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py docker compose run --rm --user "$(id -u):$(id -g)" -e TRAFIKVERKET_API_KEY data-import data-import/geo-se.py +docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpstatus.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/language.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/mimetype.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-se.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-us.py +docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/territory.py +docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/timezone.py +docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/useragent.py ``` To release, head `CHANGELOG.md` with the version's section in place of `Unreleased` diff --git a/cmd/fejkdata/main.go b/cmd/fejkdata/main.go index 57ccd79..327e5c3 100644 --- a/cmd/fejkdata/main.go +++ b/cmd/fejkdata/main.go @@ -24,7 +24,7 @@ import ( const usage = `Usage: fejkdata [flags] a category, or a dotted path into one (person, person.last); - a table's row by key or name: 'misc.country[SE]', 'misc.country[Sweden].tld' + a table's row by key or name: 'misc.territory[SE]', 'misc.territory[Sweden].tld'