From 7b70530a8aaf2542ce00dbe08bd9234fe5b6c820 Mon Sep 17 00:00:00 2001 From: Lilleman auf Larv Date: Sun, 11 Oct 2026 12:15:41 +0200 Subject: [PATCH] Answer architecture round 5: the shipped packages are enus, svse, geose, geous and misc, each holding its files at its root under its manifest's namespace, and the errors use the wording a bench of the Audience chose --- AGENTS.md | 1 + CHANGELOG.md | 11 ++--- DATA-LICENSES.md | 18 ++++---- README.md | 44 ++++++++++--------- data-import/currency.py | 4 +- data-import/geo-se.py | 2 +- data-import/geo-us.py | 2 +- data-import/geo.py | 4 +- data-import/httpmethod.py | 4 +- data-import/httpstatus.py | 4 +- data-import/language.py | 4 +- data-import/loglevel.py | 4 +- data-import/mimetype.py | 4 +- data-import/names-se.py | 4 +- data-import/names-us.py | 4 +- data-import/port.py | 4 +- data-import/protocol.py | 4 +- data-import/territory.py | 4 +- data-import/timezone.py | 6 +-- data-import/tld.py | 4 +- data-import/useragent.py | 4 +- data.go | 1 + data/en_US/fs.go | 9 ---- data/{en_US => enus}/.fejkdata.json | 3 +- data/{en_US/en_US => enus}/address.json | 0 data/{en_US/en_US => enus}/color.json | 0 data/{en_US/en_US => enus}/company.json | 0 data/{en_US/en_US => enus}/date.json | 0 data/{en_US/en_US => enus}/email.json | 0 data/{en_US/en_US => enus}/first-name.json | 0 data/{en_US/en_US => enus}/first-name.tsv | 0 data/enus/fs.go | 9 ++++ data/{en_US/en_US => enus}/ip.json | 0 data/{en_US/en_US => enus}/itin.json | 0 data/{en_US/en_US => enus}/last-name.json | 0 data/{en_US/en_US => enus}/last-name.tsv | 0 data/{en_US/en_US => enus}/person.json | 0 .../{en_US/en_US => enus}/phone-exchange.json | 0 data/{en_US/en_US => enus}/phone.json | 0 data/{en_US/en_US => enus}/price.json | 0 data/{en_US/en_US => enus}/sentence.json | 0 data/{en_US/en_US => enus}/sex.json | 0 data/{en_US/en_US => enus}/sex.tsv | 0 data/{en_US/en_US => enus}/ssn.json | 0 data/{en_US/en_US => enus}/time.json | 0 data/{en_US/en_US => enus}/title.json | 0 data/{en_US/en_US => enus}/title.tsv | 0 data/{en_US/en_US => enus}/url.json | 0 data/{en_US/en_US => enus}/username.json | 0 data/{en_US/en_US => enus}/version.json | 0 data/{en_US/en_US => enus}/word.json | 0 data/geo/SE/fs.go | 9 ---- data/geo/US/fs.go | 9 ---- data/{geo/SE => geose}/.fejkdata.json | 3 +- data/{geo/SE/geo/SE => geose}/address.json | 0 data/geose/fs.go | 9 ++++ data/{geo/SE/geo/SE => geose}/locality.json | 0 data/{geo/SE/geo/SE => geose}/locality.tsv | 0 .../SE/geo/SE => geose}/municipality.json | 0 .../{geo/SE/geo/SE => geose}/municipality.tsv | 0 .../{geo/SE/geo/SE => geose}/postal-code.json | 0 data/{geo/SE/geo/SE => geose}/postal-code.tsv | 0 data/{geo/SE/geo/SE => geose}/region.json | 0 data/{geo/SE/geo/SE => geose}/region.tsv | 0 data/{geo/SE/geo/SE => geose}/street.json | 0 data/{geo/SE/geo/SE => geose}/street.tsv | 0 data/{geo/US => geous}/.fejkdata.json | 3 +- data/{geo/US/geo/US => geous}/address.json | 0 data/geous/fs.go | 9 ++++ data/{geo/US/geo/US => geous}/locality.json | 0 data/{geo/US/geo/US => geous}/locality.tsv | 0 .../US/geo/US => geous}/municipality.json | 0 .../{geo/US/geo/US => geous}/municipality.tsv | 0 .../{geo/US/geo/US => geous}/postal-code.json | 0 data/{geo/US/geo/US => geous}/postal-code.tsv | 0 data/{geo/US/geo/US => geous}/region.json | 0 data/{geo/US/geo/US => geous}/region.tsv | 0 data/{geo/US/geo/US => geous}/street.json | 0 data/{geo/US/geo/US => geous}/street.tsv | 0 data/misc/.fejkdata.json | 3 +- data/misc/{misc => }/car.json | 0 data/misc/{misc => }/car.tsv | 0 data/misc/{misc => }/coordinate.json | 0 data/misc/{misc => }/creditcard.json | 0 data/misc/{misc => }/currency.json | 0 data/misc/{misc => }/currency.tsv | 0 data/misc/{misc => }/datetime.json | 0 data/misc/{misc => }/emoji.json | 0 data/misc/fs.go | 6 +-- data/misc/{misc => }/httpmethod.json | 0 data/misc/{misc => }/httpmethod.tsv | 0 data/misc/{misc => }/httpstatus.json | 0 data/misc/{misc => }/httpstatus.tsv | 0 data/misc/{misc => }/language.json | 0 data/misc/{misc => }/language.tsv | 0 data/misc/{misc => }/loglevel.json | 0 data/misc/{misc => }/loglevel.tsv | 0 data/misc/{misc => }/mac.json | 0 data/misc/{misc => }/mimetype.json | 0 data/misc/{misc => }/mimetype.tsv | 0 data/misc/{misc => }/objectid.json | 0 data/misc/{misc => }/port.json | 0 data/misc/{misc => }/port.tsv | 0 data/misc/{misc => }/protocol.json | 0 data/misc/{misc => }/protocol.tsv | 0 data/misc/{misc => }/territory.json | 0 data/misc/{misc => }/territory.tsv | 0 data/misc/{misc => }/timezone.json | 0 data/misc/{misc => }/timezone.tsv | 0 data/misc/{misc => }/tld.json | 0 data/misc/{misc => }/tld.tsv | 0 data/misc/{misc => }/useragent.json | 0 data/misc/{misc => }/useragent.tsv | 0 data/misc/{misc => }/uuid.json | 0 data/modules.go | 13 +++--- data/sv_SE/fs.go | 9 ---- data/{sv_SE => svse}/.fejkdata.json | 3 +- data/{sv_SE/sv_SE => svse}/address.json | 0 data/{sv_SE/sv_SE => svse}/birth-number.json | 0 data/{sv_SE/sv_SE => svse}/birth-number.tsv | 0 data/{sv_SE/sv_SE => svse}/color.json | 0 data/{sv_SE/sv_SE => svse}/company.json | 0 data/{sv_SE/sv_SE => svse}/date.json | 0 data/{sv_SE/sv_SE => svse}/email.json | 0 data/{sv_SE/sv_SE => svse}/first-name.json | 0 data/{sv_SE/sv_SE => svse}/first-name.tsv | 0 data/svse/fs.go | 9 ++++ data/{sv_SE/sv_SE => svse}/ip.json | 0 data/{sv_SE/sv_SE => svse}/last-name.json | 0 data/{sv_SE/sv_SE => svse}/last-name.tsv | 0 data/{sv_SE/sv_SE => svse}/person.json | 0 data/{sv_SE/sv_SE => svse}/personnummer.json | 0 data/{sv_SE/sv_SE => svse}/phone.json | 0 data/{sv_SE/sv_SE => svse}/price.json | 0 .../sv_SE => svse}/samordningsnummer.json | 0 data/{sv_SE/sv_SE => svse}/sentence.json | 0 data/{sv_SE/sv_SE => svse}/sex.json | 0 data/{sv_SE/sv_SE => svse}/sex.tsv | 0 data/{sv_SE/sv_SE => svse}/time.json | 0 data/{sv_SE/sv_SE => svse}/title.json | 0 data/{sv_SE/sv_SE => svse}/title.tsv | 0 data/{sv_SE/sv_SE => svse}/url.json | 0 data/{sv_SE/sv_SE => svse}/username.json | 0 data/{sv_SE/sv_SE => svse}/version.json | 0 data/{sv_SE/sv_SE => svse}/word.json | 0 docs/comprehension-history.md | 24 ++++++++++ docs/decisions.md | 15 ++++++- fejkdata.go | 2 +- internal/datafiles/datafiles.go | 26 ++++++++--- internal/datafiles/manifest.go | 27 +++++++++++- reference.go | 4 +- todo.md | 4 +- 152 files changed, 217 insertions(+), 132 deletions(-) delete mode 100644 data/en_US/fs.go rename data/{en_US => enus}/.fejkdata.json (94%) rename data/{en_US/en_US => enus}/address.json (100%) rename data/{en_US/en_US => enus}/color.json (100%) rename data/{en_US/en_US => enus}/company.json (100%) rename data/{en_US/en_US => enus}/date.json (100%) rename data/{en_US/en_US => enus}/email.json (100%) rename data/{en_US/en_US => enus}/first-name.json (100%) rename data/{en_US/en_US => enus}/first-name.tsv (100%) create mode 100644 data/enus/fs.go rename data/{en_US/en_US => enus}/ip.json (100%) rename data/{en_US/en_US => enus}/itin.json (100%) rename data/{en_US/en_US => enus}/last-name.json (100%) rename data/{en_US/en_US => enus}/last-name.tsv (100%) rename data/{en_US/en_US => enus}/person.json (100%) rename data/{en_US/en_US => enus}/phone-exchange.json (100%) rename data/{en_US/en_US => enus}/phone.json (100%) rename data/{en_US/en_US => enus}/price.json (100%) rename data/{en_US/en_US => enus}/sentence.json (100%) rename data/{en_US/en_US => enus}/sex.json (100%) rename data/{en_US/en_US => enus}/sex.tsv (100%) rename data/{en_US/en_US => enus}/ssn.json (100%) rename data/{en_US/en_US => enus}/time.json (100%) rename data/{en_US/en_US => enus}/title.json (100%) rename data/{en_US/en_US => enus}/title.tsv (100%) rename data/{en_US/en_US => enus}/url.json (100%) rename data/{en_US/en_US => enus}/username.json (100%) rename data/{en_US/en_US => enus}/version.json (100%) rename data/{en_US/en_US => enus}/word.json (100%) delete mode 100644 data/geo/SE/fs.go delete mode 100644 data/geo/US/fs.go rename data/{geo/SE => geose}/.fejkdata.json (98%) rename data/{geo/SE/geo/SE => geose}/address.json (100%) create mode 100644 data/geose/fs.go rename data/{geo/SE/geo/SE => geose}/locality.json (100%) rename data/{geo/SE/geo/SE => geose}/locality.tsv (100%) rename data/{geo/SE/geo/SE => geose}/municipality.json (100%) rename data/{geo/SE/geo/SE => geose}/municipality.tsv (100%) rename data/{geo/SE/geo/SE => geose}/postal-code.json (100%) rename data/{geo/SE/geo/SE => geose}/postal-code.tsv (100%) rename data/{geo/SE/geo/SE => geose}/region.json (100%) rename data/{geo/SE/geo/SE => geose}/region.tsv (100%) rename data/{geo/SE/geo/SE => geose}/street.json (100%) rename data/{geo/SE/geo/SE => geose}/street.tsv (100%) rename data/{geo/US => geous}/.fejkdata.json (98%) rename data/{geo/US/geo/US => geous}/address.json (100%) create mode 100644 data/geous/fs.go rename data/{geo/US/geo/US => geous}/locality.json (100%) rename data/{geo/US/geo/US => geous}/locality.tsv (100%) rename data/{geo/US/geo/US => geous}/municipality.json (100%) rename data/{geo/US/geo/US => geous}/municipality.tsv (100%) rename data/{geo/US/geo/US => geous}/postal-code.json (100%) rename data/{geo/US/geo/US => geous}/postal-code.tsv (100%) rename data/{geo/US/geo/US => geous}/region.json (100%) rename data/{geo/US/geo/US => geous}/region.tsv (100%) rename data/{geo/US/geo/US => geous}/street.json (100%) rename data/{geo/US/geo/US => geous}/street.tsv (100%) rename data/misc/{misc => }/car.json (100%) rename data/misc/{misc => }/car.tsv (100%) rename data/misc/{misc => }/coordinate.json (100%) rename data/misc/{misc => }/creditcard.json (100%) rename data/misc/{misc => }/currency.json (100%) rename data/misc/{misc => }/currency.tsv (100%) rename data/misc/{misc => }/datetime.json (100%) rename data/misc/{misc => }/emoji.json (100%) rename data/misc/{misc => }/httpmethod.json (100%) rename data/misc/{misc => }/httpmethod.tsv (100%) rename data/misc/{misc => }/httpstatus.json (100%) rename data/misc/{misc => }/httpstatus.tsv (100%) rename data/misc/{misc => }/language.json (100%) rename data/misc/{misc => }/language.tsv (100%) rename data/misc/{misc => }/loglevel.json (100%) rename data/misc/{misc => }/loglevel.tsv (100%) rename data/misc/{misc => }/mac.json (100%) rename data/misc/{misc => }/mimetype.json (100%) rename data/misc/{misc => }/mimetype.tsv (100%) rename data/misc/{misc => }/objectid.json (100%) rename data/misc/{misc => }/port.json (100%) rename data/misc/{misc => }/port.tsv (100%) rename data/misc/{misc => }/protocol.json (100%) rename data/misc/{misc => }/protocol.tsv (100%) rename data/misc/{misc => }/territory.json (100%) rename data/misc/{misc => }/territory.tsv (100%) rename data/misc/{misc => }/timezone.json (100%) rename data/misc/{misc => }/timezone.tsv (100%) rename data/misc/{misc => }/tld.json (100%) rename data/misc/{misc => }/tld.tsv (100%) rename data/misc/{misc => }/useragent.json (100%) rename data/misc/{misc => }/useragent.tsv (100%) rename data/misc/{misc => }/uuid.json (100%) delete mode 100644 data/sv_SE/fs.go rename data/{sv_SE => svse}/.fejkdata.json (94%) rename data/{sv_SE/sv_SE => svse}/address.json (100%) rename data/{sv_SE/sv_SE => svse}/birth-number.json (100%) rename data/{sv_SE/sv_SE => svse}/birth-number.tsv (100%) rename data/{sv_SE/sv_SE => svse}/color.json (100%) rename data/{sv_SE/sv_SE => svse}/company.json (100%) rename data/{sv_SE/sv_SE => svse}/date.json (100%) rename data/{sv_SE/sv_SE => svse}/email.json (100%) rename data/{sv_SE/sv_SE => svse}/first-name.json (100%) rename data/{sv_SE/sv_SE => svse}/first-name.tsv (100%) create mode 100644 data/svse/fs.go rename data/{sv_SE/sv_SE => svse}/ip.json (100%) rename data/{sv_SE/sv_SE => svse}/last-name.json (100%) rename data/{sv_SE/sv_SE => svse}/last-name.tsv (100%) rename data/{sv_SE/sv_SE => svse}/person.json (100%) rename data/{sv_SE/sv_SE => svse}/personnummer.json (100%) rename data/{sv_SE/sv_SE => svse}/phone.json (100%) rename data/{sv_SE/sv_SE => svse}/price.json (100%) rename data/{sv_SE/sv_SE => svse}/samordningsnummer.json (100%) rename data/{sv_SE/sv_SE => svse}/sentence.json (100%) rename data/{sv_SE/sv_SE => svse}/sex.json (100%) rename data/{sv_SE/sv_SE => svse}/sex.tsv (100%) rename data/{sv_SE/sv_SE => svse}/time.json (100%) rename data/{sv_SE/sv_SE => svse}/title.json (100%) rename data/{sv_SE/sv_SE => svse}/title.tsv (100%) rename data/{sv_SE/sv_SE => svse}/url.json (100%) rename data/{sv_SE/sv_SE => svse}/username.json (100%) rename data/{sv_SE/sv_SE => svse}/version.json (100%) rename data/{sv_SE/sv_SE => svse}/word.json (100%) diff --git a/AGENTS.md b/AGENTS.md index dc1af32..bb4711b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,6 +47,7 @@ In [docs/decisions.md](docs/decisions.md): - `FakeTemplate` and `NewTemplate` both stay - The shipped data is embedded, not discovered - The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0 +- A shipped package is named by its namespace in lower case without separators, and its manifest mounts it there with `namespace` - A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads - Every `{…}` draws afresh, and only a name keeps a pick - Reference sigils follow the filesystem diff --git a/CHANGELOG.md b/CHANGELOG.md index c79fd4f..68377f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,13 +126,14 @@ replacement, and each removed path, column or flag. `sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record that gives it one. - The shipped data comes in Go packages, one per module, each exporting `FS`: - `github.com/larvit/fejkdata/data/en_US`, `…/sv_SE`, `…/misc`, `…/geo/SE` and - `…/geo/US`. `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one, + `github.com/larvit/fejkdata/data/enus`, `…/svse`, `…/misc`, `…/geose` and + `…/geous`. `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one, and the CLI carries them all. `New` loads the modules `WithDataFS(modules...)` names and the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option - is given. A manifest's `reads` lists the modules its source reads by default, and a - reference to a path nothing provides fails to load, naming every module the loaded - manifests read, or saying there are none. + is given. A manifest's `namespace` mounts its source under a dot path, and its `reads` lists the + modules its source reads by default; a reference to a path nothing provides fails to + load, asking whether a module is missing and naming every module the loaded manifests + read, or saying there are none. - A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each category on the first call reaching it, as each shipped module does. Data failing to load, in `New` or on that first call, gives an error matching `ErrLoad`, and the CLI exits 1. diff --git a/DATA-LICENSES.md b/DATA-LICENSES.md index f8e527f..95b8471 100644 --- a/DATA-LICENSES.md +++ b/DATA-LICENSES.md @@ -6,15 +6,15 @@ Every shipped dataset, its source, its licence and the attribution it asks for. | Table | Source | Licence | Attribution | Rebuild | |-------|--------|---------|-------------|---------| -| `geo/SE/locality.tsv`, `postal-code.tsv` | [GeoNames](https://www.geonames.org/) postal codes for SE; populations from SCB tätorter 2023 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); CC0 1.0 | "Postal codes from GeoNames, www.geonames.org" | `data-import/geo-se.py` | -| `geo/SE/region.tsv`, `municipality.tsv` | [SCB](https://www.scb.se/) län and kommun codes 2026 and population 2024 | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) | none required | `data-import/geo-se.py` | -| `geo/SE/street.tsv` | [Trafikverket NVDB](https://www.trafikverket.se/) Gatunamn, through the open API | CC0 1.0 | none required | `data-import/geo-se.py` | -| `geo/US/region.tsv`, `municipality.tsv`, `locality.tsv` | [Census Bureau](https://www.census.gov/) Gazetteer 2026 and population estimates 2025 | [public domain](https://www.usa.gov/government-works) | none required | `data-import/geo-us.py` | -| `geo/US/postal-code.tsv`, `street.tsv` | Census Bureau ZCTA to place relationships 2020 and TIGER/Line 2025 address ranges and feature names | public domain | none required | `data-import/geo-us.py` | -| `sv_SE/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` | -| `en_US/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` | -| `en_US/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` | -| `sv_SE/sex.tsv`, `sv_SE/birth-number.tsv`, `sv_SE/title.tsv`, `en_US/sex.tsv`, `en_US/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 14) | — | — | — | +| `geose/locality.tsv`, `postal-code.tsv` | [GeoNames](https://www.geonames.org/) postal codes for SE; populations from SCB tätorter 2023 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); CC0 1.0 | "Postal codes from GeoNames, www.geonames.org" | `data-import/geo-se.py` | +| `geose/region.tsv`, `municipality.tsv` | [SCB](https://www.scb.se/) län and kommun codes 2026 and population 2024 | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) | none required | `data-import/geo-se.py` | +| `geose/street.tsv` | [Trafikverket NVDB](https://www.trafikverket.se/) Gatunamn, through the open API | CC0 1.0 | none required | `data-import/geo-se.py` | +| `geous/region.tsv`, `municipality.tsv`, `locality.tsv` | [Census Bureau](https://www.census.gov/) Gazetteer 2026 and population estimates 2025 | [public domain](https://www.usa.gov/government-works) | none required | `data-import/geo-us.py` | +| `geous/postal-code.tsv`, `street.tsv` | Census Bureau ZCTA to place relationships 2020 and TIGER/Line 2025 address ranges and feature names | public domain | none required | `data-import/geo-us.py` | +| `svse/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` | +| `enus/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` | +| `enus/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` | +| `svse/sex.tsv`, `svse/birth-number.tsv`, `svse/title.tsv`, `enus/sex.tsv`, `enus/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 14) | — | — | — | | `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` | | `misc/httpmethod.tsv` | [IANA HTTP Method Registry](https://www.iana.org/assignments/http-methods/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpmethod.py` | | `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` | diff --git a/README.md b/README.md index 8f0296a..6afb845 100644 --- a/README.md +++ b/README.md @@ -160,15 +160,14 @@ it to a name and read the name ## Data -The shipped data comes in modules, each a Go package under [`data/`](data): one per -locale (`en_US`, `sv_SE`), one per country's tree of places (`geo/SE`, `geo/US`), and -the locale-neutral `misc`. Each embeds its files, so nothing is read from disk. The CLI -carries every module, and a library loads those it passes to `WithDataFS` -([Library](#library)). A -directory is a namespace: each JSON file is a category named after the file, each -subdirectory a dot-path segment, so `mydata/sv_SE/person.json` is `sv_SE.person` and -replaces the shipped one. A module's tree spells its namespace the same way, so -`sv_SE`'s person is `data/sv_SE/sv_SE/person.json`. +The shipped data comes in modules, each a Go package under [`data/`](data) named by its +namespace in lower case without separators: one per locale (`enus` for `en_US`, `svse` +for `sv_SE`), one per country's tree of places (`geose` for `geo.SE`, `geous` for +`geo.US`), and the locale-neutral `misc`. Each embeds its files, so nothing is read from +disk. The CLI carries every module, and a library loads those it passes to `WithDataFS` +([Library](#library)). A directory is a namespace: each JSON file is a category named +after the file, each subdirectory a dot-path segment, so `mydata/sv_SE/person.json` is +`sv_SE.person` and replaces the shipped one. Sources merge in order; matching folders combine, any other clash is won by the last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"`, `/` or ` as `, which binds a [name](#names), nor be `-`, which a struct tag reserves. A dot-prefixed @@ -185,19 +184,24 @@ every category of its source, so a category file it leaves out never loads. The index maps each category's dot path to an entry: `parent`, the table it links to, which needs its own entry in the same folder; and `paths`, every path `List` advertises below the category, `""` for the category itself. Each shipped module's manifest, such as -[`data/sv_SE/.fejkdata.json`](data/sv_SE/.fejkdata.json), is one, so delete it from a +[`data/svse/.fejkdata.json`](data/svse/.fejkdata.json), is one, so delete it from a copy you edit, or keep its index current. A table whose parent column differs from its entry's `parent` fails at first reach, and a table linking under another source's table needs a source without an index. `//go:embed` of a directory leaves the manifest out, and the source then loads whole in `New`; name it in the pattern too. +A manifest's `namespace` is the dot path its source's root sits under: a folder holding +`person.json` with `"namespace": "sv_SE"` provides `sv_SE.person`, its subfolders adding +segments below that. Each shipped module carries one, so `data/svse/person.json` is +`sv_SE.person`. + A manifest's `reads` lists, by any name its author chooses, every module the source's references read unless another source provides the same paths, directly or through -another module; a manifest may carry it without an index. Nothing loads these modules: -pass them yourself. A shipped module names a Go package: `sv_SE`'s manifest reads -`github.com/larvit/fejkdata/data/geo/SE`. Data whose reference names a path nothing -provides fails to load, naming every module the loaded manifests read, or saying they -name none. +another module. Nothing loads these modules: pass them yourself. A shipped module names a +Go package: `svse`'s manifest reads `github.com/larvit/fejkdata/data/geose`. Data whose +reference names a path nothing provides fails to load, asking whether a module is +missing and naming every module the loaded manifests read, or saying they name none. +`namespace` and `reads` need no index. Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`, `ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`, @@ -366,14 +370,14 @@ only the ones you read, pass each package's `FS` with the modules its manifest r ```go import ( "github.com/larvit/fejkdata" - "github.com/larvit/fejkdata/data/geo/SE" - "github.com/larvit/fejkdata/data/sv_SE" + "github.com/larvit/fejkdata/data/geose" + "github.com/larvit/fejkdata/data/svse" ) -f, err := fejkdata.New(fejkdata.WithDataFS(sv_SE.FS, SE.FS)) +f, err := fejkdata.New(fejkdata.WithDataFS(svse.FS, geose.FS)) ``` -`en_US` reads `geo/US`; `misc` reads none. `New` fails where no `WithDataFS` or +`enus` reads `geous`; `misc` reads none. `New` fails where no `WithDataFS` or `WithDataPath` names the data to load. `New` refuses data that breaks the grammar, could mean two things or cannot render a valid @@ -793,7 +797,7 @@ A reference renders a node from elsewhere in the data — the path `Fake` takes, across every loaded source — so one category borrows another. The sigil says where the path starts, as in a filesystem: `{/en_US.person}` from the data root, `{.username}` from the folder this file sits in, `{..username}` from the folder -above. `data/sv_SE/sv_SE/email.json` can therefore read its own locale's `username` +above. `sv_SE.email` can therefore read its own locale's `username` without naming `sv_SE`: ```json diff --git a/data-import/currency.py b/data-import/currency.py index c8bd05f..3417a7e 100644 --- a/data-import/currency.py +++ b/data-import/currency.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/currency.tsv from datasets/currency-codes (PDDL) and CLDR's symbols (Unicode). +"""Rebuild data/misc/currency.tsv from datasets/currency-codes (PDDL) and CLDR's symbols (Unicode). data-import/currency.py [--source URL_OR_FILE] [--symbols URL_OR_FILE ...] [--cache DIR] [--out FILE] @@ -19,7 +19,7 @@ SYMBOLS = [ "https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/en.xml", "https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/root.xml", ] -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "currency.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "currency.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["code", "decimals", "name", "numeric", "symbol"] diff --git a/data-import/geo-se.py b/data-import/geo-se.py index 8f2012b..9e1f254 100644 --- a/data-import/geo-se.py +++ b/data-import/geo-se.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/geo/SE/geo/SE/*.tsv from SCB (CC0), GeoNames (CC BY 4.0) and Trafikverket NVDB (CC0). +"""Rebuild data/geose/*.tsv from SCB (CC0), GeoNames (CC BY 4.0) and Trafikverket NVDB (CC0). TRAFIKVERKET_API_KEY=… data-import/geo-se.py [--key-file FILE] [--cache DIR] [--streets-per-locality N] [--out DIR] """ diff --git a/data-import/geo-us.py b/data-import/geo-us.py index a5fd319..bc78af1 100644 --- a/data-import/geo-us.py +++ b/data-import/geo-us.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/geo/US/geo/US/*.tsv from the Census Bureau's Gazetteer, population estimates, ZCTA relationships and TIGER/Line files (public domain). +"""Rebuild data/geous/*.tsv from the Census Bureau's Gazetteer, population estimates, ZCTA relationships and TIGER/Line files (public domain). data-import/geo-us.py [--cache DIR] [--min-population N] [--streets-per-locality N] [--out DIR] """ diff --git a/data-import/geo.py b/data-import/geo.py index 8e69ad8..53b6100 100644 --- a/data-import/geo.py +++ b/data-import/geo.py @@ -5,14 +5,14 @@ import math import sys from pathlib import Path -DATA = Path(__file__).resolve().parent.parent / "data" / "geo" +DATA = Path(__file__).resolve().parent.parent / "data" CACHE = Path(__file__).resolve().parent / "cache" def parser(doc, country, **own): """A parser for the options every country's import takes, plus the script's own: each keyword names one option, with "_" standing for "-".""" - options = {"cache": {"default": str(CACHE)}, "out": {"default": str(DATA / country / "geo" / country)}, "streets_per_locality": {"type": int, "default": 10}, **own} + options = {"cache": {"default": str(CACHE)}, "out": {"default": str(DATA / ("geo" + country.lower()))}, "streets_per_locality": {"type": int, "default": 10}, **own} p = argparse.ArgumentParser(description=doc.splitlines()[0]) for name in sorted(options): p.add_argument("--" + name.replace("_", "-"), **options[name]) diff --git a/data-import/httpmethod.py b/data-import/httpmethod.py index 0462ddf..0919eef 100644 --- a/data-import/httpmethod.py +++ b/data-import/httpmethod.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/httpmethod.tsv from the IANA HTTP Method Registry. +"""Rebuild data/misc/httpmethod.tsv from the IANA HTTP Method Registry. data-import/httpmethod.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] @@ -18,7 +18,7 @@ import source import tsv SOURCE = "https://www.iana.org/assignments/http-methods/methods.csv" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "httpmethod.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpmethod.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["idempotent", "method", "safe"] CORE = re.compile(r"\[RFC9110, Section 9\.3\.|\[RFC5789, Section 2\]") diff --git a/data-import/httpstatus.py b/data-import/httpstatus.py index 6ea9cfc..d2ed736 100644 --- a/data-import/httpstatus.py +++ b/data-import/httpstatus.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/httpstatus.tsv from the IANA HTTP Status Code Registry. +"""Rebuild data/misc/httpstatus.tsv from the IANA HTTP Status Code Registry. data-import/httpstatus.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] @@ -14,7 +14,7 @@ import source import tsv SOURCE = "https://www.iana.org/assignments/http-status-codes/http-status-codes-1.csv" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "httpstatus.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpstatus.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["code", "reason"] diff --git a/data-import/language.py b/data-import/language.py index 6018ae2..203872b 100644 --- a/data-import/language.py +++ b/data-import/language.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/language.tsv from datasets/language-codes (PDDL), the LoC ISO 639-2 register. +"""Rebuild data/misc/language.tsv from datasets/language-codes (PDDL), the LoC ISO 639-2 register. data-import/language.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] @@ -15,7 +15,7 @@ import source import tsv SOURCE = "https://raw.githubusercontent.com/datasets/language-codes/main/data/language-codes-full.csv" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "language.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "language.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["code", "code3", "name"] diff --git a/data-import/loglevel.py b/data-import/loglevel.py index d080a16..d9da6d5 100644 --- a/data-import/loglevel.py +++ b/data-import/loglevel.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/loglevel.tsv from RFC 5424 and POSIX . +"""Rebuild data/misc/loglevel.tsv from RFC 5424 and POSIX . data-import/loglevel.py [--source URL_OR_FILE] [--posix URL_OR_FILE] [--cache DIR] [--out FILE] @@ -18,7 +18,7 @@ import tsv SOURCE = "https://www.rfc-editor.org/rfc/rfc5424.txt" POSIX = "https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/syslog.h.html" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "loglevel.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "loglevel.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["code", "keyword", "severity"] TABLE = re.compile(r"Table 1\.\s+Syslog Message Facilities(.*?)Table 2\.\s+Syslog Message Severities", re.S) diff --git a/data-import/mimetype.py b/data-import/mimetype.py index 45c96ee..b749f66 100644 --- a/data-import/mimetype.py +++ b/data-import/mimetype.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/mimetype.tsv from mime-db (MIT), the IANA media type registry with extensions. +"""Rebuild data/misc/mimetype.tsv from mime-db (MIT), the IANA media type registry with extensions. data-import/mimetype.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] @@ -13,7 +13,7 @@ import source import tsv SOURCE = "https://raw.githubusercontent.com/jshttp/mime-db/master/db.json" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "mimetype.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "mimetype.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["ext", "type"] FIXUPS = {"application/mp4": "mp4s", "audio/mpeg": "mp3", "audio/ogg": "ogg", "video/quicktime": "mov"} diff --git a/data-import/names-se.py b/data-import/names-se.py index 74ecde4..e59c198 100644 --- a/data-import/names-se.py +++ b/data-import/names-se.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/sv_SE/sv_SE/first-name.tsv and last-name.tsv from SCB's whole-population name counts (CC0, "Källa: SCB"). +"""Rebuild data/svse/first-name.tsv and last-name.tsv from SCB's whole-population name counts (CC0, "Källa: SCB"). data-import/names-se.py [--source URL_OR_FILE] [--first N] [--last N] [--cache DIR] [--out DIR] @@ -14,7 +14,7 @@ import tsv import xlsx SOURCE = "https://www.scb.se/contentassets/9fe7dbb460994c72b835163dbc491ef9/namn-med-minst-tva-barare-31-december-2022.xlsx" -OUT = Path(__file__).resolve().parent.parent / "data" / "sv_SE" / "sv_SE" +OUT = Path(__file__).resolve().parent.parent / "data" / "svse" CACHE = Path(__file__).resolve().parent / "cache" SHEETS = {"f": "Tilltalsnamn kvinnor", "m": "Tilltalsnamn män"} SURNAMES = "Efternamn" diff --git a/data-import/names-us.py b/data-import/names-us.py index 993053c..94b81a1 100644 --- a/data-import/names-us.py +++ b/data-import/names-us.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/en_US/en_US/first-name.tsv and last-name.tsv from SSA baby names and the Census 2010 surnames (public domain). +"""Rebuild data/enus/first-name.tsv and last-name.tsv from SSA baby names and the Census 2010 surnames (public domain). data-import/names-us.py [--names URL_OR_FILE] [--surnames URL_OR_FILE] [--from-year YEAR] [--first N] [--last N] [--cache DIR] [--out DIR] @@ -20,7 +20,7 @@ import tsv NAMES = "https://raw.githubusercontent.com/hackerb9/ssa-baby-names/master/alldata.txt" SURNAMES = "https://www2.census.gov/topics/genealogy/2010surnames/names.zip" -OUT = Path(__file__).resolve().parent.parent / "data" / "en_US" / "en_US" +OUT = Path(__file__).resolve().parent.parent / "data" / "enus" CACHE = Path(__file__).resolve().parent / "cache" MC = re.compile(r"^Mc([a-z])") PLACEHOLDERS = {"Baby", "Female", "Infant", "Male", "Notnamed", "Unknown", "Unnamed"} diff --git a/data-import/port.py b/data-import/port.py index 68a3a0b..9525a3a 100644 --- a/data-import/port.py +++ b/data-import/port.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry. +"""Rebuild data/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry. data-import/port.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] @@ -17,7 +17,7 @@ import source import tsv SOURCE = "https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.csv" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "port.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "port.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["number", "service"] FLOOR = 4000 diff --git a/data-import/protocol.py b/data-import/protocol.py index 8b45159..c4cb73d 100644 --- a/data-import/protocol.py +++ b/data-import/protocol.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/protocol.tsv from the IANA Protocol Numbers registry. +"""Rebuild data/misc/protocol.tsv from the IANA Protocol Numbers registry. data-import/protocol.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] @@ -17,7 +17,7 @@ import source import tsv SOURCE = "https://www.iana.org/assignments/protocol-numbers/protocol-numbers-1.csv" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "protocol.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "protocol.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["keyword", "name", "number"] SKIP = ("Reserved", "deprecated") diff --git a/data-import/territory.py b/data-import/territory.py index 9a203ce..be8723b 100644 --- a/data-import/territory.py +++ b/data-import/territory.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/territory.tsv from datasets/country-codes (PDDL). +"""Rebuild data/misc/territory.tsv from datasets/country-codes (PDDL). data-import/territory.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] """ @@ -14,7 +14,7 @@ import source import tsv SOURCE = "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "territory.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "territory.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["alpha2", "alpha3", "calling-code", "capital", "country", "currency", "flag", "languages", "name", "numeric", "tld"] SOVEREIGN = re.compile(r"^(?:Part of|Territor(?:y|ies) of|Crown dependency of|Commonwealth of|Associated with) ([A-Z]{2})$") diff --git a/data-import/timezone.py b/data-import/timezone.py index bbdb895..e5535c7 100644 --- a/data-import/timezone.py +++ b/data-import/timezone.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/timezone.tsv from the IANA tzdb tarball (public domain) and GeoNames (CC BY 4.0). +"""Rebuild data/misc/timezone.tsv from the IANA tzdb tarball (public domain) and GeoNames (CC BY 4.0). data-import/timezone.py [--source URL_OR_FILE] [--cities URL_OR_FILE] [--cache DIR] [--out FILE] [--territories FILE] @@ -22,8 +22,8 @@ import tsv SOURCE = "https://data.iana.org/time-zones/tzdata-latest.tar.gz" CITIES = "https://download.geonames.org/export/dump/cities15000.zip" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "timezone.tsv" -TERRITORIES = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "territory.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "timezone.tsv" +TERRITORIES = Path(__file__).resolve().parent.parent / "data" / "misc" / "territory.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["offset", "territory", "weight", "zone"] REGIONS = ["africa", "antarctica", "asia", "australasia", "backward", "etcetera", "europe", "northamerica", "southamerica"] diff --git a/data-import/tld.py b/data-import/tld.py index c4da318..4ff8820 100644 --- a/data-import/tld.py +++ b/data-import/tld.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/tld.tsv from the IANA Root Zone Database. +"""Rebuild data/misc/tld.tsv from the IANA Root Zone Database. data-import/tld.py [--source URL_OR_FILE] [--zone URL_OR_FILE] [--cache DIR] [--out FILE] @@ -20,7 +20,7 @@ import tsv SOURCE = "https://www.iana.org/domains/root/db" ZONE = "https://data.iana.org/TLD/tlds-alpha-by-domain.txt" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "tld.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "tld.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["tld", "type", "unicode"] TYPES = ("country-code", "generic", "generic-restricted", "infrastructure", "sponsored") diff --git a/data-import/useragent.py b/data-import/useragent.py index 45b2805..5262d93 100644 --- a/data-import/useragent.py +++ b/data-import/useragent.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Rebuild data/misc/misc/useragent.tsv from microlinkhq/top-user-agents (MIT). +"""Rebuild data/misc/useragent.tsv from microlinkhq/top-user-agents (MIT). data-import/useragent.py [--source URL] [--cache DIR] [--out FILE] @@ -16,7 +16,7 @@ import source import tsv SOURCE = "https://raw.githubusercontent.com/microlinkhq/top-user-agents/master/src" -OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "useragent.tsv" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "useragent.tsv" CACHE = Path(__file__).resolve().parent / "cache" COLUMNS = ["browser", "device", "os", "ua"] # First match wins: a Chromium fork this list does not name ships as Chrome. diff --git a/data.go b/data.go index 2cd962f..684cb49 100644 --- a/data.go +++ b/data.go @@ -34,6 +34,7 @@ func loadSources(sources []datafiles.Source) (folder, []string, error) { } func addSource(root *folder, src datafiles.Source, m datafiles.Manifest) error { + src = src.Mounted(m.Namespace) if m.Index != nil { placeIndex(root, src, m.Index) return nil diff --git a/data/en_US/fs.go b/data/en_US/fs.go deleted file mode 100644 index b577339..0000000 --- a/data/en_US/fs.go +++ /dev/null @@ -1,9 +0,0 @@ -// Package en_US is fejkdata's en_US locale module. -package en_US - -import "embed" - -// FS holds the en_US categories and the module's manifest; pass it to fejkdata.WithDataFS. -// -//go:embed en_US .fejkdata.json -var FS embed.FS diff --git a/data/en_US/.fejkdata.json b/data/enus/.fejkdata.json similarity index 94% rename from data/en_US/.fejkdata.json rename to data/enus/.fejkdata.json index 6c20291..7f9c520 100644 --- a/data/en_US/.fejkdata.json +++ b/data/enus/.fejkdata.json @@ -23,5 +23,6 @@ "en_US.version": {"paths":["","n","pre","suffix"]}, "en_US.word": {"paths":[""]} }, - "reads": ["github.com/larvit/fejkdata/data/geo/US"] + "namespace": "en_US", + "reads": ["github.com/larvit/fejkdata/data/geous"] } diff --git a/data/en_US/en_US/address.json b/data/enus/address.json similarity index 100% rename from data/en_US/en_US/address.json rename to data/enus/address.json diff --git a/data/en_US/en_US/color.json b/data/enus/color.json similarity index 100% rename from data/en_US/en_US/color.json rename to data/enus/color.json diff --git a/data/en_US/en_US/company.json b/data/enus/company.json similarity index 100% rename from data/en_US/en_US/company.json rename to data/enus/company.json diff --git a/data/en_US/en_US/date.json b/data/enus/date.json similarity index 100% rename from data/en_US/en_US/date.json rename to data/enus/date.json diff --git a/data/en_US/en_US/email.json b/data/enus/email.json similarity index 100% rename from data/en_US/en_US/email.json rename to data/enus/email.json diff --git a/data/en_US/en_US/first-name.json b/data/enus/first-name.json similarity index 100% rename from data/en_US/en_US/first-name.json rename to data/enus/first-name.json diff --git a/data/en_US/en_US/first-name.tsv b/data/enus/first-name.tsv similarity index 100% rename from data/en_US/en_US/first-name.tsv rename to data/enus/first-name.tsv diff --git a/data/enus/fs.go b/data/enus/fs.go new file mode 100644 index 0000000..b153735 --- /dev/null +++ b/data/enus/fs.go @@ -0,0 +1,9 @@ +// Package enus is fejkdata's en_US locale module, whose categories are en_US.*. Its address reads geous's places. +package enus + +import "embed" + +// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. +// +//go:embed *.json *.tsv .fejkdata.json +var FS embed.FS diff --git a/data/en_US/en_US/ip.json b/data/enus/ip.json similarity index 100% rename from data/en_US/en_US/ip.json rename to data/enus/ip.json diff --git a/data/en_US/en_US/itin.json b/data/enus/itin.json similarity index 100% rename from data/en_US/en_US/itin.json rename to data/enus/itin.json diff --git a/data/en_US/en_US/last-name.json b/data/enus/last-name.json similarity index 100% rename from data/en_US/en_US/last-name.json rename to data/enus/last-name.json diff --git a/data/en_US/en_US/last-name.tsv b/data/enus/last-name.tsv similarity index 100% rename from data/en_US/en_US/last-name.tsv rename to data/enus/last-name.tsv diff --git a/data/en_US/en_US/person.json b/data/enus/person.json similarity index 100% rename from data/en_US/en_US/person.json rename to data/enus/person.json diff --git a/data/en_US/en_US/phone-exchange.json b/data/enus/phone-exchange.json similarity index 100% rename from data/en_US/en_US/phone-exchange.json rename to data/enus/phone-exchange.json diff --git a/data/en_US/en_US/phone.json b/data/enus/phone.json similarity index 100% rename from data/en_US/en_US/phone.json rename to data/enus/phone.json diff --git a/data/en_US/en_US/price.json b/data/enus/price.json similarity index 100% rename from data/en_US/en_US/price.json rename to data/enus/price.json diff --git a/data/en_US/en_US/sentence.json b/data/enus/sentence.json similarity index 100% rename from data/en_US/en_US/sentence.json rename to data/enus/sentence.json diff --git a/data/en_US/en_US/sex.json b/data/enus/sex.json similarity index 100% rename from data/en_US/en_US/sex.json rename to data/enus/sex.json diff --git a/data/en_US/en_US/sex.tsv b/data/enus/sex.tsv similarity index 100% rename from data/en_US/en_US/sex.tsv rename to data/enus/sex.tsv diff --git a/data/en_US/en_US/ssn.json b/data/enus/ssn.json similarity index 100% rename from data/en_US/en_US/ssn.json rename to data/enus/ssn.json diff --git a/data/en_US/en_US/time.json b/data/enus/time.json similarity index 100% rename from data/en_US/en_US/time.json rename to data/enus/time.json diff --git a/data/en_US/en_US/title.json b/data/enus/title.json similarity index 100% rename from data/en_US/en_US/title.json rename to data/enus/title.json diff --git a/data/en_US/en_US/title.tsv b/data/enus/title.tsv similarity index 100% rename from data/en_US/en_US/title.tsv rename to data/enus/title.tsv diff --git a/data/en_US/en_US/url.json b/data/enus/url.json similarity index 100% rename from data/en_US/en_US/url.json rename to data/enus/url.json diff --git a/data/en_US/en_US/username.json b/data/enus/username.json similarity index 100% rename from data/en_US/en_US/username.json rename to data/enus/username.json diff --git a/data/en_US/en_US/version.json b/data/enus/version.json similarity index 100% rename from data/en_US/en_US/version.json rename to data/enus/version.json diff --git a/data/en_US/en_US/word.json b/data/enus/word.json similarity index 100% rename from data/en_US/en_US/word.json rename to data/enus/word.json diff --git a/data/geo/SE/fs.go b/data/geo/SE/fs.go deleted file mode 100644 index 2baa2a9..0000000 --- a/data/geo/SE/fs.go +++ /dev/null @@ -1,9 +0,0 @@ -// Package SE is fejkdata's Swedish geography module. -package SE - -import "embed" - -// FS holds the geo.SE categories and the module's manifest; pass it to fejkdata.WithDataFS. -// -//go:embed geo .fejkdata.json -var FS embed.FS diff --git a/data/geo/US/fs.go b/data/geo/US/fs.go deleted file mode 100644 index 3d4c606..0000000 --- a/data/geo/US/fs.go +++ /dev/null @@ -1,9 +0,0 @@ -// Package US is fejkdata's US geography module. -package US - -import "embed" - -// FS holds the geo.US categories and the module's manifest; pass it to fejkdata.WithDataFS. -// -//go:embed geo .fejkdata.json -var FS embed.FS diff --git a/data/geo/SE/.fejkdata.json b/data/geose/.fejkdata.json similarity index 98% rename from data/geo/SE/.fejkdata.json rename to data/geose/.fejkdata.json index 85d74a1..2ea3174 100644 --- a/data/geo/SE/.fejkdata.json +++ b/data/geose/.fejkdata.json @@ -6,5 +6,6 @@ "geo.SE.postal-code": {"parent":"locality","paths":["","code","locality"]}, "geo.SE.region": {"paths":["","code","municipality","municipality.code","municipality.locality","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.locality","municipality.locality.street.name","municipality.locality.street.segments","municipality.name","municipality.population","municipality.region","name","population","timezone"]}, "geo.SE.street": {"parent":"locality","paths":["","locality","name","segments"]} - } + }, + "namespace": "geo.SE" } diff --git a/data/geo/SE/geo/SE/address.json b/data/geose/address.json similarity index 100% rename from data/geo/SE/geo/SE/address.json rename to data/geose/address.json diff --git a/data/geose/fs.go b/data/geose/fs.go new file mode 100644 index 0000000..6f0cc20 --- /dev/null +++ b/data/geose/fs.go @@ -0,0 +1,9 @@ +// Package geose is fejkdata's Swedish places module, whose categories are geo.SE.*. +package geose + +import "embed" + +// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. +// +//go:embed *.json *.tsv .fejkdata.json +var FS embed.FS diff --git a/data/geo/SE/geo/SE/locality.json b/data/geose/locality.json similarity index 100% rename from data/geo/SE/geo/SE/locality.json rename to data/geose/locality.json diff --git a/data/geo/SE/geo/SE/locality.tsv b/data/geose/locality.tsv similarity index 100% rename from data/geo/SE/geo/SE/locality.tsv rename to data/geose/locality.tsv diff --git a/data/geo/SE/geo/SE/municipality.json b/data/geose/municipality.json similarity index 100% rename from data/geo/SE/geo/SE/municipality.json rename to data/geose/municipality.json diff --git a/data/geo/SE/geo/SE/municipality.tsv b/data/geose/municipality.tsv similarity index 100% rename from data/geo/SE/geo/SE/municipality.tsv rename to data/geose/municipality.tsv diff --git a/data/geo/SE/geo/SE/postal-code.json b/data/geose/postal-code.json similarity index 100% rename from data/geo/SE/geo/SE/postal-code.json rename to data/geose/postal-code.json diff --git a/data/geo/SE/geo/SE/postal-code.tsv b/data/geose/postal-code.tsv similarity index 100% rename from data/geo/SE/geo/SE/postal-code.tsv rename to data/geose/postal-code.tsv diff --git a/data/geo/SE/geo/SE/region.json b/data/geose/region.json similarity index 100% rename from data/geo/SE/geo/SE/region.json rename to data/geose/region.json diff --git a/data/geo/SE/geo/SE/region.tsv b/data/geose/region.tsv similarity index 100% rename from data/geo/SE/geo/SE/region.tsv rename to data/geose/region.tsv diff --git a/data/geo/SE/geo/SE/street.json b/data/geose/street.json similarity index 100% rename from data/geo/SE/geo/SE/street.json rename to data/geose/street.json diff --git a/data/geo/SE/geo/SE/street.tsv b/data/geose/street.tsv similarity index 100% rename from data/geo/SE/geo/SE/street.tsv rename to data/geose/street.tsv diff --git a/data/geo/US/.fejkdata.json b/data/geous/.fejkdata.json similarity index 98% rename from data/geo/US/.fejkdata.json rename to data/geous/.fejkdata.json index 50cc778..0611ed0 100644 --- a/data/geo/US/.fejkdata.json +++ b/data/geous/.fejkdata.json @@ -6,5 +6,6 @@ "geo.US.postal-code": {"parent":"locality","paths":["","addresses","code","locality"]}, "geo.US.region": {"paths":["","abbr","code","municipality","municipality.code","municipality.locality","municipality.locality.code","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.addresses","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.addresses","municipality.locality.street.locality","municipality.locality.street.name","municipality.name","municipality.population","municipality.region","name","population","timezone"]}, "geo.US.street": {"parent":"locality","paths":["","addresses","locality","name"]} - } + }, + "namespace": "geo.US" } diff --git a/data/geo/US/geo/US/address.json b/data/geous/address.json similarity index 100% rename from data/geo/US/geo/US/address.json rename to data/geous/address.json diff --git a/data/geous/fs.go b/data/geous/fs.go new file mode 100644 index 0000000..1874e6e --- /dev/null +++ b/data/geous/fs.go @@ -0,0 +1,9 @@ +// Package geous is fejkdata's US places module, whose categories are geo.US.*. +package geous + +import "embed" + +// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. +// +//go:embed *.json *.tsv .fejkdata.json +var FS embed.FS diff --git a/data/geo/US/geo/US/locality.json b/data/geous/locality.json similarity index 100% rename from data/geo/US/geo/US/locality.json rename to data/geous/locality.json diff --git a/data/geo/US/geo/US/locality.tsv b/data/geous/locality.tsv similarity index 100% rename from data/geo/US/geo/US/locality.tsv rename to data/geous/locality.tsv diff --git a/data/geo/US/geo/US/municipality.json b/data/geous/municipality.json similarity index 100% rename from data/geo/US/geo/US/municipality.json rename to data/geous/municipality.json diff --git a/data/geo/US/geo/US/municipality.tsv b/data/geous/municipality.tsv similarity index 100% rename from data/geo/US/geo/US/municipality.tsv rename to data/geous/municipality.tsv diff --git a/data/geo/US/geo/US/postal-code.json b/data/geous/postal-code.json similarity index 100% rename from data/geo/US/geo/US/postal-code.json rename to data/geous/postal-code.json diff --git a/data/geo/US/geo/US/postal-code.tsv b/data/geous/postal-code.tsv similarity index 100% rename from data/geo/US/geo/US/postal-code.tsv rename to data/geous/postal-code.tsv diff --git a/data/geo/US/geo/US/region.json b/data/geous/region.json similarity index 100% rename from data/geo/US/geo/US/region.json rename to data/geous/region.json diff --git a/data/geo/US/geo/US/region.tsv b/data/geous/region.tsv similarity index 100% rename from data/geo/US/geo/US/region.tsv rename to data/geous/region.tsv diff --git a/data/geo/US/geo/US/street.json b/data/geous/street.json similarity index 100% rename from data/geo/US/geo/US/street.json rename to data/geous/street.json diff --git a/data/geo/US/geo/US/street.tsv b/data/geous/street.tsv similarity index 100% rename from data/geo/US/geo/US/street.tsv rename to data/geous/street.tsv diff --git a/data/misc/.fejkdata.json b/data/misc/.fejkdata.json index 04289f2..094913f 100644 --- a/data/misc/.fejkdata.json +++ b/data/misc/.fejkdata.json @@ -20,5 +20,6 @@ "misc.tld": {"paths":["","tld","type","unicode"]}, "misc.useragent": {"paths":["","browser","device","os","ua"]}, "misc.uuid": {"paths":["","variant"]} - } + }, + "namespace": "misc" } diff --git a/data/misc/misc/car.json b/data/misc/car.json similarity index 100% rename from data/misc/misc/car.json rename to data/misc/car.json diff --git a/data/misc/misc/car.tsv b/data/misc/car.tsv similarity index 100% rename from data/misc/misc/car.tsv rename to data/misc/car.tsv diff --git a/data/misc/misc/coordinate.json b/data/misc/coordinate.json similarity index 100% rename from data/misc/misc/coordinate.json rename to data/misc/coordinate.json diff --git a/data/misc/misc/creditcard.json b/data/misc/creditcard.json similarity index 100% rename from data/misc/misc/creditcard.json rename to data/misc/creditcard.json diff --git a/data/misc/misc/currency.json b/data/misc/currency.json similarity index 100% rename from data/misc/misc/currency.json rename to data/misc/currency.json diff --git a/data/misc/misc/currency.tsv b/data/misc/currency.tsv similarity index 100% rename from data/misc/misc/currency.tsv rename to data/misc/currency.tsv diff --git a/data/misc/misc/datetime.json b/data/misc/datetime.json similarity index 100% rename from data/misc/misc/datetime.json rename to data/misc/datetime.json diff --git a/data/misc/misc/emoji.json b/data/misc/emoji.json similarity index 100% rename from data/misc/misc/emoji.json rename to data/misc/emoji.json diff --git a/data/misc/fs.go b/data/misc/fs.go index 352da1e..b7810ca 100644 --- a/data/misc/fs.go +++ b/data/misc/fs.go @@ -1,9 +1,9 @@ -// Package misc is fejkdata's misc module. +// Package misc is fejkdata's locale-neutral module, whose categories are misc.*. package misc import "embed" -// FS holds the misc categories and the module's manifest; pass it to fejkdata.WithDataFS. +// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. // -//go:embed misc .fejkdata.json +//go:embed *.json *.tsv .fejkdata.json var FS embed.FS diff --git a/data/misc/misc/httpmethod.json b/data/misc/httpmethod.json similarity index 100% rename from data/misc/misc/httpmethod.json rename to data/misc/httpmethod.json diff --git a/data/misc/misc/httpmethod.tsv b/data/misc/httpmethod.tsv similarity index 100% rename from data/misc/misc/httpmethod.tsv rename to data/misc/httpmethod.tsv diff --git a/data/misc/misc/httpstatus.json b/data/misc/httpstatus.json similarity index 100% rename from data/misc/misc/httpstatus.json rename to data/misc/httpstatus.json diff --git a/data/misc/misc/httpstatus.tsv b/data/misc/httpstatus.tsv similarity index 100% rename from data/misc/misc/httpstatus.tsv rename to data/misc/httpstatus.tsv diff --git a/data/misc/misc/language.json b/data/misc/language.json similarity index 100% rename from data/misc/misc/language.json rename to data/misc/language.json diff --git a/data/misc/misc/language.tsv b/data/misc/language.tsv similarity index 100% rename from data/misc/misc/language.tsv rename to data/misc/language.tsv diff --git a/data/misc/misc/loglevel.json b/data/misc/loglevel.json similarity index 100% rename from data/misc/misc/loglevel.json rename to data/misc/loglevel.json diff --git a/data/misc/misc/loglevel.tsv b/data/misc/loglevel.tsv similarity index 100% rename from data/misc/misc/loglevel.tsv rename to data/misc/loglevel.tsv diff --git a/data/misc/misc/mac.json b/data/misc/mac.json similarity index 100% rename from data/misc/misc/mac.json rename to data/misc/mac.json diff --git a/data/misc/misc/mimetype.json b/data/misc/mimetype.json similarity index 100% rename from data/misc/misc/mimetype.json rename to data/misc/mimetype.json diff --git a/data/misc/misc/mimetype.tsv b/data/misc/mimetype.tsv similarity index 100% rename from data/misc/misc/mimetype.tsv rename to data/misc/mimetype.tsv diff --git a/data/misc/misc/objectid.json b/data/misc/objectid.json similarity index 100% rename from data/misc/misc/objectid.json rename to data/misc/objectid.json diff --git a/data/misc/misc/port.json b/data/misc/port.json similarity index 100% rename from data/misc/misc/port.json rename to data/misc/port.json diff --git a/data/misc/misc/port.tsv b/data/misc/port.tsv similarity index 100% rename from data/misc/misc/port.tsv rename to data/misc/port.tsv diff --git a/data/misc/misc/protocol.json b/data/misc/protocol.json similarity index 100% rename from data/misc/misc/protocol.json rename to data/misc/protocol.json diff --git a/data/misc/misc/protocol.tsv b/data/misc/protocol.tsv similarity index 100% rename from data/misc/misc/protocol.tsv rename to data/misc/protocol.tsv diff --git a/data/misc/misc/territory.json b/data/misc/territory.json similarity index 100% rename from data/misc/misc/territory.json rename to data/misc/territory.json diff --git a/data/misc/misc/territory.tsv b/data/misc/territory.tsv similarity index 100% rename from data/misc/misc/territory.tsv rename to data/misc/territory.tsv diff --git a/data/misc/misc/timezone.json b/data/misc/timezone.json similarity index 100% rename from data/misc/misc/timezone.json rename to data/misc/timezone.json diff --git a/data/misc/misc/timezone.tsv b/data/misc/timezone.tsv similarity index 100% rename from data/misc/misc/timezone.tsv rename to data/misc/timezone.tsv diff --git a/data/misc/misc/tld.json b/data/misc/tld.json similarity index 100% rename from data/misc/misc/tld.json rename to data/misc/tld.json diff --git a/data/misc/misc/tld.tsv b/data/misc/tld.tsv similarity index 100% rename from data/misc/misc/tld.tsv rename to data/misc/tld.tsv diff --git a/data/misc/misc/useragent.json b/data/misc/useragent.json similarity index 100% rename from data/misc/misc/useragent.json rename to data/misc/useragent.json diff --git a/data/misc/misc/useragent.tsv b/data/misc/useragent.tsv similarity index 100% rename from data/misc/misc/useragent.tsv rename to data/misc/useragent.tsv diff --git a/data/misc/misc/uuid.json b/data/misc/uuid.json similarity index 100% rename from data/misc/misc/uuid.json rename to data/misc/uuid.json diff --git a/data/modules.go b/data/modules.go index 931f3ae..25d3796 100644 --- a/data/modules.go +++ b/data/modules.go @@ -1,20 +1,19 @@ -// Package data lists every module fejkdata ships. A module's tree spells its namespace: -// data/geo/SE embeds geo/SE/…, whose categories are geo.SE.*. +// Package data lists every module fejkdata ships. package data import ( "io/fs" - "github.com/larvit/fejkdata/data/en_US" - "github.com/larvit/fejkdata/data/geo/SE" - "github.com/larvit/fejkdata/data/geo/US" + "github.com/larvit/fejkdata/data/enus" + "github.com/larvit/fejkdata/data/geose" + "github.com/larvit/fejkdata/data/geous" "github.com/larvit/fejkdata/data/misc" - "github.com/larvit/fejkdata/data/sv_SE" + "github.com/larvit/fejkdata/data/svse" ) //go:generate env REGENERATE=1 go test -run ^TestShippedManifestsAreCurrent$ .. // Modules returns every module fejkdata ships, to pass to fejkdata.WithDataFS. func Modules() []fs.FS { - return []fs.FS{en_US.FS, SE.FS, US.FS, misc.FS, sv_SE.FS} + return []fs.FS{enus.FS, geose.FS, geous.FS, misc.FS, svse.FS} } diff --git a/data/sv_SE/fs.go b/data/sv_SE/fs.go deleted file mode 100644 index 0042c35..0000000 --- a/data/sv_SE/fs.go +++ /dev/null @@ -1,9 +0,0 @@ -// Package sv_SE is fejkdata's sv_SE locale module. -package sv_SE - -import "embed" - -// FS holds the sv_SE categories and the module's manifest; pass it to fejkdata.WithDataFS. -// -//go:embed sv_SE .fejkdata.json -var FS embed.FS diff --git a/data/sv_SE/.fejkdata.json b/data/svse/.fejkdata.json similarity index 94% rename from data/sv_SE/.fejkdata.json rename to data/svse/.fejkdata.json index 137addf..a8c83c6 100644 --- a/data/sv_SE/.fejkdata.json +++ b/data/svse/.fejkdata.json @@ -23,5 +23,6 @@ "sv_SE.version": {"paths":["","n","pre","suffix"]}, "sv_SE.word": {"paths":[""]} }, - "reads": ["github.com/larvit/fejkdata/data/geo/SE"] + "namespace": "sv_SE", + "reads": ["github.com/larvit/fejkdata/data/geose"] } diff --git a/data/sv_SE/sv_SE/address.json b/data/svse/address.json similarity index 100% rename from data/sv_SE/sv_SE/address.json rename to data/svse/address.json diff --git a/data/sv_SE/sv_SE/birth-number.json b/data/svse/birth-number.json similarity index 100% rename from data/sv_SE/sv_SE/birth-number.json rename to data/svse/birth-number.json diff --git a/data/sv_SE/sv_SE/birth-number.tsv b/data/svse/birth-number.tsv similarity index 100% rename from data/sv_SE/sv_SE/birth-number.tsv rename to data/svse/birth-number.tsv diff --git a/data/sv_SE/sv_SE/color.json b/data/svse/color.json similarity index 100% rename from data/sv_SE/sv_SE/color.json rename to data/svse/color.json diff --git a/data/sv_SE/sv_SE/company.json b/data/svse/company.json similarity index 100% rename from data/sv_SE/sv_SE/company.json rename to data/svse/company.json diff --git a/data/sv_SE/sv_SE/date.json b/data/svse/date.json similarity index 100% rename from data/sv_SE/sv_SE/date.json rename to data/svse/date.json diff --git a/data/sv_SE/sv_SE/email.json b/data/svse/email.json similarity index 100% rename from data/sv_SE/sv_SE/email.json rename to data/svse/email.json diff --git a/data/sv_SE/sv_SE/first-name.json b/data/svse/first-name.json similarity index 100% rename from data/sv_SE/sv_SE/first-name.json rename to data/svse/first-name.json diff --git a/data/sv_SE/sv_SE/first-name.tsv b/data/svse/first-name.tsv similarity index 100% rename from data/sv_SE/sv_SE/first-name.tsv rename to data/svse/first-name.tsv diff --git a/data/svse/fs.go b/data/svse/fs.go new file mode 100644 index 0000000..d7c6f8c --- /dev/null +++ b/data/svse/fs.go @@ -0,0 +1,9 @@ +// Package svse is fejkdata's sv_SE locale module, whose categories are sv_SE.*. Its address reads geose's places. +package svse + +import "embed" + +// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. +// +//go:embed *.json *.tsv .fejkdata.json +var FS embed.FS diff --git a/data/sv_SE/sv_SE/ip.json b/data/svse/ip.json similarity index 100% rename from data/sv_SE/sv_SE/ip.json rename to data/svse/ip.json diff --git a/data/sv_SE/sv_SE/last-name.json b/data/svse/last-name.json similarity index 100% rename from data/sv_SE/sv_SE/last-name.json rename to data/svse/last-name.json diff --git a/data/sv_SE/sv_SE/last-name.tsv b/data/svse/last-name.tsv similarity index 100% rename from data/sv_SE/sv_SE/last-name.tsv rename to data/svse/last-name.tsv diff --git a/data/sv_SE/sv_SE/person.json b/data/svse/person.json similarity index 100% rename from data/sv_SE/sv_SE/person.json rename to data/svse/person.json diff --git a/data/sv_SE/sv_SE/personnummer.json b/data/svse/personnummer.json similarity index 100% rename from data/sv_SE/sv_SE/personnummer.json rename to data/svse/personnummer.json diff --git a/data/sv_SE/sv_SE/phone.json b/data/svse/phone.json similarity index 100% rename from data/sv_SE/sv_SE/phone.json rename to data/svse/phone.json diff --git a/data/sv_SE/sv_SE/price.json b/data/svse/price.json similarity index 100% rename from data/sv_SE/sv_SE/price.json rename to data/svse/price.json diff --git a/data/sv_SE/sv_SE/samordningsnummer.json b/data/svse/samordningsnummer.json similarity index 100% rename from data/sv_SE/sv_SE/samordningsnummer.json rename to data/svse/samordningsnummer.json diff --git a/data/sv_SE/sv_SE/sentence.json b/data/svse/sentence.json similarity index 100% rename from data/sv_SE/sv_SE/sentence.json rename to data/svse/sentence.json diff --git a/data/sv_SE/sv_SE/sex.json b/data/svse/sex.json similarity index 100% rename from data/sv_SE/sv_SE/sex.json rename to data/svse/sex.json diff --git a/data/sv_SE/sv_SE/sex.tsv b/data/svse/sex.tsv similarity index 100% rename from data/sv_SE/sv_SE/sex.tsv rename to data/svse/sex.tsv diff --git a/data/sv_SE/sv_SE/time.json b/data/svse/time.json similarity index 100% rename from data/sv_SE/sv_SE/time.json rename to data/svse/time.json diff --git a/data/sv_SE/sv_SE/title.json b/data/svse/title.json similarity index 100% rename from data/sv_SE/sv_SE/title.json rename to data/svse/title.json diff --git a/data/sv_SE/sv_SE/title.tsv b/data/svse/title.tsv similarity index 100% rename from data/sv_SE/sv_SE/title.tsv rename to data/svse/title.tsv diff --git a/data/sv_SE/sv_SE/url.json b/data/svse/url.json similarity index 100% rename from data/sv_SE/sv_SE/url.json rename to data/svse/url.json diff --git a/data/sv_SE/sv_SE/username.json b/data/svse/username.json similarity index 100% rename from data/sv_SE/sv_SE/username.json rename to data/svse/username.json diff --git a/data/sv_SE/sv_SE/version.json b/data/svse/version.json similarity index 100% rename from data/sv_SE/sv_SE/version.json rename to data/svse/version.json diff --git a/data/sv_SE/sv_SE/word.json b/data/svse/word.json similarity index 100% rename from data/sv_SE/sv_SE/word.json rename to data/svse/word.json diff --git a/docs/comprehension-history.md b/docs/comprehension-history.md index 171bc3a..fbd3363 100644 --- a/docs/comprehension-history.md +++ b/docs/comprehension-history.md @@ -368,3 +368,27 @@ Junior A, decided by: - Locality: `data.go:12` `loadSources`: default modules pass through four files, an out-parameter map patched onto sites after the merge for walked sources and indexed.defaultModules copied in batch.load for indexed ones - Locality: `fejkdata.go:94` `New`: reads straight through with no hidden default source, which only partly offsets the above + +## 2026-10-11T09:52:08Z, PR #203 at 4c4bbcc, against bf1becb + +Ruling: worse + +| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall | +|---|---|---|---|---|---| +| Mid A | same | same | worse | same | worse | +| Inherited architect | worse | same | worse | same | worse | + +Mid A, decided by: + +- Shape, Overall: `data/geo/SE/fs.go:2` `SE`: each module embeds a tree repeating its own name, and the package names SE, US, en_US, sv_SE are unidiomatic; five thin packages plus a list replace one embed +- Shape: `reference.go:124` `withDefaultModules`: "default modules" suggests modules that load by default, though nothing loads them; "by default" is spelled out only in the README +- Locality, Shape: `inline.go:40` `NewTemplate`: errors are decorated twice, by hand, at four entry points +- Locality: `fejkdata.go:101` `New`: the hidden shipped default is gone, offsetting the above + +Inherited architect, decided by: + +- Navigation, Overall: `data/sv_SE/fs.go:8` `FS`: every data file sits under a doubled folder, data/sv_SE/sv_SE/, data/geo/SE/geo/SE/, a stop at each hop of the 3am path +- Navigation: `data/modules.go:15` `go:generate`: the module list is written twice and the manifest writer lives in a root test run from the data package +- Shape: `data/geo/SE/fs.go:2` `SE`: a package named SE reads as Sweden beside sv_SE, and "module" means a Go package here +- Shape: `reference.go:126` `withDefaultModules`: one concept spelled reads, defaultModules and "read by default", the last defined only in the README +- Locality: `fejkdata.go:114` `New`: four call sites apply loadError, withDefaultModules and noDataNote in a fixed order, offset by the hidden shipped source being gone diff --git a/docs/decisions.md b/docs/decisions.md index 1213bbc..26ae846 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -152,6 +152,17 @@ A directory a machine happens to have would make `--seed 42` machine-dependent; Valid while the shipped data stays under 100 MiB, which a gate test holds (`todo.md` item 192): `go get` fetches the whole Go module, and the `go` command refuses a Go module zip over 500 MiB (`MaxZipFile` in `golang.org/x/mod/zip`). Past it, `todo.md` item 193 moves the largest packages into nested Go modules that keep their import paths, each requiring the core release that drops the package, so no build sees it twice. +## A shipped package is named by its namespace in lower case without separators, and its manifest mounts it there with `namespace` + +2026-10-11, Lilleman auf Larv, from a bench of a Go test author, a data author and a +custom-CLI builder. Serves goals 1 and 3.3. `sv_SE` ships as `data/svse`, `en_US` as +`data/enus`, `geo.SE` as `data/geose`, `geo.US` as `data/geous`, and `misc` as `data/misc`: +Go linters flag a capital or an underscore in a package name, and `SE.FS` read as a type. +Each package holds its files at its root, and its manifest's `namespace` puts them under +the namespace, so no path repeats it. The bench also chose `reads` for what a module reads +by default and the wording of the errors naming it. Valid until a locale's lowered name +clashes with another's. + ## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads 2026-10-03, a restructure the maintainer approved; widened to any source on 2026-10-09. @@ -580,8 +591,8 @@ when a source carries it. ## A locale's `address` reads its country's `geo` tree, and the locale module's manifest reads its country's geo module 2026-09-24, Lilleman auf Larv; revised 2026-10-10, when the shipped data became Go packages. -Serves goal 6.2. `sv_SE.FS` loads alone, and `sv_SE.address` then fails at first reach, -naming `github.com/larvit/fejkdata/data/geo/SE`, which `sv_SE`'s manifest reads. +Serves goal 6.2. `svse.FS` loads alone, and `sv_SE.address` then fails at first reach, +naming `github.com/larvit/fejkdata/data/geose`, which `svse`'s manifest reads. `New` does not check that each module a manifest reads was given. A check by module name would refuse a stand-in providing the same paths (goal 9.2). A check by path needs every reference, and only loading each indexed category finds them, which goal 13 rules out. diff --git a/fejkdata.go b/fejkdata.go index 118c527..a6ab41e 100644 --- a/fejkdata.go +++ b/fejkdata.go @@ -78,7 +78,7 @@ func WithDataFS(modules ...fs.FS) Option { } } -var errNoData = errors.New("fejkdata: New was given no data to load; pass WithDataFS(data.Modules()...) for every module fejkdata ships (import github.com/larvit/fejkdata/data); WithDataFS or WithDataPath for your own; or WithDataFS() for none") +var errNoData = errors.New(`fejkdata: no data: add fejkdata.WithDataFS(data.Modules()...) (import "github.com/larvit/fejkdata/data") for all shipped data, WithDataFS/WithDataPath for your own, or WithDataFS() for none`) // ErrLoad marks data that fails to load: in New, or on the first call reaching a category // of an indexed source. diff --git a/internal/datafiles/datafiles.go b/internal/datafiles/datafiles.go index b57c63e..a250079 100644 --- a/internal/datafiles/datafiles.go +++ b/internal/datafiles/datafiles.go @@ -13,11 +13,13 @@ import ( ) // Source is one tree to load, from the root of an fs.FS. label prefixes file names in -// errors; onDisk marks label as a directory that must exist. +// errors; onDisk marks label as a directory that must exist. mount is the folders the +// tree's root sits under, in the namespace its categories are reached by. type Source struct { fsys fs.FS label string onDisk bool + mount []string } // Dir is the tree in the directory path, its errors labelled with it. @@ -28,8 +30,16 @@ func Dir(path string) Source { // FS is the tree in fsys. func FS(fsys fs.FS) Source { return Source{fsys: fsys} } -// Category is one category file: the folders above it from the source's root, its name, -// and its parsed JSON. +// Mounted is s with its root under the dot path namespace, "" for the namespace's root. +func (s Source) Mounted(namespace string) Source { + if namespace != "" { + s.mount = strings.Split(namespace, ".") + } + return s +} + +// Category is one category file: the folders above it in the namespace, its name, and its +// parsed JSON. type Category struct { Folders []string Name string @@ -62,7 +72,10 @@ func (s Source) Walk(compile func(Category) error) error { if err := s.check(); err != nil { return err } - _, err := s.walkDir(nil, compile) + _, err := s.walkDir(nil, func(c Category) error { + c.Folders = append(s.mount[:len(s.mount):len(s.mount)], c.Folders...) + return compile(c) + }) return err } @@ -123,9 +136,10 @@ func (s Source) walkFolder(dir []string, compile func(Category) error) (bool, er return true, nil } -// Load hands compile the category file name.json in the folder dir. +// Load hands compile the category file name.json in the folder dir of the namespace, which +// lies under the source's mount. func (s Source) Load(dir []string, name string, compile func(Category) error) error { - full := s.dirPath(dir) + full := s.dirPath(dir[len(s.mount):]) entries, err := fs.ReadDir(s.fsys, full) if err != nil { return fmt.Errorf("%s: %w", s.labelled(full), err) diff --git a/internal/datafiles/manifest.go b/internal/datafiles/manifest.go index 75df2da..1c68441 100644 --- a/internal/datafiles/manifest.go +++ b/internal/datafiles/manifest.go @@ -21,6 +21,8 @@ type Manifest struct { // Reads is every module the source reads by default, directly or through another; // nothing loads them. Reads []string `json:"reads"` + // Namespace is the dot path the source's root sits under, "" for the namespace's root. + Namespace string `json:"namespace"` } // IndexEntry is what an index says of a category: its parent table, "" for none, and the @@ -72,17 +74,40 @@ func decodeManifest(b []byte) (Manifest, error) { var indexErrs []error m.Index, indexErrs = decodeIndex(top[k]) errs = append(errs, indexErrs...) + case "namespace": + var err error + m.Namespace, err = decodeNamespace(top[k]) + errs = append(errs, err) case "reads": var err error m.Reads, err = decodeReads(top[k]) errs = append(errs, err) default: - errs = append(errs, fmt.Errorf("unknown key %q; a manifest holds index and reads", k)) + errs = append(errs, fmt.Errorf("unknown key %q; a manifest holds index, namespace and reads", k)) + } + } + for _, p := range sortedKeys(m.Index) { + if m.Namespace != "" && !strings.HasPrefix(p, m.Namespace+".") { + errs = append(errs, fmt.Errorf("index entry %q lies outside the namespace %q", p, m.Namespace)) } } return m, errors.Join(errs...) } +func decodeNamespace(v any) (string, error) { + namespace, isString := v.(string) + if !isString { + return "", fmt.Errorf("namespace must be a string, not %s", jsonvalue.Kind(v)) + } + if namespace == "" { + return "", nil + } + if err := checkCategoryPath(namespace); err != nil { + return "", fmt.Errorf("namespace %w", err) + } + return namespace, nil +} + // decodeReads decodes the modules a source reads, each named by a non-empty string. func decodeReads(v any) ([]string, error) { items, isList := v.([]any) diff --git a/reference.go b/reference.go index 2977158..f255a0c 100644 --- a/reference.go +++ b/reference.go @@ -129,9 +129,9 @@ func withDefaultModules(err error, modules []string) error { return err } if len(modules) == 0 { - return fmt.Errorf("%w, and no loaded module names a module it reads by default", err) + return fmt.Errorf("%w, and no loaded manifest names a module to add", err) } - return fmt.Errorf("%w; the loaded modules read by default %s", err, strings.Join(modules, ", ")) + return fmt.Errorf("%w; is a module missing? The loaded modules read %s", err, strings.Join(modules, ", ")) } // noDataNote adds to err, where a segment names nothing and f holds no data at all, that no diff --git a/todo.md b/todo.md index 5670120..3bf65ab 100644 --- a/todo.md +++ b/todo.md @@ -182,7 +182,7 @@ Pick the ten from a published ranking of languages by total speakers, such as Et ### 185. Seat a panel of the README's audience personas to pull data in through the module API and build a custom CLI, and fix what it names. -Needs item 184; 0.1.0 is not cut before it. Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Seats: the Go test author, the bulk fixture author, a custom CLI's builder, and a data author with a `-d` folder and the replace flag, each in a scratch Go module against the checkout or a pre-release tag. Bench the geo packages' names, `data.Modules()`, the error naming the import set, and the `.fejkdata.json` manifest a data author writes by hand. Decoding the manifest is most of `New`'s cost (about 0.55 ms and 2449 allocations on 2026-10-10, 2864 allocations once each module carried its own manifest). An entry lists every path below its category, its linked tables' paths too, so `geo.*.region` repeats its whole family. An entry holding only its own paths, with `List` composing a family from the entries naming it as `parent`, would shrink it. Record the ruling in `docs/`. +Needs item 184; 0.1.0 is not cut before it. Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Seats: the Go test author, the bulk fixture author, a custom CLI's builder, and a data author with a `-d` folder and the replace flag, each in a scratch Go module against the checkout or a pre-release tag. Bench `data.Modules()` and the `.fejkdata.json` manifest a data author writes by hand in use; the package names, the manifest's `reads` and `namespace` and the errors naming the import set were benched on 2026-10-11. Decoding the manifest is most of `New`'s cost (about 0.55 ms and 2449 allocations on 2026-10-10, 2864 allocations once each module carried its own manifest). An entry lists every path below its category, its linked tables' paths too, so `geo.*.region` repeats its whole family. An entry holding only its own paths, with `List` composing a family from the entries naming it as `parent`, would shrink it. Record the ruling in `docs/`. ### 16. Add locale categories: company, phone, finance, vehicle, words. @@ -411,7 +411,7 @@ Carries out the decision "The shipped data is Go packages a library imports by c ### 92. Name the hyphenated field in a `calc`'s refusal, and say to rename it: today `{calc(postal-code * 2)}` reports `no field "postal"`. -A hyphen is always subtraction in a calc, and the shipped data has hyphenated fields (`postal-code` and `street-number` in `geo/SE/address`). The refusal names the first fragment, not the field the author meant. +A hyphen is always subtraction in a calc, and the shipped data has hyphenated fields (`postal-code` and `street-number` in `geo.SE.address`). The refusal names the first fragment, not the field the author meant. ### 194. Let a data author write and check their own `.fejkdata.json` index with the library or the CLI, and fail a load whose index leaves out a category file.