Ship en_US, sv_SE, misc, geo/SE and geo/US as Go packages exporting FS, listed by data.Modules(), and fail a bare New naming WithDataFS

AGENTS.md

@@ -88,7 +88,7 @@ In [docs/decisions.md](docs/decisions.md):
 - A cell may read a reference, and each row its own
 - A path is walked once without drawing before it is walked for real
 - A country's postal codes and streets are siblings under its locality
-- A locale's `address` reads its country's `geo` tree, so a locale folder is no data set on its own
+- A locale's `address` reads its country's `geo` tree, and the locale module's manifest reads that module
 - The default embed holds every Swedish postort the import can place and give a street-delivery code and a street, and the US places of 25,000 or more
 - A locale's `address` binds its country's record once and prints it whole
 - A postort's kommun comes from its name, its tätort or its codes, never from distance

CHANGELOG.md

@@ -125,8 +125,15 @@ replacement, and each removed path, column or flag.
   its own quotes, a row of a table with no key is named as the path that selects it,
   `sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record
   that gives it one.
+- The shipped data comes in Go packages, one per module, each exporting `FS`:
+  `github.com/larvit/fejkdata/data/en_US`, `…/sv_SE`, `…/misc`, `…/geo/SE` and
+  `…/geo/US`. `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one,
+  and the CLI carries them all. `New` loads the modules `WithDataFS(modules...)` names and
+  the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option
+  is given; `WithDataFS()` loads none. A manifest's `reads` lists the modules its source
+  reads by default, and a reference naming nothing names them, or says there are none.
 - A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
-  category on the first call reaching it, as the shipped set does. Data failing to load,
+  category on the first call reaching it, as each shipped module does. Data failing to load,
   in `New` or on that first call, gives an error matching `ErrLoad`, and the CLI exits 1.
   A source without an index loads in `New`, with the shipped categories it reads. Every
   other shipped category loads on the first call reaching it, so one your replacement

README.md

@@ -160,12 +160,14 @@ it to a name and read the name
 
 ## Data
 
-The shipped set under [`data/`](data) is embedded, so the CLI and the library
+The shipped data comes in modules, each a Go package under [`data/`](data) that embeds
-work with no data on disk. It holds one folder per locale (`en_US`, `sv_SE`), a
+its files, so nothing is read from disk: one per locale (`en_US`, `sv_SE`), one per
-`geo` folder with a tree of places per country (`SE`, `US`), and a
+country's tree of places (`geo/SE`, `geo/US`), and the locale-neutral `misc`. The CLI
-locale-neutral `misc` folder. A directory is a namespace: each JSON file is a category named
+carries every module, and a library loads those it imports ([Library](#library)). A
-after the file, each subdirectory a dot-path segment, so `mydata/sv_SE/person.json`
+directory is a namespace: each JSON file is a category named after the file, each
-is `sv_SE.person` and replaces the shipped one.
+subdirectory a dot-path segment, so `mydata/sv_SE/person.json` is `sv_SE.person` and
+replaces the shipped one. A module's tree spells its namespace the same way, so
+`sv_SE`'s person is `data/sv_SE/sv_SE/person.json`.
 Sources merge in order; matching folders combine, any other clash is won by the
 last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"`, `/` or ` as `,
 which binds a [name](#names), nor be `-`, which a struct tag reserves. A dot-prefixed
@@ -181,13 +183,19 @@ every category of its source, so a category file it leaves out never loads.
 
 The index maps each category's dot path to an entry: `parent`, the table it links to,
 which needs its own entry in the same folder; and `paths`, every path `List` advertises
-below the category, `""` for the category itself. The shipped set's
+below the category, `""` for the category itself. Each shipped module's manifest, such as
-[`data/.fejkdata.json`](data/.fejkdata.json) is one, so delete it from a copy of `data/`
+[`data/sv_SE/.fejkdata.json`](data/sv_SE/.fejkdata.json), is one, so delete it from a
-you edit, or keep its index current. A table whose parent column differs from its
+copy you edit, or keep its index current. A table whose parent column differs from its
 entry's `parent` fails at first reach, and a table linking under another source's table
 needs a source without an index. `//go:embed` of a directory leaves the manifest out, and
 the source then loads whole in `New`; name it in the pattern too.
 
+A manifest's `reads` lists, by any name its author chooses, every module the source
+reads by default, directly or through another; a manifest may carry it without an
+index. A shipped module names a Go package: `sv_SE`'s manifest reads
+`github.com/larvit/fejkdata/data/geo/SE`. A reference naming nothing fails with the
+modules its category's manifest reads, or says it names none.
+
 Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
 `ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`,
 `username`, `version` and `word`, formatted per locale; `sv_SE` adds `personnummer`
@@ -322,7 +330,12 @@ go get github.com/larvit/fejkdata   # Go 1.22+
 ```
 
 ```go
-f, err := fejkdata.New(fejkdata.WithSeed(42))
+import (
+	"github.com/larvit/fejkdata"
+	"github.com/larvit/fejkdata/data"
+)
+
+f, err := fejkdata.New(fejkdata.WithDataFS(data.Modules()...), fejkdata.WithSeed(42))
 if err != nil {
 	log.Fatal(err)
 }
@@ -342,8 +355,13 @@ ok, err := fejkdata.IsTemplate(arg)       // an inline template by its shape, el
 |--------|--|
 | `WithSeed(n)` | same seed, version and data: identical sequence |
 | `WithDataPath(dir)` | layer a directory; repeat to layer several, the last wins a clash |
-| `WithDataFS(fsys)` | layer an `fs.FS`, such as your own `embed.FS` |
+| `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `FS`, or your own `embed.FS`; `WithDataFS()` loads none |
-| `WithoutShippedData()` | load only what you give |
+
+`data.Modules()` is every shipped module, and links all of them into your program. To link
+only the ones you read, pass each package's `FS` with the modules its manifest reads:
+`WithDataFS(sv_SE.FS, SE.FS)` from `github.com/larvit/fejkdata/data/sv_SE` and
+`github.com/larvit/fejkdata/data/geo/SE`. `New` fails where no `WithDataFS` or
+`WithDataPath` names the data to load.
 
 `New` refuses data that breaks the grammar, could mean two things or cannot render a valid
 value, and `NewTemplate` refuses such a template. A category of an indexed source is
@@ -762,7 +780,7 @@ A reference renders a node from elsewhere in the data — the path `Fake` takes,
 across every loaded source — so one category borrows another. The sigil says
 where the path starts, as in a filesystem: `{/en_US.person}` from the data root,
 `{.username}` from the folder this file sits in, `{..username}` from the folder
-above. `data/sv_SE/email.json` can therefore read its own locale's `username`
+above. `data/sv_SE/sv_SE/email.json` can therefore read its own locale's `username`
 without naming `sv_SE`:
 
 ```json
@@ -1003,7 +1021,6 @@ Where the project is heading; the sections before Audience document what ships t
 ```
 doc.go          the package doc, and the vocabulary the package is written in
 fejkdata.go     Generator, New, options, List
-shipped.go      the embedded data set
 index.go        a source's index: its categories left unloaded, and loading one on the first call reaching it
 node.go         the node model and JSON -> node compilation
 table.go        tables: their options, format and cells, the link to the parent beside them, and how a path passes and draws a table
@@ -1036,7 +1053,7 @@ internal/invariant/ the one phrase every package panics with when an invariant b
 internal/jsonvalue/ a decoded JSON value, its numbers kept as written, the one input a template compiles from, and the kind an error names it by
 internal/proven/ what a proof knows of a value, and the bounds a calc takes from its operands
 internal/rows/  a table's rows: the TSV, the options proved over it, the links between tables, row selection and draws, and the pin set one path or one named pick fixes
-data/           shipped data (JSON, and a TSV per table), embedded at build: locale folders, geo, misc
+data/           the shipped modules, a Go package each that embeds its categories (JSON, and a TSV per table) and its manifest, and Modules, which lists them
 data-import/    the scripts that rebuild each sourced table (see DATA-LICENSES.md), and the modules they share
 docs/           the decision log, the register research behind the shipped data, the survey of other libraries, panels' scores and rulings on how readable the code is, and what each chunk cost an agent to read
 release-tooling/ the release CI publishes from the changelog heading
@@ -1062,7 +1079,7 @@ Commands that rewrite source keep your file ownership when run with `--user`:
 
 ```sh
 docker compose run --rm --user "$(id -u):$(id -g)" fmt       # gofmt -w .
-docker compose run --rm --user "$(id -u):$(id -g)" generate  # go generate: data/.fejkdata.json from data/
+docker compose run --rm --user "$(id -u):$(id -g)" generate  # go generate: each data/ module's .fejkdata.json
 docker compose run --rm --user "$(id -u):$(id -g)" tidy      # go mod tidy
 ```
 
@@ -1093,8 +1110,8 @@ Goal 3 is scored by a panel of simulated readers, and [`AGENTS.md`](AGENTS.md) s
 what the score asks of a pull request. Goal 5 is checked against simulated template
 writers from the Audience.
 
-A change to the shipped data runs `generate` above first and commits
+A change to the shipped data runs `generate` above first and commits each module's
-[`data/.fejkdata.json`](data/.fejkdata.json) with the change, then re-pins
+`.fejkdata.json` with the change, then re-pins
 [`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) in its own commit:
 
 ```sh

cmd/fejkdata/main.go

@@ -13,6 +13,7 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"io/fs"
 	"os"
 	"runtime/debug"
 	"sort"
@@ -20,6 +21,7 @@ import (
 	"strings"
 
 	"github.com/larvit/fejkdata"
+	"github.com/larvit/fejkdata/data"
 )
 
 const usage = `Usage: <template> | fejkdata [flags]
@@ -319,10 +321,11 @@ func (in invocation) check() error {
 }
 
 func (in invocation) options() []fejkdata.Option {
-	var opts []fejkdata.Option
+	var modules []fs.FS
-	if in.noShipped {
+	if !in.noShipped {
-		opts = append(opts, fejkdata.WithoutShippedData())
+		modules = data.Modules()
 	}
+	opts := []fejkdata.Option{fejkdata.WithDataFS(modules...)}
 	for _, dir := range in.dirs {
 		opts = append(opts, fejkdata.WithDataPath(dir))
 	}

data-import/currency.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/currency.tsv from datasets/currency-codes (PDDL) and CLDR's symbols (Unicode).
+"""Rebuild data/misc/misc/currency.tsv from datasets/currency-codes (PDDL) and CLDR's symbols (Unicode).
 
     data-import/currency.py [--source URL_OR_FILE] [--symbols URL_OR_FILE ...] [--cache DIR] [--out FILE]
 
@@ -19,7 +19,7 @@ SYMBOLS = [
     "https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/en.xml",
     "https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/root.xml",
 ]
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "currency.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "currency.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["code", "decimals", "name", "numeric", "symbol"]
 

data-import/geo-se.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/geo/SE/*.tsv from SCB (CC0), GeoNames (CC BY 4.0) and Trafikverket NVDB (CC0).
+"""Rebuild data/geo/SE/geo/SE/*.tsv from SCB (CC0), GeoNames (CC BY 4.0) and Trafikverket NVDB (CC0).
 
     TRAFIKVERKET_API_KEY=… data-import/geo-se.py [--key-file FILE] [--cache DIR] [--streets-per-locality N] [--out DIR]
 """

data-import/geo-us.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/geo/US/*.tsv from the Census Bureau's Gazetteer, population estimates, ZCTA relationships and TIGER/Line files (public domain).
+"""Rebuild data/geo/US/geo/US/*.tsv from the Census Bureau's Gazetteer, population estimates, ZCTA relationships and TIGER/Line files (public domain).
 
     data-import/geo-us.py [--cache DIR] [--min-population N] [--streets-per-locality N] [--out DIR]
 """

data-import/geo.py

@@ -12,7 +12,7 @@ CACHE = Path(__file__).resolve().parent / "cache"
 def parser(doc, country, **own):
     """A parser for the options every country's import takes, plus the script's own: each keyword names one option, with
     "_" standing for "-"."""
-    options = {"cache": {"default": str(CACHE)}, "out": {"default": str(DATA / country)}, "streets_per_locality": {"type": int, "default": 10}, **own}
+    options = {"cache": {"default": str(CACHE)}, "out": {"default": str(DATA / country / "geo" / country)}, "streets_per_locality": {"type": int, "default": 10}, **own}
     p = argparse.ArgumentParser(description=doc.splitlines()[0])
     for name in sorted(options):
         p.add_argument("--" + name.replace("_", "-"), **options[name])

data-import/httpmethod.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/httpmethod.tsv from the IANA HTTP Method Registry.
+"""Rebuild data/misc/misc/httpmethod.tsv from the IANA HTTP Method Registry.
 
     data-import/httpmethod.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
 
@@ -18,7 +18,7 @@ import source
 import tsv
 
 SOURCE = "https://www.iana.org/assignments/http-methods/methods.csv"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpmethod.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "httpmethod.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["idempotent", "method", "safe"]
 CORE = re.compile(r"\[RFC9110, Section 9\.3\.|\[RFC5789, Section 2\]")

data-import/httpstatus.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/httpstatus.tsv from the IANA HTTP Status Code Registry.
+"""Rebuild data/misc/misc/httpstatus.tsv from the IANA HTTP Status Code Registry.
 
     data-import/httpstatus.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
 
@@ -14,7 +14,7 @@ import source
 import tsv
 
 SOURCE = "https://www.iana.org/assignments/http-status-codes/http-status-codes-1.csv"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpstatus.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "httpstatus.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["code", "reason"]
 

data-import/language.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/language.tsv from datasets/language-codes (PDDL), the LoC ISO 639-2 register.
+"""Rebuild data/misc/misc/language.tsv from datasets/language-codes (PDDL), the LoC ISO 639-2 register.
 
     data-import/language.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
 
@@ -15,7 +15,7 @@ import source
 import tsv
 
 SOURCE = "https://raw.githubusercontent.com/datasets/language-codes/main/data/language-codes-full.csv"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "language.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "language.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["code", "code3", "name"]
 

data-import/loglevel.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/loglevel.tsv from RFC 5424 and POSIX <syslog.h>.
+"""Rebuild data/misc/misc/loglevel.tsv from RFC 5424 and POSIX <syslog.h>.
 
     data-import/loglevel.py [--source URL_OR_FILE] [--posix URL_OR_FILE] [--cache DIR] [--out FILE]
 
@@ -18,7 +18,7 @@ import tsv
 
 SOURCE = "https://www.rfc-editor.org/rfc/rfc5424.txt"
 POSIX = "https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/syslog.h.html"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "loglevel.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "loglevel.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["code", "keyword", "severity"]
 TABLE = re.compile(r"Table 1\.\s+Syslog Message Facilities(.*?)Table 2\.\s+Syslog Message Severities", re.S)

data-import/mimetype.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/mimetype.tsv from mime-db (MIT), the IANA media type registry with extensions.
+"""Rebuild data/misc/misc/mimetype.tsv from mime-db (MIT), the IANA media type registry with extensions.
 
     data-import/mimetype.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
 
@@ -13,7 +13,7 @@ import source
 import tsv
 
 SOURCE = "https://raw.githubusercontent.com/jshttp/mime-db/master/db.json"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "mimetype.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "mimetype.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["ext", "type"]
 FIXUPS = {"application/mp4": "mp4s", "audio/mpeg": "mp3", "audio/ogg": "ogg", "video/quicktime": "mov"}

data-import/names-se.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/sv_SE/first-name.tsv and last-name.tsv from SCB's whole-population name counts (CC0, "Källa: SCB").
+"""Rebuild data/sv_SE/sv_SE/first-name.tsv and last-name.tsv from SCB's whole-population name counts (CC0, "Källa: SCB").
 
     data-import/names-se.py [--source URL_OR_FILE] [--first N] [--last N] [--cache DIR] [--out DIR]
 
@@ -14,7 +14,7 @@ import tsv
 import xlsx
 
 SOURCE = "https://www.scb.se/contentassets/9fe7dbb460994c72b835163dbc491ef9/namn-med-minst-tva-barare-31-december-2022.xlsx"
-OUT = Path(__file__).resolve().parent.parent / "data" / "sv_SE"
+OUT = Path(__file__).resolve().parent.parent / "data" / "sv_SE" / "sv_SE"
 CACHE = Path(__file__).resolve().parent / "cache"
 SHEETS = {"f": "Tilltalsnamn kvinnor", "m": "Tilltalsnamn män"}
 SURNAMES = "Efternamn"

data-import/names-us.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/en_US/first-name.tsv and last-name.tsv from SSA baby names and the Census 2010 surnames (public domain).
+"""Rebuild data/en_US/en_US/first-name.tsv and last-name.tsv from SSA baby names and the Census 2010 surnames (public domain).
 
     data-import/names-us.py [--names URL_OR_FILE] [--surnames URL_OR_FILE] [--from-year YEAR] [--first N] [--last N] [--cache DIR] [--out DIR]
 
@@ -20,7 +20,7 @@ import tsv
 
 NAMES = "https://raw.githubusercontent.com/hackerb9/ssa-baby-names/master/alldata.txt"
 SURNAMES = "https://www2.census.gov/topics/genealogy/2010surnames/names.zip"
-OUT = Path(__file__).resolve().parent.parent / "data" / "en_US"
+OUT = Path(__file__).resolve().parent.parent / "data" / "en_US" / "en_US"
 CACHE = Path(__file__).resolve().parent / "cache"
 MC = re.compile(r"^Mc([a-z])")
 PLACEHOLDERS = {"Baby", "Female", "Infant", "Male", "Notnamed", "Unknown", "Unnamed"}

data-import/port.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry.
+"""Rebuild data/misc/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry.
 
     data-import/port.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
 
@@ -17,7 +17,7 @@ import source
 import tsv
 
 SOURCE = "https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.csv"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "port.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "port.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["number", "service"]
 FLOOR = 4000

data-import/protocol.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/protocol.tsv from the IANA Protocol Numbers registry.
+"""Rebuild data/misc/misc/protocol.tsv from the IANA Protocol Numbers registry.
 
     data-import/protocol.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
 
@@ -17,7 +17,7 @@ import source
 import tsv
 
 SOURCE = "https://www.iana.org/assignments/protocol-numbers/protocol-numbers-1.csv"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "protocol.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "protocol.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["keyword", "name", "number"]
 SKIP = ("Reserved", "deprecated")

data-import/territory.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/territory.tsv from datasets/country-codes (PDDL).
+"""Rebuild data/misc/misc/territory.tsv from datasets/country-codes (PDDL).
 
     data-import/territory.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
 """
@@ -14,7 +14,7 @@ import source
 import tsv
 
 SOURCE = "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "territory.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "territory.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["alpha2", "alpha3", "calling-code", "capital", "country", "currency", "flag", "languages", "name", "numeric", "tld"]
 SOVEREIGN = re.compile(r"^(?:Part of|Territor(?:y|ies) of|Crown dependency of|Commonwealth of|Associated with) ([A-Z]{2})$")

data-import/timezone.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/timezone.tsv from the IANA tzdb tarball (public domain) and GeoNames (CC BY 4.0).
+"""Rebuild data/misc/misc/timezone.tsv from the IANA tzdb tarball (public domain) and GeoNames (CC BY 4.0).
 
     data-import/timezone.py [--source URL_OR_FILE] [--cities URL_OR_FILE] [--cache DIR] [--out FILE] [--territories FILE]
 
@@ -22,8 +22,8 @@ import tsv
 
 SOURCE = "https://data.iana.org/time-zones/tzdata-latest.tar.gz"
 CITIES = "https://download.geonames.org/export/dump/cities15000.zip"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "timezone.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "timezone.tsv"
-TERRITORIES = Path(__file__).resolve().parent.parent / "data" / "misc" / "territory.tsv"
+TERRITORIES = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "territory.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["offset", "territory", "weight", "zone"]
 REGIONS = ["africa", "antarctica", "asia", "australasia", "backward", "etcetera", "europe", "northamerica", "southamerica"]

data-import/tld.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/tld.tsv from the IANA Root Zone Database.
+"""Rebuild data/misc/misc/tld.tsv from the IANA Root Zone Database.
 
     data-import/tld.py [--source URL_OR_FILE] [--zone URL_OR_FILE] [--cache DIR] [--out FILE]
 
@@ -20,7 +20,7 @@ import tsv
 
 SOURCE = "https://www.iana.org/domains/root/db"
 ZONE = "https://data.iana.org/TLD/tlds-alpha-by-domain.txt"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "tld.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "tld.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["tld", "type", "unicode"]
 TYPES = ("country-code", "generic", "generic-restricted", "infrastructure", "sponsored")

data-import/useragent.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Rebuild data/misc/useragent.tsv from microlinkhq/top-user-agents (MIT).
+"""Rebuild data/misc/misc/useragent.tsv from microlinkhq/top-user-agents (MIT).
 
     data-import/useragent.py [--source URL] [--cache DIR] [--out FILE]
 
@@ -16,7 +16,7 @@ import source
 import tsv
 
 SOURCE = "https://raw.githubusercontent.com/microlinkhq/top-user-agents/master/src"
-OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "useragent.tsv"
+OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "useragent.tsv"
 CACHE = Path(__file__).resolve().parent / "cache"
 COLUMNS = ["browser", "device", "os", "ua"]
 # First match wins: a Chromium fork this list does not name ships as Chrome.

data.go

@@ -8,7 +8,7 @@ import (
 
 // loadSources merges sources in order, the last winning a clash. A source whose manifest
 // carries an index leaves its categories unloaded; every other source loads here, with
-// the unloaded categories it needs.
+// the unloaded categories it needs: docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads
 func loadSources(sources []datafiles.Source) (folder, error) {
 	root := folder{children: map[string]node{}}
 	for _, src := range sources {
@@ -25,13 +25,16 @@ func addSource(root *folder, src datafiles.Source) error {
 		return err
 	}
 	if m.Index != nil {
-		placeIndex(root, src, m.Index)
+		placeIndex(root, src, m)
 		return nil
 	}
 	g := &folder{children: map[string]node{}}
 	if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil {
 		return err
 	}
+	for _, s := range categorySites(g) {
+		s.in.setReads(s.name, m.Reads)
+	}
 	mergeFolder(root, g)
 	return nil
 }
@@ -70,6 +73,7 @@ func mergeFolder(dst, src *folder) {
 			}
 		}
 		dst.put(k, v)
+		dst.setReads(k, src.reads[k])
 	}
 }
 
@@ -80,8 +84,9 @@ type nodeSet func(fn func(label string, n node) error) error
 // categorySite is a loaded category, where it sits and its own dot path.
 type categorySite struct {
 	categoryAt
-	n    node
+	n     node
-	path string
+	path  string
+	reads []string
 }
 
 // categorySites lists every category under root, folders and names in sorted order.
@@ -102,7 +107,7 @@ func categorySites(root *folder) []categorySite {
 }
 
 func siteIn(dir []string, in *folder, name string) categorySite {
-	return categorySite{categoryAt: categoryAt{dir: dir, in: in, name: name}, n: in.children[name], path: categoryPath(dir, name)}
+	return categorySite{categoryAt: categoryAt{dir: dir, in: in, name: name}, n: in.children[name], path: categoryPath(dir, name), reads: in.reads[name]}
 }
 
 func categoryPath(dir []string, name string) string { return join(strings.Join(dir, "."), name) }

data/.fejkdata.json

@@ -1,80 +0,0 @@
-{
-	"index": {
-		"en_US.address": {"paths":["","locality","postal-code","region","street","street-number"]},
-		"en_US.color": {"paths":[""]},
-		"en_US.company": {"paths":["","base","suffix"]},
-		"en_US.date": {"paths":[""]},
-		"en_US.email": {"paths":["","domain","local","local.n"]},
-		"en_US.first-name": {"parent":"sex","paths":["","count","name","sex"]},
-		"en_US.ip": {"paths":[""]},
-		"en_US.itin": {"paths":["","area","group","serial"]},
-		"en_US.last-name": {"paths":["","count","name"]},
-		"en_US.person": {"paths":["","first","last","prefix","sex"]},
-		"en_US.phone": {"paths":["","area","exchange","line"]},
-		"en_US.phone-exchange": {"paths":[""]},
-		"en_US.price": {"paths":["","amt","cents"]},
-		"en_US.sentence": {"paths":["","adj","noun","prep","verb"]},
-		"en_US.sex": {"paths":["","code","first-name","first-name.count","first-name.name","first-name.sex","name","title","title.name","title.sex","title.share"]},
-		"en_US.ssn": {"paths":["","area","group","serial"]},
-		"en_US.time": {"paths":[""]},
-		"en_US.title": {"parent":"sex","paths":["","name","sex","share"]},
-		"en_US.url": {"paths":["","host","path"]},
-		"en_US.username": {"paths":["","n"]},
-		"en_US.version": {"paths":["","n","pre","suffix"]},
-		"en_US.word": {"paths":[""]},
-		"geo.SE.address": {"paths":["","locality","postal-code","street","street-number"]},
-		"geo.SE.locality": {"parent":"municipality","paths":["","lat","lon","municipality","name","population","postal-code","postal-code.code","postal-code.locality","street","street.locality","street.name","street.segments"]},
-		"geo.SE.municipality": {"parent":"region","paths":["","code","locality","locality.lat","locality.lon","locality.municipality","locality.name","locality.population","locality.postal-code","locality.postal-code.code","locality.postal-code.locality","locality.street","locality.street.locality","locality.street.name","locality.street.segments","name","population","region"]},
-		"geo.SE.postal-code": {"parent":"locality","paths":["","code","locality"]},
-		"geo.SE.region": {"paths":["","code","municipality","municipality.code","municipality.locality","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.locality","municipality.locality.street.name","municipality.locality.street.segments","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
-		"geo.SE.street": {"parent":"locality","paths":["","locality","name","segments"]},
-		"geo.US.address": {"paths":["","locality","postal-code","region","street","street-number"]},
-		"geo.US.locality": {"parent":"municipality","paths":["","code","lat","lon","municipality","name","population","postal-code","postal-code.addresses","postal-code.code","postal-code.locality","street","street.addresses","street.locality","street.name"]},
-		"geo.US.municipality": {"parent":"region","paths":["","code","locality","locality.code","locality.lat","locality.lon","locality.municipality","locality.name","locality.population","locality.postal-code","locality.postal-code.addresses","locality.postal-code.code","locality.postal-code.locality","locality.street","locality.street.addresses","locality.street.locality","locality.street.name","name","population","region"]},
-		"geo.US.postal-code": {"parent":"locality","paths":["","addresses","code","locality"]},
-		"geo.US.region": {"paths":["","abbr","code","municipality","municipality.code","municipality.locality","municipality.locality.code","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.addresses","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.addresses","municipality.locality.street.locality","municipality.locality.street.name","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
-		"geo.US.street": {"parent":"locality","paths":["","addresses","locality","name"]},
-		"misc.car": {"paths":["","make","model"]},
-		"misc.coordinate": {"paths":["","lat","lon"]},
-		"misc.creditcard": {"paths":["","d"]},
-		"misc.currency": {"paths":["","code","decimals","name","numeric","symbol"]},
-		"misc.datetime": {"paths":[""]},
-		"misc.emoji": {"paths":[""]},
-		"misc.httpmethod": {"paths":["","idempotent","method","safe"]},
-		"misc.httpstatus": {"paths":["","code","reason"]},
-		"misc.language": {"paths":["","code","code3","name"]},
-		"misc.loglevel": {"paths":["","code","keyword","severity"]},
-		"misc.mac": {"paths":[""]},
-		"misc.mimetype": {"paths":["","ext","type"]},
-		"misc.objectid": {"paths":[""]},
-		"misc.port": {"paths":["","number","service"]},
-		"misc.protocol": {"paths":["","keyword","name","number"]},
-		"misc.territory": {"paths":["","alpha2","alpha3","calling-code","capital","country","currency","flag","languages","name","numeric","timezone","timezone.offset","timezone.territory","timezone.weight","timezone.zone","tld"]},
-		"misc.timezone": {"parent":"territory","paths":["","offset","territory","weight","zone"]},
-		"misc.tld": {"paths":["","tld","type","unicode"]},
-		"misc.useragent": {"paths":["","browser","device","os","ua"]},
-		"misc.uuid": {"paths":["","variant"]},
-		"sv_SE.address": {"paths":["","locality","postal-code","street","street-number"]},
-		"sv_SE.birth-number": {"parent":"sex","paths":["","number","sex"]},
-		"sv_SE.color": {"paths":[""]},
-		"sv_SE.company": {"paths":["","base","suffix"]},
-		"sv_SE.date": {"paths":[""]},
-		"sv_SE.email": {"paths":["","domain","local","local.n"]},
-		"sv_SE.first-name": {"parent":"sex","paths":["","count","name","sex"]},
-		"sv_SE.ip": {"paths":[""]},
-		"sv_SE.last-name": {"paths":["","count","name"]},
-		"sv_SE.person": {"paths":["","first","last","prefix","sex"]},
-		"sv_SE.personnummer": {"paths":[""]},
-		"sv_SE.phone": {"paths":["","a","b","c","prefix"]},
-		"sv_SE.price": {"paths":["","amt","ore"]},
-		"sv_SE.samordningsnummer": {"paths":[""]},
-		"sv_SE.sentence": {"paths":["","adj","noun","prep","verb"]},
-		"sv_SE.sex": {"paths":["","birth-number","birth-number.number","birth-number.sex","code","first-name","first-name.count","first-name.name","first-name.sex","name"]},
-		"sv_SE.time": {"paths":[""]},
-		"sv_SE.title": {"paths":["","name","share"]},
-		"sv_SE.url": {"paths":["","host","path"]},
-		"sv_SE.username": {"paths":["","n"]},
-		"sv_SE.version": {"paths":["","n","pre","suffix"]},
-		"sv_SE.word": {"paths":[""]}
-	}
-}

data/en_US/.fejkdata.json

@@ -0,0 +1,27 @@
+{
+	"index": {
+		"en_US.address": {"paths":["","locality","postal-code","region","street","street-number"]},
+		"en_US.color": {"paths":[""]},
+		"en_US.company": {"paths":["","base","suffix"]},
+		"en_US.date": {"paths":[""]},
+		"en_US.email": {"paths":["","domain","local","local.n"]},
+		"en_US.first-name": {"parent":"sex","paths":["","count","name","sex"]},
+		"en_US.ip": {"paths":[""]},
+		"en_US.itin": {"paths":["","area","group","serial"]},
+		"en_US.last-name": {"paths":["","count","name"]},
+		"en_US.person": {"paths":["","first","last","prefix","sex"]},
+		"en_US.phone": {"paths":["","area","exchange","line"]},
+		"en_US.phone-exchange": {"paths":[""]},
+		"en_US.price": {"paths":["","amt","cents"]},
+		"en_US.sentence": {"paths":["","adj","noun","prep","verb"]},
+		"en_US.sex": {"paths":["","code","first-name","first-name.count","first-name.name","first-name.sex","name","title","title.name","title.sex","title.share"]},
+		"en_US.ssn": {"paths":["","area","group","serial"]},
+		"en_US.time": {"paths":[""]},
+		"en_US.title": {"parent":"sex","paths":["","name","sex","share"]},
+		"en_US.url": {"paths":["","host","path"]},
+		"en_US.username": {"paths":["","n"]},
+		"en_US.version": {"paths":["","n","pre","suffix"]},
+		"en_US.word": {"paths":[""]}
+	},
+	"reads": ["github.com/larvit/fejkdata/data/geo/US"]
+}

data/en_US/fs.go

@@ -0,0 +1,9 @@
+// Package en_US is fejkdata's en_US locale module.
+package en_US
+
+import "embed"
+
+// FS holds the en_US categories and the module's manifest; pass it to fejkdata.WithDataFS.
+//
+//go:embed en_US .fejkdata.json
+var FS embed.FS

data/geo/SE/.fejkdata.json

@@ -0,0 +1,10 @@
+{
+	"index": {
+		"geo.SE.address": {"paths":["","locality","postal-code","street","street-number"]},
+		"geo.SE.locality": {"parent":"municipality","paths":["","lat","lon","municipality","name","population","postal-code","postal-code.code","postal-code.locality","street","street.locality","street.name","street.segments"]},
+		"geo.SE.municipality": {"parent":"region","paths":["","code","locality","locality.lat","locality.lon","locality.municipality","locality.name","locality.population","locality.postal-code","locality.postal-code.code","locality.postal-code.locality","locality.street","locality.street.locality","locality.street.name","locality.street.segments","name","population","region"]},
+		"geo.SE.postal-code": {"parent":"locality","paths":["","code","locality"]},
+		"geo.SE.region": {"paths":["","code","municipality","municipality.code","municipality.locality","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.locality","municipality.locality.street.name","municipality.locality.street.segments","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
+		"geo.SE.street": {"parent":"locality","paths":["","locality","name","segments"]}
+	}
+}

data/geo/SE/fs.go

@@ -0,0 +1,9 @@
+// Package SE is fejkdata's Swedish geography module.
+package SE
+
+import "embed"
+
+// FS holds the geo.SE categories and the module's manifest; pass it to fejkdata.WithDataFS.
+//
+//go:embed geo .fejkdata.json
+var FS embed.FS

data/geo/US/.fejkdata.json

@@ -0,0 +1,10 @@
+{
+	"index": {
+		"geo.US.address": {"paths":["","locality","postal-code","region","street","street-number"]},
+		"geo.US.locality": {"parent":"municipality","paths":["","code","lat","lon","municipality","name","population","postal-code","postal-code.addresses","postal-code.code","postal-code.locality","street","street.addresses","street.locality","street.name"]},
+		"geo.US.municipality": {"parent":"region","paths":["","code","locality","locality.code","locality.lat","locality.lon","locality.municipality","locality.name","locality.population","locality.postal-code","locality.postal-code.addresses","locality.postal-code.code","locality.postal-code.locality","locality.street","locality.street.addresses","locality.street.locality","locality.street.name","name","population","region"]},
+		"geo.US.postal-code": {"parent":"locality","paths":["","addresses","code","locality"]},
+		"geo.US.region": {"paths":["","abbr","code","municipality","municipality.code","municipality.locality","municipality.locality.code","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.addresses","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.addresses","municipality.locality.street.locality","municipality.locality.street.name","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
+		"geo.US.street": {"parent":"locality","paths":["","addresses","locality","name"]}
+	}
+}

data/geo/US/fs.go

@@ -0,0 +1,9 @@
+// Package US is fejkdata's US geography module.
+package US
+
+import "embed"
+
+// FS holds the geo.US categories and the module's manifest; pass it to fejkdata.WithDataFS.
+//
+//go:embed geo .fejkdata.json
+var FS embed.FS

data/misc/.fejkdata.json

@@ -0,0 +1,24 @@
+{
+	"index": {
+		"misc.car": {"paths":["","make","model"]},
+		"misc.coordinate": {"paths":["","lat","lon"]},
+		"misc.creditcard": {"paths":["","d"]},
+		"misc.currency": {"paths":["","code","decimals","name","numeric","symbol"]},
+		"misc.datetime": {"paths":[""]},
+		"misc.emoji": {"paths":[""]},
+		"misc.httpmethod": {"paths":["","idempotent","method","safe"]},
+		"misc.httpstatus": {"paths":["","code","reason"]},
+		"misc.language": {"paths":["","code","code3","name"]},
+		"misc.loglevel": {"paths":["","code","keyword","severity"]},
+		"misc.mac": {"paths":[""]},
+		"misc.mimetype": {"paths":["","ext","type"]},
+		"misc.objectid": {"paths":[""]},
+		"misc.port": {"paths":["","number","service"]},
+		"misc.protocol": {"paths":["","keyword","name","number"]},
+		"misc.territory": {"paths":["","alpha2","alpha3","calling-code","capital","country","currency","flag","languages","name","numeric","timezone","timezone.offset","timezone.territory","timezone.weight","timezone.zone","tld"]},
+		"misc.timezone": {"parent":"territory","paths":["","offset","territory","weight","zone"]},
+		"misc.tld": {"paths":["","tld","type","unicode"]},
+		"misc.useragent": {"paths":["","browser","device","os","ua"]},
+		"misc.uuid": {"paths":["","variant"]}
+	}
+}

data/misc/fs.go

@@ -0,0 +1,9 @@
+// Package misc is fejkdata's misc module.
+package misc
+
+import "embed"
+
+// FS holds the misc categories and the module's manifest; pass it to fejkdata.WithDataFS.
+//
+//go:embed misc .fejkdata.json
+var FS embed.FS