From d242466859fa53c93e26adba2d1828d723a3f1fb Mon Sep 17 00:00:00 2001 From: Lilleman auf Larv Date: Sun, 11 Oct 2026 15:05:56 +0200 Subject: [PATCH] Answer architecture round 9: each data package exports Modules, a locale's importing its country's geo package as a bench chose, and a namespace mounts a tree through an fs.FS wrapper, so internal/datafiles walks file-system folders alone --- AGENTS.md | 2 +- CHANGELOG.md | 3 +- README.md | 13 ++-- data/en_US/fs.go | 16 ++++- data/geo/SE/fs.go | 10 ++- data/geo/US/fs.go | 10 ++- data/misc/fs.go | 10 ++- data/sv_SE/fs.go | 16 ++++- docs/decisions.md | 15 ++-- internal/datafiles/datafiles.go | 38 ++++------ internal/datafiles/mount.go | 123 ++++++++++++++++++++++++++++++++ todo.md | 4 +- 12 files changed, 207 insertions(+), 53 deletions(-) create mode 100644 internal/datafiles/mount.go diff --git a/AGENTS.md b/AGENTS.md index 52690e5..565c25e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -89,7 +89,7 @@ In [docs/decisions.md](docs/decisions.md): - A cell may read a reference, and each row its own - A path is walked once without drawing before it is walked for real - A country's postal codes and streets are siblings under its locality -- A locale's `address` reads its country's `geo` tree, whose module a program loads beside it +- A locale package imports its country's geo package, and its `Modules()` returns both - The default embed holds every Swedish postort the import can place and give a street-delivery code and a street, and the US places of 25,000 or more - A locale's `address` binds its country's record once and prints it whole - A postort's kommun comes from its name, its tätort or its codes, never from distance diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f6e2ac..e85df03 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -128,7 +128,8 @@ replacement, and each removed path, column or flag. - The shipped data comes in Go packages, one per module, each exporting `FS`: `github.com/larvit/fejkdata/data/en_US` (package `enus`), `…/sv_SE` (`svse`), `…/misc`, `…/geo/SE` (`geose`) and `…/geo/US` (`geous`). `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one, - and the CLI carries them all. `New` loads the modules `WithDataFS(modules...)` names and + and the CLI carries them all. Each package's `Modules()` returns its `FS` and every module + it reads: `svse.Modules()` holds the Swedish places too. `New` loads the modules `WithDataFS(modules...)` names and the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option is given. A manifest's `namespace` mounts its source under a dot path. - A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each diff --git a/README.md b/README.md index ddce852..e56ed87 100644 --- a/README.md +++ b/README.md @@ -356,23 +356,24 @@ ok, err := fejkdata.IsTemplate(arg) // an inline template by its shape, el |--------|--| | `WithSeed(n)` | same seed, version and data: identical sequence | | `WithDataPath(dir)` | layer a directory; repeat to layer several, the last wins a clash | -| `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `FS`, or your own `embed.FS`; `WithDataFS()` loads none | +| `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `Modules()`, or your own `embed.FS`; `WithDataFS()` loads none | `data.Modules()` is every shipped module, and links all of them into your program. To link -only the ones you read, pass each locale's `FS` with its country's places: +only the ones you read, pass a package's `Modules()`: its `FS` and every module it reads, a +locale's country's places among them, which its package imports: ```go import ( "github.com/larvit/fejkdata" - geose "github.com/larvit/fejkdata/data/geo/SE" svse "github.com/larvit/fejkdata/data/sv_SE" ) -f, err := fejkdata.New(fejkdata.WithDataFS(svse.FS, geose.FS)) +f, err := fejkdata.New(fejkdata.WithDataFS(svse.Modules()...)) ``` -`enus` goes with `geous`; `misc` and the geo packages stand alone. `New` fails where no `WithDataFS` or -`WithDataPath` names the data to load. +To stand in your own places, pass the locale's `FS` alone beside them: +`WithDataFS(svse.FS, myPlaces)`. `New` fails where no `WithDataFS` or `WithDataPath` names the +data to load. `New` refuses data that breaks the grammar, could mean two things or cannot render a valid value, and `NewTemplate` refuses such a template. A category of an indexed source is diff --git a/data/en_US/fs.go b/data/en_US/fs.go index b153735..c8b254e 100644 --- a/data/en_US/fs.go +++ b/data/en_US/fs.go @@ -1,9 +1,19 @@ -// Package enus is fejkdata's en_US locale module, whose categories are en_US.*. Its address reads geous's places. +// Package enus is fejkdata's en_US locale module, whose categories are en_US.*; its address reads +// geo.US, which package geous holds. package enus -import "embed" +import ( + "embed" + "io/fs" -// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. + geous "github.com/larvit/fejkdata/data/geo/US" +) + +// FS holds the locale alone, its categories and its manifest; pass Modules unless your own +// data stands in for geo.US. // //go:embed *.json *.tsv .fejkdata.json var FS embed.FS + +// Modules returns FS and every module it reads, to pass to fejkdata.WithDataFS. +func Modules() []fs.FS { return append([]fs.FS{FS}, geous.Modules()...) } diff --git a/data/geo/SE/fs.go b/data/geo/SE/fs.go index 6f0cc20..716a85b 100644 --- a/data/geo/SE/fs.go +++ b/data/geo/SE/fs.go @@ -1,9 +1,15 @@ // Package geose is fejkdata's Swedish places module, whose categories are geo.SE.*. package geose -import "embed" +import ( + "embed" + "io/fs" +) -// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. +// FS holds the module's categories and its manifest. // //go:embed *.json *.tsv .fejkdata.json var FS embed.FS + +// Modules returns FS, which reads no other module, to pass to fejkdata.WithDataFS. +func Modules() []fs.FS { return []fs.FS{FS} } diff --git a/data/geo/US/fs.go b/data/geo/US/fs.go index 1874e6e..2795456 100644 --- a/data/geo/US/fs.go +++ b/data/geo/US/fs.go @@ -1,9 +1,15 @@ // Package geous is fejkdata's US places module, whose categories are geo.US.*. package geous -import "embed" +import ( + "embed" + "io/fs" +) -// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. +// FS holds the module's categories and its manifest. // //go:embed *.json *.tsv .fejkdata.json var FS embed.FS + +// Modules returns FS, which reads no other module, to pass to fejkdata.WithDataFS. +func Modules() []fs.FS { return []fs.FS{FS} } diff --git a/data/misc/fs.go b/data/misc/fs.go index b7810ca..a4036b9 100644 --- a/data/misc/fs.go +++ b/data/misc/fs.go @@ -1,9 +1,15 @@ // Package misc is fejkdata's locale-neutral module, whose categories are misc.*. package misc -import "embed" +import ( + "embed" + "io/fs" +) -// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. +// FS holds the module's categories and its manifest. // //go:embed *.json *.tsv .fejkdata.json var FS embed.FS + +// Modules returns FS, which reads no other module, to pass to fejkdata.WithDataFS. +func Modules() []fs.FS { return []fs.FS{FS} } diff --git a/data/sv_SE/fs.go b/data/sv_SE/fs.go index d7c6f8c..50d8ed5 100644 --- a/data/sv_SE/fs.go +++ b/data/sv_SE/fs.go @@ -1,9 +1,19 @@ -// Package svse is fejkdata's sv_SE locale module, whose categories are sv_SE.*. Its address reads geose's places. +// Package svse is fejkdata's sv_SE locale module, whose categories are sv_SE.*; its address reads +// geo.SE, which package geose holds. package svse -import "embed" +import ( + "embed" + "io/fs" -// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS. + geose "github.com/larvit/fejkdata/data/geo/SE" +) + +// FS holds the locale alone, its categories and its manifest; pass Modules unless your own +// data stands in for geo.SE. // //go:embed *.json *.tsv .fejkdata.json var FS embed.FS + +// Modules returns FS and every module it reads, to pass to fejkdata.WithDataFS. +func Modules() []fs.FS { return append([]fs.FS{FS}, geose.Modules()...) } diff --git a/docs/decisions.md b/docs/decisions.md index e841534..7b77c70 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -591,12 +591,17 @@ its ZIPs would shape the two trees differently, so both draw inside the pinned l and an address agrees at that level. A street's own code is the exact pairing to add when a source carries it. -## A locale's `address` reads its country's `geo` tree, whose module a program loads beside it +## A locale package imports its country's geo package, and its `Modules()` returns both -2026-09-24, Lilleman auf Larv; revised 2026-10-10, when the shipped data became Go packages. -Serves goal 9.2. `svse.FS` loads alone, and `sv_SE.address` then fails at first reach with -`no entry "geo"`, so anything providing `geo.SE`'s paths can stand in for `geose`. Until -`todo.md` item 196 names the missing module in that error, it works against goal 6.2. +2026-09-24, Lilleman auf Larv; revised 2026-10-11, when the shipped data became Go packages, +from a bench of a Go test author, a data author and a custom-CLI builder, 3 to 0. Serves +goals 6.2 and 9.2. A locale's `address` reads its country's `geo` tree, so `svse` imports +`geose` and `svse.Modules()` returns both: the Go import graph states the dependency, and a +program passing `Modules()` cannot leave the places out. `svse.FS` alone loads for a stand-in +providing `geo.SE`'s paths, and without one `sv_SE.address` fails at first reach with +`no entry "geo"`; until `todo.md` item 196 names the missing module in that error, the bare +`FS` works against goal 6.2. Every package exports `Modules()`, so one spelling loads any of +them; a data package imports only the packages it reads. ## The default embed holds every Swedish postort the import can place and give a street-delivery code and a street, and the US places of 25,000 or more diff --git a/internal/datafiles/datafiles.go b/internal/datafiles/datafiles.go index 0d83bc3..4e5ea4f 100644 --- a/internal/datafiles/datafiles.go +++ b/internal/datafiles/datafiles.go @@ -6,22 +6,18 @@ import ( "io/fs" "os" "path" - "slices" "strings" "github.com/larvit/fejkdata/internal/grammar" - "github.com/larvit/fejkdata/internal/invariant" "github.com/larvit/fejkdata/internal/jsonvalue" ) // Source is one tree to load, from the root of an fs.FS. label prefixes file names in -// errors; onDisk marks label as a directory that must exist. mount is the folders the -// tree's root sits under, in the namespace its categories are reached by. +// errors; onDisk marks label as a directory that must exist. type Source struct { fsys fs.FS label string onDisk bool - mount []string } // Dir is the tree in the directory path, its errors labelled with it. @@ -32,16 +28,16 @@ func Dir(path string) Source { // FS is the tree in fsys. func FS(fsys fs.FS) Source { return Source{fsys: fsys} } -// Mounted is s with its root under the dot path namespace, "" for the namespace's root. +// Mounted is s with its tree under the folders the dot path namespace spells, "" for none. func (s Source) Mounted(namespace string) Source { if namespace != "" { - s.mount = strings.Split(namespace, ".") + s.fsys = mounted{fsys: s.fsys, at: strings.ReplaceAll(namespace, ".", "/")} } return s } -// Category is one category file: the folders above it in the namespace, its name, and its -// parsed JSON. +// Category is one category file: the folders above it from the source's root, its name, +// and its parsed JSON. type Category struct { Folders []string Name string @@ -52,26 +48,16 @@ type Category struct { // ReadRows reads a rows file beside the category. func (c Category) ReadRows(name string) (string, error) { return c.rows.read(name) } -// labelled is the file p of fsys as errors name it: under the source's label, or, for a -// mounted source with none, beside its namespace, "sv_SE (address.json)". func (s Source) labelled(p string) string { - switch { - case s.label != "": - return path.Join(s.label, p) - case s.mount != nil && p == ".": - return strings.Join(s.mount, ".") - case s.mount != nil: - return fmt.Sprintf("%s (%s)", strings.Join(s.mount, "."), p) + if s.label == "" { + return p } - return p + return path.Join(s.label, p) } -// dirPath is the folder that the namespace folders dir name, as fsys spells it. +// dirPath is the folder that dir names, as fsys spells it. func (s Source) dirPath(dir []string) string { - if len(dir) < len(s.mount) || !slices.Equal(dir[:len(s.mount)], s.mount) { - panic(invariant.Broken("folder %s lies outside the mount %s", strings.Join(dir, "."), strings.Join(s.mount, "."))) - } - if p := path.Join(dir[len(s.mount):]...); p != "" { + if p := path.Join(dir...); p != "" { return p } return "." @@ -84,7 +70,7 @@ func (s Source) Walk(compile func(Category) error) error { if err := s.check(); err != nil { return err } - _, err := s.walkDir(slices.Clip(s.mount), compile) + _, err := s.walkDir(nil, compile) return err } @@ -145,7 +131,7 @@ func (s Source) walkFolder(dir []string, compile func(Category) error) (bool, er return true, nil } -// Load hands compile the category file name.json in the namespace folder dir. +// Load hands compile the category file name.json in the folder dir. func (s Source) Load(dir []string, name string, compile func(Category) error) error { full := s.dirPath(dir) entries, err := fs.ReadDir(s.fsys, full) diff --git a/internal/datafiles/mount.go b/internal/datafiles/mount.go new file mode 100644 index 0000000..1e0ae9b --- /dev/null +++ b/internal/datafiles/mount.go @@ -0,0 +1,123 @@ +package datafiles + +import ( + "io" + "io/fs" + "path" + "strings" + "time" +) + +// mounted is fsys with its root moved down to the folder at, so a file f of fsys is at/f; +// the folders above at hold only the next folder towards it. +type mounted struct { + fsys fs.FS + at string +} + +func (m mounted) Open(name string) (fs.File, error) { + inner, ok := m.inner(name) + if ok && inner != "." { + return m.fsys.Open(inner) + } + if ok { + f, err := m.fsys.Open(".") + return mountRoot{File: f, name: path.Base(name)}, err + } + if next, ok := m.towards(name); ok { + return &mountDir{name: path.Base(name), next: next}, nil + } + return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist} +} + +func (m mounted) ReadDir(name string) ([]fs.DirEntry, error) { + if inner, ok := m.inner(name); ok { + return fs.ReadDir(m.fsys, inner) + } + if next, ok := m.towards(name); ok { + return []fs.DirEntry{mountEntry(next)}, nil + } + return nil, &fs.PathError{Op: "readdir", Path: name, Err: fs.ErrNotExist} +} + +// inner is name as fsys spells it, where name lies at or below at. +func (m mounted) inner(name string) (string, bool) { + if name == m.at { + return ".", true + } + rest, below := strings.CutPrefix(name, m.at+"/") + return rest, below && fs.ValidPath(name) +} + +// towards is the folder below name on the way to at, where name lies above at. +func (m mounted) towards(name string) (string, bool) { + rest := m.at + if name != "." { + var above bool + if rest, above = strings.CutPrefix(m.at, name+"/"); !above { + return "", false + } + } + next, _, _ := strings.Cut(rest, "/") + return next, true +} + +// mountRoot is fsys's root, named as the folder it is mounted at. +type mountRoot struct { + fs.File + name string +} + +func (r mountRoot) Stat() (fs.FileInfo, error) { + info, err := r.File.Stat() + return renamed{FileInfo: info, name: r.name}, err +} + +func (r mountRoot) ReadDir(n int) ([]fs.DirEntry, error) { + if d, ok := r.File.(fs.ReadDirFile); ok { + return d.ReadDir(n) + } + return nil, &fs.PathError{Op: "readdir", Path: r.name, Err: fs.ErrInvalid} +} + +type renamed struct { + fs.FileInfo + name string +} + +func (r renamed) Name() string { return r.name } + +// mountDir is a folder above the mount, holding only the folder next towards it. +type mountDir struct { + name, next string + read bool +} + +func (d *mountDir) Stat() (fs.FileInfo, error) { return mountEntry(d.name), nil } +func (d *mountDir) Read([]byte) (int, error) { + return 0, &fs.PathError{Op: "read", Path: d.name, Err: fs.ErrInvalid} +} +func (d *mountDir) Close() error { return nil } + +func (d *mountDir) ReadDir(n int) ([]fs.DirEntry, error) { + if d.read { + if n > 0 { + return nil, io.EOF + } + return nil, nil + } + d.read = true + return []fs.DirEntry{mountEntry(d.next)}, nil +} + +// mountEntry is a folder above the mount, as both its DirEntry and its FileInfo. +type mountEntry string + +func (e mountEntry) Name() string { return string(e) } +func (e mountEntry) IsDir() bool { return true } +func (e mountEntry) Type() fs.FileMode { return fs.ModeDir } +func (e mountEntry) Info() (fs.FileInfo, error) { return e, nil } +func (e mountEntry) Size() int64 { return 0 } +func (e mountEntry) Mode() fs.FileMode { return fs.ModeDir | 0o555 } +func (e mountEntry) ModTime() time.Time { return time.Time{} } +func (e mountEntry) Sys() any { return nil } diff --git a/todo.md b/todo.md index 5682623..641825e 100644 --- a/todo.md +++ b/todo.md @@ -282,7 +282,7 @@ Nothing has shipped, so "no longer paths", "where it used to fail" and "where it ### 196. Name, in the error for a reference nothing loaded provides, the module the reader's manifest reads by default that would provide it, or say none does. -Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0", and ends the locale-address decision's conflict with goal 6.2. Carved out of item 182 on 2026-10-11, after six rounds of the comprehension panel ruled each attempt worse. Tried and ruled worse: a module list carried per category from each source's manifest (two routes, `walked` and `indexed`); the union of every manifest's list on the `Generator`, noted at four entry points; and the same noted inside `loadReached`, its list threaded through `loadInline` and `structCompile`. Each named every module read, even loaded ones, until the list held namespaces and the note named only those nothing in the tree provides, which the product-owner and goals reviewers accepted. Bench the manifest key and the note's wording with the data author, since entries changed from import paths to namespaces after the 2026-10-11 bench. +Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0", and ends the locale-address decision's conflict with goal 6.2. Carved out of item 182 on 2026-10-11, after six rounds of the comprehension panel ruled each attempt worse. Tried and ruled worse: a module list carried per category from each source's manifest (two routes, `walked` and `indexed`); the union of every manifest's list on the `Generator`, noted at four entry points; and the same noted inside `loadReached`, its list threaded through `loadInline` and `structCompile`. Each named every module read, even loaded ones, until the list held namespaces and the note named only those nothing in the tree provides, which the product-owner and goals reviewers accepted. A shipped locale's `Modules()` already carries its places, so this serves a bare `FS` beside a stand-in, a `--data-path` folder and anyone else's module; the error names the module and points to `Modules()`, as the 2026-10-11 bench asked. Bench the manifest key and the note's wording with the data author, since entries changed from import paths to namespaces after the 2026-10-11 bench. ### 164. Let a table's format or a cell bind a name, which lives in that format or cell and keeps one pick through the draw. @@ -294,7 +294,7 @@ Split from item 129. cmd.exe's `echo` keeps its quotes and a trailing space, so ### 181. Fail a load where two sources define one category, unless the library's option or the CLI's flag says the later replaces the earlier, and then make every read reach the replacement. -Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Proposed spellings: `WithReplace()` and a boolean `--replace` covering every `-d`, so it goes anywhere on the line. Bench the option, the flag and the clash error with the data and hand fixture authors before merging; the error names the option or the flag. A clash is found from the categories each source's index names and the files each source without a manifest holds, without parsing them. Revises the decision "A `--data-path` override rebinds every reference to the category it replaces", ending its conflict with goal 9.3. The changelog names the flag, the option and the new refusal. +Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Proposed spellings: `WithReplace()` and a boolean `--replace` covering every `-d`, so it goes anywhere on the line. Bench the option, the flag and the clash error with the data and hand fixture authors before merging; the error names the option or the flag. A clash is found from the categories each source's index names and the files each source without a manifest holds, without parsing them. Revises the decision "A `--data-path` override rebinds every reference to the category it replaces", ending its conflict with goal 9.3. The changelog names the flag, the option and the new refusal. The same `embed.FS` passed twice, as `svse.Modules()` beside `data.Modules()` passes `geose.FS`, loads once and is no clash. ### 167. Lower the reading cost of the files a chunk reads most before its first commit.