Answer architecture round 9: each data package exports Modules, a locale's importing its country's geo package as a bench chose, and a namespace mounts a tree through an fs.FS wrapper, so internal/datafiles walks file-system folders alone

This commit is contained in:
2026-10-11 15:05:56 +02:00
committed by Lillem4n
parent d4dee46086
commit d242466859
12 changed files with 207 additions and 53 deletions
+1 -1
View File
@@ -89,7 +89,7 @@ In [docs/decisions.md](docs/decisions.md):
- A cell may read a reference, and each row its own
- A path is walked once without drawing before it is walked for real
- A country's postal codes and streets are siblings under its locality
- A locale's `address` reads its country's `geo` tree, whose module a program loads beside it
- A locale package imports its country's geo package, and its `Modules()` returns both
- The default embed holds every Swedish postort the import can place and give a street-delivery code and a street, and the US places of 25,000 or more
- A locale's `address` binds its country's record once and prints it whole
- A postort's kommun comes from its name, its tätort or its codes, never from distance
+2 -1
View File
@@ -128,7 +128,8 @@ replacement, and each removed path, column or flag.
- The shipped data comes in Go packages, one per module, each exporting `FS`:
`github.com/larvit/fejkdata/data/en_US` (package `enus`), `…/sv_SE` (`svse`), `…/misc`,
`…/geo/SE` (`geose`) and `…/geo/US` (`geous`). `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one,
and the CLI carries them all. `New` loads the modules `WithDataFS(modules...)` names and
and the CLI carries them all. Each package's `Modules()` returns its `FS` and every module
it reads: `svse.Modules()` holds the Swedish places too. `New` loads the modules `WithDataFS(modules...)` names and
the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option
is given. A manifest's `namespace` mounts its source under a dot path.
- A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
+7 -6
View File
@@ -356,23 +356,24 @@ ok, err := fejkdata.IsTemplate(arg) // an inline template by its shape, el
|--------|--|
| `WithSeed(n)` | same seed, version and data: identical sequence |
| `WithDataPath(dir)` | layer a directory; repeat to layer several, the last wins a clash |
| `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `FS`, or your own `embed.FS`; `WithDataFS()` loads none |
| `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `Modules()`, or your own `embed.FS`; `WithDataFS()` loads none |
`data.Modules()` is every shipped module, and links all of them into your program. To link
only the ones you read, pass each locale's `FS` with its country's places:
only the ones you read, pass a package's `Modules()`: its `FS` and every module it reads, a
locale's country's places among them, which its package imports:
```go
import (
"github.com/larvit/fejkdata"
geose "github.com/larvit/fejkdata/data/geo/SE"
svse "github.com/larvit/fejkdata/data/sv_SE"
)
f, err := fejkdata.New(fejkdata.WithDataFS(svse.FS, geose.FS))
f, err := fejkdata.New(fejkdata.WithDataFS(svse.Modules()...))
```
`enus` goes with `geous`; `misc` and the geo packages stand alone. `New` fails where no `WithDataFS` or
`WithDataPath` names the data to load.
To stand in your own places, pass the locale's `FS` alone beside them:
`WithDataFS(svse.FS, myPlaces)`. `New` fails where no `WithDataFS` or `WithDataPath` names the
data to load.
`New` refuses data that breaks the grammar, could mean two things or cannot render a valid
value, and `NewTemplate` refuses such a template. A category of an indexed source is
+13 -3
View File
@@ -1,9 +1,19 @@
// Package enus is fejkdata's en_US locale module, whose categories are en_US.*. Its address reads geous's places.
// Package enus is fejkdata's en_US locale module, whose categories are en_US.*; its address reads
// geo.US, which package geous holds.
package enus
import "embed"
import (
"embed"
"io/fs"
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
geous "github.com/larvit/fejkdata/data/geo/US"
)
// FS holds the locale alone, its categories and its manifest; pass Modules unless your own
// data stands in for geo.US.
//
//go:embed *.json *.tsv .fejkdata.json
var FS embed.FS
// Modules returns FS and every module it reads, to pass to fejkdata.WithDataFS.
func Modules() []fs.FS { return append([]fs.FS{FS}, geous.Modules()...) }
+8 -2
View File
@@ -1,9 +1,15 @@
// Package geose is fejkdata's Swedish places module, whose categories are geo.SE.*.
package geose
import "embed"
import (
"embed"
"io/fs"
)
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
// FS holds the module's categories and its manifest.
//
//go:embed *.json *.tsv .fejkdata.json
var FS embed.FS
// Modules returns FS, which reads no other module, to pass to fejkdata.WithDataFS.
func Modules() []fs.FS { return []fs.FS{FS} }
+8 -2
View File
@@ -1,9 +1,15 @@
// Package geous is fejkdata's US places module, whose categories are geo.US.*.
package geous
import "embed"
import (
"embed"
"io/fs"
)
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
// FS holds the module's categories and its manifest.
//
//go:embed *.json *.tsv .fejkdata.json
var FS embed.FS
// Modules returns FS, which reads no other module, to pass to fejkdata.WithDataFS.
func Modules() []fs.FS { return []fs.FS{FS} }
+8 -2
View File
@@ -1,9 +1,15 @@
// Package misc is fejkdata's locale-neutral module, whose categories are misc.*.
package misc
import "embed"
import (
"embed"
"io/fs"
)
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
// FS holds the module's categories and its manifest.
//
//go:embed *.json *.tsv .fejkdata.json
var FS embed.FS
// Modules returns FS, which reads no other module, to pass to fejkdata.WithDataFS.
func Modules() []fs.FS { return []fs.FS{FS} }
+13 -3
View File
@@ -1,9 +1,19 @@
// Package svse is fejkdata's sv_SE locale module, whose categories are sv_SE.*. Its address reads geose's places.
// Package svse is fejkdata's sv_SE locale module, whose categories are sv_SE.*; its address reads
// geo.SE, which package geose holds.
package svse
import "embed"
import (
"embed"
"io/fs"
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
geose "github.com/larvit/fejkdata/data/geo/SE"
)
// FS holds the locale alone, its categories and its manifest; pass Modules unless your own
// data stands in for geo.SE.
//
//go:embed *.json *.tsv .fejkdata.json
var FS embed.FS
// Modules returns FS and every module it reads, to pass to fejkdata.WithDataFS.
func Modules() []fs.FS { return append([]fs.FS{FS}, geose.Modules()...) }
+10 -5
View File
@@ -591,12 +591,17 @@ its ZIPs would shape the two trees differently, so both draw inside the pinned l
and an address agrees at that level. A street's own code is the exact pairing to add
when a source carries it.
## A locale's `address` reads its country's `geo` tree, whose module a program loads beside it
## A locale package imports its country's geo package, and its `Modules()` returns both
2026-09-24, Lilleman auf Larv; revised 2026-10-10, when the shipped data became Go packages.
Serves goal 9.2. `svse.FS` loads alone, and `sv_SE.address` then fails at first reach with
`no entry "geo"`, so anything providing `geo.SE`'s paths can stand in for `geose`. Until
`todo.md` item 196 names the missing module in that error, it works against goal 6.2.
2026-09-24, Lilleman auf Larv; revised 2026-10-11, when the shipped data became Go packages,
from a bench of a Go test author, a data author and a custom-CLI builder, 3 to 0. Serves
goals 6.2 and 9.2. A locale's `address` reads its country's `geo` tree, so `svse` imports
`geose` and `svse.Modules()` returns both: the Go import graph states the dependency, and a
program passing `Modules()` cannot leave the places out. `svse.FS` alone loads for a stand-in
providing `geo.SE`'s paths, and without one `sv_SE.address` fails at first reach with
`no entry "geo"`; until `todo.md` item 196 names the missing module in that error, the bare
`FS` works against goal 6.2. Every package exports `Modules()`, so one spelling loads any of
them; a data package imports only the packages it reads.
## The default embed holds every Swedish postort the import can place and give a street-delivery code and a street, and the US places of 25,000 or more
+12 -26
View File
@@ -6,22 +6,18 @@ import (
"io/fs"
"os"
"path"
"slices"
"strings"
"github.com/larvit/fejkdata/internal/grammar"
"github.com/larvit/fejkdata/internal/invariant"
"github.com/larvit/fejkdata/internal/jsonvalue"
)
// Source is one tree to load, from the root of an fs.FS. label prefixes file names in
// errors; onDisk marks label as a directory that must exist. mount is the folders the
// tree's root sits under, in the namespace its categories are reached by.
// errors; onDisk marks label as a directory that must exist.
type Source struct {
fsys fs.FS
label string
onDisk bool
mount []string
}
// Dir is the tree in the directory path, its errors labelled with it.
@@ -32,16 +28,16 @@ func Dir(path string) Source {
// FS is the tree in fsys.
func FS(fsys fs.FS) Source { return Source{fsys: fsys} }
// Mounted is s with its root under the dot path namespace, "" for the namespace's root.
// Mounted is s with its tree under the folders the dot path namespace spells, "" for none.
func (s Source) Mounted(namespace string) Source {
if namespace != "" {
s.mount = strings.Split(namespace, ".")
s.fsys = mounted{fsys: s.fsys, at: strings.ReplaceAll(namespace, ".", "/")}
}
return s
}
// Category is one category file: the folders above it in the namespace, its name, and its
// parsed JSON.
// Category is one category file: the folders above it from the source's root, its name,
// and its parsed JSON.
type Category struct {
Folders []string
Name string
@@ -52,26 +48,16 @@ type Category struct {
// ReadRows reads a rows file beside the category.
func (c Category) ReadRows(name string) (string, error) { return c.rows.read(name) }
// labelled is the file p of fsys as errors name it: under the source's label, or, for a
// mounted source with none, beside its namespace, "sv_SE (address.json)".
func (s Source) labelled(p string) string {
switch {
case s.label != "":
return path.Join(s.label, p)
case s.mount != nil && p == ".":
return strings.Join(s.mount, ".")
case s.mount != nil:
return fmt.Sprintf("%s (%s)", strings.Join(s.mount, "."), p)
if s.label == "" {
return p
}
return p
return path.Join(s.label, p)
}
// dirPath is the folder that the namespace folders dir name, as fsys spells it.
// dirPath is the folder that dir names, as fsys spells it.
func (s Source) dirPath(dir []string) string {
if len(dir) < len(s.mount) || !slices.Equal(dir[:len(s.mount)], s.mount) {
panic(invariant.Broken("folder %s lies outside the mount %s", strings.Join(dir, "."), strings.Join(s.mount, ".")))
}
if p := path.Join(dir[len(s.mount):]...); p != "" {
if p := path.Join(dir...); p != "" {
return p
}
return "."
@@ -84,7 +70,7 @@ func (s Source) Walk(compile func(Category) error) error {
if err := s.check(); err != nil {
return err
}
_, err := s.walkDir(slices.Clip(s.mount), compile)
_, err := s.walkDir(nil, compile)
return err
}
@@ -145,7 +131,7 @@ func (s Source) walkFolder(dir []string, compile func(Category) error) (bool, er
return true, nil
}
// Load hands compile the category file name.json in the namespace folder dir.
// Load hands compile the category file name.json in the folder dir.
func (s Source) Load(dir []string, name string, compile func(Category) error) error {
full := s.dirPath(dir)
entries, err := fs.ReadDir(s.fsys, full)
+123
View File
@@ -0,0 +1,123 @@
package datafiles
import (
"io"
"io/fs"
"path"
"strings"
"time"
)
// mounted is fsys with its root moved down to the folder at, so a file f of fsys is at/f;
// the folders above at hold only the next folder towards it.
type mounted struct {
fsys fs.FS
at string
}
func (m mounted) Open(name string) (fs.File, error) {
inner, ok := m.inner(name)
if ok && inner != "." {
return m.fsys.Open(inner)
}
if ok {
f, err := m.fsys.Open(".")
return mountRoot{File: f, name: path.Base(name)}, err
}
if next, ok := m.towards(name); ok {
return &mountDir{name: path.Base(name), next: next}, nil
}
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
}
func (m mounted) ReadDir(name string) ([]fs.DirEntry, error) {
if inner, ok := m.inner(name); ok {
return fs.ReadDir(m.fsys, inner)
}
if next, ok := m.towards(name); ok {
return []fs.DirEntry{mountEntry(next)}, nil
}
return nil, &fs.PathError{Op: "readdir", Path: name, Err: fs.ErrNotExist}
}
// inner is name as fsys spells it, where name lies at or below at.
func (m mounted) inner(name string) (string, bool) {
if name == m.at {
return ".", true
}
rest, below := strings.CutPrefix(name, m.at+"/")
return rest, below && fs.ValidPath(name)
}
// towards is the folder below name on the way to at, where name lies above at.
func (m mounted) towards(name string) (string, bool) {
rest := m.at
if name != "." {
var above bool
if rest, above = strings.CutPrefix(m.at, name+"/"); !above {
return "", false
}
}
next, _, _ := strings.Cut(rest, "/")
return next, true
}
// mountRoot is fsys's root, named as the folder it is mounted at.
type mountRoot struct {
fs.File
name string
}
func (r mountRoot) Stat() (fs.FileInfo, error) {
info, err := r.File.Stat()
return renamed{FileInfo: info, name: r.name}, err
}
func (r mountRoot) ReadDir(n int) ([]fs.DirEntry, error) {
if d, ok := r.File.(fs.ReadDirFile); ok {
return d.ReadDir(n)
}
return nil, &fs.PathError{Op: "readdir", Path: r.name, Err: fs.ErrInvalid}
}
type renamed struct {
fs.FileInfo
name string
}
func (r renamed) Name() string { return r.name }
// mountDir is a folder above the mount, holding only the folder next towards it.
type mountDir struct {
name, next string
read bool
}
func (d *mountDir) Stat() (fs.FileInfo, error) { return mountEntry(d.name), nil }
func (d *mountDir) Read([]byte) (int, error) {
return 0, &fs.PathError{Op: "read", Path: d.name, Err: fs.ErrInvalid}
}
func (d *mountDir) Close() error { return nil }
func (d *mountDir) ReadDir(n int) ([]fs.DirEntry, error) {
if d.read {
if n > 0 {
return nil, io.EOF
}
return nil, nil
}
d.read = true
return []fs.DirEntry{mountEntry(d.next)}, nil
}
// mountEntry is a folder above the mount, as both its DirEntry and its FileInfo.
type mountEntry string
func (e mountEntry) Name() string { return string(e) }
func (e mountEntry) IsDir() bool { return true }
func (e mountEntry) Type() fs.FileMode { return fs.ModeDir }
func (e mountEntry) Info() (fs.FileInfo, error) { return e, nil }
func (e mountEntry) Size() int64 { return 0 }
func (e mountEntry) Mode() fs.FileMode { return fs.ModeDir | 0o555 }
func (e mountEntry) ModTime() time.Time { return time.Time{} }
func (e mountEntry) Sys() any { return nil }
+2 -2
View File
@@ -282,7 +282,7 @@ Nothing has shipped, so "no longer paths", "where it used to fail" and "where it
### 196. Name, in the error for a reference nothing loaded provides, the module the reader's manifest reads by default that would provide it, or say none does.
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0", and ends the locale-address decision's conflict with goal 6.2. Carved out of item 182 on 2026-10-11, after six rounds of the comprehension panel ruled each attempt worse. Tried and ruled worse: a module list carried per category from each source's manifest (two routes, `walked` and `indexed`); the union of every manifest's list on the `Generator`, noted at four entry points; and the same noted inside `loadReached`, its list threaded through `loadInline` and `structCompile`. Each named every module read, even loaded ones, until the list held namespaces and the note named only those nothing in the tree provides, which the product-owner and goals reviewers accepted. Bench the manifest key and the note's wording with the data author, since entries changed from import paths to namespaces after the 2026-10-11 bench.
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0", and ends the locale-address decision's conflict with goal 6.2. Carved out of item 182 on 2026-10-11, after six rounds of the comprehension panel ruled each attempt worse. Tried and ruled worse: a module list carried per category from each source's manifest (two routes, `walked` and `indexed`); the union of every manifest's list on the `Generator`, noted at four entry points; and the same noted inside `loadReached`, its list threaded through `loadInline` and `structCompile`. Each named every module read, even loaded ones, until the list held namespaces and the note named only those nothing in the tree provides, which the product-owner and goals reviewers accepted. A shipped locale's `Modules()` already carries its places, so this serves a bare `FS` beside a stand-in, a `--data-path` folder and anyone else's module; the error names the module and points to `Modules()`, as the 2026-10-11 bench asked. Bench the manifest key and the note's wording with the data author, since entries changed from import paths to namespaces after the 2026-10-11 bench.
### 164. Let a table's format or a cell bind a name, which lives in that format or cell and keeps one pick through the draw.
@@ -294,7 +294,7 @@ Split from item 129. cmd.exe's `echo` keeps its quotes and a trailing space, so
### 181. Fail a load where two sources define one category, unless the library's option or the CLI's flag says the later replaces the earlier, and then make every read reach the replacement.
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Proposed spellings: `WithReplace()` and a boolean `--replace` covering every `-d`, so it goes anywhere on the line. Bench the option, the flag and the clash error with the data and hand fixture authors before merging; the error names the option or the flag. A clash is found from the categories each source's index names and the files each source without a manifest holds, without parsing them. Revises the decision "A `--data-path` override rebinds every reference to the category it replaces", ending its conflict with goal 9.3. The changelog names the flag, the option and the new refusal.
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Proposed spellings: `WithReplace()` and a boolean `--replace` covering every `-d`, so it goes anywhere on the line. Bench the option, the flag and the clash error with the data and hand fixture authors before merging; the error names the option or the flag. A clash is found from the categories each source's index names and the files each source without a manifest holds, without parsing them. Revises the decision "A `--data-path` override rebinds every reference to the category it replaces", ending its conflict with goal 9.3. The changelog names the flag, the option and the new refusal. The same `embed.FS` passed twice, as `svse.Modules()` beside `data.Modules()` passes `geose.FS`, loads once and is no clash.
### 167. Lower the reading cost of the files a chunk reads most before its first commit.