Answer architecture round 9: each data package exports Modules, a locale's importing its country's geo package as a bench chose, and a namespace mounts a tree through an fs.FS wrapper, so internal/datafiles walks file-system folders alone
This commit is contained in:
@@ -89,7 +89,7 @@ In [docs/decisions.md](docs/decisions.md):
|
||||
- A cell may read a reference, and each row its own
|
||||
- A path is walked once without drawing before it is walked for real
|
||||
- A country's postal codes and streets are siblings under its locality
|
||||
- A locale's `address` reads its country's `geo` tree, whose module a program loads beside it
|
||||
- A locale package imports its country's geo package, and its `Modules()` returns both
|
||||
- The default embed holds every Swedish postort the import can place and give a street-delivery code and a street, and the US places of 25,000 or more
|
||||
- A locale's `address` binds its country's record once and prints it whole
|
||||
- A postort's kommun comes from its name, its tätort or its codes, never from distance
|
||||
|
||||
+2
-1
@@ -128,7 +128,8 @@ replacement, and each removed path, column or flag.
|
||||
- The shipped data comes in Go packages, one per module, each exporting `FS`:
|
||||
`github.com/larvit/fejkdata/data/en_US` (package `enus`), `…/sv_SE` (`svse`), `…/misc`,
|
||||
`…/geo/SE` (`geose`) and `…/geo/US` (`geous`). `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one,
|
||||
and the CLI carries them all. `New` loads the modules `WithDataFS(modules...)` names and
|
||||
and the CLI carries them all. Each package's `Modules()` returns its `FS` and every module
|
||||
it reads: `svse.Modules()` holds the Swedish places too. `New` loads the modules `WithDataFS(modules...)` names and
|
||||
the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option
|
||||
is given. A manifest's `namespace` mounts its source under a dot path.
|
||||
- A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
|
||||
|
||||
@@ -356,23 +356,24 @@ ok, err := fejkdata.IsTemplate(arg) // an inline template by its shape, el
|
||||
|--------|--|
|
||||
| `WithSeed(n)` | same seed, version and data: identical sequence |
|
||||
| `WithDataPath(dir)` | layer a directory; repeat to layer several, the last wins a clash |
|
||||
| `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `FS`, or your own `embed.FS`; `WithDataFS()` loads none |
|
||||
| `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `Modules()`, or your own `embed.FS`; `WithDataFS()` loads none |
|
||||
|
||||
`data.Modules()` is every shipped module, and links all of them into your program. To link
|
||||
only the ones you read, pass each locale's `FS` with its country's places:
|
||||
only the ones you read, pass a package's `Modules()`: its `FS` and every module it reads, a
|
||||
locale's country's places among them, which its package imports:
|
||||
|
||||
```go
|
||||
import (
|
||||
"github.com/larvit/fejkdata"
|
||||
geose "github.com/larvit/fejkdata/data/geo/SE"
|
||||
svse "github.com/larvit/fejkdata/data/sv_SE"
|
||||
)
|
||||
|
||||
f, err := fejkdata.New(fejkdata.WithDataFS(svse.FS, geose.FS))
|
||||
f, err := fejkdata.New(fejkdata.WithDataFS(svse.Modules()...))
|
||||
```
|
||||
|
||||
`enus` goes with `geous`; `misc` and the geo packages stand alone. `New` fails where no `WithDataFS` or
|
||||
`WithDataPath` names the data to load.
|
||||
To stand in your own places, pass the locale's `FS` alone beside them:
|
||||
`WithDataFS(svse.FS, myPlaces)`. `New` fails where no `WithDataFS` or `WithDataPath` names the
|
||||
data to load.
|
||||
|
||||
`New` refuses data that breaks the grammar, could mean two things or cannot render a valid
|
||||
value, and `NewTemplate` refuses such a template. A category of an indexed source is
|
||||
|
||||
+13
-3
@@ -1,9 +1,19 @@
|
||||
// Package enus is fejkdata's en_US locale module, whose categories are en_US.*. Its address reads geous's places.
|
||||
// Package enus is fejkdata's en_US locale module, whose categories are en_US.*; its address reads
|
||||
// geo.US, which package geous holds.
|
||||
package enus
|
||||
|
||||
import "embed"
|
||||
import (
|
||||
"embed"
|
||||
"io/fs"
|
||||
|
||||
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
|
||||
geous "github.com/larvit/fejkdata/data/geo/US"
|
||||
)
|
||||
|
||||
// FS holds the locale alone, its categories and its manifest; pass Modules unless your own
|
||||
// data stands in for geo.US.
|
||||
//
|
||||
//go:embed *.json *.tsv .fejkdata.json
|
||||
var FS embed.FS
|
||||
|
||||
// Modules returns FS and every module it reads, to pass to fejkdata.WithDataFS.
|
||||
func Modules() []fs.FS { return append([]fs.FS{FS}, geous.Modules()...) }
|
||||
|
||||
+8
-2
@@ -1,9 +1,15 @@
|
||||
// Package geose is fejkdata's Swedish places module, whose categories are geo.SE.*.
|
||||
package geose
|
||||
|
||||
import "embed"
|
||||
import (
|
||||
"embed"
|
||||
"io/fs"
|
||||
)
|
||||
|
||||
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
|
||||
// FS holds the module's categories and its manifest.
|
||||
//
|
||||
//go:embed *.json *.tsv .fejkdata.json
|
||||
var FS embed.FS
|
||||
|
||||
// Modules returns FS, which reads no other module, to pass to fejkdata.WithDataFS.
|
||||
func Modules() []fs.FS { return []fs.FS{FS} }
|
||||
|
||||
+8
-2
@@ -1,9 +1,15 @@
|
||||
// Package geous is fejkdata's US places module, whose categories are geo.US.*.
|
||||
package geous
|
||||
|
||||
import "embed"
|
||||
import (
|
||||
"embed"
|
||||
"io/fs"
|
||||
)
|
||||
|
||||
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
|
||||
// FS holds the module's categories and its manifest.
|
||||
//
|
||||
//go:embed *.json *.tsv .fejkdata.json
|
||||
var FS embed.FS
|
||||
|
||||
// Modules returns FS, which reads no other module, to pass to fejkdata.WithDataFS.
|
||||
func Modules() []fs.FS { return []fs.FS{FS} }
|
||||
|
||||
+8
-2
@@ -1,9 +1,15 @@
|
||||
// Package misc is fejkdata's locale-neutral module, whose categories are misc.*.
|
||||
package misc
|
||||
|
||||
import "embed"
|
||||
import (
|
||||
"embed"
|
||||
"io/fs"
|
||||
)
|
||||
|
||||
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
|
||||
// FS holds the module's categories and its manifest.
|
||||
//
|
||||
//go:embed *.json *.tsv .fejkdata.json
|
||||
var FS embed.FS
|
||||
|
||||
// Modules returns FS, which reads no other module, to pass to fejkdata.WithDataFS.
|
||||
func Modules() []fs.FS { return []fs.FS{FS} }
|
||||
|
||||
+13
-3
@@ -1,9 +1,19 @@
|
||||
// Package svse is fejkdata's sv_SE locale module, whose categories are sv_SE.*. Its address reads geose's places.
|
||||
// Package svse is fejkdata's sv_SE locale module, whose categories are sv_SE.*; its address reads
|
||||
// geo.SE, which package geose holds.
|
||||
package svse
|
||||
|
||||
import "embed"
|
||||
import (
|
||||
"embed"
|
||||
"io/fs"
|
||||
|
||||
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
|
||||
geose "github.com/larvit/fejkdata/data/geo/SE"
|
||||
)
|
||||
|
||||
// FS holds the locale alone, its categories and its manifest; pass Modules unless your own
|
||||
// data stands in for geo.SE.
|
||||
//
|
||||
//go:embed *.json *.tsv .fejkdata.json
|
||||
var FS embed.FS
|
||||
|
||||
// Modules returns FS and every module it reads, to pass to fejkdata.WithDataFS.
|
||||
func Modules() []fs.FS { return append([]fs.FS{FS}, geose.Modules()...) }
|
||||
|
||||
+10
-5
@@ -591,12 +591,17 @@ its ZIPs would shape the two trees differently, so both draw inside the pinned l
|
||||
and an address agrees at that level. A street's own code is the exact pairing to add
|
||||
when a source carries it.
|
||||
|
||||
## A locale's `address` reads its country's `geo` tree, whose module a program loads beside it
|
||||
## A locale package imports its country's geo package, and its `Modules()` returns both
|
||||
|
||||
2026-09-24, Lilleman auf Larv; revised 2026-10-10, when the shipped data became Go packages.
|
||||
Serves goal 9.2. `svse.FS` loads alone, and `sv_SE.address` then fails at first reach with
|
||||
`no entry "geo"`, so anything providing `geo.SE`'s paths can stand in for `geose`. Until
|
||||
`todo.md` item 196 names the missing module in that error, it works against goal 6.2.
|
||||
2026-09-24, Lilleman auf Larv; revised 2026-10-11, when the shipped data became Go packages,
|
||||
from a bench of a Go test author, a data author and a custom-CLI builder, 3 to 0. Serves
|
||||
goals 6.2 and 9.2. A locale's `address` reads its country's `geo` tree, so `svse` imports
|
||||
`geose` and `svse.Modules()` returns both: the Go import graph states the dependency, and a
|
||||
program passing `Modules()` cannot leave the places out. `svse.FS` alone loads for a stand-in
|
||||
providing `geo.SE`'s paths, and without one `sv_SE.address` fails at first reach with
|
||||
`no entry "geo"`; until `todo.md` item 196 names the missing module in that error, the bare
|
||||
`FS` works against goal 6.2. Every package exports `Modules()`, so one spelling loads any of
|
||||
them; a data package imports only the packages it reads.
|
||||
|
||||
## The default embed holds every Swedish postort the import can place and give a street-delivery code and a street, and the US places of 25,000 or more
|
||||
|
||||
|
||||
@@ -6,22 +6,18 @@ import (
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/larvit/fejkdata/internal/grammar"
|
||||
"github.com/larvit/fejkdata/internal/invariant"
|
||||
"github.com/larvit/fejkdata/internal/jsonvalue"
|
||||
)
|
||||
|
||||
// Source is one tree to load, from the root of an fs.FS. label prefixes file names in
|
||||
// errors; onDisk marks label as a directory that must exist. mount is the folders the
|
||||
// tree's root sits under, in the namespace its categories are reached by.
|
||||
// errors; onDisk marks label as a directory that must exist.
|
||||
type Source struct {
|
||||
fsys fs.FS
|
||||
label string
|
||||
onDisk bool
|
||||
mount []string
|
||||
}
|
||||
|
||||
// Dir is the tree in the directory path, its errors labelled with it.
|
||||
@@ -32,16 +28,16 @@ func Dir(path string) Source {
|
||||
// FS is the tree in fsys.
|
||||
func FS(fsys fs.FS) Source { return Source{fsys: fsys} }
|
||||
|
||||
// Mounted is s with its root under the dot path namespace, "" for the namespace's root.
|
||||
// Mounted is s with its tree under the folders the dot path namespace spells, "" for none.
|
||||
func (s Source) Mounted(namespace string) Source {
|
||||
if namespace != "" {
|
||||
s.mount = strings.Split(namespace, ".")
|
||||
s.fsys = mounted{fsys: s.fsys, at: strings.ReplaceAll(namespace, ".", "/")}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Category is one category file: the folders above it in the namespace, its name, and its
|
||||
// parsed JSON.
|
||||
// Category is one category file: the folders above it from the source's root, its name,
|
||||
// and its parsed JSON.
|
||||
type Category struct {
|
||||
Folders []string
|
||||
Name string
|
||||
@@ -52,26 +48,16 @@ type Category struct {
|
||||
// ReadRows reads a rows file beside the category.
|
||||
func (c Category) ReadRows(name string) (string, error) { return c.rows.read(name) }
|
||||
|
||||
// labelled is the file p of fsys as errors name it: under the source's label, or, for a
|
||||
// mounted source with none, beside its namespace, "sv_SE (address.json)".
|
||||
func (s Source) labelled(p string) string {
|
||||
switch {
|
||||
case s.label != "":
|
||||
return path.Join(s.label, p)
|
||||
case s.mount != nil && p == ".":
|
||||
return strings.Join(s.mount, ".")
|
||||
case s.mount != nil:
|
||||
return fmt.Sprintf("%s (%s)", strings.Join(s.mount, "."), p)
|
||||
if s.label == "" {
|
||||
return p
|
||||
}
|
||||
return p
|
||||
return path.Join(s.label, p)
|
||||
}
|
||||
|
||||
// dirPath is the folder that the namespace folders dir name, as fsys spells it.
|
||||
// dirPath is the folder that dir names, as fsys spells it.
|
||||
func (s Source) dirPath(dir []string) string {
|
||||
if len(dir) < len(s.mount) || !slices.Equal(dir[:len(s.mount)], s.mount) {
|
||||
panic(invariant.Broken("folder %s lies outside the mount %s", strings.Join(dir, "."), strings.Join(s.mount, ".")))
|
||||
}
|
||||
if p := path.Join(dir[len(s.mount):]...); p != "" {
|
||||
if p := path.Join(dir...); p != "" {
|
||||
return p
|
||||
}
|
||||
return "."
|
||||
@@ -84,7 +70,7 @@ func (s Source) Walk(compile func(Category) error) error {
|
||||
if err := s.check(); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := s.walkDir(slices.Clip(s.mount), compile)
|
||||
_, err := s.walkDir(nil, compile)
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -145,7 +131,7 @@ func (s Source) walkFolder(dir []string, compile func(Category) error) (bool, er
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// Load hands compile the category file name.json in the namespace folder dir.
|
||||
// Load hands compile the category file name.json in the folder dir.
|
||||
func (s Source) Load(dir []string, name string, compile func(Category) error) error {
|
||||
full := s.dirPath(dir)
|
||||
entries, err := fs.ReadDir(s.fsys, full)
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
package datafiles
|
||||
|
||||
import (
|
||||
"io"
|
||||
"io/fs"
|
||||
"path"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// mounted is fsys with its root moved down to the folder at, so a file f of fsys is at/f;
|
||||
// the folders above at hold only the next folder towards it.
|
||||
type mounted struct {
|
||||
fsys fs.FS
|
||||
at string
|
||||
}
|
||||
|
||||
func (m mounted) Open(name string) (fs.File, error) {
|
||||
inner, ok := m.inner(name)
|
||||
if ok && inner != "." {
|
||||
return m.fsys.Open(inner)
|
||||
}
|
||||
if ok {
|
||||
f, err := m.fsys.Open(".")
|
||||
return mountRoot{File: f, name: path.Base(name)}, err
|
||||
}
|
||||
if next, ok := m.towards(name); ok {
|
||||
return &mountDir{name: path.Base(name), next: next}, nil
|
||||
}
|
||||
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
|
||||
}
|
||||
|
||||
func (m mounted) ReadDir(name string) ([]fs.DirEntry, error) {
|
||||
if inner, ok := m.inner(name); ok {
|
||||
return fs.ReadDir(m.fsys, inner)
|
||||
}
|
||||
if next, ok := m.towards(name); ok {
|
||||
return []fs.DirEntry{mountEntry(next)}, nil
|
||||
}
|
||||
return nil, &fs.PathError{Op: "readdir", Path: name, Err: fs.ErrNotExist}
|
||||
}
|
||||
|
||||
// inner is name as fsys spells it, where name lies at or below at.
|
||||
func (m mounted) inner(name string) (string, bool) {
|
||||
if name == m.at {
|
||||
return ".", true
|
||||
}
|
||||
rest, below := strings.CutPrefix(name, m.at+"/")
|
||||
return rest, below && fs.ValidPath(name)
|
||||
}
|
||||
|
||||
// towards is the folder below name on the way to at, where name lies above at.
|
||||
func (m mounted) towards(name string) (string, bool) {
|
||||
rest := m.at
|
||||
if name != "." {
|
||||
var above bool
|
||||
if rest, above = strings.CutPrefix(m.at, name+"/"); !above {
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
next, _, _ := strings.Cut(rest, "/")
|
||||
return next, true
|
||||
}
|
||||
|
||||
// mountRoot is fsys's root, named as the folder it is mounted at.
|
||||
type mountRoot struct {
|
||||
fs.File
|
||||
name string
|
||||
}
|
||||
|
||||
func (r mountRoot) Stat() (fs.FileInfo, error) {
|
||||
info, err := r.File.Stat()
|
||||
return renamed{FileInfo: info, name: r.name}, err
|
||||
}
|
||||
|
||||
func (r mountRoot) ReadDir(n int) ([]fs.DirEntry, error) {
|
||||
if d, ok := r.File.(fs.ReadDirFile); ok {
|
||||
return d.ReadDir(n)
|
||||
}
|
||||
return nil, &fs.PathError{Op: "readdir", Path: r.name, Err: fs.ErrInvalid}
|
||||
}
|
||||
|
||||
type renamed struct {
|
||||
fs.FileInfo
|
||||
name string
|
||||
}
|
||||
|
||||
func (r renamed) Name() string { return r.name }
|
||||
|
||||
// mountDir is a folder above the mount, holding only the folder next towards it.
|
||||
type mountDir struct {
|
||||
name, next string
|
||||
read bool
|
||||
}
|
||||
|
||||
func (d *mountDir) Stat() (fs.FileInfo, error) { return mountEntry(d.name), nil }
|
||||
func (d *mountDir) Read([]byte) (int, error) {
|
||||
return 0, &fs.PathError{Op: "read", Path: d.name, Err: fs.ErrInvalid}
|
||||
}
|
||||
func (d *mountDir) Close() error { return nil }
|
||||
|
||||
func (d *mountDir) ReadDir(n int) ([]fs.DirEntry, error) {
|
||||
if d.read {
|
||||
if n > 0 {
|
||||
return nil, io.EOF
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
d.read = true
|
||||
return []fs.DirEntry{mountEntry(d.next)}, nil
|
||||
}
|
||||
|
||||
// mountEntry is a folder above the mount, as both its DirEntry and its FileInfo.
|
||||
type mountEntry string
|
||||
|
||||
func (e mountEntry) Name() string { return string(e) }
|
||||
func (e mountEntry) IsDir() bool { return true }
|
||||
func (e mountEntry) Type() fs.FileMode { return fs.ModeDir }
|
||||
func (e mountEntry) Info() (fs.FileInfo, error) { return e, nil }
|
||||
func (e mountEntry) Size() int64 { return 0 }
|
||||
func (e mountEntry) Mode() fs.FileMode { return fs.ModeDir | 0o555 }
|
||||
func (e mountEntry) ModTime() time.Time { return time.Time{} }
|
||||
func (e mountEntry) Sys() any { return nil }
|
||||
@@ -282,7 +282,7 @@ Nothing has shipped, so "no longer paths", "where it used to fail" and "where it
|
||||
|
||||
### 196. Name, in the error for a reference nothing loaded provides, the module the reader's manifest reads by default that would provide it, or say none does.
|
||||
|
||||
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0", and ends the locale-address decision's conflict with goal 6.2. Carved out of item 182 on 2026-10-11, after six rounds of the comprehension panel ruled each attempt worse. Tried and ruled worse: a module list carried per category from each source's manifest (two routes, `walked` and `indexed`); the union of every manifest's list on the `Generator`, noted at four entry points; and the same noted inside `loadReached`, its list threaded through `loadInline` and `structCompile`. Each named every module read, even loaded ones, until the list held namespaces and the note named only those nothing in the tree provides, which the product-owner and goals reviewers accepted. Bench the manifest key and the note's wording with the data author, since entries changed from import paths to namespaces after the 2026-10-11 bench.
|
||||
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0", and ends the locale-address decision's conflict with goal 6.2. Carved out of item 182 on 2026-10-11, after six rounds of the comprehension panel ruled each attempt worse. Tried and ruled worse: a module list carried per category from each source's manifest (two routes, `walked` and `indexed`); the union of every manifest's list on the `Generator`, noted at four entry points; and the same noted inside `loadReached`, its list threaded through `loadInline` and `structCompile`. Each named every module read, even loaded ones, until the list held namespaces and the note named only those nothing in the tree provides, which the product-owner and goals reviewers accepted. A shipped locale's `Modules()` already carries its places, so this serves a bare `FS` beside a stand-in, a `--data-path` folder and anyone else's module; the error names the module and points to `Modules()`, as the 2026-10-11 bench asked. Bench the manifest key and the note's wording with the data author, since entries changed from import paths to namespaces after the 2026-10-11 bench.
|
||||
|
||||
### 164. Let a table's format or a cell bind a name, which lives in that format or cell and keeps one pick through the draw.
|
||||
|
||||
@@ -294,7 +294,7 @@ Split from item 129. cmd.exe's `echo` keeps its quotes and a trailing space, so
|
||||
|
||||
### 181. Fail a load where two sources define one category, unless the library's option or the CLI's flag says the later replaces the earlier, and then make every read reach the replacement.
|
||||
|
||||
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Proposed spellings: `WithReplace()` and a boolean `--replace` covering every `-d`, so it goes anywhere on the line. Bench the option, the flag and the clash error with the data and hand fixture authors before merging; the error names the option or the flag. A clash is found from the categories each source's index names and the files each source without a manifest holds, without parsing them. Revises the decision "A `--data-path` override rebinds every reference to the category it replaces", ending its conflict with goal 9.3. The changelog names the flag, the option and the new refusal.
|
||||
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Proposed spellings: `WithReplace()` and a boolean `--replace` covering every `-d`, so it goes anywhere on the line. Bench the option, the flag and the clash error with the data and hand fixture authors before merging; the error names the option or the flag. A clash is found from the categories each source's index names and the files each source without a manifest holds, without parsing them. Revises the decision "A `--data-path` override rebinds every reference to the category it replaces", ending its conflict with goal 9.3. The changelog names the flag, the option and the new refusal. The same `embed.FS` passed twice, as `svse.Modules()` beside `data.Modules()` passes `geose.FS`, loads once and is no clash.
|
||||
|
||||
### 167. Lower the reading cost of the files a chunk reads most before its first commit.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user