Answer architecture round 6: a manifest's reads names namespaces, loadReached notes only those nothing loaded provides, a mounted module labels its files by its namespace, Source.Load checks its mount, and the README keeps namespace and reads in an edited copy

This commit is contained in:
2026-10-11 12:41:11 +02:00
committed by Lillem4n
parent 3750770735
commit 89cafb1a21
13 changed files with 102 additions and 78 deletions
+3 -3
View File
@@ -131,9 +131,9 @@ replacement, and each removed path, column or flag.
and the CLI carries them all. `New` loads the modules `WithDataFS(modules...)` names and
the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option
is given. A manifest's `namespace` mounts its source under a dot path, and its `reads` lists the
modules its source reads by default; a reference to a path nothing provides fails to
load, asking whether a module is missing and naming every module the loaded manifests
read, or saying there are none.
namespaces of the modules its source reads by default; a reference to a path nothing
provides fails to load, asking whether a module is missing and naming each namespace
read that nothing loaded provides, or saying no loaded manifest names a module to add.
- A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
category on the first call reaching it, as each shipped module does. Data failing to load,
in `New` or on that first call, gives an error matching `ErrLoad`, and the CLI exits 1.
+14 -13
View File
@@ -184,23 +184,24 @@ every category of its source, so a category file it leaves out never loads.
The index maps each category's dot path to an entry: `parent`, the table it links to,
which needs its own entry in the same folder; and `paths`, every path `List` advertises
below the category, `""` for the category itself. Each shipped module's manifest, such as
[`data/svse/.fejkdata.json`](data/svse/.fejkdata.json), is one, so delete it from a
copy you edit, or keep its index current. A table whose parent column differs from its
entry's `parent` fails at first reach, and a table linking under another source's table
needs a source without an index. `//go:embed` of a directory leaves the manifest out, and
the source then loads whole in `New`; name it in the pattern too.
[`data/svse/.fejkdata.json`](data/svse/.fejkdata.json), carries one, so in a copy you
edit, delete its `index` and keep its `namespace` and `reads`, or keep the index current.
A table whose parent column differs from its entry's `parent` fails at first reach, and a
table linking under another source's table needs a source without an index. `//go:embed`
of a directory leaves the manifest out, and the source then loads whole in `New` and at
the root, with no `namespace`; name the manifest in the pattern too.
A manifest's `namespace` is the dot path its source's root sits under: a folder holding
`person.json` with `"namespace": "sv_SE"` provides `sv_SE.person`, its subfolders adding
segments below that. Each shipped module carries one, so `data/svse/person.json` is
`sv_SE.person`.
A manifest's `reads` lists, by any name its author chooses, every module the source's
references read unless another source provides the same paths, directly or through
another module. Nothing loads these modules: pass them yourself. A shipped module names a
Go package: `svse`'s manifest reads `github.com/larvit/fejkdata/data/geose`. Data whose
reference names a path nothing provides fails to load, asking whether a module is
missing and naming every module the loaded manifests read, or saying they name none.
A manifest's `reads` lists the namespace of every module the source's references read
unless another source provides the same paths, directly or through another module.
Nothing loads these modules: pass them yourself. `svse`'s manifest reads `geo.SE`. Data
whose reference names a path nothing provides fails to load; where a namespace the loaded
manifests read is one nothing loaded provides, the error asks whether a module is missing
and names it, and otherwise it says no loaded manifest names a module to add.
`namespace` and `reads` need no index.
Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
@@ -305,7 +306,7 @@ What the two locales do not share: `en_US.address` carries a `region` column the
one has no use for, and `sv_SE.title` has no `parent`, so it is selected as
`sv_SE.title[dr]` rather than inside a sex.
A `geo` folder holds one tree per country under its alpha-2 code: five
The `geo` namespace holds one tree per country under its alpha-2 code: five
[linked tables](#linked-tables) named alike, and an `address` record binding one locality
and drawing its street and postal code inside it. The locale's `address` prints that
record, and each of its columns describes the address printed. Do not assert that a Swedish
@@ -377,7 +378,7 @@ import (
f, err := fejkdata.New(fejkdata.WithDataFS(svse.FS, geose.FS))
```
`enus` reads `geous`; `misc` reads none. `New` fails where no `WithDataFS` or
`enus` reads `geous`; `misc` and the geo packages read none. `New` fails where no `WithDataFS` or
`WithDataPath` names the data to load.
`New` refuses data that breaks the grammar, could mean two things or cannot render a valid
+8 -8
View File
@@ -9,12 +9,12 @@ import (
// loadSources merges sources in order, the last winning a clash. A source whose manifest
// carries an index leaves its categories unloaded; every other source loads here, with
// the unloaded categories it needs. It also returns, sorted, every module the sources'
// manifests read by default.
// the unloaded categories it needs. It also returns, sorted, every namespace the sources'
// manifests read.
// docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads
func loadSources(sources []datafiles.Source) (folder, []string, error) {
root := folder{children: map[string]node{}}
var modules []string
var reads []string
for _, src := range sources {
m, err := src.Manifest()
if err != nil {
@@ -23,14 +23,14 @@ func loadSources(sources []datafiles.Source) (folder, []string, error) {
if err := addSource(&root, src, m); err != nil {
return folder{}, nil, err
}
for _, module := range m.Reads {
if !slices.Contains(modules, module) {
modules = append(modules, module)
for _, namespace := range m.Reads {
if !slices.Contains(reads, namespace) {
reads = append(reads, namespace)
}
}
}
slices.Sort(modules)
return root, modules, loadReached(&root, categorySites(&root), nil)
slices.Sort(reads)
return root, reads, loadReached(&root, reads, categorySites(&root), nil)
}
func addSource(root *folder, src datafiles.Source, m datafiles.Manifest) error {
+1 -1
View File
@@ -24,5 +24,5 @@
"en_US.word": {"paths":[""]}
},
"namespace": "en_US",
"reads": ["github.com/larvit/fejkdata/data/geous"]
"reads": ["geo.US"]
}
+1 -1
View File
@@ -24,5 +24,5 @@
"sv_SE.word": {"paths":[""]}
},
"namespace": "sv_SE",
"reads": ["github.com/larvit/fejkdata/data/geose"]
"reads": ["geo.SE"]
}
+5 -3
View File
@@ -140,7 +140,7 @@ A directory a machine happens to have would make `--seed 42` machine-dependent;
2026-10-04, Lilleman auf Larv; revised 2026-10-09 to put the packages in the core's Go module. Serves goals 2.3, 6.2, 7.3, 9 and 10.1. Carried out by `todo.md` items 181, 184, 185 and 186. The CLI binary may grow to hundreds of MB.
- One Go package per locale, one per country's `geo/` tree, and one for `misc`, all in the core's Go module. A package registers nothing when imported.
- One Go package per locale, one per country's `geo` tree, and one for `misc`, all in the core's Go module. A package registers nothing when imported.
- A bare `New()` loads no data and fails, naming the option to add. `WithoutShippedData` goes.
- A data package's `FS` is an `fs.FS` passed to `WithDataFS`, as anyone's data is, and `New` takes several. It is an exported `var FS embed.FS`, the shape `//go:embed` gives; a function returning it would only stop a program reassigning the FS variable of a package it imports. A module carrying functions loads through an option of its own, so a data module that starts carrying functions breaks its users. Such a function reaches the randomness through an interface the root declares; `internal/drawstate`'s type stays internal.
- A module names, in a manifest, every module it reads by default, directly or through another, so the first error names the whole set to import. A manifest is optional: a `--data-path` folder without one is a module. Nothing loads a default on its own.
@@ -160,7 +160,9 @@ custom-CLI builder. Serves goals 1 and 3.3. `sv_SE` ships as `data/svse`, `en_US
Go linters flag a capital or an underscore in a package name, and `SE.FS` read as a type.
Each package holds its files at its root, and its manifest's `namespace` puts them under
the namespace, so no path repeats it. The bench also chose `reads` for what a module reads
by default and the wording of the errors naming it. Valid until a locale's lowered name
by default and the wording of the errors naming it. A `reads` entry is a namespace, so the
error names only what nothing loaded provides, and with every module loaded a typo names
none. Valid until a locale's lowered name
clashes with another's.
## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads
@@ -592,7 +594,7 @@ when a source carries it.
2026-09-24, Lilleman auf Larv; revised 2026-10-10, when the shipped data became Go packages.
Serves goal 6.2. `svse.FS` loads alone, and `sv_SE.address` then fails at first reach,
naming `github.com/larvit/fejkdata/data/geose`, which `svse`'s manifest reads.
naming `geo.SE`, which `svse`'s manifest reads and nothing loaded provides.
`New` does not check that each module a manifest reads was given. A check by module name
would refuse a stand-in providing the same paths (goal 9.2). A check by path needs every
reference, and only loading each indexed category finds them, which goal 13 rules out.
+9 -9
View File
@@ -31,15 +31,15 @@ type Generator struct {
drawState *drawstate.State
root folder // the categories as the node a path walks from, owned here so a walk allocates none
structs map[reflect.Type]structResult
// defaultModules is every module the loaded sources' manifests read by default.
defaultModules []string
// manifestReads is every namespace the loaded sources' manifests read.
manifestReads []string
}
type config struct {
seed uint64
seeded bool
sources []datafiles.Source
err error
nilModules error
// given is whether an option named the data to load, though it may name none.
given bool
}
@@ -69,7 +69,7 @@ func WithDataFS(modules ...fs.FS) Option {
return func(c *config) {
for i, fsys := range modules {
if fsys == nil {
c.err = errors.Join(c.err, fmt.Errorf("fejkdata: WithDataFS module %d is nil", i+1))
c.nilModules = errors.Join(c.nilModules, fmt.Errorf("fejkdata: WithDataFS module %d is nil", i+1))
continue
}
c.sources = append(c.sources, datafiles.FS(fsys))
@@ -106,18 +106,18 @@ func New(opts ...Option) (*Generator, error) {
if !c.given {
return nil, errNoData
}
if c.err != nil {
return nil, c.err
if c.nilModules != nil {
return nil, c.nilModules
}
root, modules, err := loadSources(c.sources)
root, reads, err := loadSources(c.sources)
if err != nil {
return nil, fmt.Errorf("fejkdata: %w", withDefaultModules(loadError{err}, modules))
return nil, fmt.Errorf("fejkdata: %w", loadError{err})
}
seed, err := c.drawSeed()
if err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
return &Generator{drawState: drawstate.New(seed), root: root, defaultModules: modules}, nil
return &Generator{drawState: drawstate.New(seed), root: root, manifestReads: reads}, nil
}
// List returns the sorted dotted paths Fake renders: each category and every field,
+7 -5
View File
@@ -58,8 +58,8 @@ func (f *Generator) loadCallerPath(path string) (string, []string, error) {
return "", nil, fmt.Errorf("fejkdata: %w", err)
}
if c, ok := categoryOn(&f.root, segs); ok && c.unloaded() {
if err := loadReached(&f.root, nil, []categoryAt{c}); err != nil {
return "", nil, fmt.Errorf("fejkdata: %w", withDefaultModules(loadError{err}, f.defaultModules))
if err := loadReached(&f.root, f.manifestReads, nil, []categoryAt{c}); err != nil {
return "", nil, fmt.Errorf("fejkdata: %w", loadError{err})
}
}
return path, segs, nil
@@ -105,8 +105,9 @@ func needs(root *folder, s categorySite) []categoryAt {
// loadReached loads the unloaded categories among wanted and every unloaded category they
// or sites need, then runs sites and all it loaded through a whole load's pipeline. On an
// error it puts back what it loaded, so the next call fails the same way.
func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error {
// error it puts back what it loaded, so the next call fails the same way, and notes which
// namespace of reads, every namespace the loaded manifests read, nothing in root provides.
func loadReached(root *folder, reads []string, sites []categorySite, wanted []categoryAt) error {
queue := wanted
for _, s := range sites {
queue = append(queue, needs(root, s)...)
@@ -118,8 +119,9 @@ func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error
}
if err != nil {
b.putBack()
return withMissingModules(err, root, reads)
}
return err
return nil
}
// batch is the categories one loadReached loaded, each beside the entry it was loaded from.
+6 -5
View File
@@ -36,8 +36,8 @@ func (f *Generator) NewTemplate(input string) (*Template, error) {
}
f.mu.Lock()
defer f.mu.Unlock()
if err := loadInline(&f.root, n, "template", false); err != nil {
return nil, f.noDataNote(fmt.Errorf("fejkdata: %w", withDefaultModules(err, f.defaultModules)))
if err := loadInline(&f.root, f.manifestReads, n, "template", false); err != nil {
return nil, f.noDataNote(fmt.Errorf("fejkdata: %w", err))
}
return &Template{g: f, n: n}, nil
}
@@ -84,13 +84,14 @@ func inputValue(input string) jsonvalue.Value {
return raw
}
// loadInline loads the unloaded categories n reads, then runs n through the pipeline against root.
func loadInline(root *folder, n node, label string, typedByGo bool) error {
// loadInline loads the unloaded categories n reads, then runs n through the pipeline against root;
// reads is every namespace the loaded manifests read.
func loadInline(root *folder, reads []string, n node, label string, typedByGo bool) error {
nodes := inlineNodes(n, label)
if err := refuseFolderRefs(nodes); err != nil {
return err
}
if err := loadReached(root, nil, referenced(root, nil, nodes)); err != nil {
if err := loadReached(root, reads, nil, referenced(root, nil, nodes)); err != nil {
return loadError{err}
}
return pipeline{
+11 -2
View File
@@ -6,6 +6,7 @@ import (
"io/fs"
"os"
"path"
"slices"
"strings"
"github.com/larvit/fejkdata/internal/grammar"
@@ -30,10 +31,15 @@ func Dir(path string) Source {
// FS is the tree in fsys.
func FS(fsys fs.FS) Source { return Source{fsys: fsys} }
// Mounted is s with its root under the dot path namespace, "" for the namespace's root.
// Mounted is s with its root under the dot path namespace, "" for the namespace's root. A
// source with no label of its own names its files in errors by the namespace's folders.
func (s Source) Mounted(namespace string) Source {
if namespace != "" {
if namespace == "" {
return s
}
s.mount = strings.Split(namespace, ".")
if s.label == "" {
s.label = path.Join(s.mount...)
}
return s
}
@@ -139,6 +145,9 @@ func (s Source) walkFolder(dir []string, compile func(Category) error) (bool, er
// Load hands compile the category file name.json in the folder dir of the namespace, which
// lies under the source's mount.
func (s Source) Load(dir []string, name string, compile func(Category) error) error {
if len(dir) < len(s.mount) || !slices.Equal(dir[:len(s.mount)], s.mount) {
return fmt.Errorf("%s.%s lies outside the namespace %s", strings.Join(dir, "."), name, strings.Join(s.mount, "."))
}
full := s.dirPath(dir[len(s.mount):])
entries, err := fs.ReadDir(s.fsys, full)
if err != nil {
+13 -13
View File
@@ -17,12 +17,12 @@ const ManifestFile = ".fejkdata.json"
// Manifest is what a source's manifest says of it.
type Manifest struct {
// Index is every category of the source by dot path; nil where the manifest has none.
Index map[string]IndexEntry `json:"index"`
// Reads is every module the source reads by default, directly or through another;
// nothing loads them.
Reads []string `json:"reads"`
Index map[string]IndexEntry
// Namespace is the dot path the source's root sits under, "" for the namespace's root.
Namespace string `json:"namespace"`
Namespace string
// Reads is the namespace of every module the source reads unless another source
// provides the same paths, directly or through another; nothing loads them.
Reads []string
}
// IndexEntry is what an index says of a category: its parent table, "" for none, and the
@@ -108,7 +108,7 @@ func decodeNamespace(v any) (string, error) {
return namespace, nil
}
// decodeReads decodes the modules a source reads, each named by a non-empty string.
// decodeReads decodes the namespaces of the modules a source reads.
func decodeReads(v any) ([]string, error) {
items, isList := v.([]any)
if !isList {
@@ -118,17 +118,17 @@ func decodeReads(v any) ([]string, error) {
seen := map[string]bool{}
var errs []error
for i, item := range items {
module, isString := item.(string)
namespace, isString := item.(string)
switch {
case !isString:
errs = append(errs, fmt.Errorf("reads item %d must be a string, not %s", i+1, jsonvalue.Kind(item)))
case module == "":
errs = append(errs, fmt.Errorf("reads item %d is empty; it names a module", i+1))
case seen[module]:
errs = append(errs, fmt.Errorf("reads item %d repeats %q", i+1, module))
case checkCategoryPath(namespace) != nil:
errs = append(errs, fmt.Errorf("reads item %d: %w", i+1, checkCategoryPath(namespace)))
case seen[namespace]:
errs = append(errs, fmt.Errorf("reads item %d repeats %q", i+1, namespace))
default:
seen[module] = true
out = append(out, module)
seen[namespace] = true
out = append(out, namespace)
}
}
return out, errors.Join(errs...)
+14 -6
View File
@@ -121,17 +121,25 @@ type noEntry struct{ seg string }
func (e noEntry) Error() string { return fmt.Sprintf("no entry %q", e.seg) }
// withDefaultModules adds to err, where data fails to load because a segment names nothing, the
// modules the loaded sources' manifests read by default, or that they name none.
func withDefaultModules(err error, modules []string) error {
// withMissingModules adds to err, where a path names nothing, each namespace of reads that
// nothing in root provides, as the modules a loaded manifest reads and none loaded; or else
// that no loaded manifest names a module to add, the mistake being the path's own.
func withMissingModules(err error, root *folder, reads []string) error {
var ne noEntry
if !errors.Is(err, ErrLoad) || !errors.As(err, &ne) {
if !errors.As(err, &ne) {
return err
}
if len(modules) == 0 {
var missing []string
for _, namespace := range reads {
segs := strings.Split(namespace, ".")
if _, i := folderAt(root, segs); i < len(segs) {
missing = append(missing, namespace)
}
}
if len(missing) == 0 {
return fmt.Errorf("%w, and no loaded manifest names a module to add", err)
}
return fmt.Errorf("%w; is a module missing? The loaded modules read %s", err, strings.Join(modules, ", "))
return fmt.Errorf("%w; is a module missing? The loaded modules read %s, which nothing loaded provides", err, strings.Join(missing, ", "))
}
// noDataNote adds to err, where a segment names nothing and f holds no data at all, that no
+6 -5
View File
@@ -31,7 +31,7 @@ func (f *Generator) FakeStruct(v any) error {
defer f.mu.Unlock()
shape, err := f.structShapeOf(p.Elem().Type())
if err != nil {
return fmt.Errorf("fejkdata: %w", withDefaultModules(err, f.defaultModules))
return fmt.Errorf("fejkdata: %w", err)
}
shape.fill(f.drawState, p.Elem())
return nil
@@ -55,7 +55,7 @@ func (f *Generator) structShapeOf(t reflect.Type) (*structShape, error) {
if label == "" {
label = "struct"
}
sc := &structCompile{root: &f.root, visiting: map[reflect.Type]bool{}, structsLeft: maxStructs}
sc := &structCompile{root: &f.root, reads: f.manifestReads, visiting: map[reflect.Type]bool{}, structsLeft: maxStructs}
shape, err := sc.compileShape(t, label)
if f.structs == nil {
f.structs = map[reflect.Type]structResult{}
@@ -85,6 +85,7 @@ func (s *structShape) empty() bool { return s.record == nil && len(s.nested) ==
// rather than filled without end, and how many more structs it may walk.
type structCompile struct {
root *folder
reads []string
visiting map[reflect.Type]bool
structsLeft int
}
@@ -111,7 +112,7 @@ func (sc *structCompile) compileShape(t reflect.Type, label string) (*structShap
return nil, err
}
if len(c.tags) > 0 {
if err := c.shape.compileRecord(sc.root, t, label, c.tags); err != nil {
if err := c.shape.compileRecord(sc.root, sc.reads, t, label, c.tags); err != nil {
return nil, err
}
}
@@ -255,13 +256,13 @@ func checkTaggedType(sf reflect.StructField) error {
// compileRecord compiles the tagged fields of t as one record, and proves each column holds
// only what its field's Go type can.
func (s *structShape) compileRecord(root *folder, t reflect.Type, label string, tags map[string]jsonvalue.Value) error {
func (s *structShape) compileRecord(root *folder, reads []string, t reflect.Type, label string, tags map[string]jsonvalue.Value) error {
tags["format"] = jsonvalue.String("")
n, err := compileFrom(jsonvalue.Object(tags), atGoTop)
if err != nil {
return fmt.Errorf("%s: %w", label, err)
}
if err := loadInline(root, n, label, true); err != nil {
if err := loadInline(root, reads, n, label, true); err != nil {
return err
}
record, err := recordOf(n)