From c783d4af23dc9e4704dfbbcba0c3d197037de48a Mon Sep 17 00:00:00 2001 From: Lilleman auf Larv Date: Fri, 9 Oct 2026 19:26:54 +0200 Subject: [PATCH] Answer architecture round 1: one owner keeps a folder's loaded and unloaded names apart, each loaded category is checked against its index entry, addSource returns what it replaced, the manifest cache goes, ErrLoad exits 1, and the docs say what New and Fake now refuse --- AGENTS.md | 2 +- CHANGELOG.md | 7 +- README.md | 30 +++-- cmd/fejkdata/main.go | 12 +- data.go | 83 ++++++------ docs/comprehension-history.md | 23 ++++ docs/decisions.md | 12 +- fejkdata.go | 5 +- index.go | 222 ++++++++++++++++++++------------ inline.go | 2 +- internal/datafiles/datafiles.go | 98 +++++++------- node.go | 16 ++- todo.md | 7 +- 13 files changed, 319 insertions(+), 200 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 02aa787..1e98851 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,7 @@ In [docs/decisions.md](docs/decisions.md): - `FakeTemplate` and `NewTemplate` both stay - The shipped data is embedded, not discovered - The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0 -- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces +- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category depending on what it provides or a later source replaced - Every `{…}` draws afresh, and only a name keeps a pick - Reference sigils follow the filesystem - A change to what exists is a major; a minor only adds diff --git a/CHANGELOG.md b/CHANGELOG.md index bde91b9..e677fd4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,6 +126,7 @@ replacement, and each removed path, column or flag. `sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record that gives it one. - A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each - category on the first call reaching it, as the shipped set does. A source without one - loads in `New`, with the shipped categories it reads and every shipped category - reaching one it replaces; the rest of the shipped set still loads on first reach. + category on the first call reaching it, as the shipped set does, and a category failing + there returns an error wrapping `ErrLoad`, on which the CLI exits 1. A source without + one loads in `New`, with the shipped categories it reads and every shipped category + depending on one it replaces; the rest of the shipped set still loads on first reach. diff --git a/README.md b/README.md index 8e87b36..9607294 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ template holding one from a quoted heredoc or a file. The rest of stdin's rules `--name value` and `--name=value` both work, a short flag's value attaches or follows (`-n3`, `-n 3`) and short flags bundle (`-hn 3`) — see [Decisions](docs/decisions.md#flags-follow-getopt_long); flags go anywhere, `--` ends them. Exit codes: `0` success, `1` runtime error (missing -dir, a lone reference to nothing under `--format`), `2` misuse — a bad flag, an argument +dir, data that fails to load, a lone reference to nothing under `--format`), `2` misuse — a bad flag, an argument other than a flag, an unreadable stdin, nothing piped in, or a template that does not compile. From a checkout: `go run ./cmd/fejkdata …`. @@ -173,10 +173,15 @@ entry is never a category or a folder, so a data directory can also be a checkou a category may name a hidden rows file beside it. A source whose root holds `.fejkdata.json` with an `index` loads each category on the -first call reaching it, and its author proves each loads. The shipped set's +first call reaching it, and its author proves each loads. Any other source loads in +`New`, and so does every indexed category that depends on what such a source provides +or on a category a later source replaced. The index names every category of its source, +so a category file it leaves out never loads. The shipped set's [`data/.fejkdata.json`](data/.fejkdata.json) shows the shape: each category's dot path, -its `parent` table, the `paths` `List` advertises below it, and the categories it -`reads`. `//go:embed` of a directory leaves the manifest out; name it in the pattern too. +its `parent` table, every path `List` advertises below it, `""` for the category itself, +and the other categories its templates `reads`. A category whose entry says otherwise +fails at first reach, naming the entry it calls for. `//go:embed` of a directory leaves +the manifest out; name it in the pattern too. Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`, `ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`, @@ -336,13 +341,14 @@ ok, err := fejkdata.IsTemplate(arg) // an inline template by its shape, el | `WithoutShippedData()` | load only what you give | `New` refuses data that breaks the grammar, could mean two things or cannot render a valid -value, and `NewTemplate` refuses such a template. On a loaded generator: +value, and `NewTemplate` refuses such a template. A category of an indexed source is +checked on the first call reaching it instead, and a failure there wraps `ErrLoad`, the +same error every call. On a loaded generator: -- `Fake` fails only for a path that names nothing, could name two things, reads one - draw of a level carrying a `repeat`, or reaches a category of an indexed source that - fails to load, with the same error every call. +- `Fake` fails only for a path that names nothing, could name two things, or reads one + draw of a level carrying a `repeat`, with the same error every call, or with `ErrLoad`. - `FakeStruct` fails only for a non-struct argument or a type its tags do not describe, - with the same error every call. + with the same error every call, or with `ErrLoad`. - `Template.Fake` cannot fail. A path, wherever a call takes one, may start with one `/`, as a reference does. @@ -842,9 +848,7 @@ The target is about a microsecond per value (goal 13). A source with an [index](#data), as the shipped set has, loads nothing in `New`: a category loads on the first `Fake`, `FakeRecord`, `FakeStruct` or template that reads -it, with every category it reads and its whole table family. A source without one loads -in `New`, with the indexed categories it reads and every indexed category reaching one -it replaces, so each mistake it causes is `New`'s error. Each file is parsed, validated and weight-indexed once, bar a +it, with every category it reads and its whole table family. Each file is parsed, validated and weight-indexed once, bar a table's name and parent columns, which are mapped on the first `Fake` that selects by name or descends through it. A `Fake` call then costs about what its output costs: an unweighted pick is O(1) whatever the list's length, a weighted one @@ -866,7 +870,7 @@ and regenerate seeded fixtures and expected values when you raise that version. |---------|-------|-------| | Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another; change a table's key, name, weight or parent column, or remove a row | a path outside a record's columns, a locale, a value in a list, a row | | Records (a table is one) | remove, rename, retype or add a column; let a column be null | a record, as a new category | -| Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin | +| Data format | a fence: a spelling `New` rejects that it accepted, in a category or a manifest; a template option, since it reserves a field name | a builtin | | CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format | | Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option | diff --git a/cmd/fejkdata/main.go b/cmd/fejkdata/main.go index 77b9c27..056faae 100644 --- a/cmd/fejkdata/main.go +++ b/cmd/fejkdata/main.go @@ -340,7 +340,7 @@ func (in invocation) write(f *fejkdata.Generator, src input, w io.Writer) error } t, err := f.NewTemplate(src.template) if err != nil { - return inputError{err} + return templateError(err) } out := bufio.NewWriter(w) for i := 0; i < in.repeat; i++ { @@ -359,7 +359,7 @@ func (in invocation) recordStream(f *fejkdata.Generator, src input) (func() (*fe if src.record == "" { t, err := f.NewRecordTemplate(src.template) if err != nil { - return nil, "", inputError{spacedReference(src.template, err)} + return nil, "", templateError(spacedReference(src.template, err)) } record = func() (*fejkdata.Record, error) { return t.Fake(), nil } } @@ -621,6 +621,14 @@ func readStdin(stdin io.Reader) (input, error) { return parseInput(string(raw)), nil } +// templateError is misuse unless the data the template reads failed to load. +func templateError(err error) error { + if errors.Is(err, fejkdata.ErrLoad) { + return err + } + return inputError{err} +} + func fail(stderr io.Writer, err error) int { var te inputError if errors.As(err, &te) { diff --git a/data.go b/data.go index 4a0be88..7351834 100644 --- a/data.go +++ b/data.go @@ -7,64 +7,65 @@ import ( ) // loadSources merges sources in order, the last winning a clash. A source whose manifest -// carries an index leaves its categories unloaded; every other source's categories load -// here, with what they reach and every unloaded category leading to one of them or to one a -// later source replaced. +// carries an index leaves its categories unloaded; every other source loads here. So do +// the unloaded categories that depend on one it loaded or on one a later source replaced, +// so that a reader such a change breaks fails New, not a first reach. func loadSources(sources []datafiles.Source) (folder, error) { root := folder{children: map[string]node{}} - targets := map[string]bool{} + var changed []string for _, src := range sources { - before := categoriesUnder(&root) - provides, err := place(&root, src) + replaced, err := addSource(&root, src) if err != nil { return folder{}, err } - if len(before) == 0 { - continue - } - after := categoriesUnder(&root) - for p := range before { - if provides(p) || !after[p] { - targets[p] = true - } - } + changed = append(changed, replaced...) } sites := categorySites(&root) for _, s := range sites { - targets[s.path] = true + changed = append(changed, s.path) } - return root, loadReached(&root, sites, reaching(&root, targets)) + return root, loadReached(&root, sites, dependents(&root, changed)) } -// place adds the categories of src to root, and returns whether src provides a path. -func place(root *folder, src datafiles.Source) (func(string) bool, error) { +// addSource adds the categories of src to root, and returns the path of each category +// already there that src replaced or put out of reach. +func addSource(root *folder, src datafiles.Source) ([]string, error) { m, err := src.Manifest() if err != nil { return nil, err } + before := standing(root) if m.Index != nil { placeIndex(root, src, m.Index) - return func(p string) bool { _, ok := m.Index[p]; return ok }, nil + } else { + g := &folder{children: map[string]node{}} + if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil { + return nil, err + } + mergeFolder(root, g) } - g := &folder{children: map[string]node{}} - if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil { - return nil, err + if len(before) == 0 { + return nil, nil } - mergeFolder(root, g) - provided := categoriesUnder(g) - return func(p string) bool { return provided[p] }, nil + after := standing(root) + var replaced []string + for p, was := range before { + if after[p] != was { + replaced = append(replaced, p) + } + } + return replaced, nil } -// categoriesUnder is the path of every category under root, loaded or not. -func categoriesUnder(root *folder) map[string]bool { - out := map[string]bool{} +// standing is what stands at the path of every category under root: its node, or its +// unloaded entry. +func standing(root *folder) map[string]any { + out := map[string]any{} for _, s := range categorySites(root) { - out[s.path] = true + out[s.path] = s.n } - unloaded := map[string]categoryAt{} - unloadedUnder(root, nil, unloaded) - for p := range unloaded { - out[p] = true + for p, c := range unloadedUnder(root) { + out[p] = c.in.unloaded[c.name] } return out } @@ -75,7 +76,7 @@ func compileInto(place func(dir []string) *folder) func(datafiles.Category) erro if err != nil { return err } - place(c.Folders).children[c.Name] = n + place(c.Folders).put(c.Name, n) return nil } } @@ -87,8 +88,7 @@ func madeFolder(root *folder, dir []string) *folder { sub, isFolder := g.children[seg].(*folder) if !isFolder { sub = &folder{children: map[string]node{}} - g.children[seg] = sub - delete(g.unloaded, seg) + g.put(seg, sub) } g = sub } @@ -103,8 +103,7 @@ func mergeFolder(dst, src *folder) { continue } } - dst.children[k] = v - delete(dst.unloaded, k) + dst.put(k, v) } } @@ -116,10 +115,10 @@ type nodeSet func(fn func(label string, n node) error) error // folder's path, its name and its own dot path. type categorySite struct { dir []string - name string - path string in *folder n node + name string + path string } // categorySites lists every category under root, folders and names in sorted order. @@ -140,7 +139,7 @@ func categorySites(root *folder) []categorySite { } func siteIn(dir []string, in *folder, name string) categorySite { - return categorySite{dir: dir, name: name, path: categoryPath(dir, name), in: in, n: in.children[name]} + return categorySite{dir: dir, in: in, n: in.children[name], name: name, path: categoryPath(dir, name)} } func categoryPath(dir []string, name string) string { return join(strings.Join(dir, "."), name) } diff --git a/docs/comprehension-history.md b/docs/comprehension-history.md index b6d8510..9a3d38d 100644 --- a/docs/comprehension-history.md +++ b/docs/comprehension-history.md @@ -179,3 +179,26 @@ Ruling: same |---|---|---|---|---|---| | Mid A | same | same | same | same | same | | Inherited architect | same | same | same | same | same | + +## 2026-10-09T17:21:57Z, PR #202 at 65b2de8, against 2652779 + +Ruling: worse + +| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall | +|---|---|---|---|---|---| +| Mid A | same | worse | same | same | worse | +| Inherited architect | same | worse | worse | worse | worse | + +Mid A, decided by: + +- Locality, Overall: `data.go:13` `loadSources`: one pass mixes lazy and eager loading through before/after snapshots, a `provides` closure and a reverse graph search, and the skip of the first source is unexplained +- Locality: `data.go:107` `mergeFolder`: a category must sit in one of two maps, `children` or `unloaded`, kept by deletes in four functions across two files +- Locality: `internal/datafiles/datafiles.go:83` `embeddedManifests`: a process-wide cache in a package that was otherwise pure + +Inherited architect, decided by: + +- Locality, Self-sufficiency: `data.go:13` `loadSources`: nothing says why replaced paths drag their readers in, or what the `!after[p]` case covers +- Locality: `index.go:113` `loadReached`: its rollback claims the next call fails the same way, which holds only while no parent table was linked to a rolled-back child +- Shape: `index.go:159` `reaching`: four near-synonyms, `reaching`, `reachedFrom`, `loadReached` and `referenced`, and `referenced` and `categoryUnder` return loaded categories too +- Shape: `data.go:40` `place`: shares its name with `compileInto`'s parameter, and returns a closure answering one question two ways +- Locality: `internal/datafiles/datafiles.go:83` `embeddedManifests`: hidden global state for a small saving diff --git a/docs/decisions.md b/docs/decisions.md index 9fd40ae..608f2d8 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -154,7 +154,7 @@ still lives in `data/`; `--data-path` layers over it. Valid while the shipped data stays under 100 MiB, which a gate test holds (`todo.md` item 192): `go get` fetches the whole Go module, and the `go` command refuses a Go module zip over 500 MiB (`MaxZipFile` in `golang.org/x/mod/zip`). Past it, `todo.md` item 193 moves the largest packages into nested Go modules that keep their import paths, each requiring the core release that drops the package, so no build sees it twice. -## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces +## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category depending on what it provides or a later source replaced 2026-10-03, a restructure the maintainer approved; widened to any source on 2026-10-09. @@ -168,10 +168,12 @@ load. `go generate` writes the shipped set's manifest from `data/`; and each shipped category alone into a fresh `New`, so none fails at first reach. Goal 6: a source without an index loads whole in `New`, so each mistake in it is `New`'s -error. So does every indexed category it reads, and every indexed category leading -through references or table links to one that it or any later source replaced, so a -replacement breaking a reader fails `New`. Goal 6.3: an indexed source from anyone else -may fail at first reach; `Fake` then returns the load's error, the same every call. +error. So does every indexed category it reads, and every indexed category depending, +through references or table links, on a category it provides or a later source +replaced, so a replacement or a stand-in breaking a reader fails `New`. Goal 6.3: an +indexed source from anyone else may fail at first reach; `Fake` then returns the load's +error wrapping `ErrLoad`, the same every call. A category whose index entry disagrees +with it fails there too, so a stale index never decides what loads. Valid while CI proves the shipped set whole and each category alone. diff --git a/fejkdata.go b/fejkdata.go index 76fc960..c749b59 100644 --- a/fejkdata.go +++ b/fejkdata.go @@ -73,8 +73,7 @@ func WithoutShippedData() Option { // namespace segment. It errors on a missing directory, invalid JSON or invalid data; a // generator with no data at all renders templates that read none. A source whose // .fejkdata.json carries an index, as the shipped set's does, loads each category on the -// first call reaching it. Every other source loads here, with what it reads and every -// indexed category reaching what it replaces. +// first call reaching it, and fails there with [ErrLoad]. func New(opts ...Option) (*Generator, error) { c := config{shipped: true} for _, opt := range opts { @@ -91,7 +90,7 @@ func New(opts ...Option) (*Generator, error) { return &Generator{drawState: drawstate.New(seed), root: root}, nil } -// load builds the tree New starts from. docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads-and-every-indexed-category-reaching-what-it-replaces +// load builds the tree New starts from. docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads-and-every-indexed-category-depending-on-what-it-provides-or-a-later-source-replaced func (c config) load() (folder, error) { var sources []datafiles.Source if c.shipped { diff --git a/index.go b/index.go index 74abd1e..1456d4e 100644 --- a/index.go +++ b/index.go @@ -1,17 +1,25 @@ package fejkdata import ( + "encoding/json" + "errors" "fmt" + "slices" + "sort" "strings" "github.com/larvit/fejkdata/internal/datafiles" "github.com/larvit/fejkdata/internal/grammar" ) +// ErrLoad marks the error of a category that fails to load on the first call reaching it, +// which only a source whose manifest carries an index defers to then. +var ErrLoad = errors.New("data fails to load") + // indexed is a category an index names and no call has reached yet, and the source holding it. type indexed struct { - src datafiles.Source entry datafiles.IndexEntry + src datafiles.Source } // placeIndex sets every category index names unloaded in root, in place of what root holds @@ -19,17 +27,12 @@ type indexed struct { func placeIndex(root *folder, src datafiles.Source, index map[string]datafiles.IndexEntry) { for _, p := range sortedNames(index) { segs := strings.Split(p, ".") - g := madeFolder(root, segs[:len(segs)-1]) - name := segs[len(segs)-1] - delete(g.children, name) - if g.unloaded == nil { - g.unloaded = map[string]indexed{} - } - g.unloaded[name] = indexed{src: src, entry: index[p]} + madeFolder(root, segs[:len(segs)-1]).putUnloaded(segs[len(segs)-1], &indexed{entry: index[p], src: src}) } } -// categoryAt is where a category sits: the folder holding it, that folder's path, and its name. +// categoryAt is where a category, loaded or not, sits: the folder holding it, that +// folder's path, and its name. type categoryAt struct { dir []string in *folder @@ -38,8 +41,8 @@ type categoryAt struct { func (c categoryAt) path() string { return categoryPath(c.dir, c.name) } -// categoryUnder is the category, loaded or not, that a path from root names or descends into. -func categoryUnder(root *folder, segs []string) (categoryAt, bool) { +// categoryOn is the category, loaded or not, that a path from root names or passes. +func categoryOn(root *folder, segs []string) (categoryAt, bool) { g, i := folderAt(root, segs) if i == len(segs) { return categoryAt{}, false @@ -51,7 +54,7 @@ func categoryUnder(root *folder, segs []string) (categoryAt, bool) { } // loadCallerPath strips the / a caller's path may start with, splits the path, and loads the -// unloaded category it names or descends into. The caller walks it next, and a walk loads nothing. +// unloaded category it names or passes. The caller walks it next, and a walk loads nothing. func (f *Generator) loadCallerPath(path string) (string, []string, error) { path, err := grammar.CallerPath(path) if err != nil { @@ -61,16 +64,16 @@ func (f *Generator) loadCallerPath(path string) (string, []string, error) { if err != nil { return "", nil, fmt.Errorf("fejkdata: %w", err) } - if c, ok := categoryUnder(&f.root, segs); ok { + if c, ok := categoryOn(&f.root, segs); ok { if err := loadReached(&f.root, nil, []categoryAt{c}); err != nil { - return "", nil, fmt.Errorf("fejkdata: %w", err) + return "", nil, fmt.Errorf("fejkdata: %w: %w", ErrLoad, err) } } return path, segs, nil } -// referenced is every category the templates of nodes reference, each reference read from -// the folder dir. +// referenced is every category, loaded or not, that the templates of nodes reference, each +// reference read from the folder dir. func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt { var out []categoryAt _ = nodes(func(_ string, n node) error { @@ -83,7 +86,7 @@ func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt { if err != nil { continue // resolveRefs reports it } - if c, ok := categoryUnder(root, segs); ok { + if c, ok := categoryOn(root, segs); ok { out = append(out, c) } } @@ -92,12 +95,12 @@ func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt { return out } -// linkedTables is the table beside s that its parent column names, and every unloaded -// table beside s whose parent s is. -func linkedTables(s categorySite) []categoryAt { - var out []categoryAt - if t, isTable := s.n.(*table); isTable && t.rows.Options().Parent != "" { - out = append(out, categoryAt{dir: s.dir, in: s.in, name: t.rows.Options().Parent}) +// dependencies is every category, loaded or not, that s references, the table beside s +// that its parent column names, and every unloaded table beside s whose parent s is. +func dependencies(root *folder, s categorySite) []categoryAt { + out := referenced(root, s.dir, siteNodes([]categorySite{s})) + if parent := parentOf(s.n); parent != "" { + out = append(out, categoryAt{dir: s.dir, in: s.in, name: parent}) } for _, name := range sortedNames(s.in.unloaded) { if s.in.unloaded[name].entry.Parent == s.name { @@ -107,61 +110,14 @@ func linkedTables(s categorySite) []categoryAt { return out } -// loadReached loads the unloaded categories among wanted, and every unloaded category they -// or sites reference or link to, then runs sites and all it loaded through a whole load's -// pipeline. On an error it unloads again what it loaded, so the next call fails the same way. -func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error { - queue := wanted - for _, s := range sites { - queue = append(queue, reachedFrom(root, s)...) - } - var loaded []categorySite - var was []indexed - unload := func() { - for i, s := range loaded { - delete(s.in.children, s.name) - s.in.unloaded[s.name] = was[i] - } - } - for ; len(queue) > 0; queue = queue[1:] { - c := queue[0] - x, unloaded := c.in.unloaded[c.name] - if !unloaded { - continue - } - if err := x.src.Load(c.dir, c.name, compileInto(func([]string) *folder { return c.in })); err != nil { - unload() - return err - } - delete(c.in.unloaded, c.name) - s := siteIn(c.dir, c.in, c.name) - loaded, was = append(loaded, s), append(was, x) - queue = append(queue, reachedFrom(root, s)...) - } - sites = append(sites[:len(sites):len(sites)], loaded...) - if len(sites) == 0 { - return nil - } - if err := categoryPipeline(sites, root.children).run(); err != nil { - unload() - return err - } - return nil -} - -func reachedFrom(root *folder, s categorySite) []categoryAt { - return append(referenced(root, s.dir, siteNodes([]categorySite{s})), linkedTables(s)...) -} - -// reaching is every unloaded category under root whose index entry leads to a path in -// targets: through its reads, its parent or a child table, and through other unloaded +// dependents is every unloaded category under root whose index entry leads to one of +// paths: through its reads, its parent or a child table, and through other unloaded // categories. -func reaching(root *folder, targets map[string]bool) []categoryAt { - if len(targets) == 0 { +func dependents(root *folder, paths []string) []categoryAt { + if len(paths) == 0 { return nil } - unloaded := map[string]categoryAt{} - unloadedUnder(root, nil, unloaded) + unloaded := unloadedUnder(root) from := map[string][]string{} // a path, and the unloaded categories leading to it for _, p := range sortedNames(unloaded) { c := unloaded[p] @@ -177,7 +133,9 @@ func reaching(root *folder, targets map[string]bool) []categoryAt { } var out []categoryAt seen := map[string]bool{} - for queue := sortedNames(targets); len(queue) > 0; queue = queue[1:] { + queue := slices.Clone(paths) + sort.Strings(queue) + for ; len(queue) > 0; queue = queue[1:] { for _, p := range from[queue[0]] { c, isUnloaded := unloaded[p] if !isUnloaded || seen[p] { @@ -191,14 +149,110 @@ func reaching(root *folder, targets map[string]bool) []categoryAt { return out } -func unloadedUnder(g *folder, dir []string, out map[string]categoryAt) { - for name := range g.unloaded { - c := categoryAt{dir: dir, in: g, name: name} - out[c.path()] = c - } - for name, n := range g.children { - if sub, isFolder := n.(*folder); isFolder { - unloadedUnder(sub, append(dir[:len(dir):len(dir)], name), out) +func unloadedUnder(root *folder) map[string]categoryAt { + out := map[string]categoryAt{} + var walk func(g *folder, dir []string) + walk = func(g *folder, dir []string) { + for name := range g.unloaded { + c := categoryAt{dir: dir, in: g, name: name} + out[c.path()] = c + } + for name, n := range g.children { + if sub, isFolder := n.(*folder); isFolder { + walk(sub, append(dir[:len(dir):len(dir)], name)) + } } } + walk(root, nil) + return out +} + +// loadReached loads the unloaded categories among wanted, and every unloaded category they +// or sites depend on, then runs sites and all it loaded through a whole load's pipeline. +// On an error it puts back what it loaded, so the next call fails the same way. Checking +// each parent column against its entry before linking keeps that whole: a table then +// loads with its parent, so no table loaded earlier links to one put back. +func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error { + queue := wanted + for _, s := range sites { + queue = append(queue, dependencies(root, s)...) + } + type load struct { + site categorySite + was *indexed + } + var loads []load + putBack := func() { + for _, l := range loads { + l.site.in.putUnloaded(l.site.name, l.was) + } + } + for ; len(queue) > 0; queue = queue[1:] { + c := queue[0] + x, unloaded := c.in.unloaded[c.name] + if !unloaded { + continue + } + err := x.src.Load(c.dir, c.name, compileInto(func([]string) *folder { return c.in })) + if err == nil { + loads = append(loads, load{site: siteIn(c.dir, c.in, c.name), was: x}) + if parentOf(c.in.children[c.name]) != x.entry.Parent { + err = staleEntry(root, loads[len(loads)-1].site, x) + } + } + if err != nil { + putBack() + return err + } + queue = append(queue, dependencies(root, loads[len(loads)-1].site)...) + } + for _, l := range loads { + sites = append(sites[:len(sites):len(sites)], l.site) + } + if len(sites) == 0 { + return nil + } + err := categoryPipeline(sites, root.children).run() + for _, l := range loads { + if err == nil && !sameEntry(indexEntry(root, l.site), l.was.entry) { + err = staleEntry(root, l.site, l.was) + } + } + if err != nil { + putBack() + } + return err +} + +// indexEntry is what an index says of the loaded category s. +func indexEntry(root *folder, s categorySite) datafiles.IndexEntry { + e := datafiles.IndexEntry{Parent: parentOf(s.n), Paths: paths(s.n, false)} + sort.Strings(e.Paths) + seen := map[string]bool{s.path: true} + for _, c := range referenced(root, s.dir, siteNodes([]categorySite{s})) { + if p := c.path(); !seen[p] { + seen[p] = true + e.Reads = append(e.Reads, p) + } + } + sort.Strings(e.Reads) + return e +} + +func sameEntry(a, b datafiles.IndexEntry) bool { + return a.Parent == b.Parent && slices.Equal(a.Paths, b.Paths) && slices.Equal(a.Reads, b.Reads) +} + +// staleEntry names the entry x holds for s, and the entry s calls for. Before linking, +// the paths and reads it names may still lack what s's family adds. +func staleEntry(root *folder, s categorySite, x *indexed) error { + want, _ := json.Marshal(indexEntry(root, s)) + return fmt.Errorf("%s: the index entry for %s is stale; the category calls for %s", x.src.ManifestPath(), s.path, want) +} + +func parentOf(n node) string { + if t, isTable := n.(*table); isTable { + return t.rows.Options().Parent + } + return "" } diff --git a/inline.go b/inline.go index 6eeb03b..a4344c5 100644 --- a/inline.go +++ b/inline.go @@ -91,7 +91,7 @@ func loadInline(root *folder, n node, label string, typedByGo bool) error { return err } if err := loadReached(root, nil, referenced(root, nil, nodes)); err != nil { - return err + return fmt.Errorf("%w: %w", ErrLoad, err) } return pipeline{ nodes: nodes, diff --git a/internal/datafiles/datafiles.go b/internal/datafiles/datafiles.go index fa74d23..cb6148b 100644 --- a/internal/datafiles/datafiles.go +++ b/internal/datafiles/datafiles.go @@ -3,7 +3,6 @@ package datafiles import ( "bytes" - "embed" "encoding/json" "errors" "fmt" @@ -12,7 +11,6 @@ import ( "os" "path" "strings" - "sync" "github.com/larvit/fejkdata/internal/grammar" "github.com/larvit/fejkdata/internal/jsonvalue" @@ -79,47 +77,28 @@ type IndexEntry struct { Reads []string `json:"reads,omitempty"` } -// embeddedManifests holds each manifest read from an embed.FS, whose files never change. -var embeddedManifests sync.Map - -type embeddedManifest struct { - fsys embed.FS - base string -} - // Manifest reads the manifest at the source's root, the zero Manifest where there is none. func (s Source) Manifest() (Manifest, error) { - e, embedded := s.fsys.(embed.FS) - key := embeddedManifest{fsys: e, base: s.base} - if m, read := embeddedManifests.Load(key); embedded && read { - return m.(Manifest), nil + if err := s.check(); err != nil { + return Manifest{}, err } - m, err := s.readManifest() - if embedded && err == nil { - embeddedManifests.Store(key, m) - } - return m, err -} - -func (s Source) readManifest() (Manifest, error) { - if s.onDisk && s.label == "" { - return Manifest{}, nil // Walk refuses the empty data path - } - full := path.Join(s.base, ManifestFile) - b, err := fs.ReadFile(s.fsys, full) + b, err := fs.ReadFile(s.fsys, path.Join(s.base, ManifestFile)) if errors.Is(err, fs.ErrNotExist) { return Manifest{}, nil } if err != nil { - return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err) + return Manifest{}, fmt.Errorf("%s: %w", s.ManifestPath(), err) } m, err := decodeManifest(b) if err != nil { - return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err) + return Manifest{}, fmt.Errorf("%s: %w", s.ManifestPath(), err) } return m, nil } +// ManifestPath is the manifest's path as errors name it. +func (s Source) ManifestPath() string { return s.labelled(path.Join(s.base, ManifestFile)) } + func decodeManifest(b []byte) (Manifest, error) { dec := json.NewDecoder(bytes.NewReader(b)) dec.DisallowUnknownFields() @@ -130,36 +109,67 @@ func decodeManifest(b []byte) (Manifest, error) { if _, err := dec.Token(); err != io.EOF { return Manifest{}, fmt.Errorf("more than one JSON value") } - for p := range m.Index { - for _, seg := range strings.Split(p, ".") { - if err := grammar.CheckIdentifier(seg); err != nil { - return Manifest{}, fmt.Errorf("index names %q: %w", p, err) + for p, e := range m.Index { + for _, q := range append([]string{p}, e.Reads...) { + if err := checkCategoryPath(q); err != nil { + return Manifest{}, fmt.Errorf("index entry %q: %w", p, err) } } + if e.Parent == "" { + continue + } + if err := grammar.CheckIdentifier(e.Parent); err != nil { + return Manifest{}, fmt.Errorf("index entry %q: parent %w", p, err) + } + sibling := e.Parent + if dot := strings.LastIndexByte(p, '.'); dot >= 0 { + sibling = p[:dot+1] + e.Parent + } + if _, ok := m.Index[sibling]; !ok { + return Manifest{}, fmt.Errorf("index entry %q: parent %q names no entry beside it", p, e.Parent) + } } return m, nil } +func checkCategoryPath(p string) error { + for _, seg := range strings.Split(p, ".") { + if err := grammar.CheckIdentifier(seg); err != nil { + return fmt.Errorf("%q: %w", p, err) + } + } + return nil +} + // Walk hands compile every category of the tree, folders and files in name order. A // folder holding no category anywhere below it is skipped; a hidden file or folder is no // category, though a category may name a hidden rows file beside it. func (s Source) Walk(compile func(Category) error) error { - if s.onDisk { - if s.label == "" { - return fmt.Errorf("a data path is empty") - } - info, err := os.Stat(s.label) - if err != nil { - return fmt.Errorf("data path %s: %w", s.label, err) - } - if !info.IsDir() { - return fmt.Errorf("%s is not a directory", s.label) - } + if err := s.check(); err != nil { + return err } _, err := s.walkDir(nil, compile) return err } +// check refuses a data path that is empty or names no directory. +func (s Source) check() error { + if !s.onDisk { + return nil + } + if s.label == "" { + return fmt.Errorf("a data path is empty") + } + info, err := os.Stat(s.label) + if err != nil { + return fmt.Errorf("data path %s: %w", s.label, err) + } + if !info.IsDir() { + return fmt.Errorf("%s is not a directory", s.label) + } + return nil +} + // walkDir walks the folder that dir names, and reports whether it handed over any category. func (s Source) walkDir(dir []string, compile func(Category) error) (bool, error) { full := s.dirPath(dir) diff --git a/node.go b/node.go index 5ae9ca6..b25035b 100644 --- a/node.go +++ b/node.go @@ -20,13 +20,27 @@ type node interface{ isNode() } // folder is a namespace of named children, built from a directory of JSON files // and subdirectories. It has no value of its own: descend into a named child by // dot path; rendering one is an error (see Fake). +// A name stands in children or in unloaded, never both; put and putUnloaded keep it so. type folder struct { children map[string]node - unloaded map[string]indexed // the categories here an index names and no call has reached yet + unloaded map[string]*indexed // the categories here an index names and no call has reached yet } func (*folder) isNode() {} +func (g *folder) put(name string, n node) { + g.children[name] = n + delete(g.unloaded, name) +} + +func (g *folder) putUnloaded(name string, x *indexed) { + delete(g.children, name) + if g.unloaded == nil { + g.unloaded = map[string]*indexed{} + } + g.unloaded[name] = x +} + // choice draws one of its items. cum holds cumulative weights for a weighted // draw; when nil the choice is uniform and selection is O(1). shared is the set of // relative dot paths every item carries, counting paths inside a repeated template. diff --git a/todo.md b/todo.md index a070750..ec34a89 100644 --- a/todo.md +++ b/todo.md @@ -6,7 +6,7 @@ `Bar = 9` -`Next ID = 194` +`Next ID = 195` | Goal | W | |---|---| @@ -106,6 +106,7 @@ | 178 | 0.2.0 | | **Plan with the systems architect which `internal/` packages the engine splits into, by area, and how each type stops being written by more than one stage, then file the moves as items.** | 1 | 3 | 2 | 9 | 3, 3.2 | 18.4 | | 193 | 0.2.0 | decision | **Move the largest shipped packages into nested Go modules that keep their import paths, once item 192's gate fails.** | 6 | 6 | 6 | 7 | 2.3, 2 | 17.8 | | 92 | 0.2.0 | | **Name the hyphenated field in a `calc`'s refusal, and say to rename it: today `{calc(postal-code * 2)}` reports `no field "postal"`.** | 1 | 2 | 5 | 6 | 6 | 17.5 | +| 194 | 0.2.0 | | **Let a data author write and check their own `.fejkdata.json` index with the library or the CLI, and fail a load whose index leaves out a category file.** | 2 | 4 | 5 | 7 | 6.3, 9.1 | 17.5 | | 53 | 0.2.0 | | **Let a table column carry a `datatype`, so `--format json` writes `"safe": true` and `--format sql` a boolean.** | 4 | 5 | 6 | 6 | 4 | 16.9 | | 55 | 0.2.0 | | **Document `NewRecordTemplate`, `ErrNoColumns` and `MaxRepeat` in the README's Library section.** | 1 | 2 | 6 | 7 | 12 | 16.8 | | 18 | 0.2.0 | | **Give the address records one column set across countries: `region` and `municipality` as columns on `geo.SE.address` too.** | 4 | 4 | 6 | 6 | 5 | 16.6 | @@ -422,6 +423,10 @@ Carries out the decision "The shipped data is Go packages a library imports by c A hyphen is always subtraction in a calc, and the shipped data has hyphenated fields (`postal-code` and `street-number` in `geo/SE/address`). The refusal names the first fragment, not the field the author meant. +### 194. Let a data author write and check their own `.fejkdata.json` index with the library or the CLI, and fail a load whose index leaves out a category file. + +Goal 6.3 has an author prove their module, and goal 9.1 has fejkdata's data load as anyone's; today only a test of fejkdata's own writes and checks an index (`TestShippedManifestIsCurrent` over `indexEntry`). A category file the index leaves out never loads, silently. The principles audit of item 180's chunk found it. + ### 53. Let a table column carry a `datatype`, so `--format json` writes `"safe": true` and `--format sql` a boolean. Today only a JSON field takes one, so `misc.httpmethod`'s booleans are typed in Go and text everywhere else. Typing a shipped column changes what `json` and `sql` write, so the capability is a minor and applying it to `misc.httpmethod` is a major.