diff --git a/AGENTS.md b/AGENTS.md index 02aa787..1e98851 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,7 @@ In [docs/decisions.md](docs/decisions.md): - `FakeTemplate` and `NewTemplate` both stay - The shipped data is embedded, not discovered - The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0 -- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces +- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category depending on what it provides or a later source replaced - Every `{…}` draws afresh, and only a name keeps a pick - Reference sigils follow the filesystem - A change to what exists is a major; a minor only adds diff --git a/CHANGELOG.md b/CHANGELOG.md index bde91b9..e677fd4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,6 +126,7 @@ replacement, and each removed path, column or flag. `sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record that gives it one. - A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each - category on the first call reaching it, as the shipped set does. A source without one - loads in `New`, with the shipped categories it reads and every shipped category - reaching one it replaces; the rest of the shipped set still loads on first reach. + category on the first call reaching it, as the shipped set does, and a category failing + there returns an error wrapping `ErrLoad`, on which the CLI exits 1. A source without + one loads in `New`, with the shipped categories it reads and every shipped category + depending on one it replaces; the rest of the shipped set still loads on first reach. diff --git a/README.md b/README.md index 8e87b36..9607294 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ template holding one from a quoted heredoc or a file. The rest of stdin's rules `--name value` and `--name=value` both work, a short flag's value attaches or follows (`-n3`, `-n 3`) and short flags bundle (`-hn 3`) — see [Decisions](docs/decisions.md#flags-follow-getopt_long); flags go anywhere, `--` ends them. Exit codes: `0` success, `1` runtime error (missing -dir, a lone reference to nothing under `--format`), `2` misuse — a bad flag, an argument +dir, data that fails to load, a lone reference to nothing under `--format`), `2` misuse — a bad flag, an argument other than a flag, an unreadable stdin, nothing piped in, or a template that does not compile. From a checkout: `go run ./cmd/fejkdata …`. @@ -173,10 +173,15 @@ entry is never a category or a folder, so a data directory can also be a checkou a category may name a hidden rows file beside it. A source whose root holds `.fejkdata.json` with an `index` loads each category on the -first call reaching it, and its author proves each loads. The shipped set's +first call reaching it, and its author proves each loads. Any other source loads in +`New`, and so does every indexed category that depends on what such a source provides +or on a category a later source replaced. The index names every category of its source, +so a category file it leaves out never loads. The shipped set's [`data/.fejkdata.json`](data/.fejkdata.json) shows the shape: each category's dot path, -its `parent` table, the `paths` `List` advertises below it, and the categories it -`reads`. `//go:embed` of a directory leaves the manifest out; name it in the pattern too. +its `parent` table, every path `List` advertises below it, `""` for the category itself, +and the other categories its templates `reads`. A category whose entry says otherwise +fails at first reach, naming the entry it calls for. `//go:embed` of a directory leaves +the manifest out; name it in the pattern too. Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`, `ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`, @@ -336,13 +341,14 @@ ok, err := fejkdata.IsTemplate(arg) // an inline template by its shape, el | `WithoutShippedData()` | load only what you give | `New` refuses data that breaks the grammar, could mean two things or cannot render a valid -value, and `NewTemplate` refuses such a template. On a loaded generator: +value, and `NewTemplate` refuses such a template. A category of an indexed source is +checked on the first call reaching it instead, and a failure there wraps `ErrLoad`, the +same error every call. On a loaded generator: -- `Fake` fails only for a path that names nothing, could name two things, reads one - draw of a level carrying a `repeat`, or reaches a category of an indexed source that - fails to load, with the same error every call. +- `Fake` fails only for a path that names nothing, could name two things, or reads one + draw of a level carrying a `repeat`, with the same error every call, or with `ErrLoad`. - `FakeStruct` fails only for a non-struct argument or a type its tags do not describe, - with the same error every call. + with the same error every call, or with `ErrLoad`. - `Template.Fake` cannot fail. A path, wherever a call takes one, may start with one `/`, as a reference does. @@ -842,9 +848,7 @@ The target is about a microsecond per value (goal 13). A source with an [index](#data), as the shipped set has, loads nothing in `New`: a category loads on the first `Fake`, `FakeRecord`, `FakeStruct` or template that reads -it, with every category it reads and its whole table family. A source without one loads -in `New`, with the indexed categories it reads and every indexed category reaching one -it replaces, so each mistake it causes is `New`'s error. Each file is parsed, validated and weight-indexed once, bar a +it, with every category it reads and its whole table family. Each file is parsed, validated and weight-indexed once, bar a table's name and parent columns, which are mapped on the first `Fake` that selects by name or descends through it. A `Fake` call then costs about what its output costs: an unweighted pick is O(1) whatever the list's length, a weighted one @@ -866,7 +870,7 @@ and regenerate seeded fixtures and expected values when you raise that version. |---------|-------|-------| | Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another; change a table's key, name, weight or parent column, or remove a row | a path outside a record's columns, a locale, a value in a list, a row | | Records (a table is one) | remove, rename, retype or add a column; let a column be null | a record, as a new category | -| Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin | +| Data format | a fence: a spelling `New` rejects that it accepted, in a category or a manifest; a template option, since it reserves a field name | a builtin | | CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format | | Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option | diff --git a/cmd/fejkdata/main.go b/cmd/fejkdata/main.go index 77b9c27..056faae 100644 --- a/cmd/fejkdata/main.go +++ b/cmd/fejkdata/main.go @@ -340,7 +340,7 @@ func (in invocation) write(f *fejkdata.Generator, src input, w io.Writer) error } t, err := f.NewTemplate(src.template) if err != nil { - return inputError{err} + return templateError(err) } out := bufio.NewWriter(w) for i := 0; i < in.repeat; i++ { @@ -359,7 +359,7 @@ func (in invocation) recordStream(f *fejkdata.Generator, src input) (func() (*fe if src.record == "" { t, err := f.NewRecordTemplate(src.template) if err != nil { - return nil, "", inputError{spacedReference(src.template, err)} + return nil, "", templateError(spacedReference(src.template, err)) } record = func() (*fejkdata.Record, error) { return t.Fake(), nil } } @@ -621,6 +621,14 @@ func readStdin(stdin io.Reader) (input, error) { return parseInput(string(raw)), nil } +// templateError is misuse unless the data the template reads failed to load. +func templateError(err error) error { + if errors.Is(err, fejkdata.ErrLoad) { + return err + } + return inputError{err} +} + func fail(stderr io.Writer, err error) int { var te inputError if errors.As(err, &te) { diff --git a/data.go b/data.go index 4a0be88..7351834 100644 --- a/data.go +++ b/data.go @@ -7,64 +7,65 @@ import ( ) // loadSources merges sources in order, the last winning a clash. A source whose manifest -// carries an index leaves its categories unloaded; every other source's categories load -// here, with what they reach and every unloaded category leading to one of them or to one a -// later source replaced. +// carries an index leaves its categories unloaded; every other source loads here. So do +// the unloaded categories that depend on one it loaded or on one a later source replaced, +// so that a reader such a change breaks fails New, not a first reach. func loadSources(sources []datafiles.Source) (folder, error) { root := folder{children: map[string]node{}} - targets := map[string]bool{} + var changed []string for _, src := range sources { - before := categoriesUnder(&root) - provides, err := place(&root, src) + replaced, err := addSource(&root, src) if err != nil { return folder{}, err } - if len(before) == 0 { - continue - } - after := categoriesUnder(&root) - for p := range before { - if provides(p) || !after[p] { - targets[p] = true - } - } + changed = append(changed, replaced...) } sites := categorySites(&root) for _, s := range sites { - targets[s.path] = true + changed = append(changed, s.path) } - return root, loadReached(&root, sites, reaching(&root, targets)) + return root, loadReached(&root, sites, dependents(&root, changed)) } -// place adds the categories of src to root, and returns whether src provides a path. -func place(root *folder, src datafiles.Source) (func(string) bool, error) { +// addSource adds the categories of src to root, and returns the path of each category +// already there that src replaced or put out of reach. +func addSource(root *folder, src datafiles.Source) ([]string, error) { m, err := src.Manifest() if err != nil { return nil, err } + before := standing(root) if m.Index != nil { placeIndex(root, src, m.Index) - return func(p string) bool { _, ok := m.Index[p]; return ok }, nil + } else { + g := &folder{children: map[string]node{}} + if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil { + return nil, err + } + mergeFolder(root, g) } - g := &folder{children: map[string]node{}} - if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil { - return nil, err + if len(before) == 0 { + return nil, nil } - mergeFolder(root, g) - provided := categoriesUnder(g) - return func(p string) bool { return provided[p] }, nil + after := standing(root) + var replaced []string + for p, was := range before { + if after[p] != was { + replaced = append(replaced, p) + } + } + return replaced, nil } -// categoriesUnder is the path of every category under root, loaded or not. -func categoriesUnder(root *folder) map[string]bool { - out := map[string]bool{} +// standing is what stands at the path of every category under root: its node, or its +// unloaded entry. +func standing(root *folder) map[string]any { + out := map[string]any{} for _, s := range categorySites(root) { - out[s.path] = true + out[s.path] = s.n } - unloaded := map[string]categoryAt{} - unloadedUnder(root, nil, unloaded) - for p := range unloaded { - out[p] = true + for p, c := range unloadedUnder(root) { + out[p] = c.in.unloaded[c.name] } return out } @@ -75,7 +76,7 @@ func compileInto(place func(dir []string) *folder) func(datafiles.Category) erro if err != nil { return err } - place(c.Folders).children[c.Name] = n + place(c.Folders).put(c.Name, n) return nil } } @@ -87,8 +88,7 @@ func madeFolder(root *folder, dir []string) *folder { sub, isFolder := g.children[seg].(*folder) if !isFolder { sub = &folder{children: map[string]node{}} - g.children[seg] = sub - delete(g.unloaded, seg) + g.put(seg, sub) } g = sub } @@ -103,8 +103,7 @@ func mergeFolder(dst, src *folder) { continue } } - dst.children[k] = v - delete(dst.unloaded, k) + dst.put(k, v) } } @@ -116,10 +115,10 @@ type nodeSet func(fn func(label string, n node) error) error // folder's path, its name and its own dot path. type categorySite struct { dir []string - name string - path string in *folder n node + name string + path string } // categorySites lists every category under root, folders and names in sorted order. @@ -140,7 +139,7 @@ func categorySites(root *folder) []categorySite { } func siteIn(dir []string, in *folder, name string) categorySite { - return categorySite{dir: dir, name: name, path: categoryPath(dir, name), in: in, n: in.children[name]} + return categorySite{dir: dir, in: in, n: in.children[name], name: name, path: categoryPath(dir, name)} } func categoryPath(dir []string, name string) string { return join(strings.Join(dir, "."), name) } diff --git a/docs/comprehension-history.md b/docs/comprehension-history.md index b6d8510..9a3d38d 100644 --- a/docs/comprehension-history.md +++ b/docs/comprehension-history.md @@ -179,3 +179,26 @@ Ruling: same |---|---|---|---|---|---| | Mid A | same | same | same | same | same | | Inherited architect | same | same | same | same | same | + +## 2026-10-09T17:21:57Z, PR #202 at 65b2de8, against 2652779 + +Ruling: worse + +| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall | +|---|---|---|---|---|---| +| Mid A | same | worse | same | same | worse | +| Inherited architect | same | worse | worse | worse | worse | + +Mid A, decided by: + +- Locality, Overall: `data.go:13` `loadSources`: one pass mixes lazy and eager loading through before/after snapshots, a `provides` closure and a reverse graph search, and the skip of the first source is unexplained +- Locality: `data.go:107` `mergeFolder`: a category must sit in one of two maps, `children` or `unloaded`, kept by deletes in four functions across two files +- Locality: `internal/datafiles/datafiles.go:83` `embeddedManifests`: a process-wide cache in a package that was otherwise pure + +Inherited architect, decided by: + +- Locality, Self-sufficiency: `data.go:13` `loadSources`: nothing says why replaced paths drag their readers in, or what the `!after[p]` case covers +- Locality: `index.go:113` `loadReached`: its rollback claims the next call fails the same way, which holds only while no parent table was linked to a rolled-back child +- Shape: `index.go:159` `reaching`: four near-synonyms, `reaching`, `reachedFrom`, `loadReached` and `referenced`, and `referenced` and `categoryUnder` return loaded categories too +- Shape: `data.go:40` `place`: shares its name with `compileInto`'s parameter, and returns a closure answering one question two ways +- Locality: `internal/datafiles/datafiles.go:83` `embeddedManifests`: hidden global state for a small saving diff --git a/docs/decisions.md b/docs/decisions.md index 9fd40ae..608f2d8 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -154,7 +154,7 @@ still lives in `data/`; `--data-path` layers over it. Valid while the shipped data stays under 100 MiB, which a gate test holds (`todo.md` item 192): `go get` fetches the whole Go module, and the `go` command refuses a Go module zip over 500 MiB (`MaxZipFile` in `golang.org/x/mod/zip`). Past it, `todo.md` item 193 moves the largest packages into nested Go modules that keep their import paths, each requiring the core release that drops the package, so no build sees it twice. -## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces +## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category depending on what it provides or a later source replaced 2026-10-03, a restructure the maintainer approved; widened to any source on 2026-10-09. @@ -168,10 +168,12 @@ load. `go generate` writes the shipped set's manifest from `data/`; and each shipped category alone into a fresh `New`, so none fails at first reach. Goal 6: a source without an index loads whole in `New`, so each mistake in it is `New`'s -error. So does every indexed category it reads, and every indexed category leading -through references or table links to one that it or any later source replaced, so a -replacement breaking a reader fails `New`. Goal 6.3: an indexed source from anyone else -may fail at first reach; `Fake` then returns the load's error, the same every call. +error. So does every indexed category it reads, and every indexed category depending, +through references or table links, on a category it provides or a later source +replaced, so a replacement or a stand-in breaking a reader fails `New`. Goal 6.3: an +indexed source from anyone else may fail at first reach; `Fake` then returns the load's +error wrapping `ErrLoad`, the same every call. A category whose index entry disagrees +with it fails there too, so a stale index never decides what loads. Valid while CI proves the shipped set whole and each category alone. diff --git a/fejkdata.go b/fejkdata.go index 76fc960..c749b59 100644 --- a/fejkdata.go +++ b/fejkdata.go @@ -73,8 +73,7 @@ func WithoutShippedData() Option { // namespace segment. It errors on a missing directory, invalid JSON or invalid data; a // generator with no data at all renders templates that read none. A source whose // .fejkdata.json carries an index, as the shipped set's does, loads each category on the -// first call reaching it. Every other source loads here, with what it reads and every -// indexed category reaching what it replaces. +// first call reaching it, and fails there with [ErrLoad]. func New(opts ...Option) (*Generator, error) { c := config{shipped: true} for _, opt := range opts { @@ -91,7 +90,7 @@ func New(opts ...Option) (*Generator, error) { return &Generator{drawState: drawstate.New(seed), root: root}, nil } -// load builds the tree New starts from. docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads-and-every-indexed-category-reaching-what-it-replaces +// load builds the tree New starts from. docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads-and-every-indexed-category-depending-on-what-it-provides-or-a-later-source-replaced func (c config) load() (folder, error) { var sources []datafiles.Source if c.shipped { diff --git a/index.go b/index.go index 74abd1e..1456d4e 100644 --- a/index.go +++ b/index.go @@ -1,17 +1,25 @@ package fejkdata import ( + "encoding/json" + "errors" "fmt" + "slices" + "sort" "strings" "github.com/larvit/fejkdata/internal/datafiles" "github.com/larvit/fejkdata/internal/grammar" ) +// ErrLoad marks the error of a category that fails to load on the first call reaching it, +// which only a source whose manifest carries an index defers to then. +var ErrLoad = errors.New("data fails to load") + // indexed is a category an index names and no call has reached yet, and the source holding it. type indexed struct { - src datafiles.Source entry datafiles.IndexEntry + src datafiles.Source } // placeIndex sets every category index names unloaded in root, in place of what root holds @@ -19,17 +27,12 @@ type indexed struct { func placeIndex(root *folder, src datafiles.Source, index map[string]datafiles.IndexEntry) { for _, p := range sortedNames(index) { segs := strings.Split(p, ".") - g := madeFolder(root, segs[:len(segs)-1]) - name := segs[len(segs)-1] - delete(g.children, name) - if g.unloaded == nil { - g.unloaded = map[string]indexed{} - } - g.unloaded[name] = indexed{src: src, entry: index[p]} + madeFolder(root, segs[:len(segs)-1]).putUnloaded(segs[len(segs)-1], &indexed{entry: index[p], src: src}) } } -// categoryAt is where a category sits: the folder holding it, that folder's path, and its name. +// categoryAt is where a category, loaded or not, sits: the folder holding it, that +// folder's path, and its name. type categoryAt struct { dir []string in *folder @@ -38,8 +41,8 @@ type categoryAt struct { func (c categoryAt) path() string { return categoryPath(c.dir, c.name) } -// categoryUnder is the category, loaded or not, that a path from root names or descends into. -func categoryUnder(root *folder, segs []string) (categoryAt, bool) { +// categoryOn is the category, loaded or not, that a path from root names or passes. +func categoryOn(root *folder, segs []string) (categoryAt, bool) { g, i := folderAt(root, segs) if i == len(segs) { return categoryAt{}, false @@ -51,7 +54,7 @@ func categoryUnder(root *folder, segs []string) (categoryAt, bool) { } // loadCallerPath strips the / a caller's path may start with, splits the path, and loads the -// unloaded category it names or descends into. The caller walks it next, and a walk loads nothing. +// unloaded category it names or passes. The caller walks it next, and a walk loads nothing. func (f *Generator) loadCallerPath(path string) (string, []string, error) { path, err := grammar.CallerPath(path) if err != nil { @@ -61,16 +64,16 @@ func (f *Generator) loadCallerPath(path string) (string, []string, error) { if err != nil { return "", nil, fmt.Errorf("fejkdata: %w", err) } - if c, ok := categoryUnder(&f.root, segs); ok { + if c, ok := categoryOn(&f.root, segs); ok { if err := loadReached(&f.root, nil, []categoryAt{c}); err != nil { - return "", nil, fmt.Errorf("fejkdata: %w", err) + return "", nil, fmt.Errorf("fejkdata: %w: %w", ErrLoad, err) } } return path, segs, nil } -// referenced is every category the templates of nodes reference, each reference read from -// the folder dir. +// referenced is every category, loaded or not, that the templates of nodes reference, each +// reference read from the folder dir. func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt { var out []categoryAt _ = nodes(func(_ string, n node) error { @@ -83,7 +86,7 @@ func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt { if err != nil { continue // resolveRefs reports it } - if c, ok := categoryUnder(root, segs); ok { + if c, ok := categoryOn(root, segs); ok { out = append(out, c) } } @@ -92,12 +95,12 @@ func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt { return out } -// linkedTables is the table beside s that its parent column names, and every unloaded -// table beside s whose parent s is. -func linkedTables(s categorySite) []categoryAt { - var out []categoryAt - if t, isTable := s.n.(*table); isTable && t.rows.Options().Parent != "" { - out = append(out, categoryAt{dir: s.dir, in: s.in, name: t.rows.Options().Parent}) +// dependencies is every category, loaded or not, that s references, the table beside s +// that its parent column names, and every unloaded table beside s whose parent s is. +func dependencies(root *folder, s categorySite) []categoryAt { + out := referenced(root, s.dir, siteNodes([]categorySite{s})) + if parent := parentOf(s.n); parent != "" { + out = append(out, categoryAt{dir: s.dir, in: s.in, name: parent}) } for _, name := range sortedNames(s.in.unloaded) { if s.in.unloaded[name].entry.Parent == s.name { @@ -107,61 +110,14 @@ func linkedTables(s categorySite) []categoryAt { return out } -// loadReached loads the unloaded categories among wanted, and every unloaded category they -// or sites reference or link to, then runs sites and all it loaded through a whole load's -// pipeline. On an error it unloads again what it loaded, so the next call fails the same way. -func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error { - queue := wanted - for _, s := range sites { - queue = append(queue, reachedFrom(root, s)...) - } - var loaded []categorySite - var was []indexed - unload := func() { - for i, s := range loaded { - delete(s.in.children, s.name) - s.in.unloaded[s.name] = was[i] - } - } - for ; len(queue) > 0; queue = queue[1:] { - c := queue[0] - x, unloaded := c.in.unloaded[c.name] - if !unloaded { - continue - } - if err := x.src.Load(c.dir, c.name, compileInto(func([]string) *folder { return c.in })); err != nil { - unload() - return err - } - delete(c.in.unloaded, c.name) - s := siteIn(c.dir, c.in, c.name) - loaded, was = append(loaded, s), append(was, x) - queue = append(queue, reachedFrom(root, s)...) - } - sites = append(sites[:len(sites):len(sites)], loaded...) - if len(sites) == 0 { - return nil - } - if err := categoryPipeline(sites, root.children).run(); err != nil { - unload() - return err - } - return nil -} - -func reachedFrom(root *folder, s categorySite) []categoryAt { - return append(referenced(root, s.dir, siteNodes([]categorySite{s})), linkedTables(s)...) -} - -// reaching is every unloaded category under root whose index entry leads to a path in -// targets: through its reads, its parent or a child table, and through other unloaded +// dependents is every unloaded category under root whose index entry leads to one of +// paths: through its reads, its parent or a child table, and through other unloaded // categories. -func reaching(root *folder, targets map[string]bool) []categoryAt { - if len(targets) == 0 { +func dependents(root *folder, paths []string) []categoryAt { + if len(paths) == 0 { return nil } - unloaded := map[string]categoryAt{} - unloadedUnder(root, nil, unloaded) + unloaded := unloadedUnder(root) from := map[string][]string{} // a path, and the unloaded categories leading to it for _, p := range sortedNames(unloaded) { c := unloaded[p] @@ -177,7 +133,9 @@ func reaching(root *folder, targets map[string]bool) []categoryAt { } var out []categoryAt seen := map[string]bool{} - for queue := sortedNames(targets); len(queue) > 0; queue = queue[1:] { + queue := slices.Clone(paths) + sort.Strings(queue) + for ; len(queue) > 0; queue = queue[1:] { for _, p := range from[queue[0]] { c, isUnloaded := unloaded[p] if !isUnloaded || seen[p] { @@ -191,14 +149,110 @@ func reaching(root *folder, targets map[string]bool) []categoryAt { return out } -func unloadedUnder(g *folder, dir []string, out map[string]categoryAt) { - for name := range g.unloaded { - c := categoryAt{dir: dir, in: g, name: name} - out[c.path()] = c - } - for name, n := range g.children { - if sub, isFolder := n.(*folder); isFolder { - unloadedUnder(sub, append(dir[:len(dir):len(dir)], name), out) +func unloadedUnder(root *folder) map[string]categoryAt { + out := map[string]categoryAt{} + var walk func(g *folder, dir []string) + walk = func(g *folder, dir []string) { + for name := range g.unloaded { + c := categoryAt{dir: dir, in: g, name: name} + out[c.path()] = c + } + for name, n := range g.children { + if sub, isFolder := n.(*folder); isFolder { + walk(sub, append(dir[:len(dir):len(dir)], name)) + } } } + walk(root, nil) + return out +} + +// loadReached loads the unloaded categories among wanted, and every unloaded category they +// or sites depend on, then runs sites and all it loaded through a whole load's pipeline. +// On an error it puts back what it loaded, so the next call fails the same way. Checking +// each parent column against its entry before linking keeps that whole: a table then +// loads with its parent, so no table loaded earlier links to one put back. +func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error { + queue := wanted + for _, s := range sites { + queue = append(queue, dependencies(root, s)...) + } + type load struct { + site categorySite + was *indexed + } + var loads []load + putBack := func() { + for _, l := range loads { + l.site.in.putUnloaded(l.site.name, l.was) + } + } + for ; len(queue) > 0; queue = queue[1:] { + c := queue[0] + x, unloaded := c.in.unloaded[c.name] + if !unloaded { + continue + } + err := x.src.Load(c.dir, c.name, compileInto(func([]string) *folder { return c.in })) + if err == nil { + loads = append(loads, load{site: siteIn(c.dir, c.in, c.name), was: x}) + if parentOf(c.in.children[c.name]) != x.entry.Parent { + err = staleEntry(root, loads[len(loads)-1].site, x) + } + } + if err != nil { + putBack() + return err + } + queue = append(queue, dependencies(root, loads[len(loads)-1].site)...) + } + for _, l := range loads { + sites = append(sites[:len(sites):len(sites)], l.site) + } + if len(sites) == 0 { + return nil + } + err := categoryPipeline(sites, root.children).run() + for _, l := range loads { + if err == nil && !sameEntry(indexEntry(root, l.site), l.was.entry) { + err = staleEntry(root, l.site, l.was) + } + } + if err != nil { + putBack() + } + return err +} + +// indexEntry is what an index says of the loaded category s. +func indexEntry(root *folder, s categorySite) datafiles.IndexEntry { + e := datafiles.IndexEntry{Parent: parentOf(s.n), Paths: paths(s.n, false)} + sort.Strings(e.Paths) + seen := map[string]bool{s.path: true} + for _, c := range referenced(root, s.dir, siteNodes([]categorySite{s})) { + if p := c.path(); !seen[p] { + seen[p] = true + e.Reads = append(e.Reads, p) + } + } + sort.Strings(e.Reads) + return e +} + +func sameEntry(a, b datafiles.IndexEntry) bool { + return a.Parent == b.Parent && slices.Equal(a.Paths, b.Paths) && slices.Equal(a.Reads, b.Reads) +} + +// staleEntry names the entry x holds for s, and the entry s calls for. Before linking, +// the paths and reads it names may still lack what s's family adds. +func staleEntry(root *folder, s categorySite, x *indexed) error { + want, _ := json.Marshal(indexEntry(root, s)) + return fmt.Errorf("%s: the index entry for %s is stale; the category calls for %s", x.src.ManifestPath(), s.path, want) +} + +func parentOf(n node) string { + if t, isTable := n.(*table); isTable { + return t.rows.Options().Parent + } + return "" } diff --git a/inline.go b/inline.go index 6eeb03b..a4344c5 100644 --- a/inline.go +++ b/inline.go @@ -91,7 +91,7 @@ func loadInline(root *folder, n node, label string, typedByGo bool) error { return err } if err := loadReached(root, nil, referenced(root, nil, nodes)); err != nil { - return err + return fmt.Errorf("%w: %w", ErrLoad, err) } return pipeline{ nodes: nodes, diff --git a/internal/datafiles/datafiles.go b/internal/datafiles/datafiles.go index fa74d23..cb6148b 100644 --- a/internal/datafiles/datafiles.go +++ b/internal/datafiles/datafiles.go @@ -3,7 +3,6 @@ package datafiles import ( "bytes" - "embed" "encoding/json" "errors" "fmt" @@ -12,7 +11,6 @@ import ( "os" "path" "strings" - "sync" "github.com/larvit/fejkdata/internal/grammar" "github.com/larvit/fejkdata/internal/jsonvalue" @@ -79,47 +77,28 @@ type IndexEntry struct { Reads []string `json:"reads,omitempty"` } -// embeddedManifests holds each manifest read from an embed.FS, whose files never change. -var embeddedManifests sync.Map - -type embeddedManifest struct { - fsys embed.FS - base string -} - // Manifest reads the manifest at the source's root, the zero Manifest where there is none. func (s Source) Manifest() (Manifest, error) { - e, embedded := s.fsys.(embed.FS) - key := embeddedManifest{fsys: e, base: s.base} - if m, read := embeddedManifests.Load(key); embedded && read { - return m.(Manifest), nil + if err := s.check(); err != nil { + return Manifest{}, err } - m, err := s.readManifest() - if embedded && err == nil { - embeddedManifests.Store(key, m) - } - return m, err -} - -func (s Source) readManifest() (Manifest, error) { - if s.onDisk && s.label == "" { - return Manifest{}, nil // Walk refuses the empty data path - } - full := path.Join(s.base, ManifestFile) - b, err := fs.ReadFile(s.fsys, full) + b, err := fs.ReadFile(s.fsys, path.Join(s.base, ManifestFile)) if errors.Is(err, fs.ErrNotExist) { return Manifest{}, nil } if err != nil { - return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err) + return Manifest{}, fmt.Errorf("%s: %w", s.ManifestPath(), err) } m, err := decodeManifest(b) if err != nil { - return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err) + return Manifest{}, fmt.Errorf("%s: %w", s.ManifestPath(), err) } return m, nil } +// ManifestPath is the manifest's path as errors name it. +func (s Source) ManifestPath() string { return s.labelled(path.Join(s.base, ManifestFile)) } + func decodeManifest(b []byte) (Manifest, error) { dec := json.NewDecoder(bytes.NewReader(b)) dec.DisallowUnknownFields() @@ -130,36 +109,67 @@ func decodeManifest(b []byte) (Manifest, error) { if _, err := dec.Token(); err != io.EOF { return Manifest{}, fmt.Errorf("more than one JSON value") } - for p := range m.Index { - for _, seg := range strings.Split(p, ".") { - if err := grammar.CheckIdentifier(seg); err != nil { - return Manifest{}, fmt.Errorf("index names %q: %w", p, err) + for p, e := range m.Index { + for _, q := range append([]string{p}, e.Reads...) { + if err := checkCategoryPath(q); err != nil { + return Manifest{}, fmt.Errorf("index entry %q: %w", p, err) } } + if e.Parent == "" { + continue + } + if err := grammar.CheckIdentifier(e.Parent); err != nil { + return Manifest{}, fmt.Errorf("index entry %q: parent %w", p, err) + } + sibling := e.Parent + if dot := strings.LastIndexByte(p, '.'); dot >= 0 { + sibling = p[:dot+1] + e.Parent + } + if _, ok := m.Index[sibling]; !ok { + return Manifest{}, fmt.Errorf("index entry %q: parent %q names no entry beside it", p, e.Parent) + } } return m, nil } +func checkCategoryPath(p string) error { + for _, seg := range strings.Split(p, ".") { + if err := grammar.CheckIdentifier(seg); err != nil { + return fmt.Errorf("%q: %w", p, err) + } + } + return nil +} + // Walk hands compile every category of the tree, folders and files in name order. A // folder holding no category anywhere below it is skipped; a hidden file or folder is no // category, though a category may name a hidden rows file beside it. func (s Source) Walk(compile func(Category) error) error { - if s.onDisk { - if s.label == "" { - return fmt.Errorf("a data path is empty") - } - info, err := os.Stat(s.label) - if err != nil { - return fmt.Errorf("data path %s: %w", s.label, err) - } - if !info.IsDir() { - return fmt.Errorf("%s is not a directory", s.label) - } + if err := s.check(); err != nil { + return err } _, err := s.walkDir(nil, compile) return err } +// check refuses a data path that is empty or names no directory. +func (s Source) check() error { + if !s.onDisk { + return nil + } + if s.label == "" { + return fmt.Errorf("a data path is empty") + } + info, err := os.Stat(s.label) + if err != nil { + return fmt.Errorf("data path %s: %w", s.label, err) + } + if !info.IsDir() { + return fmt.Errorf("%s is not a directory", s.label) + } + return nil +} + // walkDir walks the folder that dir names, and reports whether it handed over any category. func (s Source) walkDir(dir []string, compile func(Category) error) (bool, error) { full := s.dirPath(dir) diff --git a/node.go b/node.go index 5ae9ca6..b25035b 100644 --- a/node.go +++ b/node.go @@ -20,13 +20,27 @@ type node interface{ isNode() } // folder is a namespace of named children, built from a directory of JSON files // and subdirectories. It has no value of its own: descend into a named child by // dot path; rendering one is an error (see Fake). +// A name stands in children or in unloaded, never both; put and putUnloaded keep it so. type folder struct { children map[string]node - unloaded map[string]indexed // the categories here an index names and no call has reached yet + unloaded map[string]*indexed // the categories here an index names and no call has reached yet } func (*folder) isNode() {} +func (g *folder) put(name string, n node) { + g.children[name] = n + delete(g.unloaded, name) +} + +func (g *folder) putUnloaded(name string, x *indexed) { + delete(g.children, name) + if g.unloaded == nil { + g.unloaded = map[string]*indexed{} + } + g.unloaded[name] = x +} + // choice draws one of its items. cum holds cumulative weights for a weighted // draw; when nil the choice is uniform and selection is O(1). shared is the set of // relative dot paths every item carries, counting paths inside a repeated template. diff --git a/todo.md b/todo.md index a070750..ec34a89 100644 --- a/todo.md +++ b/todo.md @@ -6,7 +6,7 @@ `Bar = 9` -`Next ID = 194` +`Next ID = 195` | Goal | W | |---|---| @@ -106,6 +106,7 @@ | 178 | 0.2.0 | | **Plan with the systems architect which `internal/` packages the engine splits into, by area, and how each type stops being written by more than one stage, then file the moves as items.** | 1 | 3 | 2 | 9 | 3, 3.2 | 18.4 | | 193 | 0.2.0 | decision | **Move the largest shipped packages into nested Go modules that keep their import paths, once item 192's gate fails.** | 6 | 6 | 6 | 7 | 2.3, 2 | 17.8 | | 92 | 0.2.0 | | **Name the hyphenated field in a `calc`'s refusal, and say to rename it: today `{calc(postal-code * 2)}` reports `no field "postal"`.** | 1 | 2 | 5 | 6 | 6 | 17.5 | +| 194 | 0.2.0 | | **Let a data author write and check their own `.fejkdata.json` index with the library or the CLI, and fail a load whose index leaves out a category file.** | 2 | 4 | 5 | 7 | 6.3, 9.1 | 17.5 | | 53 | 0.2.0 | | **Let a table column carry a `datatype`, so `--format json` writes `"safe": true` and `--format sql` a boolean.** | 4 | 5 | 6 | 6 | 4 | 16.9 | | 55 | 0.2.0 | | **Document `NewRecordTemplate`, `ErrNoColumns` and `MaxRepeat` in the README's Library section.** | 1 | 2 | 6 | 7 | 12 | 16.8 | | 18 | 0.2.0 | | **Give the address records one column set across countries: `region` and `municipality` as columns on `geo.SE.address` too.** | 4 | 4 | 6 | 6 | 5 | 16.6 | @@ -422,6 +423,10 @@ Carries out the decision "The shipped data is Go packages a library imports by c A hyphen is always subtraction in a calc, and the shipped data has hyphenated fields (`postal-code` and `street-number` in `geo/SE/address`). The refusal names the first fragment, not the field the author meant. +### 194. Let a data author write and check their own `.fejkdata.json` index with the library or the CLI, and fail a load whose index leaves out a category file. + +Goal 6.3 has an author prove their module, and goal 9.1 has fejkdata's data load as anyone's; today only a test of fejkdata's own writes and checks an index (`TestShippedManifestIsCurrent` over `indexEntry`). A category file the index leaves out never loads, silently. The principles audit of item 180's chunk found it. + ### 53. Let a table column carry a `datatype`, so `--format json` writes `"safe": true` and `--format sql` a boolean. Today only a JSON field takes one, so `misc.httpmethod`'s booleans are typed in Go and text everywhere else. Typing a shipped column changes what `json` and `sql` write, so the capability is a minor and applying it to `misc.httpmethod` is a major.