AGENTS.md

@@ -47,7 +47,7 @@ In [docs/decisions.md](docs/decisions.md):
 - `FakeTemplate` and `NewTemplate` both stay
 - The shipped data is embedded, not discovered
 - The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0
-- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces
+- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category depending on what it provides or a later source replaced
 - Every `{…}` draws afresh, and only a name keeps a pick
 - Reference sigils follow the filesystem
 - A change to what exists is a major; a minor only adds

CHANGELOG.md

@@ -126,6 +126,7 @@ replacement, and each removed path, column or flag.
   `sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record
   that gives it one.
 - A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
-  category on the first call reaching it, as the shipped set does. A source without one
+  category on the first call reaching it, as the shipped set does, and a category failing
-  loads in `New`, with the shipped categories it reads and every shipped category
+  there returns an error wrapping `ErrLoad`, on which the CLI exits 1. A source without
-  reaching one it replaces; the rest of the shipped set still loads on first reach.
+  one loads in `New`, with the shipped categories it reads and every shipped category
+  depending on one it replaces; the rest of the shipped set still loads on first reach.

README.md

@@ -64,7 +64,7 @@ template holding one from a quoted heredoc or a file. The rest of stdin's rules
 `--name value` and `--name=value` both work, a short flag's value attaches or
 follows (`-n3`, `-n 3`) and short flags bundle (`-hn 3`) — see
 [Decisions](docs/decisions.md#flags-follow-getopt_long); flags go anywhere, `--` ends them. Exit codes: `0` success, `1` runtime error (missing
-dir, a lone reference to nothing under `--format`), `2` misuse — a bad flag, an argument
+dir, data that fails to load, a lone reference to nothing under `--format`), `2` misuse — a bad flag, an argument
 other than a flag, an unreadable stdin, nothing piped in, or a
 template that does not compile. From a checkout:
 `go run ./cmd/fejkdata …`.
@@ -173,10 +173,15 @@ entry is never a category or a folder, so a data directory can also be a checkou
 a category may name a hidden rows file beside it.
 
 A source whose root holds `.fejkdata.json` with an `index` loads each category on the
-first call reaching it, and its author proves each loads. The shipped set's
+first call reaching it, and its author proves each loads. Any other source loads in
+`New`, and so does every indexed category that depends on what such a source provides
+or on a category a later source replaced. The index names every category of its source,
+so a category file it leaves out never loads. The shipped set's
 [`data/.fejkdata.json`](data/.fejkdata.json) shows the shape: each category's dot path,
-its `parent` table, the `paths` `List` advertises below it, and the categories it
+its `parent` table, every path `List` advertises below it, `""` for the category itself,
-`reads`. `//go:embed` of a directory leaves the manifest out; name it in the pattern too.
+and the other categories its templates `reads`. A category whose entry says otherwise
+fails at first reach, naming the entry it calls for. `//go:embed` of a directory leaves
+the manifest out; name it in the pattern too.
 
 Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
 `ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`,
@@ -336,13 +341,14 @@ ok, err := fejkdata.IsTemplate(arg)       // an inline template by its shape, el
 | `WithoutShippedData()` | load only what you give |
 
 `New` refuses data that breaks the grammar, could mean two things or cannot render a valid
-value, and `NewTemplate` refuses such a template. On a loaded generator:
+value, and `NewTemplate` refuses such a template. A category of an indexed source is
+checked on the first call reaching it instead, and a failure there wraps `ErrLoad`, the
+same error every call. On a loaded generator:
 
-- `Fake` fails only for a path that names nothing, could name two things, reads one
+- `Fake` fails only for a path that names nothing, could name two things, or reads one
-  draw of a level carrying a `repeat`, or reaches a category of an indexed source that
+  draw of a level carrying a `repeat`, with the same error every call, or with `ErrLoad`.
-  fails to load, with the same error every call.
 - `FakeStruct` fails only for a non-struct argument or a type its tags do not describe,
-  with the same error every call.
+  with the same error every call, or with `ErrLoad`.
 - `Template.Fake` cannot fail.
 
 A path, wherever a call takes one, may start with one `/`, as a reference does.
@@ -842,9 +848,7 @@ The target is about a microsecond per value (goal 13).
 
 A source with an [index](#data), as the shipped set has, loads nothing in `New`: a
 category loads on the first `Fake`, `FakeRecord`, `FakeStruct` or template that reads
-it, with every category it reads and its whole table family. A source without one loads
+it, with every category it reads and its whole table family. Each file is parsed, validated and weight-indexed once, bar a
-in `New`, with the indexed categories it reads and every indexed category reaching one
-it replaces, so each mistake it causes is `New`'s error. Each file is parsed, validated and weight-indexed once, bar a
 table's name and parent columns, which are mapped on the first `Fake` that selects by
 name or descends through it. A `Fake` call then costs about what its output
 costs: an unweighted pick is O(1) whatever the list's length, a weighted one
@@ -866,7 +870,7 @@ and regenerate seeded fixtures and expected values when you raise that version.
 |---------|-------|-------|
 | Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another; change a table's key, name, weight or parent column, or remove a row | a path outside a record's columns, a locale, a value in a list, a row |
 | Records (a table is one) | remove, rename, retype or add a column; let a column be null | a record, as a new category |
-| Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin |
+| Data format | a fence: a spelling `New` rejects that it accepted, in a category or a manifest; a template option, since it reserves a field name | a builtin |
 | CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format |
 | Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option |
 

cmd/fejkdata/main.go

@@ -340,7 +340,7 @@ func (in invocation) write(f *fejkdata.Generator, src input, w io.Writer) error
 	}
 	t, err := f.NewTemplate(src.template)
 	if err != nil {
-		return inputError{err}
+		return templateError(err)
 	}
 	out := bufio.NewWriter(w)
 	for i := 0; i < in.repeat; i++ {
@@ -359,7 +359,7 @@ func (in invocation) recordStream(f *fejkdata.Generator, src input) (func() (*fe
 	if src.record == "" {
 		t, err := f.NewRecordTemplate(src.template)
 		if err != nil {
-			return nil, "", inputError{spacedReference(src.template, err)}
+			return nil, "", templateError(spacedReference(src.template, err))
 		}
 		record = func() (*fejkdata.Record, error) { return t.Fake(), nil }
 	}
@@ -621,6 +621,14 @@ func readStdin(stdin io.Reader) (input, error) {
 	return parseInput(string(raw)), nil
 }
 
+// templateError is misuse unless the data the template reads failed to load.
+func templateError(err error) error {
+	if errors.Is(err, fejkdata.ErrLoad) {
+		return err
+	}
+	return inputError{err}
+}
+
 func fail(stderr io.Writer, err error) int {
 	var te inputError
 	if errors.As(err, &te) {

data.go

@@ -7,64 +7,65 @@ import (
 )
 
 // loadSources merges sources in order, the last winning a clash. A source whose manifest
-// carries an index leaves its categories unloaded; every other source's categories load
+// carries an index leaves its categories unloaded; every other source loads here. So do
-// here, with what they reach and every unloaded category leading to one of them or to one a
+// the unloaded categories that depend on one it loaded or on one a later source replaced,
-// later source replaced.
+// so that a reader such a change breaks fails New, not a first reach.
 func loadSources(sources []datafiles.Source) (folder, error) {
 	root := folder{children: map[string]node{}}
-	targets := map[string]bool{}
+	var changed []string
 	for _, src := range sources {
-		before := categoriesUnder(&root)
+		replaced, err := addSource(&root, src)
-		provides, err := place(&root, src)
 		if err != nil {
 			return folder{}, err
 		}
-		if len(before) == 0 {
+		changed = append(changed, replaced...)
-			continue
-		}
-		after := categoriesUnder(&root)
-		for p := range before {
-			if provides(p) || !after[p] {
-				targets[p] = true
-			}
-		}
 	}
 	sites := categorySites(&root)
 	for _, s := range sites {
-		targets[s.path] = true
+		changed = append(changed, s.path)
 	}
-	return root, loadReached(&root, sites, reaching(&root, targets))
+	return root, loadReached(&root, sites, dependents(&root, changed))
 }
 
-// place adds the categories of src to root, and returns whether src provides a path.
+// addSource adds the categories of src to root, and returns the path of each category
-func place(root *folder, src datafiles.Source) (func(string) bool, error) {
+// already there that src replaced or put out of reach.
+func addSource(root *folder, src datafiles.Source) ([]string, error) {
 	m, err := src.Manifest()
 	if err != nil {
 		return nil, err
 	}
+	before := standing(root)
 	if m.Index != nil {
 		placeIndex(root, src, m.Index)
-		return func(p string) bool { _, ok := m.Index[p]; return ok }, nil
+	} else {
+		g := &folder{children: map[string]node{}}
+		if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil {
+			return nil, err
+		}
+		mergeFolder(root, g)
 	}
-	g := &folder{children: map[string]node{}}
+	if len(before) == 0 {
-	if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil {
+		return nil, nil
-		return nil, err
 	}
-	mergeFolder(root, g)
+	after := standing(root)
-	provided := categoriesUnder(g)
+	var replaced []string
-	return func(p string) bool { return provided[p] }, nil
+	for p, was := range before {
+		if after[p] != was {
+			replaced = append(replaced, p)
+		}
+	}
+	return replaced, nil
 }
 
-// categoriesUnder is the path of every category under root, loaded or not.
+// standing is what stands at the path of every category under root: its node, or its
-func categoriesUnder(root *folder) map[string]bool {
+// unloaded entry.
-	out := map[string]bool{}
+func standing(root *folder) map[string]any {
+	out := map[string]any{}
 	for _, s := range categorySites(root) {
-		out[s.path] = true
+		out[s.path] = s.n
 	}
-	unloaded := map[string]categoryAt{}
+	for p, c := range unloadedUnder(root) {
-	unloadedUnder(root, nil, unloaded)
+		out[p] = c.in.unloaded[c.name]
-	for p := range unloaded {
-		out[p] = true
 	}
 	return out
 }
@@ -75,7 +76,7 @@ func compileInto(place func(dir []string) *folder) func(datafiles.Category) erro
 		if err != nil {
 			return err
 		}
-		place(c.Folders).children[c.Name] = n
+		place(c.Folders).put(c.Name, n)
 		return nil
 	}
 }
@@ -87,8 +88,7 @@ func madeFolder(root *folder, dir []string) *folder {
 		sub, isFolder := g.children[seg].(*folder)
 		if !isFolder {
 			sub = &folder{children: map[string]node{}}
-			g.children[seg] = sub
+			g.put(seg, sub)
-			delete(g.unloaded, seg)
 		}
 		g = sub
 	}
@@ -103,8 +103,7 @@ func mergeFolder(dst, src *folder) {
 				continue
 			}
 		}
-		dst.children[k] = v
+		dst.put(k, v)
-		delete(dst.unloaded, k)
 	}
 }
 
@@ -116,10 +115,10 @@ type nodeSet func(fn func(label string, n node) error) error
 // folder's path, its name and its own dot path.
 type categorySite struct {
 	dir  []string
-	name string
-	path string
 	in   *folder
 	n    node
+	name string
+	path string
 }
 
 // categorySites lists every category under root, folders and names in sorted order.
@@ -140,7 +139,7 @@ func categorySites(root *folder) []categorySite {
 }
 
 func siteIn(dir []string, in *folder, name string) categorySite {
-	return categorySite{dir: dir, name: name, path: categoryPath(dir, name), in: in, n: in.children[name]}
+	return categorySite{dir: dir, in: in, n: in.children[name], name: name, path: categoryPath(dir, name)}
 }
 
 func categoryPath(dir []string, name string) string { return join(strings.Join(dir, "."), name) }

docs/comprehension-history.md

@@ -179,3 +179,26 @@ Ruling: same
 |---|---|---|---|---|---|
 | Mid A | same | same | same | same | same |
 | Inherited architect | same | same | same | same | same |
+
+## 2026-10-09T17:21:57Z, PR #202 at 65b2de8, against 2652779
+
+Ruling: worse
+
+| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
+|---|---|---|---|---|---|
+| Mid A | same | worse | same | same | worse |
+| Inherited architect | same | worse | worse | worse | worse |
+
+Mid A, decided by:
+
+- Locality, Overall: `data.go:13` `loadSources`: one pass mixes lazy and eager loading through before/after snapshots, a `provides` closure and a reverse graph search, and the skip of the first source is unexplained
+- Locality: `data.go:107` `mergeFolder`: a category must sit in one of two maps, `children` or `unloaded`, kept by deletes in four functions across two files
+- Locality: `internal/datafiles/datafiles.go:83` `embeddedManifests`: a process-wide cache in a package that was otherwise pure
+
+Inherited architect, decided by:
+
+- Locality, Self-sufficiency: `data.go:13` `loadSources`: nothing says why replaced paths drag their readers in, or what the `!after[p]` case covers
+- Locality: `index.go:113` `loadReached`: its rollback claims the next call fails the same way, which holds only while no parent table was linked to a rolled-back child
+- Shape: `index.go:159` `reaching`: four near-synonyms, `reaching`, `reachedFrom`, `loadReached` and `referenced`, and `referenced` and `categoryUnder` return loaded categories too
+- Shape: `data.go:40` `place`: shares its name with `compileInto`'s parameter, and returns a closure answering one question two ways
+- Locality: `internal/datafiles/datafiles.go:83` `embeddedManifests`: hidden global state for a small saving

docs/decisions.md

@@ -154,7 +154,7 @@ still lives in `data/`; `--data-path` layers over it.
 
 Valid while the shipped data stays under 100 MiB, which a gate test holds (`todo.md` item 192): `go get` fetches the whole Go module, and the `go` command refuses a Go module zip over 500 MiB (`MaxZipFile` in `golang.org/x/mod/zip`). Past it, `todo.md` item 193 moves the largest packages into nested Go modules that keep their import paths, each requiring the core release that drops the package, so no build sees it twice.
 
-## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces
+## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category depending on what it provides or a later source replaced
 
 2026-10-03, a restructure the maintainer approved; widened to any source on 2026-10-09.
 
@@ -168,10 +168,12 @@ load. `go generate` writes the shipped set's manifest from `data/`;
 and each shipped category alone into a fresh `New`, so none fails at first reach.
 
 Goal 6: a source without an index loads whole in `New`, so each mistake in it is `New`'s
-error. So does every indexed category it reads, and every indexed category leading
+error. So does every indexed category it reads, and every indexed category depending,
-through references or table links to one that it or any later source replaced, so a
+through references or table links, on a category it provides or a later source
-replacement breaking a reader fails `New`. Goal 6.3: an indexed source from anyone else
+replaced, so a replacement or a stand-in breaking a reader fails `New`. Goal 6.3: an
-may fail at first reach; `Fake` then returns the load's error, the same every call.
+indexed source from anyone else may fail at first reach; `Fake` then returns the load's
+error wrapping `ErrLoad`, the same every call. A category whose index entry disagrees
+with it fails there too, so a stale index never decides what loads.
 
 Valid while CI proves the shipped set whole and each category alone.
 

fejkdata.go

@@ -73,8 +73,7 @@ func WithoutShippedData() Option {
 // namespace segment. It errors on a missing directory, invalid JSON or invalid data; a
 // generator with no data at all renders templates that read none. A source whose
 // .fejkdata.json carries an index, as the shipped set's does, loads each category on the
-// first call reaching it. Every other source loads here, with what it reads and every
+// first call reaching it, and fails there with [ErrLoad].
-// indexed category reaching what it replaces.
 func New(opts ...Option) (*Generator, error) {
 	c := config{shipped: true}
 	for _, opt := range opts {
@@ -91,7 +90,7 @@ func New(opts ...Option) (*Generator, error) {
 	return &Generator{drawState: drawstate.New(seed), root: root}, nil
 }
 
-// load builds the tree New starts from. docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads-and-every-indexed-category-reaching-what-it-replaces
+// load builds the tree New starts from. docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads-and-every-indexed-category-depending-on-what-it-provides-or-a-later-source-replaced
 func (c config) load() (folder, error) {
 	var sources []datafiles.Source
 	if c.shipped {

index.go

@@ -1,17 +1,25 @@
 package fejkdata
 
 import (
+	"encoding/json"
+	"errors"
 	"fmt"
+	"slices"
+	"sort"
 	"strings"
 
 	"github.com/larvit/fejkdata/internal/datafiles"
 	"github.com/larvit/fejkdata/internal/grammar"
 )
 
+// ErrLoad marks the error of a category that fails to load on the first call reaching it,
+// which only a source whose manifest carries an index defers to then.
+var ErrLoad = errors.New("data fails to load")
+
 // indexed is a category an index names and no call has reached yet, and the source holding it.
 type indexed struct {
-	src   datafiles.Source
 	entry datafiles.IndexEntry
+	src   datafiles.Source
 }
 
 // placeIndex sets every category index names unloaded in root, in place of what root holds
@@ -19,17 +27,12 @@ type indexed struct {
 func placeIndex(root *folder, src datafiles.Source, index map[string]datafiles.IndexEntry) {
 	for _, p := range sortedNames(index) {
 		segs := strings.Split(p, ".")
-		g := madeFolder(root, segs[:len(segs)-1])
+		madeFolder(root, segs[:len(segs)-1]).putUnloaded(segs[len(segs)-1], &indexed{entry: index[p], src: src})
-		name := segs[len(segs)-1]
-		delete(g.children, name)
-		if g.unloaded == nil {
-			g.unloaded = map[string]indexed{}
-		}
-		g.unloaded[name] = indexed{src: src, entry: index[p]}
 	}
 }
 
-// categoryAt is where a category sits: the folder holding it, that folder's path, and its name.
+// categoryAt is where a category, loaded or not, sits: the folder holding it, that
+// folder's path, and its name.
 type categoryAt struct {
 	dir  []string
 	in   *folder
@@ -38,8 +41,8 @@ type categoryAt struct {
 
 func (c categoryAt) path() string { return categoryPath(c.dir, c.name) }
 
-// categoryUnder is the category, loaded or not, that a path from root names or descends into.
+// categoryOn is the category, loaded or not, that a path from root names or passes.
-func categoryUnder(root *folder, segs []string) (categoryAt, bool) {
+func categoryOn(root *folder, segs []string) (categoryAt, bool) {
 	g, i := folderAt(root, segs)
 	if i == len(segs) {
 		return categoryAt{}, false
@@ -51,7 +54,7 @@ func categoryUnder(root *folder, segs []string) (categoryAt, bool) {
 }
 
 // loadCallerPath strips the / a caller's path may start with, splits the path, and loads the
-// unloaded category it names or descends into. The caller walks it next, and a walk loads nothing.
+// unloaded category it names or passes. The caller walks it next, and a walk loads nothing.
 func (f *Generator) loadCallerPath(path string) (string, []string, error) {
 	path, err := grammar.CallerPath(path)
 	if err != nil {
@@ -61,16 +64,16 @@ func (f *Generator) loadCallerPath(path string) (string, []string, error) {
 	if err != nil {
 		return "", nil, fmt.Errorf("fejkdata: %w", err)
 	}
-	if c, ok := categoryUnder(&f.root, segs); ok {
+	if c, ok := categoryOn(&f.root, segs); ok {
 		if err := loadReached(&f.root, nil, []categoryAt{c}); err != nil {
-			return "", nil, fmt.Errorf("fejkdata: %w", err)
+			return "", nil, fmt.Errorf("fejkdata: %w: %w", ErrLoad, err)
 		}
 	}
 	return path, segs, nil
 }
 
-// referenced is every category the templates of nodes reference, each reference read from
+// referenced is every category, loaded or not, that the templates of nodes reference, each
-// the folder dir.
+// reference read from the folder dir.
 func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt {
 	var out []categoryAt
 	_ = nodes(func(_ string, n node) error {
@@ -83,7 +86,7 @@ func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt {
 			if err != nil {
 				continue // resolveRefs reports it
 			}
-			if c, ok := categoryUnder(root, segs); ok {
+			if c, ok := categoryOn(root, segs); ok {
 				out = append(out, c)
 			}
 		}
@@ -92,12 +95,12 @@ func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt {
 	return out
 }
 
-// linkedTables is the table beside s that its parent column names, and every unloaded
+// dependencies is every category, loaded or not, that s references, the table beside s
-// table beside s whose parent s is.
+// that its parent column names, and every unloaded table beside s whose parent s is.
-func linkedTables(s categorySite) []categoryAt {
+func dependencies(root *folder, s categorySite) []categoryAt {
-	var out []categoryAt
+	out := referenced(root, s.dir, siteNodes([]categorySite{s}))
-	if t, isTable := s.n.(*table); isTable && t.rows.Options().Parent != "" {
+	if parent := parentOf(s.n); parent != "" {
-		out = append(out, categoryAt{dir: s.dir, in: s.in, name: t.rows.Options().Parent})
+		out = append(out, categoryAt{dir: s.dir, in: s.in, name: parent})
 	}
 	for _, name := range sortedNames(s.in.unloaded) {
 		if s.in.unloaded[name].entry.Parent == s.name {
@@ -107,61 +110,14 @@ func linkedTables(s categorySite) []categoryAt {
 	return out
 }
 
-// loadReached loads the unloaded categories among wanted, and every unloaded category they
+// dependents is every unloaded category under root whose index entry leads to one of
-// or sites reference or link to, then runs sites and all it loaded through a whole load's
+// paths: through its reads, its parent or a child table, and through other unloaded
-// pipeline. On an error it unloads again what it loaded, so the next call fails the same way.
-func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error {
-	queue := wanted
-	for _, s := range sites {
-		queue = append(queue, reachedFrom(root, s)...)
-	}
-	var loaded []categorySite
-	var was []indexed
-	unload := func() {
-		for i, s := range loaded {
-			delete(s.in.children, s.name)
-			s.in.unloaded[s.name] = was[i]
-		}
-	}
-	for ; len(queue) > 0; queue = queue[1:] {
-		c := queue[0]
-		x, unloaded := c.in.unloaded[c.name]
-		if !unloaded {
-			continue
-		}
-		if err := x.src.Load(c.dir, c.name, compileInto(func([]string) *folder { return c.in })); err != nil {
-			unload()
-			return err
-		}
-		delete(c.in.unloaded, c.name)
-		s := siteIn(c.dir, c.in, c.name)
-		loaded, was = append(loaded, s), append(was, x)
-		queue = append(queue, reachedFrom(root, s)...)
-	}
-	sites = append(sites[:len(sites):len(sites)], loaded...)
-	if len(sites) == 0 {
-		return nil
-	}
-	if err := categoryPipeline(sites, root.children).run(); err != nil {
-		unload()
-		return err
-	}
-	return nil
-}
-
-func reachedFrom(root *folder, s categorySite) []categoryAt {
-	return append(referenced(root, s.dir, siteNodes([]categorySite{s})), linkedTables(s)...)
-}
-
-// reaching is every unloaded category under root whose index entry leads to a path in
-// targets: through its reads, its parent or a child table, and through other unloaded
 // categories.
-func reaching(root *folder, targets map[string]bool) []categoryAt {
+func dependents(root *folder, paths []string) []categoryAt {
-	if len(targets) == 0 {
+	if len(paths) == 0 {
 		return nil
 	}
-	unloaded := map[string]categoryAt{}
+	unloaded := unloadedUnder(root)
-	unloadedUnder(root, nil, unloaded)
 	from := map[string][]string{} // a path, and the unloaded categories leading to it
 	for _, p := range sortedNames(unloaded) {
 		c := unloaded[p]
@@ -177,7 +133,9 @@ func reaching(root *folder, targets map[string]bool) []categoryAt {
 	}
 	var out []categoryAt
 	seen := map[string]bool{}
-	for queue := sortedNames(targets); len(queue) > 0; queue = queue[1:] {
+	queue := slices.Clone(paths)
+	sort.Strings(queue)
+	for ; len(queue) > 0; queue = queue[1:] {
 		for _, p := range from[queue[0]] {
 			c, isUnloaded := unloaded[p]
 			if !isUnloaded || seen[p] {
@@ -191,14 +149,110 @@ func reaching(root *folder, targets map[string]bool) []categoryAt {
 	return out
 }
 
-func unloadedUnder(g *folder, dir []string, out map[string]categoryAt) {
+func unloadedUnder(root *folder) map[string]categoryAt {
-	for name := range g.unloaded {
+	out := map[string]categoryAt{}
-		c := categoryAt{dir: dir, in: g, name: name}
+	var walk func(g *folder, dir []string)
-		out[c.path()] = c
+	walk = func(g *folder, dir []string) {
-	}
+		for name := range g.unloaded {
-	for name, n := range g.children {
+			c := categoryAt{dir: dir, in: g, name: name}
-		if sub, isFolder := n.(*folder); isFolder {
+			out[c.path()] = c
-			unloadedUnder(sub, append(dir[:len(dir):len(dir)], name), out)
+		}
+		for name, n := range g.children {
+			if sub, isFolder := n.(*folder); isFolder {
+				walk(sub, append(dir[:len(dir):len(dir)], name))
+			}
 		}
 	}
+	walk(root, nil)
+	return out
+}
+
+// loadReached loads the unloaded categories among wanted, and every unloaded category they
+// or sites depend on, then runs sites and all it loaded through a whole load's pipeline.
+// On an error it puts back what it loaded, so the next call fails the same way. Checking
+// each parent column against its entry before linking keeps that whole: a table then
+// loads with its parent, so no table loaded earlier links to one put back.
+func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error {
+	queue := wanted
+	for _, s := range sites {
+		queue = append(queue, dependencies(root, s)...)
+	}
+	type load struct {
+		site categorySite
+		was  *indexed
+	}
+	var loads []load
+	putBack := func() {
+		for _, l := range loads {
+			l.site.in.putUnloaded(l.site.name, l.was)
+		}
+	}
+	for ; len(queue) > 0; queue = queue[1:] {
+		c := queue[0]
+		x, unloaded := c.in.unloaded[c.name]
+		if !unloaded {
+			continue
+		}
+		err := x.src.Load(c.dir, c.name, compileInto(func([]string) *folder { return c.in }))
+		if err == nil {
+			loads = append(loads, load{site: siteIn(c.dir, c.in, c.name), was: x})
+			if parentOf(c.in.children[c.name]) != x.entry.Parent {
+				err = staleEntry(root, loads[len(loads)-1].site, x)
+			}
+		}
+		if err != nil {
+			putBack()
+			return err
+		}
+		queue = append(queue, dependencies(root, loads[len(loads)-1].site)...)
+	}
+	for _, l := range loads {
+		sites = append(sites[:len(sites):len(sites)], l.site)
+	}
+	if len(sites) == 0 {
+		return nil
+	}
+	err := categoryPipeline(sites, root.children).run()
+	for _, l := range loads {
+		if err == nil && !sameEntry(indexEntry(root, l.site), l.was.entry) {
+			err = staleEntry(root, l.site, l.was)
+		}
+	}
+	if err != nil {
+		putBack()
+	}
+	return err
+}
+
+// indexEntry is what an index says of the loaded category s.
+func indexEntry(root *folder, s categorySite) datafiles.IndexEntry {
+	e := datafiles.IndexEntry{Parent: parentOf(s.n), Paths: paths(s.n, false)}
+	sort.Strings(e.Paths)
+	seen := map[string]bool{s.path: true}
+	for _, c := range referenced(root, s.dir, siteNodes([]categorySite{s})) {
+		if p := c.path(); !seen[p] {
+			seen[p] = true
+			e.Reads = append(e.Reads, p)
+		}
+	}
+	sort.Strings(e.Reads)
+	return e
+}
+
+func sameEntry(a, b datafiles.IndexEntry) bool {
+	return a.Parent == b.Parent && slices.Equal(a.Paths, b.Paths) && slices.Equal(a.Reads, b.Reads)
+}
+
+// staleEntry names the entry x holds for s, and the entry s calls for. Before linking,
+// the paths and reads it names may still lack what s's family adds.
+func staleEntry(root *folder, s categorySite, x *indexed) error {
+	want, _ := json.Marshal(indexEntry(root, s))
+	return fmt.Errorf("%s: the index entry for %s is stale; the category calls for %s", x.src.ManifestPath(), s.path, want)
+}
+
+func parentOf(n node) string {
+	if t, isTable := n.(*table); isTable {
+		return t.rows.Options().Parent
+	}
+	return ""
 }

inline.go

@@ -91,7 +91,7 @@ func loadInline(root *folder, n node, label string, typedByGo bool) error {
 		return err
 	}
 	if err := loadReached(root, nil, referenced(root, nil, nodes)); err != nil {
-		return err
+		return fmt.Errorf("%w: %w", ErrLoad, err)
 	}
 	return pipeline{
 		nodes:     nodes,

internal/datafiles/datafiles.go

@@ -3,7 +3,6 @@ package datafiles
 
 import (
 	"bytes"
-	"embed"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -12,7 +11,6 @@ import (
 	"os"
 	"path"
 	"strings"
-	"sync"
 
 	"github.com/larvit/fejkdata/internal/grammar"
 	"github.com/larvit/fejkdata/internal/jsonvalue"
@@ -79,47 +77,28 @@ type IndexEntry struct {
 	Reads  []string `json:"reads,omitempty"`
 }
 
-// embeddedManifests holds each manifest read from an embed.FS, whose files never change.
-var embeddedManifests sync.Map
-
-type embeddedManifest struct {
-	fsys embed.FS
-	base string
-}
-
 // Manifest reads the manifest at the source's root, the zero Manifest where there is none.
 func (s Source) Manifest() (Manifest, error) {
-	e, embedded := s.fsys.(embed.FS)
+	if err := s.check(); err != nil {
-	key := embeddedManifest{fsys: e, base: s.base}
+		return Manifest{}, err
-	if m, read := embeddedManifests.Load(key); embedded && read {
-		return m.(Manifest), nil
 	}
-	m, err := s.readManifest()
+	b, err := fs.ReadFile(s.fsys, path.Join(s.base, ManifestFile))
-	if embedded && err == nil {
-		embeddedManifests.Store(key, m)
-	}
-	return m, err
-}
-
-func (s Source) readManifest() (Manifest, error) {
-	if s.onDisk && s.label == "" {
-		return Manifest{}, nil // Walk refuses the empty data path
-	}
-	full := path.Join(s.base, ManifestFile)
-	b, err := fs.ReadFile(s.fsys, full)
 	if errors.Is(err, fs.ErrNotExist) {
 		return Manifest{}, nil
 	}
 	if err != nil {
-		return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err)
+		return Manifest{}, fmt.Errorf("%s: %w", s.ManifestPath(), err)
 	}
 	m, err := decodeManifest(b)
 	if err != nil {
-		return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err)
+		return Manifest{}, fmt.Errorf("%s: %w", s.ManifestPath(), err)
 	}
 	return m, nil
 }
 
+// ManifestPath is the manifest's path as errors name it.
+func (s Source) ManifestPath() string { return s.labelled(path.Join(s.base, ManifestFile)) }
+
 func decodeManifest(b []byte) (Manifest, error) {
 	dec := json.NewDecoder(bytes.NewReader(b))
 	dec.DisallowUnknownFields()
@@ -130,36 +109,67 @@ func decodeManifest(b []byte) (Manifest, error) {
 	if _, err := dec.Token(); err != io.EOF {
 		return Manifest{}, fmt.Errorf("more than one JSON value")
 	}
-	for p := range m.Index {
+	for p, e := range m.Index {
-		for _, seg := range strings.Split(p, ".") {
+		for _, q := range append([]string{p}, e.Reads...) {
-			if err := grammar.CheckIdentifier(seg); err != nil {
+			if err := checkCategoryPath(q); err != nil {
-				return Manifest{}, fmt.Errorf("index names %q: %w", p, err)
+				return Manifest{}, fmt.Errorf("index entry %q: %w", p, err)
 			}
 		}
+		if e.Parent == "" {
+			continue
+		}
+		if err := grammar.CheckIdentifier(e.Parent); err != nil {
+			return Manifest{}, fmt.Errorf("index entry %q: parent %w", p, err)
+		}
+		sibling := e.Parent
+		if dot := strings.LastIndexByte(p, '.'); dot >= 0 {
+			sibling = p[:dot+1] + e.Parent
+		}
+		if _, ok := m.Index[sibling]; !ok {
+			return Manifest{}, fmt.Errorf("index entry %q: parent %q names no entry beside it", p, e.Parent)
+		}
 	}
 	return m, nil
 }
 
+func checkCategoryPath(p string) error {
+	for _, seg := range strings.Split(p, ".") {
+		if err := grammar.CheckIdentifier(seg); err != nil {
+			return fmt.Errorf("%q: %w", p, err)
+		}
+	}
+	return nil
+}
+
 // Walk hands compile every category of the tree, folders and files in name order. A
 // folder holding no category anywhere below it is skipped; a hidden file or folder is no
 // category, though a category may name a hidden rows file beside it.
 func (s Source) Walk(compile func(Category) error) error {
-	if s.onDisk {
+	if err := s.check(); err != nil {
-		if s.label == "" {
+		return err
-			return fmt.Errorf("a data path is empty")
-		}
-		info, err := os.Stat(s.label)
-		if err != nil {
-			return fmt.Errorf("data path %s: %w", s.label, err)
-		}
-		if !info.IsDir() {
-			return fmt.Errorf("%s is not a directory", s.label)
-		}
 	}
 	_, err := s.walkDir(nil, compile)
 	return err
 }
 
+// check refuses a data path that is empty or names no directory.
+func (s Source) check() error {
+	if !s.onDisk {
+		return nil
+	}
+	if s.label == "" {
+		return fmt.Errorf("a data path is empty")
+	}
+	info, err := os.Stat(s.label)
+	if err != nil {
+		return fmt.Errorf("data path %s: %w", s.label, err)
+	}
+	if !info.IsDir() {
+		return fmt.Errorf("%s is not a directory", s.label)
+	}
+	return nil
+}
+
 // walkDir walks the folder that dir names, and reports whether it handed over any category.
 func (s Source) walkDir(dir []string, compile func(Category) error) (bool, error) {
 	full := s.dirPath(dir)

node.go

@@ -20,13 +20,27 @@ type node interface{ isNode() }
 // folder is a namespace of named children, built from a directory of JSON files
 // and subdirectories. It has no value of its own: descend into a named child by
 // dot path; rendering one is an error (see Fake).
+// A name stands in children or in unloaded, never both; put and putUnloaded keep it so.
 type folder struct {
 	children map[string]node
-	unloaded map[string]indexed // the categories here an index names and no call has reached yet
+	unloaded map[string]*indexed // the categories here an index names and no call has reached yet
 }
 
 func (*folder) isNode() {}
 
+func (g *folder) put(name string, n node) {
+	g.children[name] = n
+	delete(g.unloaded, name)
+}
+
+func (g *folder) putUnloaded(name string, x *indexed) {
+	delete(g.children, name)
+	if g.unloaded == nil {
+		g.unloaded = map[string]*indexed{}
+	}
+	g.unloaded[name] = x
+}
+
 // choice draws one of its items. cum holds cumulative weights for a weighted
 // draw; when nil the choice is uniform and selection is O(1). shared is the set of
 // relative dot paths every item carries, counting paths inside a repeated template.

todo.md

@@ -6,7 +6,7 @@
 
 `Bar = 9`
 
-`Next ID = 194`
+`Next ID = 195`
 
 | Goal | W |
 |---|---|
@@ -106,6 +106,7 @@
 | 178 | 0.2.0 |  | **Plan with the systems architect which `internal/` packages the engine splits into, by area, and how each type stops being written by more than one stage, then file the moves as items.** | 1 | 3 | 2 | 9 | 3, 3.2 | 18.4 |
 | 193 | 0.2.0 | decision | **Move the largest shipped packages into nested Go modules that keep their import paths, once item 192's gate fails.** | 6 | 6 | 6 | 7 | 2.3, 2 | 17.8 |
 | 92 | 0.2.0 |  | **Name the hyphenated field in a `calc`'s refusal, and say to rename it: today `{calc(postal-code * 2)}` reports `no field "postal"`.** | 1 | 2 | 5 | 6 | 6 | 17.5 |
+| 194 | 0.2.0 |  | **Let a data author write and check their own `.fejkdata.json` index with the library or the CLI, and fail a load whose index leaves out a category file.** | 2 | 4 | 5 | 7 | 6.3, 9.1 | 17.5 |
 | 53 | 0.2.0 |  | **Let a table column carry a `datatype`, so `--format json` writes `"safe": true` and `--format sql` a boolean.** | 4 | 5 | 6 | 6 | 4 | 16.9 |
 | 55 | 0.2.0 |  | **Document `NewRecordTemplate`, `ErrNoColumns` and `MaxRepeat` in the README's Library section.** | 1 | 2 | 6 | 7 | 12 | 16.8 |
 | 18 | 0.2.0 |  | **Give the address records one column set across countries: `region` and `municipality` as columns on `geo.SE.address` too.** | 4 | 4 | 6 | 6 | 5 | 16.6 |
@@ -422,6 +423,10 @@ Carries out the decision "The shipped data is Go packages a library imports by c
 
 A hyphen is always subtraction in a calc, and the shipped data has hyphenated fields (`postal-code` and `street-number` in `geo/SE/address`). The refusal names the first fragment, not the field the author meant.
 
+### 194. Let a data author write and check their own `.fejkdata.json` index with the library or the CLI, and fail a load whose index leaves out a category file.
+
+Goal 6.3 has an author prove their module, and goal 9.1 has fejkdata's data load as anyone's; today only a test of fejkdata's own writes and checks an index (`TestShippedManifestIsCurrent` over `indexEntry`). A category file the index leaves out never loads, silently. The principles audit of item 180's chunk found it.
+
 ### 53. Let a table column carry a `datatype`, so `--format json` writes `"safe": true` and `--format sql` a boolean.
 
 Today only a JSON field takes one, so `misc.httpmethod`'s booleans are typed in Go and text everywhere else. Typing a shipped column changes what `json` and `sql` write, so the capability is a minor and applying it to `misc.httpmethod` is a major.