Answer architecture round 1: one owner keeps a folder's loaded and unloaded names apart, each loaded category is checked against its index entry, addSource returns what it replaced, the manifest cache goes, ErrLoad exits 1, and the docs say what New and Fake now refuse

AGENTS.md

@@ -47,7 +47,7 @@ In [docs/decisions.md](docs/decisions.md):
 - `FakeTemplate` and `NewTemplate` both stay
 - The shipped data is embedded, not discovered
 - The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0
-- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces
+- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category depending on what it provides or a later source replaced
 - Every `{…}` draws afresh, and only a name keeps a pick
 - Reference sigils follow the filesystem
 - A change to what exists is a major; a minor only adds

CHANGELOG.md

@@ -126,6 +126,7 @@ replacement, and each removed path, column or flag.
   `sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record
   that gives it one.
 - A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
-  category on the first call reaching it, as the shipped set does. A source without one
+  category on the first call reaching it, as the shipped set does, and a category failing
-  loads in `New`, with the shipped categories it reads and every shipped category
+  there returns an error wrapping `ErrLoad`, on which the CLI exits 1. A source without
-  reaching one it replaces; the rest of the shipped set still loads on first reach.
+  one loads in `New`, with the shipped categories it reads and every shipped category
+  depending on one it replaces; the rest of the shipped set still loads on first reach.

README.md

@@ -64,7 +64,7 @@ template holding one from a quoted heredoc or a file. The rest of stdin's rules
 `--name value` and `--name=value` both work, a short flag's value attaches or
 follows (`-n3`, `-n 3`) and short flags bundle (`-hn 3`) — see
 [Decisions](docs/decisions.md#flags-follow-getopt_long); flags go anywhere, `--` ends them. Exit codes: `0` success, `1` runtime error (missing
-dir, a lone reference to nothing under `--format`), `2` misuse — a bad flag, an argument
+dir, data that fails to load, a lone reference to nothing under `--format`), `2` misuse — a bad flag, an argument
 other than a flag, an unreadable stdin, nothing piped in, or a
 template that does not compile. From a checkout:
 `go run ./cmd/fejkdata …`.
@@ -173,10 +173,15 @@ entry is never a category or a folder, so a data directory can also be a checkou
 a category may name a hidden rows file beside it.
 
 A source whose root holds `.fejkdata.json` with an `index` loads each category on the
-first call reaching it, and its author proves each loads. The shipped set's
+first call reaching it, and its author proves each loads. Any other source loads in
+`New`, and so does every indexed category that depends on what such a source provides
+or on a category a later source replaced. The index names every category of its source,
+so a category file it leaves out never loads. The shipped set's
 [`data/.fejkdata.json`](data/.fejkdata.json) shows the shape: each category's dot path,
-its `parent` table, the `paths` `List` advertises below it, and the categories it
+its `parent` table, every path `List` advertises below it, `""` for the category itself,
-`reads`. `//go:embed` of a directory leaves the manifest out; name it in the pattern too.
+and the other categories its templates `reads`. A category whose entry says otherwise
+fails at first reach, naming the entry it calls for. `//go:embed` of a directory leaves
+the manifest out; name it in the pattern too.
 
 Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
 `ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`,
@@ -336,13 +341,14 @@ ok, err := fejkdata.IsTemplate(arg)       // an inline template by its shape, el
 | `WithoutShippedData()` | load only what you give |
 
 `New` refuses data that breaks the grammar, could mean two things or cannot render a valid
-value, and `NewTemplate` refuses such a template. On a loaded generator:
+value, and `NewTemplate` refuses such a template. A category of an indexed source is
+checked on the first call reaching it instead, and a failure there wraps `ErrLoad`, the
+same error every call. On a loaded generator:
 
-- `Fake` fails only for a path that names nothing, could name two things, reads one
+- `Fake` fails only for a path that names nothing, could name two things, or reads one
-  draw of a level carrying a `repeat`, or reaches a category of an indexed source that
+  draw of a level carrying a `repeat`, with the same error every call, or with `ErrLoad`.
-  fails to load, with the same error every call.
 - `FakeStruct` fails only for a non-struct argument or a type its tags do not describe,
-  with the same error every call.
+  with the same error every call, or with `ErrLoad`.
 - `Template.Fake` cannot fail.
 
 A path, wherever a call takes one, may start with one `/`, as a reference does.
@@ -842,9 +848,7 @@ The target is about a microsecond per value (goal 13).
 
 A source with an [index](#data), as the shipped set has, loads nothing in `New`: a
 category loads on the first `Fake`, `FakeRecord`, `FakeStruct` or template that reads
-it, with every category it reads and its whole table family. A source without one loads
+it, with every category it reads and its whole table family. Each file is parsed, validated and weight-indexed once, bar a
-in `New`, with the indexed categories it reads and every indexed category reaching one
-it replaces, so each mistake it causes is `New`'s error. Each file is parsed, validated and weight-indexed once, bar a
 table's name and parent columns, which are mapped on the first `Fake` that selects by
 name or descends through it. A `Fake` call then costs about what its output
 costs: an unweighted pick is O(1) whatever the list's length, a weighted one
@@ -866,7 +870,7 @@ and regenerate seeded fixtures and expected values when you raise that version.
 |---------|-------|-------|
 | Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another; change a table's key, name, weight or parent column, or remove a row | a path outside a record's columns, a locale, a value in a list, a row |
 | Records (a table is one) | remove, rename, retype or add a column; let a column be null | a record, as a new category |
-| Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin |
+| Data format | a fence: a spelling `New` rejects that it accepted, in a category or a manifest; a template option, since it reserves a field name | a builtin |
 | CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format |
 | Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option |
 

cmd/fejkdata/main.go

@@ -340,7 +340,7 @@ func (in invocation) write(f *fejkdata.Generator, src input, w io.Writer) error
 	}
 	t, err := f.NewTemplate(src.template)
 	if err != nil {
-		return inputError{err}
+		return templateError(err)
 	}
 	out := bufio.NewWriter(w)
 	for i := 0; i < in.repeat; i++ {
@@ -359,7 +359,7 @@ func (in invocation) recordStream(f *fejkdata.Generator, src input) (func() (*fe
 	if src.record == "" {
 		t, err := f.NewRecordTemplate(src.template)
 		if err != nil {
-			return nil, "", inputError{spacedReference(src.template, err)}
+			return nil, "", templateError(spacedReference(src.template, err))
 		}
 		record = func() (*fejkdata.Record, error) { return t.Fake(), nil }
 	}
@@ -621,6 +621,14 @@ func readStdin(stdin io.Reader) (input, error) {
 	return parseInput(string(raw)), nil
 }
 
+// templateError is misuse unless the data the template reads failed to load.
+func templateError(err error) error {
+	if errors.Is(err, fejkdata.ErrLoad) {
+		return err
+	}
+	return inputError{err}
+}
+
 func fail(stderr io.Writer, err error) int {
 	var te inputError
 	if errors.As(err, &te) {

data.go

@@ -7,64 +7,65 @@ import (
 )
 
 // loadSources merges sources in order, the last winning a clash. A source whose manifest
-// carries an index leaves its categories unloaded; every other source's categories load
+// carries an index leaves its categories unloaded; every other source loads here. So do
-// here, with what they reach and every unloaded category leading to one of them or to one a
+// the unloaded categories that depend on one it loaded or on one a later source replaced,
-// later source replaced.
+// so that a reader such a change breaks fails New, not a first reach.
 func loadSources(sources []datafiles.Source) (folder, error) {
 	root := folder{children: map[string]node{}}
-	targets := map[string]bool{}
+	var changed []string
 	for _, src := range sources {
-		before := categoriesUnder(&root)
+		replaced, err := addSource(&root, src)
-		provides, err := place(&root, src)
 		if err != nil {
 			return folder{}, err
 		}
-		if len(before) == 0 {
+		changed = append(changed, replaced...)
-			continue
-		}
-		after := categoriesUnder(&root)
-		for p := range before {
-			if provides(p) || !after[p] {
-				targets[p] = true
-			}
-		}
 	}
 	sites := categorySites(&root)
 	for _, s := range sites {
-		targets[s.path] = true
+		changed = append(changed, s.path)
 	}
-	return root, loadReached(&root, sites, reaching(&root, targets))
+	return root, loadReached(&root, sites, dependents(&root, changed))
 }
 
-// place adds the categories of src to root, and returns whether src provides a path.
+// addSource adds the categories of src to root, and returns the path of each category
-func place(root *folder, src datafiles.Source) (func(string) bool, error) {
+// already there that src replaced or put out of reach.
+func addSource(root *folder, src datafiles.Source) ([]string, error) {
 	m, err := src.Manifest()
 	if err != nil {
 		return nil, err
 	}
+	before := standing(root)
 	if m.Index != nil {
 		placeIndex(root, src, m.Index)
-		return func(p string) bool { _, ok := m.Index[p]; return ok }, nil
+	} else {
+		g := &folder{children: map[string]node{}}
+		if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil {
+			return nil, err
+		}
+		mergeFolder(root, g)
 	}
-	g := &folder{children: map[string]node{}}
+	if len(before) == 0 {
-	if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil {
+		return nil, nil
-		return nil, err
 	}
-	mergeFolder(root, g)
+	after := standing(root)
-	provided := categoriesUnder(g)
+	var replaced []string
-	return func(p string) bool { return provided[p] }, nil
+	for p, was := range before {
+		if after[p] != was {
+			replaced = append(replaced, p)
+		}
+	}
+	return replaced, nil
 }
 
-// categoriesUnder is the path of every category under root, loaded or not.
+// standing is what stands at the path of every category under root: its node, or its
-func categoriesUnder(root *folder) map[string]bool {
+// unloaded entry.
-	out := map[string]bool{}
+func standing(root *folder) map[string]any {
+	out := map[string]any{}
 	for _, s := range categorySites(root) {
-		out[s.path] = true
+		out[s.path] = s.n
 	}
-	unloaded := map[string]categoryAt{}
+	for p, c := range unloadedUnder(root) {
-	unloadedUnder(root, nil, unloaded)
+		out[p] = c.in.unloaded[c.name]
-	for p := range unloaded {
-		out[p] = true
 	}
 	return out
 }
@@ -75,7 +76,7 @@ func compileInto(place func(dir []string) *folder) func(datafiles.Category) erro
 		if err != nil {
 			return err
 		}
-		place(c.Folders).children[c.Name] = n
+		place(c.Folders).put(c.Name, n)
 		return nil
 	}
 }
@@ -87,8 +88,7 @@ func madeFolder(root *folder, dir []string) *folder {
 		sub, isFolder := g.children[seg].(*folder)
 		if !isFolder {
 			sub = &folder{children: map[string]node{}}
-			g.children[seg] = sub
+			g.put(seg, sub)
-			delete(g.unloaded, seg)
 		}
 		g = sub
 	}
@@ -103,8 +103,7 @@ func mergeFolder(dst, src *folder) {
 				continue
 			}
 		}
-		dst.children[k] = v
+		dst.put(k, v)
-		delete(dst.unloaded, k)
 	}
 }
 
@@ -116,10 +115,10 @@ type nodeSet func(fn func(label string, n node) error) error
 // folder's path, its name and its own dot path.
 type categorySite struct {
 	dir  []string
-	name string
-	path string
 	in   *folder
 	n    node
+	name string
+	path string
 }
 
 // categorySites lists every category under root, folders and names in sorted order.
@@ -140,7 +139,7 @@ func categorySites(root *folder) []categorySite {
 }
 
 func siteIn(dir []string, in *folder, name string) categorySite {
-	return categorySite{dir: dir, name: name, path: categoryPath(dir, name), in: in, n: in.children[name]}
+	return categorySite{dir: dir, in: in, n: in.children[name], name: name, path: categoryPath(dir, name)}
 }
 
 func categoryPath(dir []string, name string) string { return join(strings.Join(dir, "."), name) }

docs/comprehension-history.md

@@ -179,3 +179,26 @@ Ruling: same
 |---|---|---|---|---|---|
 | Mid A | same | same | same | same | same |
 | Inherited architect | same | same | same | same | same |
+
+## 2026-10-09T17:21:57Z, PR #202 at 65b2de8, against 2652779
+
+Ruling: worse
+
+| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
+|---|---|---|---|---|---|
+| Mid A | same | worse | same | same | worse |
+| Inherited architect | same | worse | worse | worse | worse |
+
+Mid A, decided by:
+
+- Locality, Overall: `data.go:13` `loadSources`: one pass mixes lazy and eager loading through before/after snapshots, a `provides` closure and a reverse graph search, and the skip of the first source is unexplained
+- Locality: `data.go:107` `mergeFolder`: a category must sit in one of two maps, `children` or `unloaded`, kept by deletes in four functions across two files
+- Locality: `internal/datafiles/datafiles.go:83` `embeddedManifests`: a process-wide cache in a package that was otherwise pure
+
+Inherited architect, decided by:
+
+- Locality, Self-sufficiency: `data.go:13` `loadSources`: nothing says why replaced paths drag their readers in, or what the `!after[p]` case covers
+- Locality: `index.go:113` `loadReached`: its rollback claims the next call fails the same way, which holds only while no parent table was linked to a rolled-back child
+- Shape: `index.go:159` `reaching`: four near-synonyms, `reaching`, `reachedFrom`, `loadReached` and `referenced`, and `referenced` and `categoryUnder` return loaded categories too
+- Shape: `data.go:40` `place`: shares its name with `compileInto`'s parameter, and returns a closure answering one question two ways
+- Locality: `internal/datafiles/datafiles.go:83` `embeddedManifests`: hidden global state for a small saving

docs/decisions.md

@@ -154,7 +154,7 @@ still lives in `data/`; `--data-path` layers over it.
 
 Valid while the shipped data stays under 100 MiB, which a gate test holds (`todo.md` item 192): `go get` fetches the whole Go module, and the `go` command refuses a Go module zip over 500 MiB (`MaxZipFile` in `golang.org/x/mod/zip`). Past it, `todo.md` item 193 moves the largest packages into nested Go modules that keep their import paths, each requiring the core release that drops the package, so no build sees it twice.
 
-## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces
+## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category depending on what it provides or a later source replaced
 
 2026-10-03, a restructure the maintainer approved; widened to any source on 2026-10-09.
 
@@ -168,10 +168,12 @@ load. `go generate` writes the shipped set's manifest from `data/`;
 and each shipped category alone into a fresh `New`, so none fails at first reach.
 
 Goal 6: a source without an index loads whole in `New`, so each mistake in it is `New`'s
-error. So does every indexed category it reads, and every indexed category leading
+error. So does every indexed category it reads, and every indexed category depending,
-through references or table links to one that it or any later source replaced, so a
+through references or table links, on a category it provides or a later source
-replacement breaking a reader fails `New`. Goal 6.3: an indexed source from anyone else
+replaced, so a replacement or a stand-in breaking a reader fails `New`. Goal 6.3: an
-may fail at first reach; `Fake` then returns the load's error, the same every call.
+indexed source from anyone else may fail at first reach; `Fake` then returns the load's
+error wrapping `ErrLoad`, the same every call. A category whose index entry disagrees
+with it fails there too, so a stale index never decides what loads.
 
 Valid while CI proves the shipped set whole and each category alone.
 

fejkdata.go

@@ -73,8 +73,7 @@ func WithoutShippedData() Option {
 // namespace segment. It errors on a missing directory, invalid JSON or invalid data; a
 // generator with no data at all renders templates that read none. A source whose
 // .fejkdata.json carries an index, as the shipped set's does, loads each category on the
-// first call reaching it. Every other source loads here, with what it reads and every
+// first call reaching it, and fails there with [ErrLoad].
-// indexed category reaching what it replaces.
 func New(opts ...Option) (*Generator, error) {
 	c := config{shipped: true}
 	for _, opt := range opts {
@@ -91,7 +90,7 @@ func New(opts ...Option) (*Generator, error) {
 	return &Generator{drawState: drawstate.New(seed), root: root}, nil
 }
 
-// load builds the tree New starts from. docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads-and-every-indexed-category-reaching-what-it-replaces
+// load builds the tree New starts from. docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads-and-every-indexed-category-depending-on-what-it-provides-or-a-later-source-replaced
 func (c config) load() (folder, error) {
 	var sources []datafiles.Source
 	if c.shipped {

index.go

@@ -1,17 +1,25 @@
 package fejkdata
 
 import (
+	"encoding/json"
+	"errors"
 	"fmt"
+	"slices"
+	"sort"
 	"strings"
 
 	"github.com/larvit/fejkdata/internal/datafiles"
 	"github.com/larvit/fejkdata/internal/grammar"
 )
 
+// ErrLoad marks the error of a category that fails to load on the first call reaching it,
+// which only a source whose manifest carries an index defers to then.
+var ErrLoad = errors.New("data fails to load")
+
 // indexed is a category an index names and no call has reached yet, and the source holding it.
 type indexed struct {
-	src   datafiles.Source
 	entry datafiles.IndexEntry
+	src   datafiles.Source
 }
 
 // placeIndex sets every category index names unloaded in root, in place of what root holds
@@ -19,17 +27,12 @@ type indexed struct {
 func placeIndex(root *folder, src datafiles.Source, index map[string]datafiles.IndexEntry) {
 	for _, p := range sortedNames(index) {
 		segs := strings.Split(p, ".")
-		g := madeFolder(root, segs[:len(segs)-1])
+		madeFolder(root, segs[:len(segs)-1]).putUnloaded(segs[len(segs)-1], &indexed{entry: index[p], src: src})
-		name := segs[len(segs)-1]
-		delete(g.children, name)
-		if g.unloaded == nil {
-			g.unloaded = map[string]indexed{}
-		}
-		g.unloaded[name] = indexed{src: src, entry: index[p]}
 	}
 }
 
-// categoryAt is where a category sits: the folder holding it, that folder's path, and its name.
+// categoryAt is where a category, loaded or not, sits: the folder holding it, that
+// folder's path, and its name.
 type categoryAt struct {
 	dir  []string
 	in   *folder
@@ -38,8 +41,8 @@ type categoryAt struct {
 
 func (c categoryAt) path() string { return categoryPath(c.dir, c.name) }
 
-// categoryUnder is the category, loaded or not, that a path from root names or descends into.
+// categoryOn is the category, loaded or not, that a path from root names or passes.
-func categoryUnder(root *folder, segs []string) (categoryAt, bool) {
+func categoryOn(root *folder, segs []string) (categoryAt, bool) {
 	g, i := folderAt(root, segs)
 	if i == len(segs) {
 		return categoryAt{}, false
@@ -51,7 +54,7 @@ func categoryUnder(root *folder, segs []string) (categoryAt, bool) {
 }
 
 // loadCallerPath strips the / a caller's path may start with, splits the path, and loads the
-// unloaded category it names or descends into. The caller walks it next, and a walk loads nothing.
+// unloaded category it names or passes. The caller walks it next, and a walk loads nothing.
 func (f *Generator) loadCallerPath(path string) (string, []string, error) {
 	path, err := grammar.CallerPath(path)
 	if err != nil {
@@ -61,16 +64,16 @@ func (f *Generator) loadCallerPath(path string) (string, []string, error) {
 	if err != nil {
 		return "", nil, fmt.Errorf("fejkdata: %w", err)
 	}
-	if c, ok := categoryUnder(&f.root, segs); ok {
+	if c, ok := categoryOn(&f.root, segs); ok {
 		if err := loadReached(&f.root, nil, []categoryAt{c}); err != nil {
-			return "", nil, fmt.Errorf("fejkdata: %w", err)
+			return "", nil, fmt.Errorf("fejkdata: %w: %w", ErrLoad, err)
 		}
 	}
 	return path, segs, nil
 }
 
-// referenced is every category the templates of nodes reference, each reference read from
+// referenced is every category, loaded or not, that the templates of nodes reference, each
-// the folder dir.
+// reference read from the folder dir.
 func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt {
 	var out []categoryAt
 	_ = nodes(func(_ string, n node) error {
@@ -83,7 +86,7 @@ func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt {
 			if err != nil {
 				continue // resolveRefs reports it
 			}
-			if c, ok := categoryUnder(root, segs); ok {
+			if c, ok := categoryOn(root, segs); ok {
 				out = append(out, c)
 			}
 		}
@@ -92,12 +95,12 @@ func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt {
 	return out
 }
 
-// linkedTables is the table beside s that its parent column names, and every unloaded
+// dependencies is every category, loaded or not, that s references, the table beside s
-// table beside s whose parent s is.
+// that its parent column names, and every unloaded table beside s whose parent s is.
-func linkedTables(s categorySite) []categoryAt {
+func dependencies(root *folder, s categorySite) []categoryAt {
-	var out []categoryAt
+	out := referenced(root, s.dir, siteNodes([]categorySite{s}))
-	if t, isTable := s.n.(*table); isTable && t.rows.Options().Parent != "" {
+	if parent := parentOf(s.n); parent != "" {
-		out = append(out, categoryAt{dir: s.dir, in: s.in, name: t.rows.Options().Parent})
+		out = append(out, categoryAt{dir: s.dir, in: s.in, name: parent})
 	}
 	for _, name := range sortedNames(s.in.unloaded) {
 		if s.in.unloaded[name].entry.Parent == s.name {
@@ -107,61 +110,14 @@ func linkedTables(s categorySite) []categoryAt {
 	return out
 }
 
-// loadReached loads the unloaded categories among wanted, and every unloaded category they
+// dependents is every unloaded category under root whose index entry leads to one of
-// or sites reference or link to, then runs sites and all it loaded through a whole load's
+// paths: through its reads, its parent or a child table, and through other unloaded
-// pipeline. On an error it unloads again what it loaded, so the next call fails the same way.
-func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error {
-	queue := wanted
-	for _, s := range sites {
-		queue = append(queue, reachedFrom(root, s)...)
-	}
-	var loaded []categorySite
-	var was []indexed
-	unload := func() {
-		for i, s := range loaded {
-			delete(s.in.children, s.name)
-			s.in.unloaded[s.name] = was[i]
-		}
-	}
-	for ; len(queue) > 0; queue = queue[1:] {
-		c := queue[0]
-		x, unloaded := c.in.unloaded[c.name]
-		if !unloaded {
-			continue
-		}
-		if err := x.src.Load(c.dir, c.name, compileInto(func([]string) *folder { return c.in })); err != nil {
-			unload()
-			return err
-		}
-		delete(c.in.unloaded, c.name)
-		s := siteIn(c.dir, c.in, c.name)
-		loaded, was = append(loaded, s), append(was, x)
-		queue = append(queue, reachedFrom(root, s)...)
-	}
-	sites = append(sites[:len(sites):len(sites)], loaded...)
-	if len(sites) == 0 {
-		return nil
-	}
-	if err := categoryPipeline(sites, root.children).run(); err != nil {
-		unload()
-		return err
-	}
-	return nil
-}
-
-func reachedFrom(root *folder, s categorySite) []categoryAt {
-	return append(referenced(root, s.dir, siteNodes([]categorySite{s})), linkedTables(s)...)
-}
-
-// reaching is every unloaded category under root whose index entry leads to a path in
-// targets: through its reads, its parent or a child table, and through other unloaded
 // categories.
-func reaching(root *folder, targets map[string]bool) []categoryAt {
+func dependents(root *folder, paths []string) []categoryAt {
-	if len(targets) == 0 {
+	if len(paths) == 0 {
 		return nil
 	}
-	unloaded := map[string]categoryAt{}
+	unloaded := unloadedUnder(root)
-	unloadedUnder(root, nil, unloaded)
 	from := map[string][]string{} // a path, and the unloaded categories leading to it
 	for _, p := range sortedNames(unloaded) {
 		c := unloaded[p]
@@ -177,7 +133,9 @@ func reaching(root *folder, targets map[string]bool) []categoryAt {
 	}
 	var out []categoryAt
 	seen := map[string]bool{}
-	for queue := sortedNames(targets); len(queue) > 0; queue = queue[1:] {
+	queue := slices.Clone(paths)
+	sort.Strings(queue)
+	for ; len(queue) > 0; queue = queue[1:] {
 		for _, p := range from[queue[0]] {
 			c, isUnloaded := unloaded[p]
 			if !isUnloaded || seen[p] {
@@ -191,14 +149,110 @@ func reaching(root *folder, targets map[string]bool) []categoryAt {
 	return out
 }
 
-func unloadedUnder(g *folder, dir []string, out map[string]categoryAt) {
+func unloadedUnder(root *folder) map[string]categoryAt {
-	for name := range g.unloaded {
+	out := map[string]categoryAt{}
-		c := categoryAt{dir: dir, in: g, name: name}
+	var walk func(g *folder, dir []string)
-		out[c.path()] = c
+	walk = func(g *folder, dir []string) {
-	}
+		for name := range g.unloaded {
-	for name, n := range g.children {
+			c := categoryAt{dir: dir, in: g, name: name}
-		if sub, isFolder := n.(*folder); isFolder {
+			out[c.path()] = c
-			unloadedUnder(sub, append(dir[:len(dir):len(dir)], name), out)
+		}
+		for name, n := range g.children {
+			if sub, isFolder := n.(*folder); isFolder {
+				walk(sub, append(dir[:len(dir):len(dir)], name))
+			}
 		}
 	}
+	walk(root, nil)
+	return out
+}
+
+// loadReached loads the unloaded categories among wanted, and every unloaded category they
+// or sites depend on, then runs sites and all it loaded through a whole load's pipeline.
+// On an error it puts back what it loaded, so the next call fails the same way. Checking
+// each parent column against its entry before linking keeps that whole: a table then
+// loads with its parent, so no table loaded earlier links to one put back.
+func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error {
+	queue := wanted
+	for _, s := range sites {
+		queue = append(queue, dependencies(root, s)...)
+	}
+	type load struct {
+		site categorySite
+		was  *indexed
+	}
+	var loads []load
+	putBack := func() {
+		for _, l := range loads {
+			l.site.in.putUnloaded(l.site.name, l.was)
+		}
+	}
+	for ; len(queue) > 0; queue = queue[1:] {
+		c := queue[0]
+		x, unloaded := c.in.unloaded[c.name]
+		if !unloaded {
+			continue
+		}
+		err := x.src.Load(c.dir, c.name, compileInto(func([]string) *folder { return c.in }))
+		if err == nil {
+			loads = append(loads, load{site: siteIn(c.dir, c.in, c.name), was: x})
+			if parentOf(c.in.children[c.name]) != x.entry.Parent {
+				err = staleEntry(root, loads[len(loads)-1].site, x)
+			}
+		}
+		if err != nil {
+			putBack()
+			return err
+		}
+		queue = append(queue, dependencies(root, loads[len(loads)-1].site)...)
+	}
+	for _, l := range loads {
+		sites = append(sites[:len(sites):len(sites)], l.site)
+	}
+	if len(sites) == 0 {
+		return nil
+	}
+	err := categoryPipeline(sites, root.children).run()
+	for _, l := range loads {
+		if err == nil && !sameEntry(indexEntry(root, l.site), l.was.entry) {
+			err = staleEntry(root, l.site, l.was)
+		}
+	}
+	if err != nil {
+		putBack()
+	}
+	return err
+}
+
+// indexEntry is what an index says of the loaded category s.
+func indexEntry(root *folder, s categorySite) datafiles.IndexEntry {
+	e := datafiles.IndexEntry{Parent: parentOf(s.n), Paths: paths(s.n, false)}
+	sort.Strings(e.Paths)
+	seen := map[string]bool{s.path: true}
+	for _, c := range referenced(root, s.dir, siteNodes([]categorySite{s})) {
+		if p := c.path(); !seen[p] {
+			seen[p] = true
+			e.Reads = append(e.Reads, p)
+		}
+	}
+	sort.Strings(e.Reads)
+	return e
+}
+
+func sameEntry(a, b datafiles.IndexEntry) bool {
+	return a.Parent == b.Parent && slices.Equal(a.Paths, b.Paths) && slices.Equal(a.Reads, b.Reads)
+}
+
+// staleEntry names the entry x holds for s, and the entry s calls for. Before linking,
+// the paths and reads it names may still lack what s's family adds.
+func staleEntry(root *folder, s categorySite, x *indexed) error {
+	want, _ := json.Marshal(indexEntry(root, s))
+	return fmt.Errorf("%s: the index entry for %s is stale; the category calls for %s", x.src.ManifestPath(), s.path, want)
+}
+
+func parentOf(n node) string {
+	if t, isTable := n.(*table); isTable {
+		return t.rows.Options().Parent
+	}
+	return ""
 }

inline.go

@@ -91,7 +91,7 @@ func loadInline(root *folder, n node, label string, typedByGo bool) error {
 		return err
 	}
 	if err := loadReached(root, nil, referenced(root, nil, nodes)); err != nil {
-		return err
+		return fmt.Errorf("%w: %w", ErrLoad, err)
 	}
 	return pipeline{
 		nodes:     nodes,

internal/datafiles/datafiles.go

@@ -3,7 +3,6 @@ package datafiles
 
 import (
 	"bytes"
-	"embed"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -12,7 +11,6 @@ import (
 	"os"
 	"path"
 	"strings"
-	"sync"
 
 	"github.com/larvit/fejkdata/internal/grammar"
 	"github.com/larvit/fejkdata/internal/jsonvalue"
@@ -79,47 +77,28 @@ type IndexEntry struct {
 	Reads  []string `json:"reads,omitempty"`
 }
 
-// embeddedManifests holds each manifest read from an embed.FS, whose files never change.
-var embeddedManifests sync.Map
-
-type embeddedManifest struct {
-	fsys embed.FS
-	base string
-}
-
 // Manifest reads the manifest at the source's root, the zero Manifest where there is none.
 func (s Source) Manifest() (Manifest, error) {
-	e, embedded := s.fsys.(embed.FS)
+	if err := s.check(); err != nil {
-	key := embeddedManifest{fsys: e, base: s.base}
+		return Manifest{}, err
-	if m, read := embeddedManifests.Load(key); embedded && read {
-		return m.(Manifest), nil
 	}
-	m, err := s.readManifest()
+	b, err := fs.ReadFile(s.fsys, path.Join(s.base, ManifestFile))
-	if embedded && err == nil {
-		embeddedManifests.Store(key, m)
-	}
-	return m, err
-}
-
-func (s Source) readManifest() (Manifest, error) {
-	if s.onDisk && s.label == "" {
-		return Manifest{}, nil // Walk refuses the empty data path
-	}
-	full := path.Join(s.base, ManifestFile)
-	b, err := fs.ReadFile(s.fsys, full)
 	if errors.Is(err, fs.ErrNotExist) {
 		return Manifest{}, nil
 	}
 	if err != nil {
-		return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err)
+		return Manifest{}, fmt.Errorf("%s: %w", s.ManifestPath(), err)
 	}
 	m, err := decodeManifest(b)
 	if err != nil {
-		return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err)
+		return Manifest{}, fmt.Errorf("%s: %w", s.ManifestPath(), err)
 	}
 	return m, nil
 }
 
+// ManifestPath is the manifest's path as errors name it.
+func (s Source) ManifestPath() string { return s.labelled(path.Join(s.base, ManifestFile)) }
+
 func decodeManifest(b []byte) (Manifest, error) {
 	dec := json.NewDecoder(bytes.NewReader(b))
 	dec.DisallowUnknownFields()
@@ -130,36 +109,67 @@ func decodeManifest(b []byte) (Manifest, error) {
 	if _, err := dec.Token(); err != io.EOF {
 		return Manifest{}, fmt.Errorf("more than one JSON value")
 	}
-	for p := range m.Index {
+	for p, e := range m.Index {
-		for _, seg := range strings.Split(p, ".") {
+		for _, q := range append([]string{p}, e.Reads...) {
-			if err := grammar.CheckIdentifier(seg); err != nil {
+			if err := checkCategoryPath(q); err != nil {
-				return Manifest{}, fmt.Errorf("index names %q: %w", p, err)
+				return Manifest{}, fmt.Errorf("index entry %q: %w", p, err)
 			}
 		}
+		if e.Parent == "" {
+			continue
+		}
+		if err := grammar.CheckIdentifier(e.Parent); err != nil {
+			return Manifest{}, fmt.Errorf("index entry %q: parent %w", p, err)
+		}
+		sibling := e.Parent
+		if dot := strings.LastIndexByte(p, '.'); dot >= 0 {
+			sibling = p[:dot+1] + e.Parent
+		}
+		if _, ok := m.Index[sibling]; !ok {
+			return Manifest{}, fmt.Errorf("index entry %q: parent %q names no entry beside it", p, e.Parent)
+		}
 	}
 	return m, nil
 }
 
+func checkCategoryPath(p string) error {
+	for _, seg := range strings.Split(p, ".") {
+		if err := grammar.CheckIdentifier(seg); err != nil {
+			return fmt.Errorf("%q: %w", p, err)
+		}
+	}
+	return nil
+}
+
 // Walk hands compile every category of the tree, folders and files in name order. A
 // folder holding no category anywhere below it is skipped; a hidden file or folder is no
 // category, though a category may name a hidden rows file beside it.
 func (s Source) Walk(compile func(Category) error) error {
-	if s.onDisk {
+	if err := s.check(); err != nil {
-		if s.label == "" {
+		return err
-			return fmt.Errorf("a data path is empty")
-		}
-		info, err := os.Stat(s.label)
-		if err != nil {
-			return fmt.Errorf("data path %s: %w", s.label, err)
-		}
-		if !info.IsDir() {
-			return fmt.Errorf("%s is not a directory", s.label)
-		}
 	}
 	_, err := s.walkDir(nil, compile)
 	return err
 }
 
+// check refuses a data path that is empty or names no directory.
+func (s Source) check() error {
+	if !s.onDisk {
+		return nil
+	}
+	if s.label == "" {
+		return fmt.Errorf("a data path is empty")
+	}
+	info, err := os.Stat(s.label)
+	if err != nil {
+		return fmt.Errorf("data path %s: %w", s.label, err)
+	}
+	if !info.IsDir() {
+		return fmt.Errorf("%s is not a directory", s.label)
+	}
+	return nil
+}
+
 // walkDir walks the folder that dir names, and reports whether it handed over any category.
 func (s Source) walkDir(dir []string, compile func(Category) error) (bool, error) {
 	full := s.dirPath(dir)

node.go

@@ -20,13 +20,27 @@ type node interface{ isNode() }
 // folder is a namespace of named children, built from a directory of JSON files
 // and subdirectories. It has no value of its own: descend into a named child by
 // dot path; rendering one is an error (see Fake).
+// A name stands in children or in unloaded, never both; put and putUnloaded keep it so.
 type folder struct {
 	children map[string]node
-	unloaded map[string]indexed // the categories here an index names and no call has reached yet
+	unloaded map[string]*indexed // the categories here an index names and no call has reached yet
 }
 
 func (*folder) isNode() {}
 
+func (g *folder) put(name string, n node) {
+	g.children[name] = n
+	delete(g.unloaded, name)
+}
+
+func (g *folder) putUnloaded(name string, x *indexed) {
+	delete(g.children, name)
+	if g.unloaded == nil {
+		g.unloaded = map[string]*indexed{}
+	}
+	g.unloaded[name] = x
+}
+
 // choice draws one of its items. cum holds cumulative weights for a weighted
 // draw; when nil the choice is uniform and selection is O(1). shared is the set of
 // relative dot paths every item carries, counting paths inside a repeated template.

todo.md

@@ -6,7 +6,7 @@
 
 `Bar = 9`
 
-`Next ID = 194`
+`Next ID = 195`
 
 | Goal | W |
 |---|---|
@@ -106,6 +106,7 @@
 | 178 | 0.2.0 |  | **Plan with the systems architect which `internal/` packages the engine splits into, by area, and how each type stops being written by more than one stage, then file the moves as items.** | 1 | 3 | 2 | 9 | 3, 3.2 | 18.4 |
 | 193 | 0.2.0 | decision | **Move the largest shipped packages into nested Go modules that keep their import paths, once item 192's gate fails.** | 6 | 6 | 6 | 7 | 2.3, 2 | 17.8 |
 | 92 | 0.2.0 |  | **Name the hyphenated field in a `calc`'s refusal, and say to rename it: today `{calc(postal-code * 2)}` reports `no field "postal"`.** | 1 | 2 | 5 | 6 | 6 | 17.5 |
+| 194 | 0.2.0 |  | **Let a data author write and check their own `.fejkdata.json` index with the library or the CLI, and fail a load whose index leaves out a category file.** | 2 | 4 | 5 | 7 | 6.3, 9.1 | 17.5 |
 | 53 | 0.2.0 |  | **Let a table column carry a `datatype`, so `--format json` writes `"safe": true` and `--format sql` a boolean.** | 4 | 5 | 6 | 6 | 4 | 16.9 |
 | 55 | 0.2.0 |  | **Document `NewRecordTemplate`, `ErrNoColumns` and `MaxRepeat` in the README's Library section.** | 1 | 2 | 6 | 7 | 12 | 16.8 |
 | 18 | 0.2.0 |  | **Give the address records one column set across countries: `region` and `municipality` as columns on `geo.SE.address` too.** | 4 | 4 | 6 | 6 | 5 | 16.6 |
@@ -422,6 +423,10 @@ Carries out the decision "The shipped data is Go packages a library imports by c
 
 A hyphen is always subtraction in a calc, and the shipped data has hyphenated fields (`postal-code` and `street-number` in `geo/SE/address`). The refusal names the first fragment, not the field the author meant.
 
+### 194. Let a data author write and check their own `.fejkdata.json` index with the library or the CLI, and fail a load whose index leaves out a category file.
+
+Goal 6.3 has an author prove their module, and goal 9.1 has fejkdata's data load as anyone's; today only a test of fejkdata's own writes and checks an index (`TestShippedManifestIsCurrent` over `indexEntry`). A category file the index leaves out never loads, silently. The principles audit of item 180's chunk found it.
+
 ### 53. Let a table column carry a `datatype`, so `--format json` writes `"safe": true` and `--format sql` a boolean.
 
 Today only a JSON field takes one, so `misc.httpmethod`'s booleans are typed in Go and text everywhere else. Typing a shipped column changes what `json` and `sql` write, so the capability is a minor and applying it to `misc.httpmethod` is a major.