Add the table node: rows TSV, key and name selection, linked tables drawn consistently, and the choice-of-rows fence

CHANGELOG.md

@@ -9,3 +9,15 @@ replacement, and each removed path, column or flag.
 ### Added
 
 - First release: the CLI, the library and the shipped data set.
+- Table categories: a category JSON naming a `rows` TSV beside it, with the options
+  `key`, `name`, `weight` and `parent`; a path selects a row by key or name,
+  `misc.country[SE]`, and descends to a linked table by name; linked tables draw
+  consistently within one render and draw group. `rows` is an option, so no
+  template may carry a field of that name.
+- `New` refuses a root choice of templates sharing one format and one set of string
+  fields, naming the rows TSV to write instead.
+- `misc.country`, `misc.currency`, `misc.language`, `misc.httpstatus` and
+  `misc.mimetype` are tables. `misc.country` is the full ISO 3166 register with the
+  columns `calling-code`, `capital`, `currency`, `flag`, `languages`, `numeric` and
+  `tld` added; `misc.currency` the current ISO 4217 list with `decimals` and
+  `numeric` added, and its symbols from CLDR. `DATA-LICENSES.md` lists each source.

README.md

@@ -17,6 +17,7 @@ fejkdata sv_SE.person.last                     # Eriksson
 fejkdata --seed 42 sv_SE.address               # the same address every run
 fejkdata -n 3 --separator ', ' sv_SE.word      # nät, barn, sol
 fejkdata --list                                # every path the data offers
+fejkdata 'misc.country[SE].capital'            # Stockholm — a table's row, selected by key or name
 fejkdata --data-path ./mydata sv_SE.word       # layer a directory over the shipped data
 fejkdata --no-shipped-data -d ./mydata --list  # only your data
 fejkdata 'name: {/sv_SE.person.last}'          # name: <a surname> — an inline template
@@ -24,7 +25,8 @@ fejkdata '{"format":"name: {x}","x":["bosse","lina"]}'  # name: bosse or name: l
 ```
 
 A path names a category, or a field inside one: each dot segment descends one
-level — folders, then the category (a JSON file), then fields. An argument that is
+level — folders, then the category (a JSON file), then fields — and `[SE]` after a
+[table](#table) selects its row. An argument that is
 a JSON object, array or string, or that carries a `{` token, is instead an
 **inline template**: a format string or a JSON value compiled and rendered on the
 spot. Its tokens reach the data by reference from the root —
@@ -32,8 +34,8 @@ spot. Its tokens reach the data by reference from the root —
 available. An inline template sits in no folder, so the folder-relative `{.name}`
 and `{..name}` are rejected naming the root spelling, and one reference alone —
 `{/sv_SE.person}` — is the path written as a template, rejected naming the path, as is
-a path written `/sv_SE.person`. A path never contains a brace, a bracket or a quote,
+a path written `/sv_SE.person`. A path never contains a brace or a quote, and a
-so the two cannot collide (see [Decisions](#decisions)).
+bracket only as a selector after a name, so the two cannot collide (see [Decisions](#decisions)).
 
 | Flag | |
 |------|--|
@@ -220,7 +222,10 @@ Each locale carries `address`, `color`, `company`, `date`, `email`, `ip`,
 `country` (ISO 3166), `creditcard` (Luhn-valid), `currency` (ISO 4217), `emoji`,
 `httpstatus`, `language` (ISO 639), `mac`, `mimetype`, `objectid`, `timezone`
 (IANA), `useragent` and `uuid` (v4). Many carry sub-fields — `misc.currency.symbol`,
-`misc.country.alpha2`, `misc.httpstatus.code` — which `--list` shows.
+`misc.country.alpha2`, `misc.httpstatus.code` — which `--list` shows. `country`,
+`currency`, `httpstatus`, `language` and `mimetype` are [tables](#table), so
+`misc.country[SE].capital` and `misc.currency[Euro].symbol` select a row;
+[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
 
 ## Data format
 
@@ -231,6 +236,7 @@ Every value is a **node**, nestable without limit:
 | string | `"Malmö"` | its text, with any `{…}` tokens expanded |
 | choice | `["a", "b", …]` | one item, picked at random |
 | template | `{"format": "…", …}` | its format, with `{name}` tokens rendering the named fields |
+| table | `{"format": "…", "rows": "x.tsv", …}` | its format over one row of the TSV beside it ([Table](#table)) |
 
 ### Format string
 
@@ -340,10 +346,105 @@ renders a null as `""`. The other items' weights skew its odds:
 load: `null` anywhere but a column, naming `""`, and a column whose items hold
 different datatypes.
 
+### Table
+
+A table is a category whose rows come from a TSV beside its JSON file: the header
+names the columns, each line below it is one row, and the format renders the row
+drawn. Save `mydata/country.tsv` and `mydata/country.json`:
+
+```tsv
+alpha2	name	population
+DK	Denmark	5900000
+NO	Norway	5500000
+SE	Sweden	10500000
+```
+
+```json
+{ "format": "{name} ({alpha2})", "rows": "country.tsv", "key": "alpha2", "name": "name", "weight": "population" }
+```
+
+```sh
+fejkdata -d ./mydata country                    # Sweden (SE), about half the time
+fejkdata -d ./mydata country.alpha2             # NO
+fejkdata -d ./mydata 'country[SE]'              # Sweden (SE)
+fejkdata -d ./mydata 'country[Norway].alpha2'   # NO
+fejkdata -d ./mydata --format csv country       # alpha2,name,population  →  SE,Sweden,10500000
+```
+
+`rows` names the TSV beside the category file; `key` names the column a path
+selects a row by, `name` a column it also selects by, `weight` a column of positive
+numbers that skews the draw, and `parent` the table a column links to
+([Linked tables](#linked-tables)). The format's `{tokens}` read the columns, and the
+columns are the [record](#records)'s columns, so `--format csv` writes the rows and
+`--list` shows `country.alpha2`. A cell is a string node: `1{digits(2)} {digits(2)}`
+in a cell draws digits and `{/misc.uuid}` reads a reference, while `{name}` in a cell
+is refused, since a cell has no sibling. `New` proves the header, the options and every
+cell token, and refuses a TSV no category names, a key that is empty or repeats, a
+weight that is not a positive number, and a key or name holding `[`, `]`, `{`, `}`,
+`"` or `|`, which a selector cannot spell; the rows are indexed on the first draw that
+selects one. The table's options are its own — `rows`, `key`, `name`, `weight` and
+`parent` — so a column may be named `name`, as one usually is.
+
+A choice of templates sharing one format and one set of string fields is a table
+written by hand, and `New` refuses it in a data file naming the TSV to write; an
+inline template has no file beside it, so there it stays a choice.
+
+### Row selection
+
+`[key]` or `[name]` after a table's name selects one row: `misc.country[SE]` and
+`misc.country[Sweden]` name one row, and `misc.country[SE].capital` reads its column.
+A key wins over a name that spells the same, and a name naming several rows is an
+error listing their keys, unless a row selected before it settles which
+([Linked tables](#linked-tables)). A selector is part of the path, so it works
+wherever a path does: `Fake`, `FakeRecord`, a `{/misc.country[SE].capital}` reference
+and a struct tag. A dot inside the brackets belongs to the key or name, so
+`city[St. Louis]` selects it. A path starts with a name, and `[` still opens a JSON
+array at the start of a CLI argument, so `'[SE]'` alone names nothing.
+
+### Linked tables
+
+A table's `parent` names a column and, by the same name, the table beside it that the
+column links to by key. Save `mydata/city.tsv` and `mydata/city.json` beside the
+`country` table above:
+
+```tsv
+name	country	population
+Copenhagen	DK	660000
+Göteborg	SE	600000
+Oslo	NO	710000
+Stockholm	SE	990000
+```
+
+```json
+{ "format": "{name}", "rows": "city.tsv", "key": "name", "parent": "country", "weight": "population" }
+```
+
+```sh
+fejkdata -d ./mydata 'country[SE].city'      # Stockholm or Göteborg
+fejkdata -d ./mydata country.city.name       # a country drawn, then a city inside it
+fejkdata -d ./mydata 'city[Oslo].country'    # NO — the link column's cell
+fejkdata -d ./mydata '{/city.name}, {/country.name}'   # Oslo, Norway — one consistent draw
+```
+
+A path descends from a row to a linked table by name, at any depth, and `--list`
+advertises each direct step. Within one render and [draw group](#draw-group), linked
+tables agree: the first table a reference path reads pins its ancestors, and a
+descendant read after it is drawn inside them, so `{/city.name}` and
+`{/country.name}` are a city and its country whichever is read first. A selected row
+pins the render the same way, so every reference path into one family of linked
+tables in one render and group selects the same rows: one that selects none beside
+one that does is refused naming the spelling that does, `{/country[SE].city.name}`
+beside `{/country[SE].name}`, and two selecting different rows are refused naming a
+`drawGroup` to draw them apart in. A bare `{/city}` beside a path into its family is
+refused too, since a bare reference draws each time. `New` also refuses a link cell
+that is no key of the parent, a parent row no child links to, a chain of parents that
+closes, and a child named like one of its parent's columns.
+
 ### Options and fields
 
 `format`, `weight`, `repeat`, `separator`, `datatype` and `drawGroup` are the only options;
-**any other key is a field** (see [Decisions](#decisions)). An object that does nothing a
+**any other key is a field** (see [Decisions](#decisions)), and `rows` makes a category
+a [table](#table), so no template carries a field of that name. An object that does nothing a
 string can't — only a `format` — is rejected naming the string, as is a one-item
 choice naming its item.
 
@@ -470,10 +571,11 @@ its groups by name; the unnamed group spans them all.
 Renders e.g. `Sara Eriksson pays Ebba Lind; signed Eriksson`: the signature reads the
 payer's draw, while the payee is drawn apart. Rejected at load, each naming nothing: a
 `drawGroup` of `""` (the default); one naming the draw group its template already draws
-in; one on a template that renders no reference path short of a `repeat` or a nested
+in; one on a template that renders no reference path — a bare reference to a
-`drawGroup`, on a `repeat` itself — each iteration renders in no draw group — or on an
+[table](#table) counts, since the group answers for the family it draws in — short of a
-inline template's root, which nothing references. So is a path reading into a level that
+`repeat` or a nested `drawGroup`, on a `repeat` itself — each iteration renders in no
-carries a `drawGroup`.
+draw group — or on an inline template's root, which nothing references. So is a path
+reading into a level that carries a `drawGroup`.
 
 ### Correlated fields
 
@@ -538,7 +640,7 @@ a minor only adds, and a major is the only release that changes what exists.
 
 | Surface | Major | Minor |
 |---------|-------|-------|
-| Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another | a path outside a record's columns, a locale, a value in a list |
+| Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another; change a table's key, name, weight or parent column, or remove a row | a path outside a record's columns, a locale, a value in a list, a row |
 | Records | remove, rename, retype or add a column; let a column be null | a record, as a new category |
 | Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin |
 | CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format |
@@ -559,8 +661,8 @@ with no breaking change. From `v2` the module path carries `/vN`, so fences ship
 batched into as few majors as possible.
 
 [`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) pins every path, each
-template category's format, the categories each category reads, and each column's
+template category's format, the categories each category reads, each column's
-datatype and nullability; a pull request that
+datatype and nullability, and each table's key, name, weight and parent columns; a pull request that
 changes it or `data/` adds its `CHANGELOG.md` entry, which CI checks. A removed,
 renamed or retyped line is a major.
 
@@ -796,6 +898,52 @@ renamed or retyped line is a major.
   almost always costs an allocation too (a lost pre-size, a per-item map, an extra
   copy). The benchmark suite (see Development) reports time for a human, not as a
   pass/fail gate.
+- **Rows live in a TSV, the shape in JSON.** `New` allocates once per node, so a
+  register of thirty thousand rows written as JSON objects would cost it a second;
+  a TSV is one allocation whose cells are substrings, and the JSON says only how a
+  row is composed. The TSV sits beside its category file, named by `rows`, so a
+  data directory stays a directory of categories, and one nothing names is a
+  load error rather than a file silently ignored.
+- **A selector is bracketed, and a dot inside it is literal.** `municipality[0180]`
+  reads as selection to anyone who has indexed an array, and `[St. Louis]` keeps a
+  name whole where a colon or a dot-separated spelling could not; zsh needs the
+  brackets quoted, which the README's examples show. A key wins over a name that
+  spells the same, since a key names one row by contract and a name may not.
+- **A table read into is pinned; a table rendered whole draws afresh.** A path into a
+  table pins its row for the render and group, as a reference path pins its level,
+  and a bare `{/city}` draws each time, as a bare reference does; so a bare table
+  beside a path into its family is refused like a bare reference beside a path into
+  it. A bare table reference still counts as a read for a `drawGroup`, since the group
+  is what draws it apart from the family's pins.
+- **Every reference path into one family selects the same rows, per render and
+  group.** A selector pins rows for the render, and a read that draws freely before it
+  could pin a row the selector contradicts, so accepting both would make the result
+  depend on which token rendered first. Requiring one selection per family per group
+  is checkable at load with no data lookup beyond the selectors themselves, and the
+  error names the rewrite. Two selectors naming one row by key and by name compare
+  equal, since the load check resolves them.
+- **A parent row with no child row is a load error.** A descendant is drawn inside the
+  nearest pinned ancestor, so every ancestor row must lead to a row at every level
+  below it, or a render could find nothing to draw. The import script drops or fills
+  such rows; the alternative, falling back to a free draw, would break the consistency
+  the link exists for without saying so.
+- **The choice-of-rows fence guards a data file's root, and requires string fields.**
+  A table is a category with a TSV beside its file, so only a root choice has the
+  spelling the fence names; a nested choice of same-shaped templates and an inline
+  one keep loading. Fields must all be strings because a cell is a string node: a
+  choice whose items carry a nested choice, as `misc.car` does, is not one table but
+  two linked ones, which a later conversion writes.
+- **A table is a record of string columns.** Its columns are the CSV header and the
+  `INSERT` column list, fixed by the TSV header, so a table is a record by
+  construction; every column is a string until a typed column option earns its place.
+- **The key index is built at load, the rest on first draw.** A link is proved
+  against the parent's keys and a key's uniqueness is a data mistake, so both are
+  load-time; the name index and the per-parent child lists serve only a draw or a
+  selection, so they wait for the first one, keeping `New` linear in the bytes read.
+- **`List` advertises direct descents only.** `region.municipality.locality` is
+  listed, and `region.locality` resolves too but is not: the set of every descent
+  through a chain of five tables is every subsequence of it, and the direct chain is
+  the one a reader can predict from the tables' parents.
 
 ## Development
 
@@ -839,6 +987,15 @@ in its own commit:
 REPIN=1 docker compose run --rm --user "$(id -u):$(id -g)" test
 ```
 
+A shipped table built from a source is rebuilt by its script under
+[`data-import/`](data-import), one command per dataset, fetching the source named in
+[`DATA-LICENSES.md`](DATA-LICENSES.md):
+
+```sh
+docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/country.py
+docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py
+```
+
 To release, head `CHANGELOG.md` with the version's section in place of `Unreleased`
 and merge: once `main` passes the gate, CI tags that commit `vX.Y.Z` and publishes
 the Gitea release with the section as its body. A top heading of `[Unreleased]`
@@ -849,7 +1006,8 @@ publishes nothing.
 ```
 fejkdata.go     Generator, New, options, the embedded data set, List
 node.go         the node model and JSON -> node compilation
-path.go         the dotted-path walk, and proving a path resolves
+table.go        tables: the rows TSV, its options and links, row selection and draws
+path.go         the dotted-path walk with its selectors, and proving a path resolves
 render.go       Fake and the recursive renderer (choices, format strings, expansions)
 record.go       records: Record, the JSON/CSV/SQL serializers, and their entry points
 struct.go       structs: FakeStruct, fake tags, and a field's Go type as its column's datatype
@@ -865,7 +1023,8 @@ datatype.go     column datatypes: DataType, where datatype and null may sit, a c
 value.go        the value proof: what a typed column or calc operand holds, checked at load
 data.go         data loading: fs.FS folders/files -> namespace tree, multi-source merge
 cmd/fejkdata/   the fejkdata CLI
-data/           shipped data (JSON), embedded at build: locale folders + a misc folder
+data/           shipped data (JSON, and a TSV per table), embedded at build: locale folders + a misc folder
+data-import/    the scripts that rebuild each sourced table (see DATA-LICENSES.md)
 release-tooling/ the release CI publishes from the changelog heading
 testdata/       the pinned shipped shape (see Versioning)
 ```

cmd/fejkdata/main.go

@@ -23,16 +23,17 @@ import (
 
 const usage = `Usage: fejkdata [flags] <path|template>
 
-  <path>                 a category, or a dotted path into one (person, person.last)
+  <path>                 a category, or a dotted path into one (person, person.last);
+                         a table's row by key or name: 'misc.country[SE]', 'misc.country[Sweden].tld'
   <template>             a format string or JSON value to render inline, e.g.
                          'name: {/sv_SE.person.last}' or '{"format":"{x}","x":["bosse","lina"]}'
 
 An argument containing a { token, or a JSON object, array or string, is a
-template; any other argument is a path (a path never contains a brace, a bracket
+template; any other argument is a path (a path never contains a brace or a quote,
-or a quote). Templates reach the data by reference from the root —
+and a bracket only as a [selector] after a table's name). Templates reach the
-{/sv_SE.person.last} — whether the data is shipped or layered with --data-path. An
+data by reference from the root — {/sv_SE.person.last} — whether the data is
-argument carrying a bracket, a closing brace or a quote but no valid JSON names
+shipped or layered with --data-path. An argument carrying a closing brace or a
-neither.
+quote but no valid JSON names neither.
 
 With --format json, ndjson, csv or sql the argument must name a record — a
 template whose fields are its columns — and the rows are written as one JSON
@@ -396,7 +397,18 @@ func defaultTable(arg string, kind argKind) string {
 	if kind == argTemplate {
 		return "records"
 	}
-	segments := strings.Split(arg, ".")
+	var names strings.Builder // the path with its [selectors] cut out
+	for depth, i := 0, 0; i < len(arg); i++ {
+		switch {
+		case arg[i] == '[':
+			depth++
+		case arg[i] == ']':
+			depth--
+		case depth == 0:
+			names.WriteByte(arg[i])
+		}
+	}
+	segments := strings.Split(names.String(), ".")
 	return segments[len(segments)-1]
 }
 

data.go

@@ -62,6 +62,9 @@ func loadData(sources []dataSource) (map[string]node, error) {
 	if len(root) == 0 {
 		return nil, fmt.Errorf("no .json data found")
 	}
+	if err := linkTables(root); err != nil {
+		return nil, err
+	}
 	if err := linkRefs(root); err != nil {
 		return nil, err
 	}
@@ -83,22 +86,56 @@ func loadDir(src dataSource, dir string) (*folder, error) {
 		return nil, fmt.Errorf("%s: %w", src.name(dir), err)
 	}
 	g := &folder{children: map[string]node{}}
+	files := &categoryFiles{src: src, dir: dir, tsv: map[string]bool{}}
+	for _, e := range entries {
+		if strings.HasSuffix(e.Name(), ".tsv") && !e.IsDir() {
+			files.tsv[e.Name()] = false
+		}
+	}
 	for _, e := range entries {
 		if strings.HasPrefix(e.Name(), ".") { // hidden: a checkout or an editor's file, never data
 			continue
 		}
 		full := path.Join(dir, e.Name())
-		load := loadFile
 		if e.IsDir() {
-			load = loadFolder
+			if err := loadFolder(src, g, full, e.Name()); err != nil {
+				return nil, err
+			}
+			continue
 		}
-		if err := load(src, g, full, e.Name()); err != nil {
+		if err := loadFile(src, g, full, e.Name(), files); err != nil {
 			return nil, err
 		}
 	}
+	for name, named := range files.tsv {
+		if !named && !strings.HasPrefix(name, ".") {
+			return nil, fmt.Errorf("%s: no category names it in its rows; a table's rows file sits beside a category file naming it", src.name(path.Join(dir, name)))
+		}
+	}
 	return g, nil
 }
 
+// categoryFiles is what a category may name beside itself: the rows files of its
+// directory, each marked once a category names it.
+type categoryFiles struct {
+	src dataSource
+	dir string
+	tsv map[string]bool
+}
+
+// rows reads the rows file a category names beside it.
+func (c *categoryFiles) rows(name string) (string, error) {
+	if _, present := c.tsv[name]; !present {
+		return "", fmt.Errorf("rows names %s, which is not beside it in %s", name, c.src.name(c.dir))
+	}
+	c.tsv[name] = true
+	b, err := fs.ReadFile(c.src.fsys, path.Join(c.dir, name))
+	if err != nil {
+		return "", fmt.Errorf("%s: %w", c.src.name(path.Join(c.dir, name)), err)
+	}
+	return string(b), nil
+}
+
 // loadFolder adds a subdirectory as a nested folder, unless nothing under it is data.
 func loadFolder(src dataSource, g *folder, full, name string) error {
 	child, err := loadDir(src, full)
@@ -117,7 +154,7 @@ func loadFolder(src dataSource, g *folder, full, name string) error {
 
 // loadFile compiles a *.json file into a category named after it; any other file
 // is skipped.
-func loadFile(src dataSource, g *folder, full, file string) error {
+func loadFile(src dataSource, g *folder, full, file string, files *categoryFiles) error {
 	if !strings.HasSuffix(file, ".json") {
 		return nil
 	}
@@ -133,7 +170,7 @@ func loadFile(src dataSource, g *folder, full, file string) error {
 	if err := json.Unmarshal(b, &raw); err != nil {
 		return fmt.Errorf("%s: %w", src.name(full), err)
 	}
-	n, err := compile(raw)
+	n, err := compileCategory(raw, name, files)
 	if err != nil {
 		return fmt.Errorf("%s: %w", src.name(full), err)
 	}

draw.go

@@ -13,21 +13,127 @@ type drawSet struct {
 }
 
 // drawScope is where a render reads its reference paths: a draw set, in the draw group of the
-// template rendering.
+// template rendering, and the row a table's format is rendering, which its columns read.
 type drawScope struct {
 	set   *drawSet
 	group string
+	t     *table
+	row   int
+}
+
+// pinned is the row the render pinned for t, if any.
+func (d *draws) pinned(t *table) (int, bool) {
+	for _, p := range d.pins[:d.npins] {
+		if p.t == t {
+			return p.row, true
+		}
+	}
+	r, ok := d.more[t]
+	return r, ok
+}
+
+// mustRow is the row pinned for t, which the walk reaching a column pinned.
+func (d *draws) mustRow(t *table) int {
+	r, ok := d.pinned(t)
+	if !ok {
+		panic(fmt.Sprintf("fejkdata: a column of %s is rendered with no row pinned", t.category))
+	}
+	return r
+}
+
+// pin pins row r of t, and the rows of t's ancestors it links to.
+func (d *draws) pin(t *table, r int) {
+	for {
+		if _, done := d.pinned(t); done {
+			return
+		}
+		if d.npins < len(d.pins) {
+			d.pins[d.npins] = tablePin{t, r}
+			d.npins++
+		} else {
+			if d.more == nil {
+				d.more = map[*table]int{}
+			}
+			d.more[t] = r
+		}
+		if t.parentT == nil {
+			return
+		}
+		t, r = t.parentT, t.parentRow(r)
+	}
+}
+
+// rowOf is the render's row of t: the one pinned, else one drawn inside the
+// nearest pinned ancestor — its parent drawn inside that first where the ancestor
+// is further up — or over the whole table, and pinned with its ancestors.
+func (d *draws) rowOf(t *table) int {
+	if r, ok := d.pinned(t); ok {
+		return r
+	}
+	r := -1
+	for a := t.parentT; a != nil && r < 0; a = a.parentT {
+		if _, ok := d.pinned(a); !ok {
+			continue
+		}
+		if t.parentT != a {
+			d.rowOf(t.parentT)
+		}
+		pr, _ := d.pinned(t.parentT)
+		r = t.drawUnder(d.s, pr)
+	}
+	if r < 0 {
+		r = t.draw(d.s)
+	}
+	d.pin(t, r)
+	return r
+}
+
+// selectRow pins the row a selector names, refusing one outside the rows pinned
+// before it.
+func (d *draws) selectRow(t *table, sel string) error {
+	r, err := t.find(sel, d)
+	if err != nil {
+		return err
+	}
+	if pr, ok := d.pinned(t); ok && pr != r {
+		return fmt.Errorf("%s is not %s, the row already drawn", t.selectorSpelling(r), t.selectorSpelling(pr))
+	}
+	for a := t.parentT; a != nil; a = a.parentT {
+		if pa, ok := d.pinned(a); ok && !t.under(r, a, pa) {
+			return fmt.Errorf("%s is not inside %s", t.selectorSpelling(r), a.selectorSpelling(pa))
+		}
+	}
+	d.pin(t, r)
+	return nil
+}
+
+// inside keeps the rows of t that sit inside every pinned ancestor.
+func (d *draws) inside(t *table, rows []int) []int {
+	for a := t.parentT; a != nil; a = a.parentT {
+		pa, ok := d.pinned(a)
+		if !ok {
+			continue
+		}
+		var kept []int
+		for _, r := range rows {
+			if t.under(r, a, pa) {
+				kept = append(kept, r)
+			}
+		}
+		rows = kept
+	}
+	return rows
 }
 
 // newDrawSet makes the unnamed group's maps where the set is declared, keeping them on that frame's
 // stack for a render that reads through them; a zero drawSet makes them on its first read instead.
-func newDrawSet() drawSet {
+func newDrawSet(s *session) drawSet {
-	return drawSet{unnamed: draws{variant: map[string]node{}, value: map[string]draw{}}}
+	return drawSet{unnamed: draws{variant: map[string]node{}, value: map[string]draw{}, s: s}}
 }
 
 // renderOnce renders n as one render, over draws of its own.
 func renderOnce(s *session, n node) string {
-	var set drawSet
+	set := drawSet{unnamed: draws{s: s}}
 	return render(s, n, drawScope{set: &set})
 }
 
@@ -39,8 +145,9 @@ func (sc drawScope) in(t *template) drawScope {
 	return sc
 }
 
-// draws is the set's draws for sc's draw group.
+// draws is the set's draws for sc's draw group. The session is passed in rather than
-func (sc drawScope) draws() *draws {
+// read out of the set: copying it from there would leak the set's maps to the heap.
+func (sc drawScope) draws(s *session) *draws {
 	if sc.group == "" {
 		return &sc.set.unnamed
 	}
@@ -49,7 +156,7 @@ func (sc drawScope) draws() *draws {
 		if sc.set.named == nil {
 			sc.set.named = map[string]*draws{}
 		}
-		d = &draws{variant: map[string]node{}, value: map[string]draw{}}
+		d = &draws{variant: map[string]node{}, value: map[string]draw{}, s: s}
 		sc.set.named[strings.Clone(sc.group)] = d // a key from sc would leak sc, and with it every render's draw set, to the heap
 	}
 	return d
@@ -186,7 +293,7 @@ func (c *drawCheck) reads(n node, stopAtGroup bool) bool {
 	}
 	r := false
 	for _, e := range renderEdges(n) {
-		if a, isRef := refRead(n, e.label); (isRef && len(a.tail) > 0) || (!repeats(e.to) && !(stopAtGroup && grouped(e.to)) && c.reads(e.to, stopAtGroup)) {
+		if a, isRef := refRead(n, e.label); (isRef && (len(a.tail) > 0 || readsTable(n, a))) || (!repeats(e.to) && !(stopAtGroup && grouped(e.to)) && c.reads(e.to, stopAtGroup)) {
 			r = true
 			break
 		}
@@ -208,6 +315,17 @@ func grouped(n node) bool {
 	return isTemplate && t.drawGroupKey != ""
 }
 
+// readsTable reports whether a reference of n names a table, whose draw the render's
+// group answers for even when read whole.
+func readsTable(n node, a arm) bool {
+	t, isTemplate := n.(*template)
+	if !isTemplate {
+		return false
+	}
+	_, isTable := t.fields[a.key].(*table)
+	return isTable
+}
+
 // refRead is the reference an edge of n reads; false when the edge reads none.
 func refRead(n node, label string) (arm, bool) {
 	t, isTemplate := n.(*template)
@@ -247,6 +365,79 @@ type drawRoute struct{ spelling, label string }
 type pathRead struct {
 	at drawAt
 	a  arm
+	tr *tableRead // set where the reference names a table
+}
+
+// tableRead is what a reference reads of a table family: the table named, the rows
+// its selectors pin along the way, and whether it lands on a row rendered whole.
+type tableRead struct {
+	head  *table
+	sels  []tableSel
+	whole bool
+}
+
+// tableSel is one selector on the way: the table it selects a row of, and the
+// selector's spelling up to its closing bracket.
+type tableSel struct {
+	t        *table
+	row      int
+	spelling string
+}
+
+// tableReadOf reads what a reference path does of a table, replaying its selectors
+// over a walk of its own — checkPath proved each names a row — so two paths naming
+// one row by key and by name compare equal.
+func tableReadOf(head node, a arm, leaf node) *tableRead {
+	t, isTable := head.(*table)
+	if !isTable {
+		return nil
+	}
+	tr := &tableRead{head: t}
+	var pins draws
+	cur, seen := t, a.name
+	for _, seg := range a.tail {
+		switch {
+		case isSelector(seg):
+			end := strings.Index(seen, "]") + 1
+			_ = pins.selectRow(cur, selectorOf(seg))
+			row, _ := pins.pinned(cur)
+			tr.sels = append(tr.sels, tableSel{cur, row, a.name[:len(a.name)-len(seen)+end]})
+			seen = seen[end:]
+		case cur.children[seg] != nil:
+			cur = cur.children[seg]
+		}
+	}
+	c, isColumn := leaf.(*column)
+	tr.whole = isColumn && c.i < 0
+	return tr
+}
+
+// family is the table a chain of parents ends at.
+func (t *table) family() *table {
+	for t.parentT != nil {
+		t = t.parentT
+	}
+	return t
+}
+
+// selected is the selector in r that pins t, or the nearest ancestor of t it pins.
+func (r *tableRead) selected(t *table) (tableSel, bool) {
+	for ; t != nil; t = t.parentT {
+		for _, s := range r.sels {
+			if s.t == t {
+				return s, true
+			}
+		}
+	}
+	return tableSel{}, false
+}
+
+func (r *tableRead) selection() string {
+	parts := make([]string, len(r.sels))
+	for i, s := range r.sels {
+		parts[i] = fmt.Sprintf("%s[%d]", s.t.category, s.row)
+	}
+	return strings.Join(parts, " ")
 }
 
 type drawVisit struct {
@@ -286,15 +477,15 @@ func (w *drawWalk) edge(from node, e renderEdge, at drawAt) {
 	if a, reads := refRead(from, e.label); reads {
 		if k := (drawKey{at.group, a.path}); !w.read[k] {
 			w.read[k] = true
-			w.reads = append(w.reads, pathRead{at, a})
+			w.reads = append(w.reads, pathRead{at, a, tableReadOf(from.(*template).fields[a.key], a, e.to)})
 		}
 	}
 	w.walk(e.to, at)
 }
 
 // check refuses what one draw per reference path cannot answer for: a read of a level beside a path
-// another read takes into it. Reads are compared in path order, so which pair is reported does not
+// another read takes into it, and two reads of one table family that select different rows. Reads
-// vary.
+// are compared in path order, so which pair is reported does not vary.
 func (w *drawWalk) check() error {
 	sort.SliceStable(w.reads, func(i, j int) bool {
 		if w.reads[i].at.group != w.reads[j].at.group {
@@ -304,14 +495,53 @@ func (w *drawWalk) check() error {
 	})
 	for i, level := range w.reads {
 		for _, into := range w.reads[i+1:] {
-			if into.at.group == level.at.group && strings.HasPrefix(into.a.path, level.a.path+".") {
+			if into.at.group != level.at.group {
+				continue
+			}
+			if strings.HasPrefix(into.a.path, level.a.path+".") && !(level.tr != nil && level.tr.whole) {
 				return overlapError(level.at.route, level.a.name, into)
 			}
+			if err := checkFamily(level, into); err != nil {
+				return err
+			}
 		}
 	}
 	return nil
 }
 
+// checkFamily refuses two reads of one table family in one group that cannot read one consistent
+// draw: a table rendered whole beside a path into the family, and two paths selecting different rows.
+func checkFamily(a, b pathRead) error {
+	if a.tr == nil || b.tr == nil || a.tr.head.family() != b.tr.head.family() {
+		return nil
+	}
+	for _, pair := range [][2]pathRead{{a, b}, {b, a}} {
+		bare, path := pair[0], pair[1]
+		if len(bare.a.tail) == 0 && len(path.a.tail) > 0 {
+			return overlapError(bare.at.route, bare.a.name, path)
+		}
+	}
+	if len(a.a.tail) == 0 || len(b.a.tail) == 0 || a.tr.selection() == b.tr.selection() {
+		return nil
+	}
+	for _, pair := range [][2]pathRead{{a, b}, {b, a}} {
+		selected, plain := pair[0], pair[1]
+		if len(plain.tr.sels) > 0 {
+			continue
+		}
+		if s, ok := selected.tr.selected(plain.tr.head); ok {
+			tail := plain.a.tail
+			if s.t != plain.tr.head {
+				tail = append([]string{plain.tr.head.category}, tail...)
+			}
+			return fmt.Errorf("%s reads %s without the row %s selects; write {%s.%s}, or draw them apart with a drawGroup",
+				plain.at.route.spelled(plain.a.name), plain.tr.head.category, selected.at.route.spelled(selected.a.name), s.spelling, strings.Join(tail, "."))
+		}
+	}
+	return fmt.Errorf("%s and %s select different rows of one table family; select the same rows in both, or draw them apart with a drawGroup",
+		a.at.route.spelled(a.a.name), b.at.route.spelled(b.a.name))
+}
+
 func overlapError(route drawRoute, ref string, into pathRead) error {
 	return fmt.Errorf("%s renders a level that %s reads a path into; name the fields you want instead, or draw them apart with a drawGroup", route.spelled(ref), into.at.route.spelled(into.a.name))
 }

fejkdata.go

@@ -46,6 +46,8 @@ type Generator struct {
 	mu         sync.Mutex
 	rand       *session
 	categories map[string]node
+	root       *folder // the categories as the node a path walks from
+	set        drawSet // one Fake's draws, owned here so a walk pinning rows keeps them off the heap
 	records    map[node]recordShape
 	structs    map[reflect.Type]structResult
 }
@@ -123,7 +125,7 @@ func New(opts ...Option) (*Generator, error) {
 	if err != nil {
 		return nil, fmt.Errorf("fejkdata: %w", err)
 	}
-	return &Generator{rand: rng, categories: cats}, nil
+	return &Generator{rand: rng, categories: cats, root: &folder{children: cats}}, nil
 }
 
 // List returns the sorted dotted paths Fake can render: every category, the dotted
@@ -166,6 +168,10 @@ func paths(n node) []string {
 		return out
 	case *null:
 		return []string{""}
+	case *table:
+		return tablePaths(n)
+	case *column:
+		return []string{""}
 	case *choice:
 		out := []string{""}
 		for p := range n.shared {
@@ -176,6 +182,24 @@ func paths(n node) []string {
 	return nil
 }
 
+// tablePaths is a table's columns, then each table linked to it under its name: the
+// direct descents, a step at a time.
+func tablePaths(t *table) []string {
+	out := append([]string{""}, t.columns...)
+	sort.Strings(out[1:])
+	children := make([]string, 0, len(t.children))
+	for name := range t.children {
+		children = append(children, name)
+	}
+	sort.Strings(children)
+	for _, name := range children {
+		for _, p := range paths(t.children[name]) {
+			out = append(out, join(name, p))
+		}
+	}
+	return out
+}
+
 // sharedPaths is the sub-paths every item carries — the only ones a path may step
 // through a choice to reach. It intersects, bailing as soon as the set
 // is empty, which is immediate for a choice of plain strings.

graph.go

@@ -3,7 +3,6 @@ package fejkdata
 import (
 	"fmt"
 	"sort"
-	"strings"
 )
 
 // walkNodes calls fn once per contained node, passing the dot path that reaches it,
@@ -61,6 +60,19 @@ func contained(n node) []namedNode {
 		return out
 	case *template:
 		return named(n.fields)
+	case *table:
+		return append([]namedNode{{node: n.format}}, named(n.fields)...)
+	case *column:
+		if n.i < 0 {
+			return nil
+		}
+		var out []namedNode
+		for r := 0; r < n.t.rows(); r++ {
+			if cell := n.t.cellNode(r, n.i); cell != nil {
+				out = append(out, namedNode{node: cell})
+			}
+		}
+		return out
 	default:
 		return nil
 	}
@@ -142,12 +154,23 @@ func renderEdges(n node) []renderEdge {
 				}
 				return nil
 			}
-			for _, name := range strings.Split(t.body, "|") {
+			for _, name := range splitOutside(t.body, '|') {
 				add(name, "")
 			}
 			return nil
 		})
 		return es
+	case *table:
+		return []renderEdge{{to: n.format, label: "format"}}
+	case *column:
+		if n.i < 0 {
+			return []renderEdge{{to: n.t.format, label: "format"}}
+		}
+		var es []renderEdge
+		for _, c := range contained(n) {
+			es = append(es, renderEdge{to: c.node, label: n.t.columns[n.i]})
+		}
+		return es
 	default:
 		return nil
 	}
@@ -158,7 +181,7 @@ func renderEdges(n node) []renderEdge {
 // already proved the tail resolves in every variant.
 func pathLeaves(n node, tail []string) []node {
 	var out []node
-	_ = walkPath(n, tail, pathWalk{
+	_, _ = walkPath(n, tail, pathWalk{
 		choice: func(c *choice, _ []string) ([]node, error) { return c.items, nil },
 		leaf:   func(n node) error { out = append(out, n); return nil },
 	})

hold.go

@@ -109,7 +109,7 @@ func operandReader(t *template, head string) string {
 // coverPath collects what holding one path pins: every choice level the path
 // passes through, whole, and the leaf it renders.
 func coverPath(n node, tail []string, into map[node]bool) {
-	_ = walkPath(n, tail, pathWalk{
+	_, _ = walkPath(n, tail, pathWalk{
 		choice: func(c *choice, _ []string) ([]node, error) { cover(c, into, false); return nil, nil },
 		leaf:   func(n node) error { cover(n, into, false); return nil },
 	})
@@ -262,6 +262,16 @@ func checkNoRepeatedRead(format string, c formatOps, refs map[string]refBinding)
 type draws struct {
 	variant map[string]node
 	value   map[string]draw
+	pins    [4]tablePin // the rows pinned, inline so a render pinning a few tables stays off the heap
+	npins   int
+	more    map[*table]int // the rows pinned past the inline four
+	s       *session       // what draws a row; nil where a walk only proves selectors
+}
+
+// tablePin is one table's pinned row.
+type tablePin struct {
+	t   *table
+	row int
 }
 
 // draw is what one read drew: its text, and whether it landed on a null.
@@ -284,12 +294,11 @@ func readField(s *session, t *template, held *draws, sc drawScope, a arm) draw {
 		}
 		return draw{text: render(s, t.fields[a.key], sc)}
 	}
-	d := readScope(held, sc, a)
+	d := readScope(s, held, sc, a)
 	if r, done := d.value[a.path]; done {
 		return r
 	}
-	var r draw
+	leaf, _ := walkPath(t.fields[a.key], a.tail, pathWalk{
-	_ = walkPath(t.fields[a.key], a.tail, pathWalk{
 		// Hold the draw at every level passed through, so two paths sharing a
 		// prefix share it.
 		choice: func(c *choice, rest []string) ([]node, error) {
@@ -307,8 +316,9 @@ func readField(s *session, t *template, held *draws, sc drawScope, a arm) draw {
 			}
 			return []node{n}, nil
 		},
-		leaf: func(n node) error { r = renderLeaf(s, n, sc); return nil },
+		pins: d,
 	})
+	r := renderLeaf(s, leaf, sc)
 	if d.value == nil {
 		d.value = map[string]draw{}
 	}
@@ -319,9 +329,9 @@ func readField(s *session, t *template, held *draws, sc drawScope, a arm) draw {
 // readScope is the draws a held read keeps its draw in: for a reference that reads a path,
 // the render's draws for its group, so its draw spans the render; for a sibling, or a
 // reference read whole, held.
-func readScope(held *draws, sc drawScope, a arm) *draws {
+func readScope(s *session, held *draws, sc drawScope, a arm) *draws {
 	if isRef(a.key) && len(a.tail) > 0 {
-		return sc.draws()
+		return sc.draws(s)
 	}
 	return held
 }
@@ -335,7 +345,7 @@ func renderLeaf(s *session, n node, sc drawScope) draw {
 	}
 	r := draw{text: render(s, n, sc)}
 	if t, _ := n.(*template); t != nil && t.readsColumn != nil {
-		r.null = sc.in(t).draws().value[t.readsColumn.a.path].null
+		r.null = sc.in(t).draws(s).value[t.readsColumn.a.path].null
 	}
 	return r
 }

inline.go

@@ -69,9 +69,15 @@ func isTemplate(arg string) (bool, error) {
 		}
 		return true, nil
 	}
-	if i := strings.IndexAny(arg, `[]}"`); i >= 0 {
+	if i := strings.IndexAny(arg, `}"`); i >= 0 {
 		return false, fmt.Errorf("%q holds a %q, which no path may, and it is not valid JSON, so it names no template either", arg, arg[i:i+1])
 	}
+	if strings.HasPrefix(strings.TrimSpace(arg), "[") {
+		return false, fmt.Errorf(`%q starts with "[" and is not valid JSON, so it names no template; a path starts with a name`, arg)
+	}
+	if _, err := splitPath(arg); err != nil {
+		return false, err
+	}
 	if path := strings.TrimLeft(arg, "/"); path != arg && path != "" {
 		return false, pathAdvice(path, fmt.Sprintf("path %s starts with /, and every path starts at the root already", arg))
 	}

node.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"math"
+	"slices"
 	"sort"
 	"strings"
 )
@@ -59,9 +60,11 @@ type template struct {
 	// held is every name drawn once per expansion: the bound levels above, plus the
 	// siblings a {calc()} reads. nil when the format holds nothing (see expand).
 	held         map[string]bool
+	heldLocal    bool        // some held name is kept by the expansion itself, so expand makes its draws
 	fromString   bool        // written as a JSON string rather than an object
 	readsColumn  *columnRead // set when the format is one reference alone reading a record's column
 	record       bool        // compiled at the top without a repeat, so its fields are record columns
+	table        *table      // the table whose format this is, whose columns are the fields
 	drawGroup    string      // the draw group it draws in, as written; "" keeps its caller's
 	drawGroupKey string      // its draw group keyed by its category once linked: what a render reads its reference paths under
 }
@@ -83,6 +86,70 @@ func compile(v any) (node, error) {
 	return compileAt(v, atTop)
 }
 
+// compileCategory compiles a data file's value, which may be a table over a rows
+// file beside it, and refuses a choice that is a table written as templates.
+func compileCategory(v any, name string, files *categoryFiles) (node, error) {
+	if m, ok := v.(map[string]any); ok {
+		if _, isTable := m["rows"]; isTable {
+			return compileTable(m, name, files)
+		}
+	}
+	if items, ok := v.([]any); ok {
+		if err := checkNotRows(items, name); err != nil {
+			return nil, err
+		}
+	}
+	return compile(v)
+}
+
+// checkNotRows refuses a choice of templates sharing one format and one set of
+// string fields: each item is a row, and the rows file is the spelling for that.
+func checkNotRows(items []any, name string) error {
+	var format string
+	var fields []string
+	weighted := false
+	for i, raw := range items {
+		f, keys, w, isRow := rowShape(raw)
+		if !isRow || i > 0 && (f != format || !slices.Equal(keys, fields)) {
+			return nil
+		}
+		format, fields, weighted = f, keys, weighted || w
+	}
+	if len(items) < 2 || len(fields) == 0 {
+		return nil
+	}
+	weight := ""
+	if weighted {
+		weight = `, a weight column, and "weight" naming it`
+	}
+	return fmt.Errorf("a choice of %d templates with one format and the fields %v is a table; write the rows in %s.tsv with the header %q%s, and the category as {\"format\": %q, \"rows\": \"%s.tsv\"}",
+		len(items), fields, name, strings.Join(fields, "\t"), weight, format, name)
+}
+
+// rowShape reads a choice item as a row: a template whose every field is a string,
+// with its format, its sorted field names, and whether it carries a weight.
+func rowShape(raw any) (format string, fields []string, weighted, isRow bool) {
+	m, ok := raw.(map[string]any)
+	if !ok {
+		return "", nil, false, false
+	}
+	format, _ = m["format"].(string)
+	for k, v := range m {
+		if k == "weight" {
+			weighted = true
+			continue
+		}
+		if _, isString := v.(string); !isString || isOption(k) && k != "format" {
+			return "", nil, false, false
+		}
+		if k != "format" {
+			fields = append(fields, k)
+		}
+	}
+	sort.Strings(fields)
+	return format, fields, weighted, true
+}
+
 // compileAt compiles a node that is no choice's item. Only a choice's items carry a
 // weight, so one here would be inert whatever its type.
 func compileAt(v any, pos position) (node, error) {
@@ -140,7 +207,7 @@ func compileString(s string) (node, error) {
 // applies the fences that need the compiled reads.
 func (t *template) compileFormat() error {
 	c := compileOps(t.format, t.refs)
-	t.ops, t.grow, t.bound, t.held = c.ops, c.grow, c.bound, c.held
+	t.ops, t.grow, t.bound, t.held, t.heldLocal = c.ops, c.grow, c.bound, c.held, c.heldLocal
 	t.fixed = true
 	for _, o := range t.ops {
 		if o.kind != 'l' {
@@ -231,6 +298,9 @@ func checkNoRepeatedItem(items []any) error {
 }
 
 func compileTemplate(m map[string]any, pos position) (node, error) {
+	if _, isTable := m["rows"]; isTable {
+		return nil, fmt.Errorf("rows names a TSV beside a category's file, so only a category is a table; an inline template has no file beside it")
+	}
 	o, err := readOptions(m, pos)
 	if err != nil {
 		return nil, err
@@ -407,7 +477,11 @@ func checkName(name string) error {
 
 // checkPathNames rejects a dotted path with a segment no name may be.
 func checkPathNames(path string) error {
-	for _, seg := range strings.Split(path, ".") {
+	segs, err := splitPath(path)
+	if err != nil {
+		return err
+	}
+	for _, seg := range names(segs) {
 		if err := checkName(seg); err != nil {
 			return fmt.Errorf("path %w", err)
 		}

path.go

@@ -6,61 +6,244 @@ import (
 	"strings"
 )
 
+// splitPath splits a dotted path into segments, a selector — [key or name] after a
+// table's name — becoming a segment of its own, brackets kept. A dot inside a
+// selector is part of the key or name.
+func splitPath(path string) ([]string, error) {
+	segs := make([]string, 0, strings.Count(path, ".")+2*strings.Count(path, "[")+1)
+	start, open := 0, -1
+	for i := 0; i < len(path); i++ {
+		switch path[i] {
+		case '[':
+			if err := checkOpen(path, i, start, open); err != nil {
+				return nil, err
+			}
+			segs = append(segs, path[start:i])
+			start, open = i, i
+		case ']':
+			if err := checkClose(path, i, open); err != nil {
+				return nil, err
+			}
+			segs = append(segs, path[start:i+1])
+			start, open = i+2, -1
+			i++
+		case '.':
+			if open < 0 {
+				segs = append(segs, path[start:i])
+				start = i + 1
+			}
+		}
+	}
+	if open >= 0 {
+		return nil, fmt.Errorf(`%q opens a selector with "[" and never closes it with "]"`, path)
+	}
+	if start <= len(path) {
+		segs = append(segs, path[start:])
+	}
+	return segs, nil
+}
+
+// checkOpen refuses a "[" at i that opens no selector: one inside a selector, or one
+// following no name.
+func checkOpen(path string, i, start, open int) error {
+	switch {
+	case open >= 0:
+		return fmt.Errorf(`%q holds a "[" inside a selector, which no key or name may`, path)
+	case i == 0:
+		return fmt.Errorf(`%q starts with "["; a path starts with a name, and a selector follows a table's name`, path)
+	case i == start:
+		return fmt.Errorf(`%q: a selector follows its table's name; write %s`, path, path[:i-1]+path[i:])
+	}
+	return nil
+}
+
+// checkClose refuses a "]" at i that closes no selector, closes an empty one, or is
+// followed by anything but a dot or the end.
+func checkClose(path string, i, open int) error {
+	switch {
+	case open < 0:
+		return fmt.Errorf(`%q holds a "]" that closes no "["`, path)
+	case i == open+1:
+		return fmt.Errorf("%q holds an empty selector; a selector names a row by key or name", path)
+	case i+1 < len(path) && path[i+1] != '.':
+		return fmt.Errorf(`%q: a "]" ends its selector, so a dot or the end must follow it`, path)
+	}
+	return nil
+}
+
+// indexOutside is the first c in s outside a [selector], or -1.
+func indexOutside(s string, c byte) int {
+	depth := 0
+	for i := 0; i < len(s); i++ {
+		switch {
+		case s[i] == '[':
+			depth++
+		case s[i] == ']' && depth > 0:
+			depth--
+		case s[i] == c && depth == 0:
+			return i
+		}
+	}
+	return -1
+}
+
+// splitOutside splits s on c outside any [selector].
+func splitOutside(s string, c byte) []string {
+	var parts []string
+	for {
+		i := indexOutside(s, c)
+		if i < 0 {
+			return append(parts, s)
+		}
+		parts, s = append(parts, s[:i]), s[i+1:]
+	}
+}
+
+// isSelector reports whether a segment is a [key or name] rather than a name.
+func isSelector(seg string) bool { return strings.HasPrefix(seg, "[") }
+
+// selectorOf is the key or name a selector segment holds.
+func selectorOf(seg string) string { return seg[1 : len(seg)-1] }
+
+// names is the segments of a path that are names, its selectors left out.
+func names(segs []string) []string {
+	out := segs[:0:0]
+	for _, s := range segs {
+		if !isSelector(s) {
+			out = append(out, s)
+		}
+	}
+	return out
+}
+
 // pathWalk is what one walk of a dotted path does at each kind of level: choice
-// returns the variants to continue into (none stops the walk); level runs at each
+// returns the variants to continue into (none stops the walk), and where it is nil
-// template a segment descends into; leaf runs where the tail ends. A nil action
+// the walk draws one with the pins' session, or stops; level runs at each template
-// is skipped.
+// a segment descends into; leaf runs where the tail ends; pins is where the walk
+// pins the table rows it selects and draws, nil skipping tables. A nil action is
+// skipped. A render's walk sets only pins, so it allocates nothing.
 type pathWalk struct {
 	choice func(c *choice, rest []string) ([]node, error)
 	level  func(t *template, rest []string) error
 	leaf   func(n node) error
+	pins   *draws
 }
 
-// walkPath descends tail from n: a folder or template by its next segment, a
+// walkPath descends tail from n and returns the node it ends at: a folder or
-// choice by w.choice, which consumes no segment. A missing segment is an error,
+// template by its next segment, a choice by w.choice, which consumes no segment, a
-// so no walk reaches past what the data holds. A table-shaped dispatch, one case
+// table by walkTable. A missing segment is an error, so no walk reaches past what
-// per node kind, kept whole on purpose.
+// the data holds. A table-shaped dispatch, one case per node kind, kept whole on
-func walkPath(n node, tail []string, w pathWalk) error {
+// purpose.
+func walkPath(n node, tail []string, w pathWalk) (node, error) {
 	if len(tail) == 0 {
-		if w.leaf != nil {
+		return n, w.atLeaf(n)
-			return w.leaf(n)
+	}
-		}
+	if t, isTable := n.(*table); isTable {
-		return nil
+		return walkTable(t, tail, w, false)
+	}
+	if isSelector(tail[0]) {
+		return nil, fmt.Errorf("%s is not a table, so it has no row to select", selectorOf(tail[0]))
 	}
 	switch n := n.(type) {
 	case *folder:
 		child, ok := n.children[tail[0]]
 		if !ok {
-			return fmt.Errorf("no entry %q", tail[0])
+			return nil, fmt.Errorf("no entry %q", tail[0])
 		}
 		return walkPath(child, tail[1:], w)
 	case *template:
 		if w.level != nil {
 			if err := w.level(n, tail); err != nil {
-				return err
+				return nil, err
 			}
 		}
 		child, ok := n.field(tail[0])
 		if !ok {
-			return fmt.Errorf("no field %q", tail[0])
+			return nil, fmt.Errorf("no field %q", tail[0])
 		}
 		return walkPath(child, tail[1:], w)
 	case *choice:
-		if w.choice == nil {
+		return walkChoice(n, tail, w)
-			return nil
+	case *column:
-		}
+		return nil, fmt.Errorf("no field %q: %q is a column, and a cell holds no fields", tail[0], n.t.columns[n.i])
-		next, err := w.choice(n, tail)
-		if err != nil {
-			return err
-		}
-		for _, item := range next {
-			if err := walkPath(item, tail, w); err != nil {
-				return err
-			}
-		}
-		return nil
 	}
-	return fmt.Errorf("no field %q", tail[0])
+	return nil, fmt.Errorf("no field %q", tail[0])
+}
+
+func (w pathWalk) atLeaf(n node) error {
+	if w.leaf != nil {
+		return w.leaf(n)
+	}
+	return nil
+}
+
+// walkChoice continues a walk into the variants w.choice returns, or into one drawn
+// by the pins' session where the walk names no choice action.
+func walkChoice(c *choice, tail []string, w pathWalk) (node, error) {
+	if w.choice == nil {
+		if w.pins == nil || w.pins.s == nil {
+			return nil, nil
+		}
+		if err := carriedByAll(c, tail); err != nil {
+			return nil, err
+		}
+		return walkPath(pick(w.pins.s, c), tail, w)
+	}
+	next, err := w.choice(c, tail)
+	if err != nil {
+		return nil, err
+	}
+	var last node
+	for _, item := range next {
+		if last, err = walkPath(item, tail, w); err != nil {
+			return nil, err
+		}
+	}
+	return last, nil
+}
+
+// walkTable descends tail from a table: a selector first, then a column or a
+// linked table by name. The walk reads a row wherever a segment follows or the
+// table was reached from another; a table reached whole, with no selector, is left
+// to a render's own draw.
+func walkTable(t *table, tail []string, w pathWalk, descended bool) (node, error) {
+	sel := ""
+	if len(tail) > 0 && isSelector(tail[0]) {
+		sel, tail = selectorOf(tail[0]), tail[1:]
+		if len(tail) > 0 && isSelector(tail[0]) {
+			return nil, fmt.Errorf("%s[%s] is selected twice; one selector names its row", t.category, sel)
+		}
+	}
+	if w.pins != nil {
+		if err := readRow(w.pins, t, sel, descended || len(tail) > 0); err != nil {
+			return nil, err
+		}
+	}
+	if len(tail) == 0 {
+		if sel == "" && !descended {
+			return walkPath(t, nil, w)
+		}
+		return walkPath(t.whole, nil, w)
+	}
+	if i, ok := t.col[tail[0]]; ok {
+		return walkPath(t.fields[t.columns[i]], tail[1:], w)
+	}
+	if child := t.descendant(tail[0]); child != nil {
+		return walkTable(child, tail[1:], w, true)
+	}
+	return nil, fmt.Errorf("no column or linked table %q in %s", tail[0], t.category)
+}
+
+// readRow pins the row a path reads of t: the one its selector names, or, where the
+// walk draws, one drawn where the path reads into the table.
+func readRow(d *draws, t *table, sel string, draw bool) error {
+	if sel != "" {
+		return d.selectRow(t, sel)
+	}
+	if draw && d.s != nil {
+		d.rowOf(t)
+	}
+	return nil
 }
 
 // carriedByAll is the choice rule a path that must resolve on every call obeys:
@@ -88,12 +271,13 @@ func unreachableInChoice(c *choice, want string) error {
 }
 
 // checkPath proves a dotted tail resolves whichever way the draws go — a choice
-// must carry the rest of the path in the set every variant shares — and that no
+// must carry the rest of the path in the set every variant shares, a selector must
-// level a path reads carries a repeat or a drawGroup, which one draw of it could not
+// name one row inside the rows selected before it — and that no level a path reads
-// apply. So a path that validates here resolves on every render, and a typo is a
+// carries a repeat or a drawGroup, which one draw of it could not apply. So a path
-// New-time error.
+// that validates here resolves on every render, and a typo is a New-time error.
 func checkPath(n node, tail []string, level string) error {
-	return walkPath(n, tail, pathWalk{
+	var pins draws
+	_, err := walkPath(n, tail, pathWalk{
 		choice: func(c *choice, rest []string) ([]node, error) {
 			if err := carriedByAll(c, rest); err != nil {
 				return nil, err
@@ -110,5 +294,7 @@ func checkPath(n node, tail []string, level string) error {
 			}
 			return nil
 		},
+		pins: &pins,
 	})
+	return err
 }

path_test.go

@@ -13,14 +13,14 @@ func TestWalkPathStopsAtAMissingSegment(t *testing.T) {
 		level: func(tm *template, rest []string) error { seen = append(seen, "level:"+rest[0]); return nil },
 		leaf:  func(n node) error { seen = append(seen, "leaf"); return nil },
 	}
-	if err := walkPath(n, []string{"a", "b"}, walk); err != nil {
+	if _, err := walkPath(n, []string{"a", "b"}, walk); err != nil {
 		t.Fatalf("walkPath(a.b) = %v", err)
 	}
 	if want := []string{"level:a", "level:b", "leaf"}; !slices.Equal(seen, want) {
 		t.Errorf("walk visited %v, want %v", seen, want)
 	}
 	seen = nil
-	err := walkPath(n, []string{"a", "nope", "deeper"}, walk)
+	_, err := walkPath(n, []string{"a", "nope", "deeper"}, walk)
 	if err == nil || !strings.Contains(err.Error(), `no field "nope"`) {
 		t.Errorf("walkPath(a.nope.deeper) = %v, want the missing segment named", err)
 	}
@@ -32,7 +32,7 @@ func TestWalkPathStopsAtAMissingSegment(t *testing.T) {
 func TestWalkPathChoiceConsumesNoSegment(t *testing.T) {
 	n := compiled(t, `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2"}]`)
 	var leaves []node
-	err := walkPath(n, []string{"f"}, pathWalk{
+	_, err := walkPath(n, []string{"f"}, pathWalk{
 		choice: func(c *choice, rest []string) ([]node, error) {
 			if len(rest) != 1 || rest[0] != "f" {
 				t.Errorf("choice saw rest %v, want [f]", rest)
@@ -74,7 +74,7 @@ func TestDescendIntoStringErrors(t *testing.T) {
 // one call and failing on the next: every variant must carry the rest of the path.
 func TestPathThroughChoice(t *testing.T) {
 	dir := writeData(t, map[string]string{
-		"every":  `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2"}]`,
+		"every":  `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2","g":"x"}]`,
 		"notall": `[{"format":"{f}","f":"1"},"plain"]`,
 		"some":   `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2","extra":"x"}]`,
 	})

record.go

@@ -122,18 +122,48 @@ func literal(c Column, quote func(string) string, nullText string) string {
 func (f *Generator) FakeRecord(path string) (*Record, error) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
-	_, n, tail, err := resolveCategory(f.categories, strings.Split(path, "."))
+	segments, err := splitPath(path)
+	if err != nil {
+		return nil, fmt.Errorf("fejkdata: %w", err)
+	}
+	_, n, tail, err := resolveCategory(f.categories, segments)
 	if err != nil {
 		return nil, fmt.Errorf("fejkdata: %s: %w", path, err)
 	}
-	if len(tail) > 0 {
+	set := newDrawSet(f.rand)
+	sc := drawScope{set: &set}
+	if t, isTable := n.(*table); isTable {
+		if n, err = tableRecord(f.rand, t, tail, sc); err != nil {
+			return nil, fmt.Errorf("fejkdata: %s: %w", path, err)
+		}
+	} else if len(tail) > 0 {
 		return nil, fmt.Errorf("fejkdata: %s descends into %q, a field; only a category-level template is a record", path, tail[0])
 	}
 	shape := f.recordShapeOf(n)
 	if shape.err != nil {
 		return nil, fmt.Errorf("fejkdata: %s %w", path, shape.err)
 	}
-	return renderRecord(f.rand, shape.t, shape.columns), nil
+	return renderRecord(f.rand, shape.t, shape.columns, sc), nil
+}
+
+// tableRecord walks a path's tail from a table to the table whose row is the record,
+// pinning the rows it selects or draws.
+func tableRecord(s *session, t *table, tail []string, sc drawScope) (node, error) {
+	n, err := descend(s, t, tail, sc)
+	if err != nil {
+		return nil, err
+	}
+	switch n := n.(type) {
+	case *table:
+		sc.draws(s).rowOf(n)
+		return n, nil
+	case *column:
+		if n.i < 0 {
+			return n.t, nil
+		}
+		return nil, fmt.Errorf("descends into %q, a column; a record is a table's row", n.t.columns[n.i])
+	}
+	return n, nil
 }
 
 // recordShape is what recordOf settled about a node: the template to project, its
@@ -171,7 +201,8 @@ type RecordTemplate struct {
 func (t *RecordTemplate) Fake() *Record {
 	t.g.mu.Lock()
 	defer t.g.mu.Unlock()
-	return renderRecord(t.g.rand, t.t, t.columns)
+	set := newDrawSet(t.g.rand)
+	return renderRecord(t.g.rand, t.t, t.columns, drawScope{set: &set})
 }
 
 // NewRecordTemplate compiles an inline record — a JSON object with a format and
@@ -200,6 +231,9 @@ func (f *Generator) FakeRecordTemplate(input string) (*Record, error) {
 // recordOf is the fence both record entry points pass. The columns come back with
 // the template, fixed for every draw the caller goes on to make.
 func recordOf(n node) (*template, []Column, error) {
+	if tb, isTable := n.(*table); isTable {
+		n = tb.format
+	}
 	t, ok := n.(*template)
 	if !ok {
 		return nil, nil, errors.New("names a choice, not a template; a record is a template whose fields are its columns")
@@ -219,10 +253,13 @@ func recordOf(n node) (*template, []Column, error) {
 	return t, columns, nil
 }
 
-// renderRecord draws each column once, in the name order recordOf fixed, as one render.
+// renderRecord draws each column once, in the name order recordOf fixed, as one render
-func renderRecord(s *session, t *template, columns []Column) *Record {
+// over sc's draws; a table's columns read the row pinned there.
-	set := newDrawSet()
+func renderRecord(s *session, t *template, columns []Column, sc drawScope) *Record {
-	sc := drawScope{set: &set}.in(t)
+	sc = sc.in(t)
+	if t.table != nil {
+		sc.t, sc.row = t.table, sc.draws(s).mustRow(t.table)
+	}
 	r := &Record{columns: append([]Column(nil), columns...)}
 	for i := range r.columns {
 		column := renderLeaf(s, t.fields[r.columns[i].Name], sc)

reference.go

@@ -35,7 +35,11 @@ func refShape(name string) (sigil, rest string, err error) {
 	if strings.HasPrefix(rest, ".") {
 		return "", "", fmt.Errorf("a reference starts with / (the root), . (this folder) or .. (the folder above)")
 	}
-	for _, seg := range strings.Split(rest, ".") {
+	segs, err := splitPath(rest)
+	if err != nil {
+		return "", "", err
+	}
+	for _, seg := range segs {
 		if seg == "" {
 			return "", "", fmt.Errorf("path has an empty segment")
 		}
@@ -59,7 +63,8 @@ func refSegments(name string, folder []string) ([]string, error) {
 		}
 		base = folder[:len(folder)-1]
 	}
-	return append(append([]string{}, base...), strings.Split(rest, ".")...), nil
+	segs, _ := splitPath(rest) // refShape proved it splits
+	return append(append([]string{}, base...), segs...), nil
 }
 
 // linkRefs resolves every reference in the assembled tree. The head of the path —
@@ -199,6 +204,9 @@ func resolveCategory(root map[string]node, segments []string) (head []string, ta
 		if !ok {
 			break
 		}
+		if isSelector(segments[i]) {
+			return nil, nil, nil, fmt.Errorf("%s is a folder, not a table, so it has no row to select", strings.Join(segments[:i], "."))
+		}
 		child, ok := g.children[segments[i]]
 		if !ok {
 			return nil, nil, nil, fmt.Errorf("no entry %q", segments[i])

render.go

@@ -2,7 +2,6 @@ package fejkdata
 
 import (
 	"fmt"
-	"sort"
 	"strings"
 )
 
@@ -20,33 +19,33 @@ type rng interface {
 func (f *Generator) Fake(path string) (string, error) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
-	n, err := descend(f.rand, &folder{children: f.categories}, strings.Split(path, "."))
+	segments, err := splitPath(path)
+	if err != nil {
+		return "", fmt.Errorf("fejkdata: %w", err)
+	}
+	f.set = drawSet{unnamed: draws{s: f.rand}}
+	sc := drawScope{set: &f.set}
+	if f.root == nil {
+		f.root = &folder{children: f.categories}
+	}
+	n, err := descend(f.rand, f.root, segments, sc)
 	if err != nil {
 		return "", fmt.Errorf("fejkdata: %s: %w", path, err)
 	}
 	if _, ok := n.(*folder); ok {
 		return "", fmt.Errorf("fejkdata: %s names a folder, not a value", path)
 	}
-	return renderOnce(f.rand, n), nil
+	return render(f.rand, n, sc), nil
 }
 
-// descend walks named fields to the node a path names. It is the one render-side
+// descend walks named fields to the node a path names, pinning the table rows it
-// step that can fail, because the path comes from the caller and may name a field
+// selects or draws in sc. It is the one render-side step that can fail, because the
-// that does not exist. A choice consumes no segment, so the rest of the path must
+// path comes from the caller and may name a field or a row that does not exist. A
-// be one every variant carries before a variant is picked — a path that resolves
+// choice consumes no segment, so the rest of the path must be one every variant
-// at all resolves on every call.
+// carries before a variant is picked — a path that resolves at all resolves on
-func descend(s *session, root node, segments []string) (node, error) {
+// every call.
-	var found node
+func descend(s *session, root node, segments []string, sc drawScope) (node, error) {
-	err := walkPath(root, segments, pathWalk{
+	return walkPath(root, segments, pathWalk{pins: sc.draws(s)})
-		choice: func(c *choice, rest []string) ([]node, error) {
-			if err := carriedByAll(c, rest); err != nil {
-				return nil, err
-			}
-			return []node{pick(s, c)}, nil
-		},
-		leaf: func(n node) error { found = n; return nil },
-	})
-	return found, err
 }
 
 // render evaluates a compiled node to a string. compile validates every node up
@@ -58,6 +57,20 @@ func render(s *session, n node, sc drawScope) string {
 		return render(s, pick(s, n), sc)
 	case *null:
 		return ""
+	case *table:
+		sc.t, sc.row = n, n.draw(s)
+		return expand(s, n.format, sc)
+	case *column:
+		if sc.t != n.t {
+			sc.t, sc.row = n.t, sc.draws(s).mustRow(n.t)
+		}
+		if n.i < 0 {
+			return expand(s, n.t.format, sc)
+		}
+		if cell := n.t.cellNode(sc.row, n.i); cell != nil {
+			return render(s, cell, sc)
+		}
+		return n.t.cell(sc.row, n.i)
 	case *template:
 		sc = sc.in(n)
 		if n.repeat == 1 {
@@ -85,20 +98,20 @@ func render(s *session, n node, sc drawScope) string {
 //
 //go:noinline
 func expandAnew(s *session, t *template) string {
-	var set drawSet
+	set := drawSet{unnamed: draws{s: s}}
 	return expand(s, t, drawScope{set: &set})
 }
 
 // pick selects one item. Uniform choices are O(1); weighted choices are an
 // O(log n) search over precomputed cumulative weights. compile guarantees a
 // non-empty choice and a finite positive total, so the index is always in range.
-func pick(r rng, c *choice) node {
+// The session is concrete rather than the rng interface, which would make the
+// walk that draws through it leak its draw set to the heap.
+func pick(s *session, c *choice) node {
 	if c.cum == nil {
-		return c.items[r.IntN(len(c.items))]
+		return c.items[s.IntN(len(c.items))]
 	}
-	x := r.Float64() * c.cum[len(c.cum)-1]
+	return c.items[pickCum(s, c.cum)]
-	i := sort.Search(len(c.cum), func(i int) bool { return c.cum[i] > x })
-	return c.items[i]
 }
 
 // expand renders a template's compiled ops. compile validated every token, so this
@@ -110,7 +123,7 @@ func expand(s *session, t *template, sc drawScope) string {
 	// repeat iteration get their own, since each is its own expansion. A reference
 	// path reads the render's draws in sc instead.
 	var held *draws
-	if len(t.held) > 0 {
+	if t.heldLocal {
 		held = &draws{
 			variant: make(map[string]node, len(t.held)),
 			value:   make(map[string]draw, len(t.held)),

shape_test.go

@@ -43,6 +43,11 @@ func shippedShape(f *Generator) string {
 			}
 		case *choice:
 			facts[prefix] = reads(n)
+		case *table:
+			facts[prefix] = "\tformat " + strconv.Quote(n.format.format) + tableFacts(n) + reads(n)
+			for _, name := range n.columns {
+				facts[join(prefix, name)] = "\tstring"
+			}
 		case *template:
 			facts[prefix] = "\tformat " + strconv.Quote(n.format) + reads(n)
 			if _, columns, err := recordOf(n); err == nil {
@@ -66,6 +71,20 @@ func shippedShape(f *Generator) string {
 	return b.String()
 }
 
+// tableFacts names the columns a table's options read.
+func tableFacts(t *table) string {
+	var b strings.Builder
+	for _, o := range []struct {
+		name string
+		col  int
+	}{{"key", t.key}, {"name", t.name}, {"weight", t.weight}, {"parent", t.parent}} {
+		if o.col >= 0 {
+			b.WriteString("\t" + o.name + " " + t.columns[o.col])
+		}
+	}
+	return b.String()
+}
+
 // reads names the categories any template under n references, sorted.
 func reads(n node) string {
 	set := map[string]bool{}
@@ -76,6 +95,11 @@ func reads(n node) string {
 			for _, it := range n.items {
 				collect(it)
 			}
+		case *table:
+			collect(n.format)
+			for _, cell := range n.tokens {
+				collect(cell)
+			}
 		case *template:
 			for _, b := range n.refs {
 				set[strings.TrimPrefix(b.key, "/")] = true

struct.go

@@ -349,7 +349,8 @@ func (p *valueProof) checkField(label string, ft reflect.Type, column node) erro
 // fill draws the record into v's tagged fields, then each nested struct as a record of its own.
 func (s *structShape) fill(sess *session, v reflect.Value) {
 	if s.record != nil {
-		for i, c := range renderRecord(sess, s.record, s.columns).columns {
+		set := newDrawSet(sess)
+		for i, c := range renderRecord(sess, s.record, s.columns, drawScope{set: &set}).columns {
 			setColumn(fieldAt(v, s.fields[i]), c)
 		}
 	}

table.go

@@ -0,0 +1,486 @@
+package fejkdata
+
+import (
+	"fmt"
+	"math"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+)
+
+// table is a category whose rows come from a TSV beside it: the header names the
+// columns, each row is one draw, and the format renders the drawn row. A column is a
+// cell of the row a render pinned; a cell carrying tokens compiles to a string node.
+type table struct {
+	category string
+	file     string
+	format   *template // fields are the column nodes
+	columns  []string
+	col      map[string]int
+	fields   map[string]node // column nodes, the format's fields
+	whole    *column         // the pinned row rendered by the format
+	cells    []string        // rows × columns, flat
+	tokens   map[int]*template
+	key      int // column index, or -1
+	name     int
+	weight   int
+	parent   int
+	cum      []float64 // cumulative weights, nil when uniform
+	byKey    map[string]int
+	parentT  *table
+	children map[string]*table
+	once     sync.Once
+	index    tableIndex
+}
+
+// tableIndex is what selection and linked draws look up, built on the first draw
+// that needs it.
+type tableIndex struct {
+	byName   map[string][]int
+	children map[string][]int     // rows by parent key
+	childCum map[string][]float64 // cumulative weights per parent key, nil when uniform
+}
+
+func (*table) isNode() {}
+
+// column is one column of a table, rendered as the cell of the row the render
+// pinned; i < 0 is the whole row rendered by the table's format.
+type column struct {
+	t *table
+	i int
+}
+
+func (*column) isNode() {}
+
+func (t *table) rows() int { return len(t.cells) / len(t.columns) }
+
+func (t *table) cell(row, col int) string { return t.cells[row*len(t.columns)+col] }
+
+// cellNode is what a cell renders: its compiled template where it carries tokens,
+// else nil for its text.
+func (t *table) cellNode(row, col int) *template { return t.tokens[row*len(t.columns)+col] }
+
+// tableOptions are the keys a table object takes; every other key is refused.
+var tableOptions = []string{"format", "key", "name", "parent", "rows", "weight"}
+
+func isTableOption(name string) bool {
+	for _, o := range tableOptions {
+		if o == name {
+			return true
+		}
+	}
+	return false
+}
+
+// inSelector is what a key or name cell may not contain: the path grammar reserves
+// brackets and braces, and a quote or a bar would make the selector no path and no
+// token arm.
+const inSelector = `[]{}"|`
+
+// compileTable compiles a category object naming a rows file into a table.
+func compileTable(m map[string]any, category string, files *categoryFiles) (*table, error) {
+	o, err := readTableOptions(m)
+	if err != nil {
+		return nil, err
+	}
+	data, err := files.rows(o.rows)
+	if err != nil {
+		return nil, err
+	}
+	t := &table{category: category, file: o.rows, key: -1, name: -1, weight: -1, parent: -1}
+	if err := t.parseRows(data); err != nil {
+		return nil, fmt.Errorf("%s: %w", o.rows, err)
+	}
+	if err := t.bindOptions(o); err != nil {
+		return nil, err
+	}
+	if err := t.checkCells(); err != nil {
+		return nil, fmt.Errorf("%s: %w", o.rows, err)
+	}
+	if err := t.compileFormat(o.format); err != nil {
+		return nil, err
+	}
+	return t, nil
+}
+
+type tableOptionValues struct {
+	format, key, name, parent, rows, weight string
+}
+
+func readTableOptions(m map[string]any) (tableOptionValues, error) {
+	var o tableOptionValues
+	for k := range m {
+		if !isTableOption(k) {
+			return o, fmt.Errorf("a table takes %s; %q is none of them", strings.Join(tableOptions, ", "), k)
+		}
+	}
+	for k, into := range map[string]*string{"format": &o.format, "key": &o.key, "name": &o.name, "parent": &o.parent, "rows": &o.rows, "weight": &o.weight} {
+		v, ok := m[k]
+		if !ok {
+			continue
+		}
+		s, isString := v.(string)
+		if !isString {
+			return o, fmt.Errorf("%s must be a string, got %T", k, v)
+		}
+		if k != "format" && s == "" {
+			return o, fmt.Errorf("%s names a column, so it cannot be empty; drop it", k)
+		}
+		*into = s
+	}
+	if _, ok := m["format"]; !ok {
+		return o, fmt.Errorf("template object missing string \"format\"")
+	}
+	if !strings.HasSuffix(o.rows, ".tsv") || strings.Contains(o.rows, "/") {
+		return o, fmt.Errorf("rows names a .tsv file beside the category, not %q", o.rows)
+	}
+	if o.key != "" && o.key == o.name {
+		return o, fmt.Errorf("name names the key column %q, which a selector already reads; drop it", o.name)
+	}
+	return o, nil
+}
+
+// parseRows reads the TSV: the header line names the columns, each following line
+// is a row of as many cells. Cells are substrings of data, so the file is held once.
+func (t *table) parseRows(data string) error {
+	data = strings.TrimSuffix(data, "\n")
+	header, rest, _ := strings.Cut(data, "\n")
+	if data == "" {
+		return fmt.Errorf("has no header line naming its columns")
+	}
+	t.columns = strings.Split(strings.TrimSuffix(header, "\r"), "\t")
+	t.col = make(map[string]int, len(t.columns))
+	t.fields = make(map[string]node, len(t.columns))
+	for i, name := range t.columns {
+		if err := checkName(name); err != nil {
+			return fmt.Errorf("column %w", err)
+		}
+		if _, dup := t.col[name]; dup {
+			return fmt.Errorf("column %q is named twice", name)
+		}
+		t.col[name] = i
+		t.fields[name] = &column{t, i}
+	}
+	if header == data {
+		return fmt.Errorf("has no rows below its header; a table is at least two rows")
+	}
+	m := len(t.columns)
+	t.cells = make([]string, 0, m*(strings.Count(rest, "\n")+1))
+	for line, n := 2, 0; rest != ""; line++ {
+		var row string
+		row, rest, _ = strings.Cut(rest, "\n")
+		row = strings.TrimSuffix(row, "\r")
+		for n = 0; n < m; n++ {
+			cell, more, tab := strings.Cut(row, "\t")
+			if !tab && n < m-1 || tab && n == m-1 {
+				return fmt.Errorf("line %d has %s cells than the %d columns", line, map[bool]string{true: "more", false: "fewer"}[tab], m)
+			}
+			t.cells = append(t.cells, cell)
+			row = more
+		}
+	}
+	if t.rows() == 1 {
+		return fmt.Errorf("has one row, which is a template; write it as one")
+	}
+	return nil
+}
+
+// bindOptions resolves each option to its column and proves what it claims of the
+// cells: a key is unique, a weight a positive number.
+func (t *table) bindOptions(o tableOptionValues) error {
+	for _, opt := range []struct {
+		name, value string
+		into        *int
+	}{{"key", o.key, &t.key}, {"name", o.name, &t.name}, {"parent", o.parent, &t.parent}, {"weight", o.weight, &t.weight}} {
+		if opt.value == "" {
+			continue
+		}
+		i, ok := t.col[opt.value]
+		if !ok {
+			return fmt.Errorf("%s names no column %q of %s; the columns are %v", opt.name, opt.value, o.rows, t.columns)
+		}
+		*opt.into = i
+	}
+	if err := t.indexKeys(); err != nil {
+		return err
+	}
+	return t.sumWeights()
+}
+
+// indexKeys proves every key names one row, and keeps the index a link is proved by.
+func (t *table) indexKeys() error {
+	if t.key < 0 {
+		return nil
+	}
+	t.byKey = make(map[string]int, t.rows())
+	for r := 0; r < t.rows(); r++ {
+		k := t.cell(r, t.key)
+		if k == "" {
+			return fmt.Errorf("%s line %d: the key is empty", t.file, r+2)
+		}
+		if first, dup := t.byKey[k]; dup {
+			return fmt.Errorf("%s line %d: key %q repeats line %d; a key selects one row", t.file, r+2, k, first+2)
+		}
+		t.byKey[k] = r
+	}
+	return nil
+}
+
+// sumWeights proves every weight is a positive number and builds the cumulative table.
+func (t *table) sumWeights() error {
+	if t.weight < 0 {
+		return nil
+	}
+	t.cum = make([]float64, t.rows())
+	total := 0.0
+	for r := range t.cum {
+		w, err := strconv.ParseFloat(t.cell(r, t.weight), 64)
+		if err != nil || math.IsInf(w, 0) || math.IsNaN(w) || w <= 0 {
+			return fmt.Errorf("%s line %d: weight %q is not a positive number", t.file, r+2, t.cell(r, t.weight))
+		}
+		total += w
+		t.cum[r] = total
+	}
+	if math.IsInf(total, 0) {
+		return fmt.Errorf("%s: the weights sum past the largest number", t.file)
+	}
+	return nil
+}
+
+// checkCells compiles every cell carrying a token, and fences what a selector reads.
+func (t *table) checkCells() error {
+	for i, cell := range t.cells {
+		row, col := i/len(t.columns), i%len(t.columns)
+		if (col == t.key || col == t.name) && strings.ContainsAny(cell, inSelector) {
+			return fmt.Errorf("line %d: %s %q contains %q, which a selector cannot spell", row+2, t.columns[col], cell, cell[strings.IndexAny(cell, inSelector):][:1])
+		}
+		if !strings.ContainsAny(cell, "{}") {
+			continue
+		}
+		n, err := compileString(cell)
+		if err != nil {
+			return fmt.Errorf("line %d, %s: %w", row+2, t.columns[col], err)
+		}
+		if t.tokens == nil {
+			t.tokens = map[int]*template{}
+		}
+		t.tokens[i] = n.(*template)
+	}
+	return nil
+}
+
+// compileFormat compiles the format over the columns as its fields.
+func (t *table) compileFormat(format string) error {
+	for _, name := range fieldTokens(format) {
+		a := splitArm(name, nil)
+		if _, ok := t.col[a.key]; !ok && !isRef(a.key) && a.key != "" {
+			return fmt.Errorf("format names no column %q of %s; the columns are %v", a.key, t.file, t.columns)
+		}
+	}
+	if err := checkTokens(format, t.fields); err != nil {
+		return err
+	}
+	t.format = &template{format: format, fields: t.fields, repeat: 1, record: true, table: t}
+	t.whole = &column{t, -1}
+	return t.format.compileFormat()
+}
+
+// linkTables binds every table's parent to the table beside it, and proves the
+// links: a parent has a key, every link cell is one, every parent row is linked
+// to, no chain of parents closes, and no child is named like a parent's column.
+func linkTables(root map[string]node) error {
+	var walk func(dir string, children map[string]node) error
+	walk = func(dir string, children map[string]node) error {
+		for _, name := range sortedNames(children) {
+			path := join(dir, name)
+			switch n := children[name].(type) {
+			case *folder:
+				if err := walk(path, n.children); err != nil {
+					return err
+				}
+			case *table:
+				n.category = name
+				if n.parent < 0 {
+					continue
+				}
+				if err := n.linkParent(path, children); err != nil {
+					return fmt.Errorf("%s: %w", path, err)
+				}
+			}
+		}
+		return nil
+	}
+	return walk("", root)
+}
+
+func (t *table) linkParent(path string, siblings map[string]node) error {
+	name := t.columns[t.parent]
+	p, ok := siblings[name].(*table)
+	switch {
+	case siblings[name] == nil:
+		return fmt.Errorf("parent %q names no table beside it", name)
+	case !ok:
+		return fmt.Errorf("parent %q is not a table; a link column reads a table's key", name)
+	case p.key < 0:
+		return fmt.Errorf("parent %q has no key column to link to", name)
+	}
+	for q, seen := p, map[*table]bool{t: true}; q != nil; q, _ = siblings[q.columns[q.parent]].(*table) {
+		if seen[q] {
+			return fmt.Errorf("parent cycle: %s reaches itself through its parents", q.category)
+		}
+		seen[q] = true
+		if q.parent < 0 {
+			break
+		}
+	}
+	if _, clash := p.col[t.category]; clash {
+		return fmt.Errorf("%q is named like a column of its parent %q, so %s.%s could read either; rename one", t.category, name, name, t.category)
+	}
+	linked := make(map[string]bool, p.rows())
+	for r := 0; r < t.rows(); r++ {
+		k := t.cell(r, t.parent)
+		if _, ok := p.byKey[k]; !ok {
+			return fmt.Errorf("%s line %d: %s %q is no key of %s", t.file, r+2, name, k, p.file)
+		}
+		linked[k] = true
+	}
+	for r := 0; r < p.rows(); r++ {
+		if k := p.cell(r, p.key); !linked[k] {
+			return fmt.Errorf("%s links no row to %s %q; every %s row needs one, or drop line %d of %s", t.file, name, k, name, r+2, p.file)
+		}
+	}
+	t.parentT = p
+	if p.children == nil {
+		p.children = map[string]*table{}
+	}
+	p.children[t.category] = t
+	return nil
+}
+
+func (t *table) indexed() *tableIndex {
+	t.once.Do(func() {
+		if t.name >= 0 {
+			t.index.byName = make(map[string][]int, t.rows())
+			for r := 0; r < t.rows(); r++ {
+				n := t.cell(r, t.name)
+				t.index.byName[n] = append(t.index.byName[n], r)
+			}
+		}
+		if t.parent >= 0 {
+			t.index.children = map[string][]int{}
+			for r := 0; r < t.rows(); r++ {
+				k := t.cell(r, t.parent)
+				t.index.children[k] = append(t.index.children[k], r)
+			}
+			if t.cum != nil {
+				t.index.childCum = make(map[string][]float64, len(t.index.children))
+				for k, rows := range t.index.children {
+					cum, total := make([]float64, len(rows)), 0.0
+					for i, r := range rows {
+						total += t.cum[r] - t.cumBefore(r)
+						cum[i] = total
+					}
+					t.index.childCum[k] = cum
+				}
+			}
+		}
+	})
+	return &t.index
+}
+
+func (t *table) cumBefore(r int) float64 {
+	if r == 0 {
+		return 0
+	}
+	return t.cum[r-1]
+}
+
+// draw picks a row over the whole table. The session is concrete rather than the rng
+// interface so that a walk holding the draws allocates nothing.
+func (t *table) draw(s *session) int {
+	if t.cum == nil {
+		return s.IntN(t.rows())
+	}
+	return pickCum(s, t.cum)
+}
+
+func pickCum(s *session, cum []float64) int {
+	x := s.Float64() * cum[len(cum)-1]
+	return sort.Search(len(cum), func(i int) bool { return cum[i] > x })
+}
+
+// drawUnder picks a row among those linked to parent row pr.
+func (t *table) drawUnder(s *session, pr int) int {
+	ix := t.indexed()
+	k := t.parentT.cell(pr, t.parentT.key)
+	rows := ix.children[k]
+	if ix.childCum == nil {
+		return rows[s.IntN(len(rows))]
+	}
+	return rows[pickCum(s, ix.childCum[k])]
+}
+
+// descendant is the table named name among those linked to t, at any depth.
+func (t *table) descendant(name string) *table {
+	if c, ok := t.children[name]; ok {
+		return c
+	}
+	for _, c := range t.children {
+		if d := c.descendant(name); d != nil {
+			return d
+		}
+	}
+	return nil
+}
+
+// parentRow is the row of t's parent that row r links to.
+func (t *table) parentRow(r int) int { return t.parentT.byKey[t.cell(r, t.parent)] }
+
+// under reports whether row r of t sits inside row pr of ancestor a.
+func (t *table) under(r int, a *table, pr int) bool {
+	for c, row := t, r; c.parentT != nil; c, row = c.parentT, c.parentRow(row) {
+		if c.parentT == a {
+			return c.parentRow(row) == pr
+		}
+	}
+	return false
+}
+
+// find is the row a selector names: by key first, then by name, where a name
+// naming several rows resolves inside the ancestors pinned in d.
+func (t *table) find(sel string, d *draws) (int, error) {
+	if t.key < 0 && t.name < 0 {
+		return 0, fmt.Errorf("%s has no key or name column to select a row by", t.category)
+	}
+	if r, ok := t.byKey[sel]; ok {
+		return r, nil
+	}
+	rows := t.indexed().byName[sel]
+	if len(rows) > 1 {
+		rows = d.inside(t, rows)
+	}
+	switch len(rows) {
+	case 1:
+		return rows[0], nil
+	case 0:
+		return 0, fmt.Errorf("no row of %s has key or name %q", t.category, sel)
+	}
+	keys := make([]string, len(rows))
+	for i, r := range rows {
+		keys[i] = t.cell(r, t.key)
+	}
+	inside := ""
+	if t.parentT != nil {
+		inside = fmt.Sprintf(", or select it inside its %s", t.parentT.category)
+	}
+	return 0, fmt.Errorf("%q names %d rows of %s; select one by key, one of %v%s", sel, len(rows), t.category, keys, inside)
+}
+
+// selectorSpelling is how a path writes a selected row, for messages.
+func (t *table) selectorSpelling(r int) string {
+	if t.key >= 0 {
+		return t.category + "[" + t.cell(r, t.key) + "]"
+	}
+	return t.category + "[" + t.cell(r, t.name) + "]"
+}

template.go

@@ -81,19 +81,20 @@ type builtin struct {
 // for a plain field or alternation body. A '(' without a trailing ')' yields
 // ok=false; checkFunc reports it as malformed at compile time.
 func funcCall(body string) (name string, args []string, ok bool) {
-	lp := strings.IndexByte(body, '(')
+	lp := indexOutside(body, '(')
 	if lp < 0 || !strings.HasSuffix(body, ")") {
 		return "", nil, false
 	}
 	return body[:lp], splitArgs(body[lp+1 : len(body)-1]), true
 }
 
-// splitArgs parses a function arg list: comma-separated, trimmed; empty -> none.
+// splitArgs parses a function arg list: comma-separated outside a selector,
+// trimmed; empty -> none.
 func splitArgs(s string) []string {
 	if strings.TrimSpace(s) == "" {
 		return nil
 	}
-	args := strings.Split(s, ",")
+	args := splitOutside(s, ',')
 	for i := range args {
 		args[i] = strings.TrimSpace(args[i])
 	}
@@ -132,10 +133,10 @@ func checkTokens(format string, fields map[string]node) error {
 		if t.kind != 'b' {
 			return nil
 		}
-		if strings.IndexByte(t.body, '(') >= 0 { // a function token, not a field
+		if indexOutside(t.body, '(') >= 0 { // a function token, not a field
 			return checkFunc(t.body, fields)
 		}
-		names := strings.Split(t.body, "|")
+		names := splitOutside(t.body, '|')
 		for _, name := range names {
 			if isRef(name) {
 				if _, _, err := refShape(name); err != nil {
@@ -187,12 +188,7 @@ func checkArm(name string, fields map[string]node, wholeToken bool) error {
 // hintableRef reports whether {/name} is a reference the grammar accepts, so the
 // hint never names a spelling that fails too.
 func hintableRef(name string) bool {
-	for _, seg := range strings.Split(name, ".") {
+	return checkPathNames(name) == nil
-		if checkName(seg) != nil {
-			return false
-		}
-	}
-	return true
 }
 
 // tokenOperands lists the fields one {token} body reads as operands, empty for a
@@ -244,8 +240,8 @@ func checkNoRepeatedArm(body string, names []string) error {
 func fieldTokens(format string) []string {
 	var names []string
 	_ = eachToken(format, func(t ftoken) error {
-		if t.kind == 'b' && strings.IndexByte(t.body, '(') < 0 {
+		if t.kind == 'b' && indexOutside(t.body, '(') < 0 {
-			names = append(names, strings.Split(t.body, "|")...)
+			names = append(names, splitOutside(t.body, '|')...)
 		}
 		return nil
 	})
@@ -279,11 +275,11 @@ func splitArm(name string, refs map[string]refBinding) arm {
 		}
 		return pathArm(name, b.key, b.tail)
 	}
-	head, tail, dotted := strings.Cut(name, ".")
+	segs, err := splitPath(name)
-	if !dotted {
+	if err != nil || len(segs) == 1 {
 		return arm{name: name, key: name, path: name}
 	}
-	return pathArm(name, head, strings.Split(tail, "."))
+	return pathArm(name, segs[0], segs[1:])
 }
 
 func pathArm(name, key string, segs []string) arm {
@@ -296,7 +292,7 @@ func pathArm(name, key string, segs []string) arm {
 
 // splitArms splits a token body's '|' alternatives.
 func splitArms(body string, refs map[string]refBinding) []arm {
-	parts := strings.Split(body, "|")
+	parts := splitOutside(body, '|')
 	arms := make([]arm, len(parts))
 	for i, p := range parts {
 		arms[i] = splitArm(p, refs)
@@ -347,11 +343,12 @@ type op struct {
 // it, for error messages. The maps are nil when the format holds nothing, so data
 // that holds nothing carries no render-time cost.
 type formatOps struct {
-	ops    []op
+	ops       []op
-	grow   int
+	grow      int
-	bound  map[string]string
+	bound     map[string]string
-	held   map[string]bool
+	held      map[string]bool
-	holder map[string]string
+	holder    map[string]string
+	heldLocal bool // some held name is kept by the expansion itself rather than the render's draws
 }
 
 func (c *formatOps) hold(a arm, label string) {
@@ -360,6 +357,9 @@ func (c *formatOps) hold(a arm, label string) {
 		c.holder = map[string]string{}
 	}
 	c.held[a.key] = true
+	if !isRef(a.key) || len(a.tail) == 0 {
+		c.heldLocal = true
+	}
 	if _, named := c.holder[a.key]; !named {
 		c.holder[a.key] = label
 	}

value.go

@@ -85,6 +85,8 @@ func (p *valueProof) of(n node) proven {
 		v = p.unite(n.items)
 	case *template:
 		v = p.template(n)
+	case *column:
+		v = p.cells(n)
 	default:
 		v = unproven(`it reads a null, which renders "" outside its own column`)
 	}
@@ -92,6 +94,28 @@ func (p *valueProof) of(n node) proven {
 	return v
 }
 
+// cells proves a table column over every cell it may render.
+func (p *valueProof) cells(c *column) proven {
+	if c.i < 0 {
+		return unproven(fmt.Sprintf("%q renders a row of %s, which is composed text", c.t.format.format, c.t.category))
+	}
+	var v proven
+	for r := 0; r < c.t.rows(); r++ {
+		var w proven
+		if cell := c.t.cellNode(r, c.i); cell != nil {
+			w = p.of(cell)
+		} else {
+			w = literalValue(c.t.cell(r, c.i))
+		}
+		if r == 0 {
+			v = w
+		} else {
+			v = v.or(w)
+		}
+	}
+	return v
+}
+
 func (p *valueProof) unite(nodes []node) proven {
 	v := p.of(nodes[0])
 	for _, n := range nodes[1:] {