Make great documentation goal 1.2, and plan the README's reorganisation and the nested-template gaps #33
@@ -47,7 +47,7 @@ In [docs/decisions.md](docs/decisions.md):
|
||||
- `FakeTemplate` and `NewTemplate` both stay
|
||||
- The shipped data is embedded, not discovered
|
||||
- The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0
|
||||
- With only the shipped set, a category loads on the first call reaching it; beside a `--data-path`, every category loads in `New`
|
||||
- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces
|
||||
- Every `{…}` draws afresh, and only a name keeps a pick
|
||||
- Reference sigils follow the filesystem
|
||||
- A change to what exists is a major; a minor only adds
|
||||
|
||||
@@ -125,3 +125,7 @@ replacement, and each removed path, column or flag.
|
||||
its own quotes, a row of a table with no key is named as the path that selects it,
|
||||
`sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record
|
||||
that gives it one.
|
||||
- A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
|
||||
category on the first call reaching it, as the shipped set does. A source without one
|
||||
loads in `New`, with the shipped categories it reads and every shipped category
|
||||
reaching one it replaces; the rest of the shipped set still loads on first reach.
|
||||
|
||||
@@ -172,6 +172,12 @@ which binds a [name](#names), nor be `-`, which a struct tag reserves. A dot-pre
|
||||
entry is never a category or a folder, so a data directory can also be a checkout, though
|
||||
a category may name a hidden rows file beside it.
|
||||
|
||||
A source whose root holds `.fejkdata.json` with an `index` loads each category on the
|
||||
first call reaching it, and its author proves each loads. The shipped set's
|
||||
[`data/.fejkdata.json`](data/.fejkdata.json) shows the shape: each category's dot path,
|
||||
its `parent` table, the `paths` `List` advertises below it, and the categories it
|
||||
`reads`. `//go:embed` of a directory leaves the manifest out; name it in the pattern too.
|
||||
|
||||
Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
|
||||
`ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`,
|
||||
`username`, `version` and `word`, formatted per locale; `sv_SE` adds `personnummer`
|
||||
@@ -332,8 +338,9 @@ ok, err := fejkdata.IsTemplate(arg) // an inline template by its shape, el
|
||||
`New` refuses data that breaks the grammar, could mean two things or cannot render a valid
|
||||
value, and `NewTemplate` refuses such a template. On a loaded generator:
|
||||
|
||||
- `Fake` fails only for a path that names nothing, could name two things, or reads one
|
||||
draw of a level carrying a `repeat`, with the same error every call.
|
||||
- `Fake` fails only for a path that names nothing, could name two things, reads one
|
||||
draw of a level carrying a `repeat`, or reaches a category of an indexed source that
|
||||
fails to load, with the same error every call.
|
||||
- `FakeStruct` fails only for a non-struct argument or a type its tags do not describe,
|
||||
with the same error every call.
|
||||
- `Template.Fake` cannot fail.
|
||||
@@ -833,10 +840,11 @@ advertises no such path; read that level whole.
|
||||
|
||||
The target is about a microsecond per value (goal 13).
|
||||
|
||||
With only the shipped set, `New` loads nothing: a category loads on the first `Fake`,
|
||||
`FakeRecord`, `FakeStruct` or template that reads it, with every category it reads and
|
||||
its whole table family. Beside `--data-path`, `WithDataPath` or `WithDataFS`, every
|
||||
category loads in `New`. Each file is parsed, validated and weight-indexed once, bar a
|
||||
A source with an [index](#data), as the shipped set has, loads nothing in `New`: a
|
||||
category loads on the first `Fake`, `FakeRecord`, `FakeStruct` or template that reads
|
||||
it, with every category it reads and its whole table family. A source without one loads
|
||||
in `New`, with the indexed categories it reads and every indexed category reaching one
|
||||
it replaces, so each mistake it causes is `New`'s error. Each file is parsed, validated and weight-indexed once, bar a
|
||||
table's name and parent columns, which are mapped on the first `Fake` that selects by
|
||||
name or descends through it. A `Fake` call then costs about what its output
|
||||
costs: an unweighted pick is O(1) whatever the list's length, a weighted one
|
||||
@@ -985,8 +993,8 @@ Where the project is heading; the sections before Audience document what ships t
|
||||
```
|
||||
doc.go the package doc, and the vocabulary the package is written in
|
||||
fejkdata.go Generator, New, options, List
|
||||
shipped.go the embedded data set, and loading a shipped category on the first call reaching it
|
||||
shippedindex.go the index of shipped categories, generated from data/
|
||||
shipped.go the embedded data set
|
||||
index.go a source's index: its categories left unloaded, and loading one on the first call reaching it
|
||||
node.go the node model and JSON -> node compilation
|
||||
table.go tables: their options, format and cells, the link to the parent beside them, and how a path passes and draws a table
|
||||
name.go names: {x as n} bindings, the scopes they live in, the reads they answer, and each binding's target
|
||||
@@ -1044,7 +1052,7 @@ Commands that rewrite source keep your file ownership when run with `--user`:
|
||||
|
||||
```sh
|
||||
docker compose run --rm --user "$(id -u):$(id -g)" fmt # gofmt -w .
|
||||
docker compose run --rm --user "$(id -u):$(id -g)" generate # go generate: shippedindex.go from data/
|
||||
docker compose run --rm --user "$(id -u):$(id -g)" generate # go generate: data/.fejkdata.json from data/
|
||||
docker compose run --rm --user "$(id -u):$(id -g)" tidy # go mod tidy
|
||||
```
|
||||
|
||||
@@ -1076,7 +1084,7 @@ what the score asks of a pull request. Goal 5 is checked against simulated templ
|
||||
writers from the Audience.
|
||||
|
||||
A change to the shipped data runs `generate` above first and commits
|
||||
[`shippedindex.go`](shippedindex.go) with the change, then re-pins
|
||||
[`data/.fejkdata.json`](data/.fejkdata.json) with the change, then re-pins
|
||||
[`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) in its own commit:
|
||||
|
||||
```sh
|
||||
|
||||
@@ -6,19 +6,67 @@ import (
|
||||
"github.com/larvit/fejkdata/internal/datafiles"
|
||||
)
|
||||
|
||||
func loadData(sources []datafiles.Source) (map[string]node, error) {
|
||||
root := map[string]node{}
|
||||
// loadSources merges sources in order, the last winning a clash. A source whose manifest
|
||||
// carries an index leaves its categories unloaded; every other source's categories load
|
||||
// here, with what they reach and every unloaded category leading to one of them or to one a
|
||||
// later source replaced.
|
||||
func loadSources(sources []datafiles.Source) (folder, error) {
|
||||
root := folder{children: map[string]node{}}
|
||||
targets := map[string]bool{}
|
||||
for _, src := range sources {
|
||||
g := &folder{children: map[string]node{}}
|
||||
if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil {
|
||||
return nil, err
|
||||
before := categoriesUnder(&root)
|
||||
provides, err := place(&root, src)
|
||||
if err != nil {
|
||||
return folder{}, err
|
||||
}
|
||||
if len(before) == 0 {
|
||||
continue
|
||||
}
|
||||
after := categoriesUnder(&root)
|
||||
for p := range before {
|
||||
if provides(p) || !after[p] {
|
||||
targets[p] = true
|
||||
}
|
||||
}
|
||||
mergeChildren(root, g.children)
|
||||
}
|
||||
if err := categoryPipeline(categorySites(&folder{children: root}), root).run(); err != nil {
|
||||
sites := categorySites(&root)
|
||||
for _, s := range sites {
|
||||
targets[s.path] = true
|
||||
}
|
||||
return root, loadReached(&root, sites, reaching(&root, targets))
|
||||
}
|
||||
|
||||
// place adds the categories of src to root, and returns whether src provides a path.
|
||||
func place(root *folder, src datafiles.Source) (func(string) bool, error) {
|
||||
m, err := src.Manifest()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return root, nil
|
||||
if m.Index != nil {
|
||||
placeIndex(root, src, m.Index)
|
||||
return func(p string) bool { _, ok := m.Index[p]; return ok }, nil
|
||||
}
|
||||
g := &folder{children: map[string]node{}}
|
||||
if err := src.Walk(compileInto(func(dir []string) *folder { return madeFolder(g, dir) })); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
mergeFolder(root, g)
|
||||
provided := categoriesUnder(g)
|
||||
return func(p string) bool { return provided[p] }, nil
|
||||
}
|
||||
|
||||
// categoriesUnder is the path of every category under root, loaded or not.
|
||||
func categoriesUnder(root *folder) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
for _, s := range categorySites(root) {
|
||||
out[s.path] = true
|
||||
}
|
||||
unloaded := map[string]categoryAt{}
|
||||
unloadedUnder(root, nil, unloaded)
|
||||
for p := range unloaded {
|
||||
out[p] = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func compileInto(place func(dir []string) *folder) func(datafiles.Category) error {
|
||||
@@ -40,21 +88,23 @@ func madeFolder(root *folder, dir []string) *folder {
|
||||
if !isFolder {
|
||||
sub = &folder{children: map[string]node{}}
|
||||
g.children[seg] = sub
|
||||
delete(g.unloaded, seg)
|
||||
}
|
||||
g = sub
|
||||
}
|
||||
return g
|
||||
}
|
||||
|
||||
func mergeChildren(dst, src map[string]node) {
|
||||
for k, v := range src {
|
||||
if dg, ok := dst[k].(*folder); ok {
|
||||
func mergeFolder(dst, src *folder) {
|
||||
for k, v := range src.children {
|
||||
if dg, ok := dst.children[k].(*folder); ok {
|
||||
if sg, ok := v.(*folder); ok {
|
||||
mergeChildren(dg.children, sg.children)
|
||||
mergeFolder(dg, sg)
|
||||
continue
|
||||
}
|
||||
}
|
||||
dst[k] = v
|
||||
dst.children[k] = v
|
||||
delete(dst.unloaded, k)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,9 +113,10 @@ func mergeChildren(dst, src map[string]node) {
|
||||
type nodeSet func(fn func(label string, n node) error) error
|
||||
|
||||
// categorySite is a loaded category and where it sits: the folder holding it, that
|
||||
// folder's path, and its own dot path.
|
||||
// folder's path, its name and its own dot path.
|
||||
type categorySite struct {
|
||||
dir []string
|
||||
name string
|
||||
path string
|
||||
in *folder
|
||||
n node
|
||||
@@ -89,7 +140,7 @@ func categorySites(root *folder) []categorySite {
|
||||
}
|
||||
|
||||
func siteIn(dir []string, in *folder, name string) categorySite {
|
||||
return categorySite{dir: dir, path: categoryPath(dir, name), in: in, n: in.children[name]}
|
||||
return categorySite{dir: dir, name: name, path: categoryPath(dir, name), in: in, n: in.children[name]}
|
||||
}
|
||||
|
||||
func categoryPath(dir []string, name string) string { return join(strings.Join(dir, "."), name) }
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"index": {
|
||||
"en_US.address": {"paths":["","locality","postal-code","region","street","street-number"],"reads":["geo.US.address"]},
|
||||
"en_US.color": {"paths":[""]},
|
||||
"en_US.company": {"paths":["","base","suffix"]},
|
||||
"en_US.date": {"paths":[""]},
|
||||
"en_US.email": {"paths":["","domain","local","local.n"]},
|
||||
"en_US.first-name": {"parent":"sex","paths":["","count","name","sex"]},
|
||||
"en_US.ip": {"paths":[""]},
|
||||
"en_US.itin": {"paths":["","area","group","serial"]},
|
||||
"en_US.last-name": {"paths":["","count","name"]},
|
||||
"en_US.person": {"paths":["","first","last","prefix","sex"],"reads":["en_US.first-name","en_US.last-name"]},
|
||||
"en_US.phone": {"paths":["","area","exchange","line"],"reads":["en_US.phone-exchange"]},
|
||||
"en_US.phone-exchange": {"paths":[""]},
|
||||
"en_US.price": {"paths":["","amt","cents"]},
|
||||
"en_US.sentence": {"paths":["","adj","noun","prep","verb"]},
|
||||
"en_US.sex": {"paths":["","code","first-name","first-name.count","first-name.name","first-name.sex","name","title","title.name","title.sex","title.share"]},
|
||||
"en_US.ssn": {"paths":["","area","group","serial"]},
|
||||
"en_US.time": {"paths":[""]},
|
||||
"en_US.title": {"parent":"sex","paths":["","name","sex","share"]},
|
||||
"en_US.url": {"paths":["","host","path"]},
|
||||
"en_US.username": {"paths":["","n"]},
|
||||
"en_US.version": {"paths":["","n","pre","suffix"]},
|
||||
"en_US.word": {"paths":[""]},
|
||||
"geo.SE.address": {"paths":["","locality","postal-code","street","street-number"],"reads":["geo.SE.locality"]},
|
||||
"geo.SE.locality": {"parent":"municipality","paths":["","lat","lon","municipality","name","population","postal-code","postal-code.code","postal-code.locality","street","street.locality","street.name","street.segments"]},
|
||||
"geo.SE.municipality": {"parent":"region","paths":["","code","locality","locality.lat","locality.lon","locality.municipality","locality.name","locality.population","locality.postal-code","locality.postal-code.code","locality.postal-code.locality","locality.street","locality.street.locality","locality.street.name","locality.street.segments","name","population","region"]},
|
||||
"geo.SE.postal-code": {"parent":"locality","paths":["","code","locality"]},
|
||||
"geo.SE.region": {"paths":["","code","municipality","municipality.code","municipality.locality","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.locality","municipality.locality.street.name","municipality.locality.street.segments","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
|
||||
"geo.SE.street": {"parent":"locality","paths":["","locality","name","segments"]},
|
||||
"geo.US.address": {"paths":["","locality","postal-code","region","street","street-number"],"reads":["geo.US.locality"]},
|
||||
"geo.US.locality": {"parent":"municipality","paths":["","code","lat","lon","municipality","name","population","postal-code","postal-code.addresses","postal-code.code","postal-code.locality","street","street.addresses","street.locality","street.name"]},
|
||||
"geo.US.municipality": {"parent":"region","paths":["","code","locality","locality.code","locality.lat","locality.lon","locality.municipality","locality.name","locality.population","locality.postal-code","locality.postal-code.addresses","locality.postal-code.code","locality.postal-code.locality","locality.street","locality.street.addresses","locality.street.locality","locality.street.name","name","population","region"]},
|
||||
"geo.US.postal-code": {"parent":"locality","paths":["","addresses","code","locality"]},
|
||||
"geo.US.region": {"paths":["","abbr","code","municipality","municipality.code","municipality.locality","municipality.locality.code","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.addresses","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.addresses","municipality.locality.street.locality","municipality.locality.street.name","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
|
||||
"geo.US.street": {"parent":"locality","paths":["","addresses","locality","name"]},
|
||||
"misc.car": {"paths":["","make","model"]},
|
||||
"misc.coordinate": {"paths":["","lat","lon"]},
|
||||
"misc.creditcard": {"paths":["","d"]},
|
||||
"misc.currency": {"paths":["","code","decimals","name","numeric","symbol"]},
|
||||
"misc.datetime": {"paths":[""]},
|
||||
"misc.emoji": {"paths":[""]},
|
||||
"misc.httpmethod": {"paths":["","idempotent","method","safe"]},
|
||||
"misc.httpstatus": {"paths":["","code","reason"]},
|
||||
"misc.language": {"paths":["","code","code3","name"]},
|
||||
"misc.loglevel": {"paths":["","code","keyword","severity"]},
|
||||
"misc.mac": {"paths":[""]},
|
||||
"misc.mimetype": {"paths":["","ext","type"]},
|
||||
"misc.objectid": {"paths":[""]},
|
||||
"misc.port": {"paths":["","number","service"]},
|
||||
"misc.protocol": {"paths":["","keyword","name","number"]},
|
||||
"misc.territory": {"paths":["","alpha2","alpha3","calling-code","capital","country","currency","flag","languages","name","numeric","timezone","timezone.offset","timezone.territory","timezone.weight","timezone.zone","tld"]},
|
||||
"misc.timezone": {"parent":"territory","paths":["","offset","territory","weight","zone"]},
|
||||
"misc.tld": {"paths":["","tld","type","unicode"]},
|
||||
"misc.useragent": {"paths":["","browser","device","os","ua"]},
|
||||
"misc.uuid": {"paths":["","variant"]},
|
||||
"sv_SE.address": {"paths":["","locality","postal-code","street","street-number"],"reads":["geo.SE.address"]},
|
||||
"sv_SE.birth-number": {"parent":"sex","paths":["","number","sex"]},
|
||||
"sv_SE.color": {"paths":[""]},
|
||||
"sv_SE.company": {"paths":["","base","suffix"]},
|
||||
"sv_SE.date": {"paths":[""]},
|
||||
"sv_SE.email": {"paths":["","domain","local","local.n"]},
|
||||
"sv_SE.first-name": {"parent":"sex","paths":["","count","name","sex"]},
|
||||
"sv_SE.ip": {"paths":[""]},
|
||||
"sv_SE.last-name": {"paths":["","count","name"]},
|
||||
"sv_SE.person": {"paths":["","first","last","prefix","sex"],"reads":["sv_SE.first-name","sv_SE.last-name","sv_SE.title"]},
|
||||
"sv_SE.personnummer": {"paths":[""],"reads":["sv_SE.birth-number"]},
|
||||
"sv_SE.phone": {"paths":["","a","b","c","prefix"]},
|
||||
"sv_SE.price": {"paths":["","amt","ore"]},
|
||||
"sv_SE.samordningsnummer": {"paths":[""],"reads":["sv_SE.birth-number"]},
|
||||
"sv_SE.sentence": {"paths":["","adj","noun","prep","verb"]},
|
||||
"sv_SE.sex": {"paths":["","birth-number","birth-number.number","birth-number.sex","code","first-name","first-name.count","first-name.name","first-name.sex","name"]},
|
||||
"sv_SE.time": {"paths":[""]},
|
||||
"sv_SE.title": {"paths":["","name","share"]},
|
||||
"sv_SE.url": {"paths":["","host","path"]},
|
||||
"sv_SE.username": {"paths":["","n"]},
|
||||
"sv_SE.version": {"paths":["","n","pre","suffix"]},
|
||||
"sv_SE.word": {"paths":[""]}
|
||||
}
|
||||
}
|
||||
+16
-13
@@ -140,7 +140,7 @@ still lives in `data/`; `--data-path` layers over it.
|
||||
|
||||
## The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0
|
||||
|
||||
2026-10-04, Lilleman auf Larv; revised 2026-10-09 to put the packages in the core's Go module. Serves goals 2.3, 6.2, 7.3, 9 and 10.1. Carried out by `todo.md` items 180, 181, 182, 184, 185 and 186; until then the shipped data is embedded whole, as the decision "The shipped data is embedded, not discovered" states. The CLI binary may grow to hundreds of MB.
|
||||
2026-10-04, Lilleman auf Larv; revised 2026-10-09 to put the packages in the core's Go module. Serves goals 2.3, 6.2, 7.3, 9 and 10.1. Carried out by `todo.md` items 181, 182, 184, 185 and 186; until then the shipped data is embedded whole, as the decision "The shipped data is embedded, not discovered" states. The CLI binary may grow to hundreds of MB.
|
||||
|
||||
- One Go package per locale, one per country's `geo/` tree, and one for `misc`, all in the core's Go module. A package registers nothing when imported.
|
||||
- A bare `New()` loads no data and fails, naming the option to add. `WithoutShippedData` goes.
|
||||
@@ -154,21 +154,24 @@ still lives in `data/`; `--data-path` layers over it.
|
||||
|
||||
Valid while the shipped data stays under 100 MiB, which a gate test holds (`todo.md` item 192): `go get` fetches the whole Go module, and the `go` command refuses a Go module zip over 500 MiB (`MaxZipFile` in `golang.org/x/mod/zip`). Past it, `todo.md` item 193 moves the largest packages into nested Go modules that keep their import paths, each requiring the core release that drops the package, so no build sees it twice.
|
||||
|
||||
## With only the shipped set, a category loads on the first call reaching it; beside a `--data-path`, every category loads in `New`
|
||||
## A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads and every indexed category reaching what it replaces
|
||||
|
||||
2026-10-03, a restructure the maintainer approved.
|
||||
2026-10-03, a restructure the maintainer approved; widened to any source on 2026-10-09.
|
||||
|
||||
Goal 13: `New` pays nothing for a category a run never reaches, which matters once the
|
||||
shipped set grows to hundreds of MB. A category loads with every category it references
|
||||
and its whole table family, and binds as a whole load does, so it renders the same as
|
||||
after a whole load. `List` reads `shippedindex.go`, which `go generate` writes from
|
||||
`data/`; `TestShippedIndexIsCurrent` fails while the index is stale. CI loads the whole
|
||||
shipped set, and each shipped category alone into a fresh `New`, so none fails at first
|
||||
reach.
|
||||
Goal 13: `New` pays nothing for a category a run never reaches. A manifest,
|
||||
`.fejkdata.json` at a source's root, may carry an index: each category's parent table,
|
||||
the paths `List` advertises below it and the categories its templates reference. A
|
||||
category of an indexed source loads with every category it references and its whole
|
||||
table family, and binds as a whole load does, so it renders the same as after a whole
|
||||
load. `go generate` writes the shipped set's manifest from `data/`;
|
||||
`TestShippedManifestIsCurrent` fails while it is stale. CI loads the whole shipped set,
|
||||
and each shipped category alone into a fresh `New`, so none fails at first reach.
|
||||
|
||||
Goal 6: beside a `--data-path`, `WithDataPath` or `WithDataFS`, every category loads in
|
||||
`New`, so each mistake in the user's data is `New`'s error. That parses the whole shipped
|
||||
set, against goal 13, until `todo.md` item 180 ends it.
|
||||
Goal 6: a source without an index loads whole in `New`, so each mistake in it is `New`'s
|
||||
error. So does every indexed category it reads, and every indexed category leading
|
||||
through references or table links to one that it or any later source replaced, so a
|
||||
replacement breaking a reader fails `New`. Goal 6.3: an indexed source from anyone else
|
||||
may fail at first reach; `Fake` then returns the load's error, the same every call.
|
||||
|
||||
Valid while CI proves the shipped set whole and each category alone.
|
||||
|
||||
|
||||
+7
-9
@@ -24,7 +24,7 @@ var _ [^uint(0)>>63 - 1]struct{}
|
||||
// It is safe for concurrent use; a seeded sequence is reproducible only when drawn
|
||||
// from one goroutine.
|
||||
type Generator struct {
|
||||
// mu guards drawState, structs and root, which gains a shipped category on the
|
||||
// mu guards drawState, structs and root, which gains an indexed category on the
|
||||
// first call reaching it.
|
||||
mu sync.Mutex
|
||||
drawState *drawstate.State
|
||||
@@ -71,8 +71,10 @@ func WithoutShippedData() Option {
|
||||
// in order with the last winning a name clash. Each JSON file becomes a category
|
||||
// named after the file (address.json -> "address") and each subdirectory a
|
||||
// namespace segment. It errors on a missing directory, invalid JSON or invalid data; a
|
||||
// generator with no data at all renders templates that read none. With only the shipped data, a category loads on the first call
|
||||
// reaching it; with WithDataPath or WithDataFS, every category loads here.
|
||||
// generator with no data at all renders templates that read none. A source whose
|
||||
// .fejkdata.json carries an index, as the shipped set's does, loads each category on the
|
||||
// first call reaching it. Every other source loads here, with what it reads and every
|
||||
// indexed category reaching what it replaces.
|
||||
func New(opts ...Option) (*Generator, error) {
|
||||
c := config{shipped: true}
|
||||
for _, opt := range opts {
|
||||
@@ -89,18 +91,14 @@ func New(opts ...Option) (*Generator, error) {
|
||||
return &Generator{drawState: drawstate.New(seed), root: root}, nil
|
||||
}
|
||||
|
||||
// load builds the tree New starts from. docs/decisions.md#with-only-the-shipped-set-a-category-loads-on-the-first-call-reaching-it-beside-a---data-path-every-category-loads-in-new
|
||||
// load builds the tree New starts from. docs/decisions.md#a-source-whose-manifest-carries-an-index-loads-each-category-on-first-reach-any-other-loads-in-new-with-what-it-reads-and-every-indexed-category-reaching-what-it-replaces
|
||||
func (c config) load() (folder, error) {
|
||||
if c.shipped && len(c.sources) == 0 {
|
||||
return unloadedTree(), nil
|
||||
}
|
||||
var sources []datafiles.Source
|
||||
if c.shipped {
|
||||
sources = append(sources, shippedSource)
|
||||
}
|
||||
sources = append(sources, c.sources...)
|
||||
cats, err := loadData(sources)
|
||||
return folder{children: cats}, err
|
||||
return loadSources(sources)
|
||||
}
|
||||
|
||||
// List returns the sorted dotted paths Fake renders: each category and every field,
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
package fejkdata
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/larvit/fejkdata/internal/datafiles"
|
||||
"github.com/larvit/fejkdata/internal/grammar"
|
||||
)
|
||||
|
||||
// indexed is a category an index names and no call has reached yet, and the source holding it.
|
||||
type indexed struct {
|
||||
src datafiles.Source
|
||||
entry datafiles.IndexEntry
|
||||
}
|
||||
|
||||
// placeIndex sets every category index names unloaded in root, in place of what root holds
|
||||
// at its path.
|
||||
func placeIndex(root *folder, src datafiles.Source, index map[string]datafiles.IndexEntry) {
|
||||
for _, p := range sortedNames(index) {
|
||||
segs := strings.Split(p, ".")
|
||||
g := madeFolder(root, segs[:len(segs)-1])
|
||||
name := segs[len(segs)-1]
|
||||
delete(g.children, name)
|
||||
if g.unloaded == nil {
|
||||
g.unloaded = map[string]indexed{}
|
||||
}
|
||||
g.unloaded[name] = indexed{src: src, entry: index[p]}
|
||||
}
|
||||
}
|
||||
|
||||
// categoryAt is where a category sits: the folder holding it, that folder's path, and its name.
|
||||
type categoryAt struct {
|
||||
dir []string
|
||||
in *folder
|
||||
name string
|
||||
}
|
||||
|
||||
func (c categoryAt) path() string { return categoryPath(c.dir, c.name) }
|
||||
|
||||
// categoryUnder is the category, loaded or not, that a path from root names or descends into.
|
||||
func categoryUnder(root *folder, segs []string) (categoryAt, bool) {
|
||||
g, i := folderAt(root, segs)
|
||||
if i == len(segs) {
|
||||
return categoryAt{}, false
|
||||
}
|
||||
if _, unloaded := g.unloaded[segs[i]]; !unloaded && g.children[segs[i]] == nil {
|
||||
return categoryAt{}, false
|
||||
}
|
||||
return categoryAt{dir: segs[:i:i], in: g, name: segs[i]}, true
|
||||
}
|
||||
|
||||
// loadCallerPath strips the / a caller's path may start with, splits the path, and loads the
|
||||
// unloaded category it names or descends into. The caller walks it next, and a walk loads nothing.
|
||||
func (f *Generator) loadCallerPath(path string) (string, []string, error) {
|
||||
path, err := grammar.CallerPath(path)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("fejkdata: %w", err)
|
||||
}
|
||||
segs, err := grammar.SplitPath(path)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("fejkdata: %w", err)
|
||||
}
|
||||
if c, ok := categoryUnder(&f.root, segs); ok {
|
||||
if err := loadReached(&f.root, nil, []categoryAt{c}); err != nil {
|
||||
return "", nil, fmt.Errorf("fejkdata: %w", err)
|
||||
}
|
||||
}
|
||||
return path, segs, nil
|
||||
}
|
||||
|
||||
// referenced is every category the templates of nodes reference, each reference read from
|
||||
// the folder dir.
|
||||
func referenced(root *folder, dir []string, nodes nodeSet) []categoryAt {
|
||||
var out []categoryAt
|
||||
_ = nodes(func(_ string, n node) error {
|
||||
t, isTemplate := n.(*template)
|
||||
if !isTemplate {
|
||||
return nil
|
||||
}
|
||||
for _, name := range refTokens(t.tokens) {
|
||||
segs, err := refSegments(name, dir)
|
||||
if err != nil {
|
||||
continue // resolveRefs reports it
|
||||
}
|
||||
if c, ok := categoryUnder(root, segs); ok {
|
||||
out = append(out, c)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// linkedTables is the table beside s that its parent column names, and every unloaded
|
||||
// table beside s whose parent s is.
|
||||
func linkedTables(s categorySite) []categoryAt {
|
||||
var out []categoryAt
|
||||
if t, isTable := s.n.(*table); isTable && t.rows.Options().Parent != "" {
|
||||
out = append(out, categoryAt{dir: s.dir, in: s.in, name: t.rows.Options().Parent})
|
||||
}
|
||||
for _, name := range sortedNames(s.in.unloaded) {
|
||||
if s.in.unloaded[name].entry.Parent == s.name {
|
||||
out = append(out, categoryAt{dir: s.dir, in: s.in, name: name})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// loadReached loads the unloaded categories among wanted, and every unloaded category they
|
||||
// or sites reference or link to, then runs sites and all it loaded through a whole load's
|
||||
// pipeline. On an error it unloads again what it loaded, so the next call fails the same way.
|
||||
func loadReached(root *folder, sites []categorySite, wanted []categoryAt) error {
|
||||
queue := wanted
|
||||
for _, s := range sites {
|
||||
queue = append(queue, reachedFrom(root, s)...)
|
||||
}
|
||||
var loaded []categorySite
|
||||
var was []indexed
|
||||
unload := func() {
|
||||
for i, s := range loaded {
|
||||
delete(s.in.children, s.name)
|
||||
s.in.unloaded[s.name] = was[i]
|
||||
}
|
||||
}
|
||||
for ; len(queue) > 0; queue = queue[1:] {
|
||||
c := queue[0]
|
||||
x, unloaded := c.in.unloaded[c.name]
|
||||
if !unloaded {
|
||||
continue
|
||||
}
|
||||
if err := x.src.Load(c.dir, c.name, compileInto(func([]string) *folder { return c.in })); err != nil {
|
||||
unload()
|
||||
return err
|
||||
}
|
||||
delete(c.in.unloaded, c.name)
|
||||
s := siteIn(c.dir, c.in, c.name)
|
||||
loaded, was = append(loaded, s), append(was, x)
|
||||
queue = append(queue, reachedFrom(root, s)...)
|
||||
}
|
||||
sites = append(sites[:len(sites):len(sites)], loaded...)
|
||||
if len(sites) == 0 {
|
||||
return nil
|
||||
}
|
||||
if err := categoryPipeline(sites, root.children).run(); err != nil {
|
||||
unload()
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func reachedFrom(root *folder, s categorySite) []categoryAt {
|
||||
return append(referenced(root, s.dir, siteNodes([]categorySite{s})), linkedTables(s)...)
|
||||
}
|
||||
|
||||
// reaching is every unloaded category under root whose index entry leads to a path in
|
||||
// targets: through its reads, its parent or a child table, and through other unloaded
|
||||
// categories.
|
||||
func reaching(root *folder, targets map[string]bool) []categoryAt {
|
||||
if len(targets) == 0 {
|
||||
return nil
|
||||
}
|
||||
unloaded := map[string]categoryAt{}
|
||||
unloadedUnder(root, nil, unloaded)
|
||||
from := map[string][]string{} // a path, and the unloaded categories leading to it
|
||||
for _, p := range sortedNames(unloaded) {
|
||||
c := unloaded[p]
|
||||
e := c.in.unloaded[c.name].entry
|
||||
for _, r := range e.Reads {
|
||||
from[r] = append(from[r], p)
|
||||
}
|
||||
if e.Parent != "" {
|
||||
parent := categoryPath(c.dir, e.Parent)
|
||||
from[parent] = append(from[parent], p)
|
||||
from[p] = append(from[p], parent)
|
||||
}
|
||||
}
|
||||
var out []categoryAt
|
||||
seen := map[string]bool{}
|
||||
for queue := sortedNames(targets); len(queue) > 0; queue = queue[1:] {
|
||||
for _, p := range from[queue[0]] {
|
||||
c, isUnloaded := unloaded[p]
|
||||
if !isUnloaded || seen[p] {
|
||||
continue
|
||||
}
|
||||
seen[p] = true
|
||||
out = append(out, c)
|
||||
queue = append(queue, p)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func unloadedUnder(g *folder, dir []string, out map[string]categoryAt) {
|
||||
for name := range g.unloaded {
|
||||
c := categoryAt{dir: dir, in: g, name: name}
|
||||
out[c.path()] = c
|
||||
}
|
||||
for name, n := range g.children {
|
||||
if sub, isFolder := n.(*folder); isFolder {
|
||||
unloadedUnder(sub, append(dir[:len(dir):len(dir)], name), out)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -84,13 +84,15 @@ func inputValue(input string) jsonvalue.Value {
|
||||
return raw
|
||||
}
|
||||
|
||||
// loadInline loads the shipped categories n reads, then runs n through the pipeline against root.
|
||||
// loadInline loads the unloaded categories n reads, then runs n through the pipeline against root.
|
||||
func loadInline(root *folder, n node, label string, typedByGo bool) error {
|
||||
nodes := inlineNodes(n, label)
|
||||
if err := refuseFolderRefs(nodes); err != nil {
|
||||
return err
|
||||
}
|
||||
loadShipped(root, unloadedReads(root, nil, nodes))
|
||||
if err := loadReached(root, nil, referenced(root, nil, nodes)); err != nil {
|
||||
return err
|
||||
}
|
||||
return pipeline{
|
||||
nodes: nodes,
|
||||
resolve: func() error { return resolveInlineTemplates(nodes, root.children) },
|
||||
|
||||
@@ -2,11 +2,17 @@
|
||||
package datafiles
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"embed"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/larvit/fejkdata/internal/grammar"
|
||||
"github.com/larvit/fejkdata/internal/jsonvalue"
|
||||
@@ -56,6 +62,84 @@ func (s Source) dirPath(dir []string) string {
|
||||
return "."
|
||||
}
|
||||
|
||||
// ManifestFile is the file at a source's root that describes the source.
|
||||
const ManifestFile = ".fejkdata.json"
|
||||
|
||||
// Manifest is what a source's manifest says of it.
|
||||
type Manifest struct {
|
||||
// Index is every category of the source by dot path; nil where the manifest has none.
|
||||
Index map[string]IndexEntry `json:"index"`
|
||||
}
|
||||
|
||||
// IndexEntry is what an index says of a category: its parent table, "" for none, the paths
|
||||
// List advertises below it, and the categories its templates reference.
|
||||
type IndexEntry struct {
|
||||
Parent string `json:"parent,omitempty"`
|
||||
Paths []string `json:"paths"`
|
||||
Reads []string `json:"reads,omitempty"`
|
||||
}
|
||||
|
||||
// embeddedManifests holds each manifest read from an embed.FS, whose files never change.
|
||||
var embeddedManifests sync.Map
|
||||
|
||||
type embeddedManifest struct {
|
||||
fsys embed.FS
|
||||
base string
|
||||
}
|
||||
|
||||
// Manifest reads the manifest at the source's root, the zero Manifest where there is none.
|
||||
func (s Source) Manifest() (Manifest, error) {
|
||||
e, embedded := s.fsys.(embed.FS)
|
||||
key := embeddedManifest{fsys: e, base: s.base}
|
||||
if m, read := embeddedManifests.Load(key); embedded && read {
|
||||
return m.(Manifest), nil
|
||||
}
|
||||
m, err := s.readManifest()
|
||||
if embedded && err == nil {
|
||||
embeddedManifests.Store(key, m)
|
||||
}
|
||||
return m, err
|
||||
}
|
||||
|
||||
func (s Source) readManifest() (Manifest, error) {
|
||||
if s.onDisk && s.label == "" {
|
||||
return Manifest{}, nil // Walk refuses the empty data path
|
||||
}
|
||||
full := path.Join(s.base, ManifestFile)
|
||||
b, err := fs.ReadFile(s.fsys, full)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return Manifest{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err)
|
||||
}
|
||||
m, err := decodeManifest(b)
|
||||
if err != nil {
|
||||
return Manifest{}, fmt.Errorf("%s: %w", s.labelled(full), err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func decodeManifest(b []byte) (Manifest, error) {
|
||||
dec := json.NewDecoder(bytes.NewReader(b))
|
||||
dec.DisallowUnknownFields()
|
||||
var m Manifest
|
||||
if err := dec.Decode(&m); err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
if _, err := dec.Token(); err != io.EOF {
|
||||
return Manifest{}, fmt.Errorf("more than one JSON value")
|
||||
}
|
||||
for p := range m.Index {
|
||||
for _, seg := range strings.Split(p, ".") {
|
||||
if err := grammar.CheckIdentifier(seg); err != nil {
|
||||
return Manifest{}, fmt.Errorf("index names %q: %w", p, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// Walk hands compile every category of the tree, folders and files in name order. A
|
||||
// folder holding no category anywhere below it is skipped; a hidden file or folder is no
|
||||
// category, though a category may name a hidden rows file beside it.
|
||||
|
||||
@@ -22,7 +22,7 @@ type node interface{ isNode() }
|
||||
// dot path; rendering one is an error (see Fake).
|
||||
type folder struct {
|
||||
children map[string]node
|
||||
unloaded map[string]shippedEntry // the shipped categories here a call has not reached yet
|
||||
unloaded map[string]indexed // the categories here an index names and no call has reached yet
|
||||
}
|
||||
|
||||
func (*folder) isNode() {}
|
||||
|
||||
@@ -237,7 +237,7 @@ func paths(n node, intoRepeats bool) []string {
|
||||
out = appendUnder(out, name, paths(n.children[name], intoRepeats))
|
||||
}
|
||||
for _, name := range sortedNames(n.unloaded) {
|
||||
out = appendUnder(out, name, n.unloaded[name].paths)
|
||||
out = appendUnder(out, name, n.unloaded[name].entry.Paths)
|
||||
}
|
||||
return out
|
||||
case *template:
|
||||
|
||||
+2
-139
@@ -2,150 +2,13 @@ package fejkdata
|
||||
|
||||
import (
|
||||
"embed"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/larvit/fejkdata/internal/datafiles"
|
||||
"github.com/larvit/fejkdata/internal/grammar"
|
||||
"github.com/larvit/fejkdata/internal/invariant"
|
||||
)
|
||||
|
||||
//go:generate env REGENERATE=1 go test -run ^TestShippedIndexIsCurrent$ .
|
||||
//go:generate env REGENERATE=1 go test -run ^TestShippedManifestIsCurrent$ .
|
||||
|
||||
//go:embed data
|
||||
//go:embed data data/.fejkdata.json
|
||||
var shippedFS embed.FS
|
||||
|
||||
var shippedSource = datafiles.FS(shippedFS, "data")
|
||||
|
||||
// shippedEntry is a shipped category's entry in shippedindex.go: its parent table, "" for
|
||||
// none, and the paths List advertises below it. After changing it, empty
|
||||
// shippedIndex's literal by hand so the package builds, then run generate.
|
||||
type shippedEntry struct {
|
||||
parent string
|
||||
paths []string
|
||||
}
|
||||
|
||||
// unloadedTree builds the folders of the shipped index, with every category in them unloaded.
|
||||
func unloadedTree() folder {
|
||||
root := folder{children: map[string]node{}}
|
||||
for p, e := range shippedIndex {
|
||||
segs := strings.Split(p, ".")
|
||||
g := madeFolder(&root, segs[:len(segs)-1])
|
||||
if g.unloaded == nil {
|
||||
g.unloaded = map[string]shippedEntry{}
|
||||
}
|
||||
g.unloaded[segs[len(segs)-1]] = e
|
||||
}
|
||||
return root
|
||||
}
|
||||
|
||||
// unloadedCategory is a shipped category not loaded yet.
|
||||
type unloadedCategory struct {
|
||||
dir []string
|
||||
in *folder
|
||||
name string
|
||||
}
|
||||
|
||||
// unloadedAt is the unloaded shipped category a path from root names, or descends into.
|
||||
func unloadedAt(root *folder, segs []string) (unloadedCategory, bool) {
|
||||
g, i := folderAt(root, segs)
|
||||
if i == len(segs) {
|
||||
return unloadedCategory{}, false
|
||||
}
|
||||
if _, unloaded := g.unloaded[segs[i]]; !unloaded {
|
||||
return unloadedCategory{}, false
|
||||
}
|
||||
return unloadedCategory{dir: segs[:i:i], in: g, name: segs[i]}, true
|
||||
}
|
||||
|
||||
// loadCallerPath strips the / a caller's path may start with, splits the path, and loads the
|
||||
// shipped category it names or descends into. The caller walks it next, and a walk loads nothing.
|
||||
func (f *Generator) loadCallerPath(path string) (string, []string, error) {
|
||||
path, err := grammar.CallerPath(path)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("fejkdata: %w", err)
|
||||
}
|
||||
segs, err := grammar.SplitPath(path)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("fejkdata: %w", err)
|
||||
}
|
||||
if u, unloaded := unloadedAt(&f.root, segs); unloaded {
|
||||
loadShipped(&f.root, []unloadedCategory{u})
|
||||
}
|
||||
return path, segs, nil
|
||||
}
|
||||
|
||||
// unloadedReads is every unloaded shipped category the templates of nodes reference, each
|
||||
// reference read from the folder dir.
|
||||
func unloadedReads(root *folder, dir []string, nodes nodeSet) []unloadedCategory {
|
||||
var out []unloadedCategory
|
||||
_ = nodes(func(_ string, n node) error {
|
||||
t, isTemplate := n.(*template)
|
||||
if !isTemplate {
|
||||
return nil
|
||||
}
|
||||
for _, name := range refTokens(t.tokens) {
|
||||
segs, err := refSegments(name, dir)
|
||||
if err != nil {
|
||||
continue // resolveRefs reports it
|
||||
}
|
||||
if u, unloaded := unloadedAt(root, segs); unloaded {
|
||||
out = append(out, u)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// loadShipped loads the shipped categories wanted and, for each category it loads, every
|
||||
// category it reads and its table family, then runs all it loaded through a whole load's pipeline.
|
||||
// TestEveryShippedCategoryLoadsAlone reaches each category, so a failure here means a stale
|
||||
// index.
|
||||
func loadShipped(root *folder, wanted []unloadedCategory) {
|
||||
var sites []categorySite
|
||||
for queue := wanted; len(queue) > 0; queue = queue[1:] {
|
||||
u := queue[0]
|
||||
e, unloaded := u.in.unloaded[u.name]
|
||||
if !unloaded {
|
||||
continue
|
||||
}
|
||||
site, err := u.load()
|
||||
if err != nil {
|
||||
panic(invariant.Broken("shipped %s: %v; after a change under data/, regenerate shippedindex.go", categoryPath(u.dir, u.name), err))
|
||||
}
|
||||
sites = append(sites, site)
|
||||
queue = append(queue, unloadedReads(root, u.dir, siteNodes([]categorySite{site}))...)
|
||||
queue = append(queue, u.linkedTables(e)...)
|
||||
}
|
||||
if len(sites) == 0 {
|
||||
return
|
||||
}
|
||||
if err := categoryPipeline(sites, root.children).run(); err != nil {
|
||||
panic(invariant.Broken("loading shipped categories: %v; after a change under data/, regenerate shippedindex.go", err))
|
||||
}
|
||||
}
|
||||
|
||||
// load parses and compiles the category, and moves it from its folder's unloaded map to
|
||||
// the folder's children.
|
||||
func (u unloadedCategory) load() (categorySite, error) {
|
||||
if err := shippedSource.Load(u.dir, u.name, compileInto(func([]string) *folder { return u.in })); err != nil {
|
||||
return categorySite{}, err
|
||||
}
|
||||
delete(u.in.unloaded, u.name)
|
||||
return siteIn(u.dir, u.in, u.name), nil
|
||||
}
|
||||
|
||||
// linkedTables is u's parent beside it, and every table beside it whose parent u is.
|
||||
func (u unloadedCategory) linkedTables(e shippedEntry) []unloadedCategory {
|
||||
var out []unloadedCategory
|
||||
if e.parent != "" {
|
||||
out = append(out, unloadedCategory{dir: u.dir, in: u.in, name: e.parent})
|
||||
}
|
||||
for _, name := range sortedNames(u.in.unloaded) {
|
||||
if u.in.unloaded[name].parent == u.name {
|
||||
out = append(out, unloadedCategory{dir: u.dir, in: u.in, name: name})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -1,82 +0,0 @@
|
||||
// Code generated by go generate. DO NOT EDIT.
|
||||
|
||||
package fejkdata
|
||||
|
||||
var shippedIndex = map[string]shippedEntry{
|
||||
"en_US.address": {paths: []string{"", "locality", "postal-code", "region", "street", "street-number"}},
|
||||
"en_US.color": {paths: []string{""}},
|
||||
"en_US.company": {paths: []string{"", "base", "suffix"}},
|
||||
"en_US.date": {paths: []string{""}},
|
||||
"en_US.email": {paths: []string{"", "domain", "local", "local.n"}},
|
||||
"en_US.first-name": {parent: "sex", paths: []string{"", "count", "name", "sex"}},
|
||||
"en_US.ip": {paths: []string{""}},
|
||||
"en_US.itin": {paths: []string{"", "area", "group", "serial"}},
|
||||
"en_US.last-name": {paths: []string{"", "count", "name"}},
|
||||
"en_US.person": {paths: []string{"", "first", "last", "prefix", "sex"}},
|
||||
"en_US.phone": {paths: []string{"", "area", "exchange", "line"}},
|
||||
"en_US.phone-exchange": {paths: []string{""}},
|
||||
"en_US.price": {paths: []string{"", "amt", "cents"}},
|
||||
"en_US.sentence": {paths: []string{"", "adj", "noun", "prep", "verb"}},
|
||||
"en_US.sex": {paths: []string{"", "code", "first-name", "first-name.count", "first-name.name", "first-name.sex", "name", "title", "title.name", "title.sex", "title.share"}},
|
||||
"en_US.ssn": {paths: []string{"", "area", "group", "serial"}},
|
||||
"en_US.time": {paths: []string{""}},
|
||||
"en_US.title": {parent: "sex", paths: []string{"", "name", "sex", "share"}},
|
||||
"en_US.url": {paths: []string{"", "host", "path"}},
|
||||
"en_US.username": {paths: []string{"", "n"}},
|
||||
"en_US.version": {paths: []string{"", "n", "pre", "suffix"}},
|
||||
"en_US.word": {paths: []string{""}},
|
||||
"geo.SE.address": {paths: []string{"", "locality", "postal-code", "street", "street-number"}},
|
||||
"geo.SE.locality": {parent: "municipality", paths: []string{"", "lat", "lon", "municipality", "name", "population", "postal-code", "postal-code.code", "postal-code.locality", "street", "street.locality", "street.name", "street.segments"}},
|
||||
"geo.SE.municipality": {parent: "region", paths: []string{"", "code", "locality", "locality.lat", "locality.lon", "locality.municipality", "locality.name", "locality.population", "locality.postal-code", "locality.postal-code.code", "locality.postal-code.locality", "locality.street", "locality.street.locality", "locality.street.name", "locality.street.segments", "name", "population", "region"}},
|
||||
"geo.SE.postal-code": {parent: "locality", paths: []string{"", "code", "locality"}},
|
||||
"geo.SE.region": {paths: []string{"", "code", "municipality", "municipality.code", "municipality.locality", "municipality.locality.lat", "municipality.locality.lon", "municipality.locality.municipality", "municipality.locality.name", "municipality.locality.population", "municipality.locality.postal-code", "municipality.locality.postal-code.code", "municipality.locality.postal-code.locality", "municipality.locality.street", "municipality.locality.street.locality", "municipality.locality.street.name", "municipality.locality.street.segments", "municipality.name", "municipality.population", "municipality.region", "name", "population", "timezone"}},
|
||||
"geo.SE.street": {parent: "locality", paths: []string{"", "locality", "name", "segments"}},
|
||||
"geo.US.address": {paths: []string{"", "locality", "postal-code", "region", "street", "street-number"}},
|
||||
"geo.US.locality": {parent: "municipality", paths: []string{"", "code", "lat", "lon", "municipality", "name", "population", "postal-code", "postal-code.addresses", "postal-code.code", "postal-code.locality", "street", "street.addresses", "street.locality", "street.name"}},
|
||||
"geo.US.municipality": {parent: "region", paths: []string{"", "code", "locality", "locality.code", "locality.lat", "locality.lon", "locality.municipality", "locality.name", "locality.population", "locality.postal-code", "locality.postal-code.addresses", "locality.postal-code.code", "locality.postal-code.locality", "locality.street", "locality.street.addresses", "locality.street.locality", "locality.street.name", "name", "population", "region"}},
|
||||
"geo.US.postal-code": {parent: "locality", paths: []string{"", "addresses", "code", "locality"}},
|
||||
"geo.US.region": {paths: []string{"", "abbr", "code", "municipality", "municipality.code", "municipality.locality", "municipality.locality.code", "municipality.locality.lat", "municipality.locality.lon", "municipality.locality.municipality", "municipality.locality.name", "municipality.locality.population", "municipality.locality.postal-code", "municipality.locality.postal-code.addresses", "municipality.locality.postal-code.code", "municipality.locality.postal-code.locality", "municipality.locality.street", "municipality.locality.street.addresses", "municipality.locality.street.locality", "municipality.locality.street.name", "municipality.name", "municipality.population", "municipality.region", "name", "population", "timezone"}},
|
||||
"geo.US.street": {parent: "locality", paths: []string{"", "addresses", "locality", "name"}},
|
||||
"misc.car": {paths: []string{"", "make", "model"}},
|
||||
"misc.coordinate": {paths: []string{"", "lat", "lon"}},
|
||||
"misc.creditcard": {paths: []string{"", "d"}},
|
||||
"misc.currency": {paths: []string{"", "code", "decimals", "name", "numeric", "symbol"}},
|
||||
"misc.datetime": {paths: []string{""}},
|
||||
"misc.emoji": {paths: []string{""}},
|
||||
"misc.httpmethod": {paths: []string{"", "idempotent", "method", "safe"}},
|
||||
"misc.httpstatus": {paths: []string{"", "code", "reason"}},
|
||||
"misc.language": {paths: []string{"", "code", "code3", "name"}},
|
||||
"misc.loglevel": {paths: []string{"", "code", "keyword", "severity"}},
|
||||
"misc.mac": {paths: []string{""}},
|
||||
"misc.mimetype": {paths: []string{"", "ext", "type"}},
|
||||
"misc.objectid": {paths: []string{""}},
|
||||
"misc.port": {paths: []string{"", "number", "service"}},
|
||||
"misc.protocol": {paths: []string{"", "keyword", "name", "number"}},
|
||||
"misc.territory": {paths: []string{"", "alpha2", "alpha3", "calling-code", "capital", "country", "currency", "flag", "languages", "name", "numeric", "timezone", "timezone.offset", "timezone.territory", "timezone.weight", "timezone.zone", "tld"}},
|
||||
"misc.timezone": {parent: "territory", paths: []string{"", "offset", "territory", "weight", "zone"}},
|
||||
"misc.tld": {paths: []string{"", "tld", "type", "unicode"}},
|
||||
"misc.useragent": {paths: []string{"", "browser", "device", "os", "ua"}},
|
||||
"misc.uuid": {paths: []string{"", "variant"}},
|
||||
"sv_SE.address": {paths: []string{"", "locality", "postal-code", "street", "street-number"}},
|
||||
"sv_SE.birth-number": {parent: "sex", paths: []string{"", "number", "sex"}},
|
||||
"sv_SE.color": {paths: []string{""}},
|
||||
"sv_SE.company": {paths: []string{"", "base", "suffix"}},
|
||||
"sv_SE.date": {paths: []string{""}},
|
||||
"sv_SE.email": {paths: []string{"", "domain", "local", "local.n"}},
|
||||
"sv_SE.first-name": {parent: "sex", paths: []string{"", "count", "name", "sex"}},
|
||||
"sv_SE.ip": {paths: []string{""}},
|
||||
"sv_SE.last-name": {paths: []string{"", "count", "name"}},
|
||||
"sv_SE.person": {paths: []string{"", "first", "last", "prefix", "sex"}},
|
||||
"sv_SE.personnummer": {paths: []string{""}},
|
||||
"sv_SE.phone": {paths: []string{"", "a", "b", "c", "prefix"}},
|
||||
"sv_SE.price": {paths: []string{"", "amt", "ore"}},
|
||||
"sv_SE.samordningsnummer": {paths: []string{""}},
|
||||
"sv_SE.sentence": {paths: []string{"", "adj", "noun", "prep", "verb"}},
|
||||
"sv_SE.sex": {paths: []string{"", "birth-number", "birth-number.number", "birth-number.sex", "code", "first-name", "first-name.count", "first-name.name", "first-name.sex", "name"}},
|
||||
"sv_SE.time": {paths: []string{""}},
|
||||
"sv_SE.title": {paths: []string{"", "name", "share"}},
|
||||
"sv_SE.url": {paths: []string{"", "host", "path"}},
|
||||
"sv_SE.username": {paths: []string{"", "n"}},
|
||||
"sv_SE.version": {paths: []string{"", "n", "pre", "suffix"}},
|
||||
"sv_SE.word": {paths: []string{""}},
|
||||
}
|
||||
@@ -86,7 +86,6 @@
|
||||
| 127 | 0.1.0 | | **Give a Swedish street only postal codes whose delivery digit means street or rural delivery, dropping codes whose delivery digit is 9, or record why they stay.** | 3 | 3 | 4 | 6 | 4 | 14.4 |
|
||||
| 110 | 0.1.0 | | **Refuse an empty segment in a path `Fake` and `FakeRecord` take, as a reference does: `sv_SE.person.` reports `path has an empty segment`.** | 1 | 1 | 3 | 6 | 6 | 13.8 |
|
||||
| 166 | 0.1.0 | | **Let a key or name cell hold a bracket, a brace, a double quote or a pipe, so the table loads, its row draws at random, and only a selector cannot reach it.** | 2 | 2 | 3 | 6 | 5.7 | 13.1 |
|
||||
| 180 | 0.1.0 | decision | **Load on first reach a source whose `.fejkdata.json` carries an index, and any other source in `New`; move the shipped index from `shippedindex.go` into that manifest.** | 6 | 6 | 5 | 8 | 13, 9.1 | 12.1 |
|
||||
| 191 | 0.1.0 | defect | **Read an existing tag through `git/ref/tags/{tag}` in `publish_release.py`, so its guard against a tag at another commit fires.** | 1 | 1 | 3 | 6 | 10 | 11.4 |
|
||||
| 58 | 0.2.0 | | **Ship prebuilt binaries, a container image, and packages for Homebrew, Scoop and the biggest Linux package managers, so the CLI needs no Go.** | 3 | 7 | 8 | 9 | 1.1, 7 | 29.2 |
|
||||
| 113 | 0.2.0 | defect | **Draw `{iban(CC)}` with its country's national check digits, and SE's with a real bank's id, so a validator checking more than mod 97 accepts it.** | 3 | 5 | 7 | 9 | 4 | 25.0 |
|
||||
@@ -182,7 +181,7 @@ Needs item 182, which moves the paths under `data/` this item writes. Pick the t
|
||||
|
||||
### 182. Ship `en_US`, `sv_SE`, `misc`, `geo/SE` and `geo/US` as Go packages exporting `FS`, listed by `data.Modules()`, and fail a bare `New()` naming `WithDataFS`.
|
||||
|
||||
Needs item 180. Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0".
|
||||
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0".
|
||||
- Each package's directory is a data root, so its tree spells its namespace: `data/sv_SE/` is `github.com/larvit/fejkdata/data/sv_SE` and holds `sv_SE/*.json`; `data/geo/SE/` holds `geo/SE/…`. Each package exports only `var FS embed.FS`, with no `init`, and carries a generated manifest whose `reads` names the modules it reads, `sv_SE` naming `geo/SE`.
|
||||
- The package `github.com/larvit/fejkdata/data` exports `Modules() []fs.FS` and imports every data package; the core imports none, so a library links only the data it imports.
|
||||
- The core stops embedding data. `WithDataFS` takes several. `WithoutShippedData` goes, and `--no-shipped-data` loads none of the modules the CLI carries. A read nothing provides fails, naming the module from the reader's manifest or saying none does, and the first error names the whole set to import.
|
||||
@@ -192,7 +191,7 @@ Needs item 180. Carries out the decision "The shipped data is Go packages a libr
|
||||
|
||||
### 185. Seat a panel of the README's audience personas to pull data in through the module API and build a custom CLI, and fix what it names.
|
||||
|
||||
Needs item 184; 0.1.0 is not cut before it. Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Seats: the Go test author, the bulk fixture author, a custom CLI's builder, and a data author with a `-d` folder and the replace flag, each in a scratch Go module against the checkout or a pre-release tag. Bench the geo packages' names, `data.Modules()`, and the error naming the import set. Record the ruling in `docs/`.
|
||||
Needs item 184; 0.1.0 is not cut before it. Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Seats: the Go test author, the bulk fixture author, a custom CLI's builder, and a data author with a `-d` folder and the replace flag, each in a scratch Go module against the checkout or a pre-release tag. Bench the geo packages' names, `data.Modules()`, the error naming the import set, and the `.fejkdata.json` manifest a data author writes by hand. Record the ruling in `docs/`.
|
||||
|
||||
### 16. Add locale categories: company, phone, finance, vehicle, words.
|
||||
|
||||
@@ -299,7 +298,7 @@ Split from item 129. cmd.exe's `echo` keeps its quotes and a trailing space, so
|
||||
|
||||
### 181. Fail a load where two sources define one category, unless the library's option or the CLI's flag says the later replaces the earlier, and then make every read reach the replacement.
|
||||
|
||||
Needs item 180. Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Proposed spellings: `WithReplace()` and a boolean `--replace` covering every `-d`, so it goes anywhere on the line. Bench the option, the flag and the clash error with the data and hand fixture authors before merging; the error names the option or the flag. A clash is found from the categories each source's index names and the files each source without a manifest holds, without parsing them. Revises the decision "A `--data-path` override rebinds every reference to the category it replaces", ending its conflict with goal 9.3. The changelog names the flag, the option and the new refusal.
|
||||
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Proposed spellings: `WithReplace()` and a boolean `--replace` covering every `-d`, so it goes anywhere on the line. Bench the option, the flag and the clash error with the data and hand fixture authors before merging; the error names the option or the flag. A clash is found from the categories each source's index names and the files each source without a manifest holds, without parsing them. Revises the decision "A `--data-path` override rebinds every reference to the category it replaces", ending its conflict with goal 9.3. The changelog names the flag, the option and the new refusal.
|
||||
|
||||
### 167. Lower the reading cost of the files a chunk reads most before its first commit.
|
||||
|
||||
@@ -325,14 +324,6 @@ Today `Fake("sv_SE.person.")` reports `no field ""`, while `{/sv_SE.person.}` re
|
||||
|
||||
The maintainer ruled so on 2026-10-09. Today `checkSelectorCells` in `internal/rows` refuses the whole table. Afterwards a path naming such a key fails with an error naming the character a selector cannot spell, and the README says a row whose name holds such a character is still selected by its key.
|
||||
|
||||
### 180. Load on first reach a source whose `.fejkdata.json` carries an index, and any other source in `New`; move the shipped index from `shippedindex.go` into that manifest.
|
||||
|
||||
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". The manifest is `.fejkdata.json` at a module's root, hidden so it is no category: `{"reads": {path prefix: module}, "index": {category: {"parent", "paths", "reads"}}}`. `go generate` writes it from the data, as it writes `shippedindex.go` today, and `TestShippedIndexIsCurrent` becomes a test that it is current.
|
||||
- `internal/datafiles` reads the manifest, and `shipped.go`'s unloaded tree is built per source from its index, so fejkdata's data loads as anyone's does (goal 9.1).
|
||||
- Each index entry carries the categories it reads. Beside a `--data-path`, `New` loads the user's categories, what they read and the shipped categories reaching a replaced one; the rest load on first reach.
|
||||
- Core tests move to `fstest.MapFS` fixtures carrying manifests.
|
||||
- Revises the decision "With only the shipped set, a category loads on the first call reaching it; beside a `--data-path`, every category loads in `New`", ending its conflict with goal 13. A module from anyone else loads on first reach too, as goal 6.3 allows.
|
||||
|
||||
### 191. Read an existing tag through `git/ref/tags/{tag}` in `publish_release.py`, so its guard against a tag at another commit fires.
|
||||
|
||||
GitHub has no `tags/{tag}` endpoint and answers 404 whether or not the tag exists. So when the tag already sits at another commit, the release job publishes on that old tag, and the guard never stops it. The guard follows an annotated tag to its commit before comparing.
|
||||
|
||||
Reference in New Issue
Block a user