larvit/fejkdata
3
0
Fork 0
Code Pull Requests 1 Actions Activity

Compare commits

base: larvit/fejkdata:main
Branches Tags
larvit/fejkdata:main larvit/fejkdata:record-output
..
compare: larvit/fejkdata:d72d326de55ba21fff1c73a55c926d0350524064
Branches Tags
larvit/fejkdata:main larvit/fejkdata:record-output

1 Commits
main .. d72d326de5

Author SHA1 Message Date
lilleman d72d326de5 Tests for weight 0, a never-numeric calc operand, the repeat product along a path, and ErrNoData
Tests / vet + fmt + tests (pull_request) Failing after 15s
Details
2026-09-02 18:22:23 +02:00
36 changed files with 1362 additions and 2740 deletions
Expand all files Collapse all files
.github/workflows/test.yml
+1 -17
View File
@@ -16,28 +16,12 @@ jobs:
- uses: actions/checkout@v6
with:
persist-credentials: false
fetch-depth: 0
- name: Relevant changes
id: changes
run: |
if [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
git fetch --no-tags origin "$GITHUB_BASE_REF"
base="origin/$GITHUB_BASE_REF"
else
base='${{ github.event.before }}'
fi
if git diff --quiet "$base"...HEAD -- '*.go' go.mod go.sum Dockerfile .dockerignore .github data README.md; then
echo "code=false" >> "$GITHUB_OUTPUT"
else
echo "code=true" >> "$GITHUB_OUTPUT"
fi
# `docker build` streams the context to the daemon. A compose bind-mount of
# `.` mounts an empty host dir instead, because the job is itself a container.
- name: Latest supported Go
if: ${{ steps.changes.outputs.code == 'true' }}
run: docker build .
# Runs even when the step above failed, so a red build says whether the
# failure is version-specific.
- name: Lowest supported Go
if: ${{ !cancelled() && steps.changes.outputs.code == 'true' }}
if: ${{ !cancelled() }}
run: docker build --build-arg GO_VERSION=1.22.12 .
.gitignore
+1
View File
@@ -1,3 +1,4 @@
.claude
*.out
__pycache__/
todo.md
AGENTS.md
-1
View File
@@ -7,4 +7,3 @@
- One spelling per result: reject the other at `New`, and let the error name the spelling to use.
- A standing choice a reader would relitigate goes under Decisions in the README, not in a comment.
- A README example is a `json` block that loads and renders as a category; `readme_test.go` runs every one.
- Cyclomatic complexity is gated at 14: the table-shaped dispatches (`eachToken`, `calc.factor`, `walkPath`) sit at 13–14 and stay whole; anything else that reaches 14 is decomposed.
Dockerfile
-1
View File
@@ -13,7 +13,6 @@ RUN go mod download
COPY . .
RUN go vet ./... && \
go run github.com/fzipp/gocyclo/cmd/gocyclo@v0.6.0 -over 14 -ignore _test . && \
{ unformatted="$(gofmt -l .)"; test -z "$unformatted" || \
{ echo "unformatted files:"; echo "$unformatted"; exit 1; }; } && \
go test -race ./...
README.md
+57 -168
View File
@@ -1,17 +1,32 @@
# fejkdata
Locale-aware fake data for tests and fixtures, generated from JSON templates. Use
it as a Go library or the CLI — no data on disk, no dependencies, and a seed makes output
reproducible.
A Go library and CLI for generating locale-aware fake data from JSON templates.
Forked from [github.com/Timewave-AB/fakes](https://github.com/Timewave-AB/fakes).
```sh
go install gitea.larvit.se/larvit/fejkdata/cmd/fejkdata@latest
fejkdata sv_SE.person # Sara Eriksson
```
## Goals
1. **Valid by construction** — every value passes the check its real consumer
applies; facts that belong together come from one draw, within a value and
across categories.
2. **Text means what it says** — a format renders as written; only `{…}` varies,
random characters included (`{digits(3)}`). One spelling per result; the wrong
one is a load error naming the right one.
3. **Every mistake is a load error** — `New` rejects; `Fake` on a loaded generator
fails only for an unknown path.
4. **Zero to a value in one command** — `go install`, then `fejkdata sv_SE.person`:
no checkout, no flag. Flags are GNU-form (`--seed 42`, `-n 3`) in any position;
the first custom template needs no escape and no option.
5. **Data lives in JSON** — a builtin only for what data can't express.
6. **Reproducible** — seed in, same stream out; no builtin reads a clock.
7. **Zero dependencies** — standard library only.
8. **Docs index the grammar** — every syntax feature is a heading; every example
runs under test and shows its output; a rule is stated once.
## CLI
```sh
go install gitea.larvit.se/larvit/fejkdata/cmd/fejkdata@latest
fejkdata sv_SE.person # Sara Eriksson
fejkdata sv_SE.person.last # Eriksson
fejkdata --seed 42 sv_SE.address # the same address every run
@@ -19,27 +34,17 @@ fejkdata -n 3 --separator ', ' sv_SE.word # nät, barn, sol
fejkdata --list # every path the data offers
fejkdata --data-path ./mydata sv_SE.word # layer a directory over the shipped data
fejkdata --no-shipped-data -d ./mydata --list # only your data
fejkdata 'name: {/sv_SE.person.last}' # name: <a surname> — an inline template
fejkdata '{"format":"name: {x}","x":["bosse","lina"]}' # name: bosse or name: lina
```
A path names a category, or a field inside one: each dot segment descends one
level — folders, then the category (a JSON file), then fields. An argument that is
a JSON object, array or string, or that carries a `{` token, is instead an
**inline template**: a format string or a JSON value compiled and rendered on the
spot. Its tokens reach the data by reference from the root —
`{/sv_SE.person.last}`, so shipped and `--data-path` categories are alike
available. An inline template sits in no folder, so the folder-relative `{.name}`
and `{..name}` are rejected naming the root spelling. A path never contains a
brace, a bracket or a quote, so the two cannot collide (see
[Decisions](#decisions)).
level — folders, then the category (a JSON file), then fields.
| Flag | |
|------|--|
| `-d`, `--data-path D` | a directory to layer over the shipped data; repeatable, the last wins a name clash |
| `--no-shipped-data` | load only the `--data-path` directories |
| `-s`, `--seed N` | reproducible output |
| `-n`, `--repeat N` | render the value N times (up to 1048576), each an independent draw, streamed |
| `-n`, `--repeat N` | render the path N times, each an independent draw |
| `--separator S` | between repeated values (default a newline) |
| `--list` | print every path, then exit |
| `--version`, `-h`, `--help` | print, then exit |
@@ -47,9 +52,7 @@ brace, a bracket or a quote, so the two cannot collide (see
`--name value` and `--name=value` both work, a short flag's value attaches or
follows (`-n3`, `-n 3`) and short flags bundle (`-hn 3`) — see
[Decisions](#decisions); flags go anywhere, `--` ends them. Exit codes: `0` success, `1` runtime error (missing
dir, unknown path), `2` misuse — a bad flag, an argument that names neither a
template nor a path, or an inline template that does not compile. From a checkout:
`go run ./cmd/fejkdata …`.
dir, unknown path), `2` misuse. From a checkout: `go run ./cmd/fejkdata …`.
### Your own data
@@ -86,9 +89,6 @@ if err != nil {
}
v, err := f.Fake("sv_SE.address") // "Kungsvägen 68\n379 17 Stockholm"
paths := f.List() // every path Fake accepts, sorted
v, err = f.FakeTemplate("name: {/sv_SE.person.last}") // compile + render in one call
t, err := f.NewTemplate(`{"format":"name: {x}","x":["bosse","lina"]}`) // compile once
v = t.Fake() // render many times, no re-parse
```
| Option | |
@@ -110,8 +110,8 @@ work with no data on disk. A directory is a namespace: each JSON file is a
category named after the file, each subdirectory a dot-path segment, so
`mydata/sv_SE/person.json` is `sv_SE.person` and replaces the shipped one.
Sources merge in order; matching folders combine, any other clash is won by the
last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"` or `/`;
dot-prefixed entries are skipped, so a data directory can also be a checkout.
last loaded. Names may not use `.`, `|`, `(`, `{` or `}`; dot-prefixed entries
are skipped, so a data directory can also be a checkout.
Each locale carries `address`, `color`, `company`, `date`, `email`, `ip`,
`person`, `phone`, `price`, `sentence`, `ssn`, `time`, `url`, `username`,
@@ -141,7 +141,7 @@ Every character is literal except a `{…}` token:
| `{name.field}` | `field` of one draw of `name` ([Correlated fields](#correlated-fields)) |
| `{a\|b}` | one of the named fields, even odds |
| `{fn(args)}` | a builtin ([Functions](#functions)) |
| `{/path}`, `{.path}`, `{..path}` | a node reached from the data root, this file's folder, or the folder above ([References](#references)) |
| `{..path}` | a node reached from the data root ([References](#references)) |
| `{{`, `}}` | a literal `{` or `}` |
```json
@@ -167,8 +167,8 @@ An item of a choice may carry a `weight` (default `1`) to skew its odds:
Renders `070-412 38 91` ten times as often as `08-…`. A string item is weighted by
writing it as `{ "format": "AB", "weight": 3 }`. Rejected at load: a weight that
is negative, non-numeric, `0` (never drawn — remove the item), `1` (the default)
or outside a choice, and a repeated item — `["a", "a", "b"]` is `[{ "format": "a", "weight": 2 }, "b"]`,
is negative, non-numeric, `1` (the default) or outside a choice, a set summing to
zero, and a repeated item — `["a", "a", "b"]` is `[{ "format": "a", "weight": 2 }, "b"]`,
and the error says so.
### Repeat
@@ -180,9 +180,7 @@ many times — each an independent draw — joined by `separator` (default `""`)
{ "format": "{word}", "repeat": 3, "separator": " ", "word": ["foo", "bar", "baz"] }
```
Renders e.g. `bar foo baz`. Rejected at load: a `separator` without a `repeat`,
a `separator` of `""` (the default), and a `repeat` that multiplies to more than
1 048 576 renders along any path of nested repeats.
Renders e.g. `bar foo baz`. A `separator` without a `repeat` is rejected.
### Options and fields
@@ -194,11 +192,9 @@ choice naming its item.
### Functions
A `{name(args)}` token calls a builtin. Arguments are checked at `New`: a bad
count, range, country or expression fails fast; an integer is written plain
(`5`, not `+5` or `05`); bounds are finite; a sample that could only ever emit one
value (`int(5,5)`, `float(1,1,2)`) is rejected naming the text to write instead;
and a length, count or decimal place beyond a sane maximum is rejected, so a
fat-fingered `hex(2000000000)` never tries to allocate gigabytes. Every builtin draws only from the seed — a
count, range, country or expression fails fast, and a length, count or decimal
place beyond a sane maximum is rejected, so a fat-fingered `hex(2000000000)`
never tries to allocate gigabytes. Every builtin draws only from the seed — a
time-based id takes its timestamp from the rng, not the clock — so seeded output
stays reproducible.
@@ -244,11 +240,8 @@ hyphenated field can't be an operand.
{ "format": "{net} x {qty} = {calc(net * qty, 2)}", "net": ["19.99", "5.00"], "qty": ["3", "7"] }
```
Renders e.g. `19.99 x 3 = 59.97`. An operand that can never be a number (`"abc"`,
or a choice of such) is rejected at load, as is a division by a constant zero
(`1/0`, or a fixed `"0"` field); an operand that sometimes is not a number yields
`NaN`, and a division by one that is not constant `Inf` — both print rather than
fail.
Renders e.g. `19.99 x 3 = 59.97`. A non-numeric operand yields `NaN` and a
division by zero `Inf`; both print rather than fail.
### Transforms
@@ -271,25 +264,21 @@ a mix.
### References
A reference renders a node from elsewhere in the data — the path `Fake` takes,
across every loaded source — so one category borrows another. The sigil says
where the path starts, as in a filesystem: `{/en_US.person}` from the data root,
`{.username}` from the folder this file sits in, `{..username}` from the folder
above. `data/sv_SE/email.json` can therefore read its own locale's `username`
without naming `sv_SE`:
`{..path}` renders a node from the **data root** — the path `Fake` takes, across
every loaded source — so one category borrows another, even across folders or
layered directories:
```json
"Hej, {/en_US.person}!"
"Hej, {..en_US.person}!"
```
Renders e.g. `Hej, Pat Smith!`. A reference into a category is held like a
[correlated](#correlated-fields) path — `{.person.first} {.person.last}` name one
person, `{lowercase(.person.first)}` reads that same draw, and `{.person.first}`
beside `{/sv_SE.person.last}` in `sv_SE` is one person too — while a bare
`{/misc.uuid} {/misc.uuid}` is two draws. Rejected at `New`: a path that is
unknown, names a folder, has no folder above, or reads a field not every variant
of a choice carries, and a reference that leads back to its own value, directly,
mutually or through a chain.
[correlated](#correlated-fields) path — `{..sv_SE.person.first} {..sv_SE.person.last}`
name one person, `{lowercase(..sv_SE.person.first)}` reads that same draw — while
a bare `{..misc.uuid} {..misc.uuid}` is two draws. Rejected at `New`: a path that
is unknown, names a folder, or reads a field not every variant of a choice
carries, and a reference that leads back to its own value, directly, mutually or
through a chain.
### Correlated fields
@@ -323,8 +312,8 @@ The sub-fields stay addressable — `Fake("address.place.locality")` renders, an
A name any token reads as a path (`{p.first}`) or as an operand (`{calc(net * 2)}`,
`{uppercase(w)}`) is drawn **once per expansion**, and every other route to it —
a bare `{p}`, a second bare `{w}`, `{/cat.net}`, a nested template rendering
`{/cat.p.last}`, at any depth — is a load error naming the spelling to use. A
a bare `{p}`, a second bare `{w}`, `{..cat.net}`, a nested template rendering
`{..cat.p.last}`, at any depth — is a load error naming the spelling to use. A
name nothing reads that way is drawn each time: `{word} {word}` differs. An
expansion is one render of one format, so each `repeat` iteration and each nested
template draws again.
@@ -341,29 +330,6 @@ then costs about what its output costs: an unweighted pick is O(1) whatever the
list's length, a weighted one O(log n), and long formats, deep nesting and many
tokens add cost in proportion to the output.
## Goals
1. **Valid by construction** — every value passes the check its real consumer
applies; facts that belong together come from one draw, within a value and
across categories.
2. **Text means what it says** — a format renders as written; only `{…}` varies,
random characters included (`{digits(3)}`). One spelling per result; the wrong
one is a load error naming the right one.
3. **Every mistake is a load error** — `New` rejects the data and `NewTemplate`
the inline template; on a loaded generator `Fake` fails only for an unknown
path, and `Template.Fake` cannot fail at all.
4. **Zero to a value in one command** — `go install`, then `fejkdata sv_SE.person`:
no checkout, no flag. Flags are GNU-form (`--seed 42`, `-n 3`) in any position;
the first custom template needs no escape and no option.
5. **Data lives in JSON** — a builtin only for what data can't express.
6. **Reproducible** — seed in, same stream out; no builtin reads a clock.
7. **Zero dependencies** — standard library only.
8. **Docs index the grammar** — every syntax feature is a heading; every example
runs under test and shows its output; a rule is stated once.
9. **Fast enough to be free** — a value renders in about a microsecond and `New`
parses and validates the whole set once upfront, so generating fixtures stays
noise against a test's own runtime.
## Decisions
- **Options and fields share one namespace.** `format`, `weight`, `repeat` and
@@ -379,87 +345,17 @@ tokens add cost in proportion to the output.
naming the double-dash spelling, and `-s=42` is rejected naming both short
spellings: `=` belongs to the long form, and reading `=42` as the value would
make `-d=./x` a directory named `=./x`.
- **An argument is a template by its shape, not by a flag.** A JSON object, array
or string, or a string carrying a `{` token, is an inline template; anything else
is a path. A name may not contain a brace, a bracket or a quote, so a path can
never collide with any of those spellings, and the leading `[` or `"` is gated on
valid JSON so a stray copied bracket never swallows an argument — it names
nothing, and says so. No `--template` flag is needed. Reserving the characters
whole — though only a leading one could collide — keeps one simple name rule
instead of a leading-position special case. The JSON string is what makes the
library's own advice reachable: the error for an object holding only a format
names `"…"`, and that spelling has to work where it is printed.
- **An inline template skips the cycle fence, and only that one.** `New` proves the
loaded tree acyclic, an inline node is a finite tree of its own, and nothing in
the tree can reference it, so no render of it reaches itself. Every other fence
runs over both, from one `checkScope`.
- **An inline template that does not compile is misuse (exit 2), including a
reference that resolves to nothing** — the whole argument is the spelling under
test, and `NewTemplate` compiles, links and validates as one step. An unknown
*path* stays a runtime error (exit 1): there the argument is well-formed and only
the data is absent.
- **A padded JSON argument is rejected, not trimmed.** Padding is the one place the
two readings disagree — a format string renders it, JSON drops it — so the
spelling that renders is named rather than silently chosen.
- **`FakeTemplate` and `NewTemplate` both stay.** They reach the same value but not
at the same cost: `NewTemplate` pays the compile and validation once and renders
many times, `FakeTemplate` is the one-shot call, and `--repeat` is exactly the
case that needs the first. The pair is `regexp.MustCompile` and `regexp.Match`,
not two spellings of one result.
- **The shipped data is embedded, not discovered.** A directory a machine happens
to have would make `--seed 42` machine-dependent. Data still lives in `data/`
as JSON; `--data-path` layers over it.
- **A bare reference draws each time; a reference path is held.** `{/p} {/p}`
is two draws, as `{word} {word}` is, while `{/p.first}` beside a nested template
rendering `{/p.first}` is a load error: a bare token is by contract an
- **A bare reference draws each time; a reference path is held.** `{..p} {..p}`
is two draws, as `{word} {word}` is, while `{..p.first}` beside a nested template
rendering `{..p.first}` is a load error: a bare token is by contract an
independent draw, a path pins its level, and any route into a pinned level from
another expansion could show another row.
- **Reference sigils follow the filesystem.** `/` is the root, `.` this file's
folder, `..` the folder above — what those spellings already mean to anyone who
has typed a path. A locale's files reach each other without naming the locale,
so a folder renames and copies without editing its references.
- **After the first tag, a new fence is a major version.** Data files are the
public API, and one spelling per result grows by tightening, so every fence
invalidates some file. Each such release names the rejected spelling and its
replacement in the changelog and in the load error, and that is the whole
migration: a fence rejects one spelling with one replacement, so the fix is
local to each site. A fence that would need a non-local rewrite ships a
converter with its release instead. Before the first tag there is no
compatibility promise.
- **A `--data-path` override rebinds every reference to the category it
replaces.** References bind against the merged tree, so once shipped data uses
`{.person}`, a consumer's `sv_SE/person.json` is what every shipped reference
into `person` reads, and `New` fails on shipped data the consumer never wrote
when that file lacks a field those references read. Accepted: overriding is the
point of layering, the error names the reference and the field, and the fix is
the consumer's file carrying the fields the shipped tree reads.
- **The repeat cap bounds renders, not bytes.** A repeat, alone or nested, may
ask for at most 1 048 576 renders; how large each render is stays what the data
asked for, so `{hex(1048576)}` repeated to the cap is a terabyte, loaded without
complaint. A byte estimate would need every builtin to declare a width to fence
a shape no data comes near, and the harm lands on the author who wrote it.
- **64-bit targets only.** The gate builds amd64, and the buffer sizing a render
pre-computes (renders × bytes) assumes a 64-bit int; on a 32-bit target it could
overflow and panic.
- **A constant zero divisor is a load error; a divisor that is not constant prints
`Inf`.** `1/0` and a fixed `"0"` field are decidable, so they join the
never-numeric operand as a load error; the fold stops where an operand varies,
so `a/(b*c)` with `b` fixed at `0` and `c` varying loads and prints `Inf` every
draw — catching it needs zero-absorbing algebra for a shape nobody writes.
- **In data, a default written out and a constant spelled as a sample are load
errors.** `weight: 1`, `repeat: 1`, `separator: ""`, `int(5,5)`, `float(1,1,2)`,
`+5` and `05` each spell what a shorter form already spells, so each is rejected
naming that form. The CLI's numbers follow the shell instead: `--seed 007` and
`--repeat +3` are 7 and 3, as every command line reads them.
- **Samples say what they emit, transforms what they do.** `{upper(2)}` is two
letters, `{uppercase(x)}` is `x` upper-cased; one name for both would turn on
whether the argument looks like a number.
- **The performance gate asserts allocations, not wall-clock time.** `AllocsPerRun`
is deterministic across machines, so a ±10% ceiling does not flake under CI load,
while time varies with the machine and its neighbours. A rendering slowdown
almost always costs an allocation too (a lost pre-size, a per-item map, an extra
copy). The benchmark suite (see Development) reports time for a human, not as a
pass/fail gate.
## Development
@@ -472,7 +368,6 @@ docker compose run --rm cover # tests with coverage
docker compose run --rm bench # benchmarks
docker compose run --rm build # compile the library
docker compose run --rm vet # go vet
docker compose run --rm cyclo # cyclomatic complexity over 14 (test files excluded)
docker compose run --rm dev # interactive shell
```
@@ -484,10 +379,8 @@ docker compose run --rm --user "$(id -u):$(id -g)" tidy # go mod tidy
```
Every pull request runs `docker build .` against both the latest and the lowest
supported Go, and must pass before it can be merged — unless it changes none of
the files the build and its tests read, nor the workflow itself, in which case
it's skipped (see [Decisions](#decisions)). That build is the whole gate — vet,
complexity, format check and tests — so run it locally before pushing:
supported Go, and must pass before it can be merged. That build is the whole
gate — vet, format check and tests — so run it locally before pushing:
```sh
docker build . # latest
@@ -500,13 +393,9 @@ GO_VERSION=1.22.12 docker compose run --rm test # the same tests, without the im
```
fejkdata.go Generator, New, options, the embedded data set, List
node.go the node model and JSON -> node compilation
path.go the dotted-path walk, and proving a path resolves
render.go Fake and the recursive renderer (choices, format strings, expansions)
inline.go inline templates: Template, NewTemplate, FakeTemplate, and their compile and link
template.go the {token} grammar: scanning, tokens, operands, validation, compiling a format
hold.go the hold: one draw per expansion for paths and operands, and its fences
reference.go reference sigils, and binding references across the tree
graph.go the render graph: edges, cycles, the repeat bound, tree walks
render.go Fake and the recursive renderer (choices, format strings, paths, held draws)
template.go the {token} grammar: scanning, arms, operands, validation
reference.go {..path} binding across the tree, the render graph, and the walks over it
builtins.go the {name()} function registry and its implementations
calc.go the {calc()} arithmetic evaluator: parser, eval, validation
data.go data loading: fs.FS folders/files -> namespace tree, multi-source merge
@@ -516,4 +405,4 @@ data/ shipped data (JSON), embedded at build: locale folders + a misc
## License
MIT — see [LICENSE](LICENSE). Forked from [github.com/Timewave-AB/fakes](https://github.com/Timewave-AB/fakes).
MIT — see [LICENSE](LICENSE).
hold_test.go → bound_test.go
+34 -43
View File
@@ -92,13 +92,13 @@ func TestReferenceNamingABoundLevelIsRejected(t *testing.T) {
// afresh beside the path that reads its held draw — the same overlap by
// another spelling.
rejected := map[string]string{
"reference names the head": `{"format":"{p.first}|{/cat.p}","p":{"format":"{first}","first":["Anna","Bo"]}}`,
"reference names the leaf": `{"format":"{p.addr}|{/cat.p.addr}","p":{"format":"x","addr":["A","B","C","D"]}}`,
"reference names the head": `{"format":"{p.first}|{..cat.p}","p":{"format":"{first}","first":["Anna","Bo"]}}`,
"reference names the leaf": `{"format":"{p.addr}|{..cat.p.addr}","p":{"format":"x","addr":["A","B","C","D"]}}`,
// The reference need not sit in the format that binds: any field it renders
// reaches the level just the same, however deep.
"reference from a sibling field": `{"format":"{p.first}|{inner}","p":[` +
`{"format":"{first}-{last}","first":"A","last":"1"},{"format":"{first}-{last}","first":"B","last":"2"}],` +
`"inner":"{/cat.p}"}`,
`"inner":"{..cat.p}"}`,
}
for name, file := range rejected {
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{"cat": file})))
@@ -108,7 +108,7 @@ func TestReferenceNamingABoundLevelIsRejected(t *testing.T) {
}
// A reference to anything this format does not bind is untouched.
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{
"cat": `{"format":"{p.first} {/surname}","p":{"format":"{first}","first":["Anna","Bo"]}}`,
"cat": `{"format":"{p.first} {..surname}","p":{"format":"{first}","first":["Anna","Bo"]}}`,
"surname": `["Eriksson","Lindqvist"]`,
}))); err != nil {
t.Errorf("New = %v, want a reference outside the bound level accepted", err)
@@ -120,7 +120,7 @@ func TestCycleReachedOnlyByAPathTokenIsRejected(t *testing.T) {
// cycle walk has to follow it there. A head whose own format names nothing
// would otherwise hide the cycle until render, where it is fatal.
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{
"a": `{"format":"{p.x}","p":{"format":"static","x":"{/a}"}}`,
"a": `{"format":"{p.x}","p":{"format":"static","x":"{..a}"}}`,
})))
if err == nil || !strings.Contains(err.Error(), "reference cycle") {
t.Fatalf("New = %v, want the cycle through {p.x} rejected", err)
@@ -131,7 +131,7 @@ func TestALevelRenderedOnlyByAPathTokenIsHeld(t *testing.T) {
// {p.a} renders q, so it is a route to the level {q.x} holds — even though p's
// own format names nothing.
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{
"thing": `{"format":"{p.a} {q.x}","p":{"format":"static","a":"{/thing.q}"},` +
"thing": `{"format":"{p.a} {q.x}","p":{"format":"static","a":"{..thing.q}"},` +
`"q":{"format":"{x}","x":["1","2"]}}`,
})))
if err == nil || !strings.Contains(err.Error(), "reads a path into") {
@@ -151,14 +151,14 @@ func TestACalcOperandIsHeldAgainstEveryRoute(t *testing.T) {
}{
"a reference beside the operand": {
map[string]string{
"cat": `{"format":"{/cat.net} x 2 = {calc(net * 2, 2)}","net":["10.00","20.00"]}`,
"cat": `{"format":"{..cat.net} x 2 = {calc(net * 2, 2)}","net":["10.00","20.00"]}`,
},
`{/cat.net} renders "net"`,
`{..cat.net} renders "net"`,
},
"a reference one level down": {
map[string]string{
"cat": `{"format":"{calc(net * 2, 2)} {q}","net":["10.00","20.00"],` +
`"q":"{/cat.net}"}`,
`"q":"{..cat.net}"}`,
},
`{q} renders "net"`,
},
@@ -167,7 +167,7 @@ func TestACalcOperandIsHeldAgainstEveryRoute(t *testing.T) {
"a reference to an operand wrapped in a choice": {
map[string]string{
"cat": `{"format":"{calc(n * 2, 2)} {q}","n":{"format":"{v}","v":["1","2"]},` +
`"q":"{/cat.n}"}`,
`"q":"{..cat.n}"}`,
},
`{q} renders "n"`,
},
@@ -176,7 +176,7 @@ func TestACalcOperandIsHeldAgainstEveryRoute(t *testing.T) {
"an operand reaching another operand": {
map[string]string{
"cat": `{"format":"{calc(a + b, 0)}","a":{"format":"{x}","x":["1","2"]},` +
`"b":"{/cat.a}"}`,
`"b":"{..cat.a}"}`,
},
`calc operand "b" renders "a"`,
},
@@ -185,15 +185,15 @@ func TestACalcOperandIsHeldAgainstEveryRoute(t *testing.T) {
// rejected, so the reference spelling has to be.
"a reference into the operand": {
map[string]string{
"cat": `{"format":"{calc(net * 2, 2)}|{/cat.net.v}",` +
"cat": `{"format":"{calc(net * 2, 2)}|{..cat.net.v}",` +
`"net":{"format":"{v}","v":["10.00","20.00"]}}`,
},
`{/cat.net.v} renders "net"`,
`{..cat.net.v} renders "net"`,
},
"a reference into the operand one level down": {
map[string]string{
"cat": `{"format":"{calc(net * 2, 2)}|{q}","net":{"format":"{v}","v":["10.00","20.00"]},` +
`"q":"{/cat.net.v}"}`,
`"q":"{..cat.net.v}"}`,
},
`{q} renders "net"`,
},
@@ -203,7 +203,7 @@ func TestACalcOperandIsHeldAgainstEveryRoute(t *testing.T) {
"a violation on the second of two held heads": {
map[string]string{
"cat": `{"format":"{calc(a + b, 0)} {w}","a":{"format":"{x}","x":["1","2"]},` +
`"b":{"format":"{y}","y":["3","4"]},"w":"{/cat.b}"}`,
`"b":{"format":"{y}","y":["3","4"]},"w":"{..cat.b}"}`,
},
`{w} renders "b"`,
},
@@ -230,24 +230,24 @@ func TestACalcOperandIsHeldAgainstEveryRoute(t *testing.T) {
"a reference to a sibling the operand never renders": {
"cat": `{"format":"{calc(net * 2, 2)} {unit}",` +
`"net":{"format":"{v}","v":["1","2"],"spare":["kg","lb"]},` +
`"unit":"{/cat.net.spare}"}`,
`"unit":"{..cat.net.spare}"}`,
},
// Two operands drawing from one source are two names, so two draws: each is
// held under its own name and shown once, and neither can disagree.
"two operands sharing one source": {
"die": `["1","2","3","4","5","6"]`,
"cat": `{"format":"{d1} + {d2} = {calc(d1 + d2, 0)}","d1":"{/die}",` +
`"d2":"{/die}"}`,
"cat": `{"format":"{d1} + {d2} = {calc(d1 + d2, 0)}","d1":"{..die}",` +
`"d2":"{..die}"}`,
},
// Likewise a reference drawing from what the operand draws from: {/common}
// Likewise a reference drawing from what the operand draws from: {..common}
// and net are two names, not two spellings of one field.
"a reference to what an operand renders through": {
"common": `["1","2"]`,
"cat": `{"format":"{calc(net * 2, 2)} {/common}","net":"{/common}"}`,
"cat": `{"format":"{calc(net * 2, 2)} {..common}","net":"{..common}"}`,
},
// A fixed string cannot disagree with itself, so it needs no fence.
"a literal operand named twice": {
"cat": `{"format":"{calc(n * 2, 0)} {/cat.n}","n":"5"}`,
"cat": `{"format":"{calc(n * 2, 0)} {..cat.n}","n":"5"}`,
},
// One node reached twice while walking the operand: the walk must not
// revisit it, and the repeat is not a second route to anything.
@@ -269,11 +269,11 @@ func TestAPathReachesEveryVariantItMightDraw(t *testing.T) {
// to a held level disagrees silently.
rejected := map[string]struct{ file, want string }{
"a cycle in a later variant": {
`{"format":"{p.x}","p":[{"format":"h","x":"safe"},{"format":"h","x":"{/cat}"}]}`,
`{"format":"{p.x}","p":[{"format":"h","x":"safe"},{"format":"h","x":"{..cat}"}]}`,
"reference cycle",
},
"a second route in a later variant": {
`{"format":"{p.x} {q.y}","p":[{"format":"h","x":"safe"},{"format":"h","x":"{/cat.q}"}],` +
`{"format":"{p.x} {q.y}","p":[{"format":"h","x":"safe"},{"format":"h","x":"{..cat.q}"}],` +
`"q":{"format":"{y}","y":["1","2"]}}`,
"reads a path into",
},
@@ -292,9 +292,9 @@ func TestALevelAPathNeverRendersIsAccepted(t *testing.T) {
// path at all. Both orders must load.
accepted := map[string]string{
"reference in the head's own format": `{"format":"{p.first} {q.a}",` +
`"p":{"format":"{first} {/thing.q}","first":["A","B"]},"q":{"format":"{a}","a":["1","2"]}}`,
`"p":{"format":"{first} {..thing.q}","first":["A","B"]},"q":{"format":"{a}","a":["1","2"]}}`,
"the mirror shape": `{"format":"{p.first} {q.a}",` +
`"p":{"format":"{first}","first":["A","B"]},"q":{"format":"{a} {/thing.p}","a":["1","2"]}}`,
`"p":{"format":"{first}","first":["A","B"]},"q":{"format":"{a} {..thing.p}","a":["1","2"]}}`,
}
for name, file := range accepted {
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{"thing": file}))); err != nil {
@@ -309,9 +309,9 @@ func TestADeepDiamondChainLoads(t *testing.T) {
files := map[string]string{"l0": `"x"`}
for i := 1; i <= 30; i++ {
files[fmt.Sprintf("l%d", i)] = fmt.Sprintf(
`{"format":"{a}{b}","a":"{/l%d}","b":"{/l%d}"}`, i-1, i-1)
`{"format":"{a}{b}","a":"{..l%d}","b":"{..l%d}"}`, i-1, i-1)
}
files["thing"] = `{"format":"{p.first} {/l30}","p":{"format":"x","first":["A","B"]}}`
files["thing"] = `{"format":"{p.first} {..l30}","p":{"format":"x","first":["A","B"]}}`
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, files))); err != nil {
t.Fatalf("New = %v, want a deep diamond chain to load", err)
}
@@ -323,7 +323,7 @@ func TestASharedNodeIsWalkedOnce(t *testing.T) {
// the shared node once per route.
f, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{
"cat": `{"format":"{p.first}|{q}","p":{"format":"{first}","first":["Anna","Bo"]},` +
`"q":{"format":"{a}{b}","a":"{/shared}","b":"{/shared}"}}`,
`"q":{"format":"{a}{b}","a":"{..shared}","b":"{..shared}"}}`,
"shared": `"x"`,
})), WithSeed(1))
if err != nil {
@@ -351,7 +351,7 @@ func TestReferenceToAMatchingStringIsAccepted(t *testing.T) {
// held one. Two unrelated literals that merely spell the same text must not
// read as the same node — the trap when comparing a value type.
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{
"cat": `{"format":"{p.city} {/other.tag}","p":{"format":"{city}","city":"Stockholm"}}`,
"cat": `{"format":"{p.city} {..other.tag}","p":{"format":"{city}","city":"Stockholm"}}`,
"other": `{"format":"x","tag":"Stockholm"}`,
}))); err != nil {
t.Fatalf("New = %v, want a reference to a matching string accepted", err)
@@ -642,11 +642,12 @@ func TestDeepPathsUnderOneHeadStayIndependentWhereTheyDiverge(t *testing.T) {
}
func TestEmptyPathSegmentIsRejected(t *testing.T) {
// "{a.}" and "{a..b}" are unfinished paths, and the segment naming nothing is
// reported as that rather than as a missing field. An empty name is rejected
// where it is authored, so no data can make these resolve.
// "{a.}", "{.b}" and "{a..b}" are unfinished paths, and the segment naming
// nothing is reported as that rather than as a missing field. An empty name is
// rejected where it is authored, so no data can make these resolve.
rejected := map[string]string{
"trailing dot": `{"format":"[{a.}]","a":"x"}`,
"leading dot": `{"format":"[{.b}]","a":{"format":"{b}","b":"V"}}`,
"double dot": `{"format":"[{a..b}]","a":"x"}`,
}
for name, file := range rejected {
@@ -666,7 +667,7 @@ func TestCycleThroughAPathTokenIsRejected(t *testing.T) {
// must be caught at New. Reaching render would be fatal: the recursion never
// terminates, and a stack overflow cannot be recovered.
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{
"a": `{"format":"{p.x}","p":{"format":"{x}","x":"{/a}"}}`,
"a": `{"format":"{p.x}","p":{"format":"{x}","x":"{..a}"}}`,
})))
if err == nil || !strings.Contains(err.Error(), "reference cycle") {
t.Fatalf("New = %v, want the cycle through {p.x} rejected", err)
@@ -711,13 +712,3 @@ func TestRepeatedBareTokenOfAHeldNameIsRejected(t *testing.T) {
}
}
}
func TestReadFieldPanicsOnAnUnheldPath(t *testing.T) {
tm, ok := compiled(t, `{"format":"{w}","w":{"format":"{x}","x":"1"}}`).(*template)
if !ok {
t.Fatal("not a template")
}
mustPanic(t, "unheld arm with a path", func() {
readField(engine(1).rand, tm, nil, arm{name: "w.x", key: "w", tail: []string{"x"}, path: "w.x"})
})
}
builtins.go
+23 -59
View File
@@ -2,7 +2,6 @@ package fejkdata
import (
"encoding/base64"
"errors"
"fmt"
"math"
"strconv"
@@ -10,12 +9,11 @@ import (
"unicode"
)
// maxLen caps sample output lengths (hex, nanoid, base64, digits, upper, lower)
// and maxDecimals float/calc decimal places, so a fat-fingered or overflowing
// argument fails at New instead of trying to allocate gigabytes — or panicking —
// at render.
// maxLen caps sample output lengths (hex, nanoid, base64) and maxDecimals caps
// float/calc decimal places, so a fat-fingered or overflowing argument fails at
// New instead of trying to allocate gigabytes — or panicking — at render.
const (
maxLen = 1 << 20
maxLen = 1 << 20 // 1,048,576 chars/bytes
maxDecimals = 1024
)
@@ -130,10 +128,12 @@ func transformArg(fields map[string]node, a []string) error {
return err
}
if isRef(leaf) {
_, _, err := refShape(leaf)
return err
if leaf == refPrefix {
return fmt.Errorf("reference has no path")
}
return checkArm(leaf, fields, false)
return nil
}
return checkArm(leaf, fields)
}
func transformOperand(a []string) []string {
@@ -177,7 +177,7 @@ var asciiFolds = map[rune]string{
func asciiFold(s string) string {
var b strings.Builder
for _, r := range s {
if r <= unicode.MaxASCII {
if r < unicode.MaxASCII {
b.WriteRune(r)
} else {
b.WriteString(asciiFolds[r])
@@ -222,31 +222,10 @@ func randChars(r rng, n int, alphabet string) string {
return string(b)
}
// plainInt parses an integer arg written the one way: no sign, no leading zero.
func plainInt(s string) (int, error) {
n, err := strconv.Atoi(s)
if errors.Is(err, strconv.ErrRange) {
return 0, fmt.Errorf("%q is past the integer range: %w", s, err)
}
if err != nil {
return 0, fmt.Errorf("%q is not an integer", s)
}
if strconv.Itoa(n) != s {
return 0, fmt.Errorf("%q is not a plain integer; write %d", s, n)
}
return n, nil
}
func posIntArg(_ map[string]node, a []string) error {
n, err := plainInt(a[0])
if errors.Is(err, strconv.ErrRange) {
return fmt.Errorf("count %q exceeds the maximum %d", a[0], maxLen)
}
if err != nil {
return fmt.Errorf("count %w", err)
}
if n < 1 {
return fmt.Errorf("count %q must be positive", a[0])
n, err := strconv.Atoi(a[0])
if err != nil || n < 1 {
return fmt.Errorf("count %q must be a positive integer", a[0])
}
if n > maxLen {
return fmt.Errorf("count %d exceeds the maximum %d", n, maxLen)
@@ -255,20 +234,14 @@ func posIntArg(_ map[string]node, a []string) error {
}
func intRangeArgs(_ map[string]node, a []string) error {
lo, err := plainInt(a[0])
if err != nil {
return fmt.Errorf("int(min,max): min %w", err)
}
hi, err := plainInt(a[1])
if err != nil {
return fmt.Errorf("int(min,max): max %w", err)
lo, e1 := strconv.Atoi(a[0])
hi, e2 := strconv.Atoi(a[1])
if e1 != nil || e2 != nil {
return fmt.Errorf("int(min,max) needs integer args, got %q,%q", a[0], a[1])
}
if lo > hi {
return fmt.Errorf("int(min,max): min %d > max %d", lo, hi)
}
if lo == hi {
return fmt.Errorf("int(%d,%d) is the constant %d; write it as text", lo, hi, lo)
}
if uint64(hi)-uint64(lo) >= uint64(math.MaxInt64) { // span hi-lo+1 would overflow int -> IntN panic
return fmt.Errorf("int(min,max): range %d..%d is too wide", lo, hi)
}
@@ -278,28 +251,19 @@ func intRangeArgs(_ map[string]node, a []string) error {
func floatArgs(_ map[string]node, a []string) error {
lo, e1 := strconv.ParseFloat(a[0], 64)
hi, e2 := strconv.ParseFloat(a[1], 64)
if e1 != nil || e2 != nil {
return fmt.Errorf("float(min,max,dp) needs numeric bounds, got %q,%q", a[0], a[1])
}
dp, err := plainInt(a[2])
if err != nil {
return fmt.Errorf("float(min,max,dp): decimals %w", err)
}
if math.IsNaN(lo) || math.IsNaN(hi) || math.IsInf(lo, 0) || math.IsInf(hi, 0) {
return fmt.Errorf("float(min,max,dp) needs finite bounds, got %q,%q", a[0], a[1])
dp, e3 := strconv.Atoi(a[2])
if e1 != nil || e2 != nil || e3 != nil {
return fmt.Errorf("float(min,max,dp) needs numeric args, got %q,%q,%q", a[0], a[1], a[2])
}
if lo > hi {
return fmt.Errorf("float(min,max,dp): min %v > max %v", lo, hi)
}
if dp < 0 || dp > maxDecimals {
return fmt.Errorf("float(min,max,dp): decimals %d out of range 0..%d", dp, maxDecimals)
}
if lo == hi {
return fmt.Errorf("float(%s,%s,%d) is the constant %q; write it as text", a[0], a[1], dp, strconv.FormatFloat(lo, 'f', dp, 64))
}
if math.IsInf(hi-lo, 0) { // an overflowing span would render as "+Inf"
return fmt.Errorf("float(min,max,dp): range %v..%v is too wide", lo, hi)
}
if dp < 0 || dp > maxDecimals {
return fmt.Errorf("float(min,max,dp): decimals %d out of range 0..%d", dp, maxDecimals)
}
return nil
}
calc.go
+3 -100
View File
@@ -70,114 +70,18 @@ func checkCalc(fields map[string]node, args []string) error {
return fmt.Errorf("calc(%q): %w", args[0], err)
}
for _, name := range calcVars(expr) {
operand, ok := fields[name]
if !ok {
if _, ok := fields[name]; !ok {
return fmt.Errorf("calc(%q): no field %q", args[0], name)
}
if text, never := neverNumeric(operand); never {
return fmt.Errorf("calc(%q): operand %q is never a number: it renders %q", args[0], name, text)
}
}
if divisor, zero := constantZeroDivisor(expr, fields); zero {
return fmt.Errorf("calc(%q) divides by %s, which is always zero", args[0], divisor)
}
if len(args) == 2 {
dp, err := plainInt(args[1])
if err != nil {
return fmt.Errorf("calc decimals %w", err)
}
if dp < 0 || dp > maxDecimals {
return fmt.Errorf("calc decimals %d must be in 0..%d", dp, maxDecimals)
if dp, err := strconv.Atoi(args[1]); err != nil || dp < 0 || dp > maxDecimals {
return fmt.Errorf("calc decimals %q must be an integer in 0..%d", args[1], maxDecimals)
}
}
return nil
}
// constantZeroDivisor finds a division whose right side is a constant zero: number
// literals and fixed operands folded, anything that varies left unknown.
func constantZeroDivisor(n calcNode, fields map[string]node) (string, bool) {
switch n := n.(type) {
case calcNeg:
return constantZeroDivisor(n.x, fields)
case calcBin:
if n.op == '/' {
if v, known := constantValue(n.r, fields); known && v == 0 {
return calcText(n.r), true
}
}
if d, zero := constantZeroDivisor(n.l, fields); zero {
return d, true
}
return constantZeroDivisor(n.r, fields)
}
return "", false
}
// constantValue evaluates an expression whose every operand is fixed.
func constantValue(n calcNode, fields map[string]node) (float64, bool) {
switch n := n.(type) {
case calcNum:
return float64(n), true
case calcVar:
t, ok := fields[string(n)].(*template)
if !ok || !t.fixed || t.repeat > 1 {
return 0, false
}
v, err := strconv.ParseFloat(strings.TrimSpace(t.lit), 64)
return v, err == nil
case calcNeg:
v, ok := constantValue(n.x, fields)
return -v, ok
case calcBin:
l, lok := constantValue(n.l, fields)
r, rok := constantValue(n.r, fields)
if !lok || !rok {
return 0, false
}
return calcBin{n.op, calcNum(l), calcNum(r)}.eval(nil), true
}
return 0, false
}
// calcText spells an expression node the way an author would read it.
func calcText(n calcNode) string {
switch n := n.(type) {
case calcNum:
return strconv.FormatFloat(float64(n), 'f', -1, 64)
case calcVar:
return string(n)
case calcNeg:
return "-" + calcText(n.x)
case calcBin:
return "(" + calcText(n.l) + " " + string(n.op) + " " + calcText(n.r) + ")"
}
return "?"
}
// neverNumeric reports a node no render of which is a number: fixed text that does
// not parse, or a choice of only such items. text is one such render.
func neverNumeric(n node) (text string, never bool) {
switch n := n.(type) {
case *template:
if !n.fixed || n.repeat > 1 {
return "", false
}
if _, err := strconv.ParseFloat(strings.TrimSpace(n.lit), 64); err != nil {
return n.lit, true
}
case *choice:
for _, it := range n.items {
t, itemNever := neverNumeric(it)
if !itemNever {
return "", false
}
text = t
}
return text, true
}
return "", false
}
// calcPrep parses the expression and decimals once, at compile time, and places each
// operand name at the position expand will read it into. checkCalc proved both args
// valid, so no step here can fail; dp -1 prints the minimal form.
@@ -310,7 +214,6 @@ func (p *calcParser) binary(next func() (calcNode, error), ops ...byte) (calcNod
}
}
// factor is a table-shaped scanner, one case per token kind, kept whole on purpose.
func (p *calcParser) factor() (calcNode, error) {
p.space()
if p.pos >= len(p.rs) {
calc_test.go
+4 -33
View File
@@ -72,17 +72,13 @@ func TestCalcFields(t *testing.T) {
}
}
// TestCalcNonNumericIsNaN pins the never-fail rule: a field that sometimes does
// not render to a number becomes NaN then, which propagates and prints visibly.
// TestCalcNonNumericIsNaN pins the never-fail rule: a field that doesn't render
// to a number becomes NaN, which propagates and prints visibly.
func TestCalcNonNumericIsNaN(t *testing.T) {
f := engine(1)
for i := 0; i < 50; i++ {
if got := mustRender(t, f, `{"format":"{calc(x * 2)}","x":["abc","1"]}`); got == "NaN" {
return
if got := mustRender(t, engine(1), `{"format":"{calc(x * 2)}","x":"abc"}`); got != "NaN" {
t.Fatalf("calc over non-numeric field = %q, want NaN", got)
}
}
t.Fatal("calc over a sometimes non-numeric field never printed NaN in 50 draws")
}
// TestCalcReproducible pins that a calc over a random operand stays seed-stable.
func TestCalcReproducible(t *testing.T) {
@@ -245,28 +241,3 @@ func TestCalcOverANeverNumericOperandIsRejected(t *testing.T) {
}
}
}
func TestCalcConstantZeroDivisorIsRejected(t *testing.T) {
for src, want := range map[string]string{
`"{calc(1/0)}"`: "divides by 0",
`"{calc(2/(1-1))}"`: "divides by (1 - 1)",
`{"format":"{calc(x/y)}","x":"1","y":"0"}`: "divides by y",
`{"format":"{calc(x/(y*2))}","x":"1","y":" 0 "}`: "divides by (y * 2)",
`{"format":"{calc((a/0)+b)}","a":"1","b":"2"}`: "divides by 0",
`{"format":"{calc(a/-0)}","a":"1"}`: "divides by -0",
} {
if _, err := compile(parse(t, src)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("compile(%s) = %v, want the constant zero divisor rejected naming %q", src, err, want)
}
}
if _, err := compile(parse(t, `{"format":"{calc(a/(b*c))}","a":"1","b":"0","c":["1","2"]}`)); err != nil {
t.Errorf("compile(a/(b*c)) = %v, want a divisor that varies accepted", err)
}
f := engine(1)
for i := 0; i < 50; i++ {
if got := mustRender(t, f, `{"format":"{calc(x/y)}","x":"1","y":["0","1"]}`); got == "+Inf" {
return
}
}
t.Fatal("a sometimes-zero divisor never printed +Inf in 50 draws")
}
cmd/fejkdata/main.go
+33 -103
View File
@@ -8,8 +8,6 @@
package main
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
@@ -21,30 +19,21 @@ import (
"gitea.larvit.se/larvit/fejkdata"
)
const usage = `Usage: fejkdata [flags] <path|template>
const usage = `Usage: fejkdata [flags] <path>
<path> a category, or a dotted path into one (person, person.last)
<template> a format string or JSON value to render inline, e.g.
'name: {/sv_SE.person.last}' or '{"format":"{x}","x":["bosse","lina"]}'
An argument containing a { token, or a JSON object, array or string, is a
template; any other argument is a path (a path never contains a brace, a bracket
or a quote). Templates reach the data by reference from the root —
{/sv_SE.person.last} — whether the data is shipped or layered with --data-path. An
argument carrying a bracket, a closing brace or a quote but no valid JSON names
neither.
-d, --data-path D a data directory to layer over the shipped data (repeatable; last wins on a clash)
-h, --help print this help, then exit
--list list the paths the data offers, then exit
--no-shipped-data load only the --data-path directories
-n, --repeat N render the value N times, 1..1048576 (default 1)
-n, --repeat N render the path N times (default 1)
-s, --seed N seed for reproducible output
--separator S string between repeated values (default newline)
--version print the version, then exit
Flags may come before or after <path|template>; -- ends the flags. A short flag's
value attaches or follows (-n3, -n 3); short flags bundle (-hn 3).
Flags may come before or after <path>; -- ends the flags. A short flag's value
attaches or follows (-n3, -n 3); short flags bundle (-hn 3).
`
type invocation struct {
@@ -54,11 +43,9 @@ type invocation struct {
noShipped bool
paths []string
repeat int
repeatSet bool
seed uint64
seeded bool
separator string
separatorSet bool
version bool
}
@@ -76,10 +63,10 @@ var flagDefs = []flagDef{
{"no-shipped-data", "", false, func(in *invocation, _ string) error { in.noShipped = true; return nil }},
{"repeat", "n", true, func(in *invocation, v string) error {
n, err := strconv.Atoi(v)
if err != nil || n < 1 || n > fejkdata.MaxRepeat {
return fmt.Errorf("--repeat needs an integer in 1..%d, got %q", fejkdata.MaxRepeat, v)
if err != nil || n < 1 {
return fmt.Errorf("--repeat needs a positive integer, got %q", v)
}
in.repeat, in.repeatSet = n, true
in.repeat = n
return nil
}},
{"seed", "s", true, func(in *invocation, v string) error {
@@ -90,7 +77,7 @@ var flagDefs = []flagDef{
in.seed, in.seeded = n, true
return nil
}},
{"separator", "", true, func(in *invocation, v string) error { in.separator, in.separatorSet = v, true; return nil }},
{"separator", "", true, func(in *invocation, v string) error { in.separator = v; return nil }},
{"version", "", false, func(in *invocation, _ string) error { in.version = true; return nil }},
}
@@ -137,8 +124,8 @@ func splitFlags(arg string) ([]flagArg, error) {
}
var flags []flagArg
letters := arg[1:]
for i, r := range letters {
letter := string(r)
for i := 0; i < len(letters); i++ {
letter := letters[i : i+1]
def := flagByShort(letter)
if def == nil {
return nil, fmt.Errorf("unknown flag -%s", letter)
@@ -147,7 +134,7 @@ func splitFlags(arg string) ([]flagArg, error) {
flags = append(flags, flagArg{def: def})
continue
}
rest := letters[i+len(letter):]
rest := letters[i+1:]
if strings.HasPrefix(rest, "=") {
return nil, fmt.Errorf("-%s takes its value attached (-%s%s) or next (-%s %s); = belongs to --%s=%s",
letter, letter, rest[1:], letter, rest[1:], def.long, rest[1:])
@@ -196,22 +183,18 @@ func parseArgs(argv []string) (invocation, error) {
return in, nil
}
// check rejects a flag combination or an argument that cannot run, and reports
// what the argument names, so its shape is settled before any data is read.
func (in invocation) check() (argKind, error) {
// check rejects a flag combination that cannot run.
func (in invocation) check() error {
if in.noShipped && len(in.dirs) == 0 {
return errors.New("--no-shipped-data needs at least one --data-path")
}
if in.list && len(in.paths) > 0 {
return argPath, errors.New("--list takes no path")
return errors.New("--list takes no path")
}
if in.list && (in.repeatSet || in.separatorSet) {
return argPath, errors.New("--list takes no --repeat or --separator")
if !in.list && len(in.paths) != 1 {
return fmt.Errorf("expected one path, got %d", len(in.paths))
}
if in.list {
return argPath, nil
}
if len(in.paths) != 1 {
return argPath, fmt.Errorf("expected one path or template, got %d", len(in.paths))
}
return classify(in.paths[0])
return nil
}
func (in invocation) options() []fejkdata.Option {
@@ -228,63 +211,17 @@ func (in invocation) options() []fejkdata.Option {
return opts
}
// write streams the argument's renders to w, repeat of them joined by the
// separator and ended by a newline. A value that renders once renders every time,
// so a render failure comes before anything is written; a write failure surfaces
// from Flush, bufio keeping the first one.
func (in invocation) write(f *fejkdata.Generator, kind argKind, w io.Writer) error {
arg := in.paths[0]
draw := func() (string, error) { return f.Fake(arg) }
if kind == argTemplate {
t, err := f.NewTemplate(arg)
// values renders the path repeat times, joined by the separator.
func (in invocation) values(f *fejkdata.Generator) (string, error) {
vals := make([]string, in.repeat)
for i := range vals {
v, err := f.Fake(in.paths[0])
if err != nil {
return templateError{err}
return "", err
}
draw = func() (string, error) { return t.Fake(), nil }
vals[i] = v
}
out := bufio.NewWriter(w)
for i := 0; i < in.repeat; i++ {
v, err := draw()
if err != nil {
return err
}
if i > 0 {
out.WriteString(in.separator)
}
out.WriteString(v)
}
out.WriteString("\n")
return out.Flush()
}
// templateError marks a render failure that is the argument's own fault — an
// inline template that does not compile. run reports it as misuse (exit 2, with a
// pointer to --help), unlike an unknown path, which is a runtime error (exit 1).
type templateError struct{ error }
func (e templateError) Unwrap() error { return e.error }
type argKind int
const (
argPath argKind = iota
argTemplate
)
// classify reads what a positional argument names by its shape: a { token, or a
// JSON object, array or string, is an inline template; anything else is a path.
func classify(arg string) (argKind, error) {
if strings.ContainsRune(arg, '{') || (isJSONStart(strings.TrimSpace(arg)) && json.Valid([]byte(arg))) {
return argTemplate, nil
}
if i := strings.IndexAny(arg, `[]}"`); i >= 0 {
return argPath, fmt.Errorf("%q holds a %q, which no path may, and it is not valid JSON, so it names no template either", arg, arg[i:i+1])
}
return argPath, nil
}
func isJSONStart(arg string) bool {
return strings.HasPrefix(arg, "[") || strings.HasPrefix(arg, `"`)
return strings.Join(vals, in.separator), nil
}
func main() { os.Exit(run(os.Args[1:], os.Stdout, os.Stderr)) }
@@ -303,14 +240,10 @@ func run(args []string, stdout, stderr io.Writer) int {
fmt.Fprintln(stdout, "fejkdata "+buildVersion())
return 0
}
kind, err := in.check()
if err != nil {
if err := in.check(); err != nil {
return misuse(stderr, err)
}
f, err := fejkdata.New(in.options()...)
if errors.Is(err, fejkdata.ErrNoData) {
return misuse(stderr, errors.New("--no-shipped-data needs at least one --data-path"))
}
if err != nil {
fmt.Fprintln(stderr, err)
return 1
@@ -321,20 +254,17 @@ func run(args []string, stdout, stderr io.Writer) int {
}
return 0
}
if err := in.write(f, kind, stdout); err != nil {
var te templateError
if errors.As(err, &te) {
return misuse(stderr, te.error)
}
out, err := in.values(f)
if err != nil {
fmt.Fprintln(stderr, err)
return 1
}
fmt.Fprintln(stdout, out)
return 0
}
func misuse(stderr io.Writer, err error) int {
// A library error already names the program, so the prefix is not doubled.
fmt.Fprintf(stderr, "fejkdata: %s\ntry 'fejkdata --help'\n", strings.TrimPrefix(err.Error(), "fejkdata: "))
fmt.Fprintf(stderr, "fejkdata: %v\ntry 'fejkdata --help'\n", err)
return 2
}
cmd/fejkdata/main_test.go
+1 -142
View File
@@ -2,9 +2,6 @@ package main
import (
"bytes"
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
@@ -93,7 +90,7 @@ func TestRunShortFlagValues(t *testing.T) {
{[]string{"-s=42", "-d", svSE, "address"}, "-s42"},
{[]string{"-s=42", "-d", svSE, "address"}, "--seed=42"},
{[]string{"-d=" + svSE, "address"}, "--data-path="},
{[]string{"-nd", "3", "-d", svSE, "word"}, `--repeat needs an integer in 1..1048576, got "d"`},
{[]string{"-nd", "3", "-d", svSE, "word"}, `--repeat needs a positive integer, got "d"`},
{[]string{"--seed=", "-d", svSE, "word"}, `--seed needs an unsigned integer, got ""`},
} {
code, out, errb := runOut(c.args...)
@@ -302,92 +299,6 @@ func TestRunShippedDataByDefault(t *testing.T) {
}
}
func TestClassify(t *testing.T) {
for arg, want := range map[string]argKind{
"sv_SE.person": argPath,
"person.last": argPath,
"name: {x}": argTemplate, // a { token: a path can never carry a brace
`{"format":"x"}`: argTemplate,
`["a","b"]`: argTemplate, // a JSON array carries no brace
`[1, 2]`: argTemplate,
` ["a","b"]`: argTemplate, // padding is the template's own error, not a shape verdict
`"hello"`: argTemplate, // a JSON string, the spelling a format-only object names
} {
got, err := classify(arg)
if err != nil || got != want {
t.Errorf("classify(%q) = %v, %v; want %v", arg, got, err, want)
}
}
for arg, want := range map[string]string{
"[abc]": `holds a "["`,
"[abc].field": `holds a "["`,
"x[1]": `holds a "["`,
"a]b": `holds a "]"`,
"a}b": `holds a "}"`,
`"abc`: `holds a "\""`,
`"a]b`: `holds a "\""`, // the opener the reader typed, not the bracket behind it
} {
_, err := classify(arg)
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("classify(%q) = %v; want it rejected naming %s", arg, err, want)
}
}
}
func TestUsageReferencesResolve(t *testing.T) {
for _, token := range regexp.MustCompile(`\{/[^}]+\}`).FindAllString(usage, -1) {
code, out, errb := runOut("--seed", "1", token)
if code != 0 || strings.TrimSpace(out) == "" {
t.Errorf("usage advertises %s: run = %d, %q, stderr %q", token, code, out, errb)
}
}
}
func TestRunShapeMisuseBeforeLoad(t *testing.T) {
code, _, errb := runOut("--no-shipped-data", "[abc]")
if code != 2 || !strings.Contains(errb, "[abc]") || strings.Contains(errb, "--data-path") {
t.Fatalf("shape misuse with no data = %d, %q; want the shape error before any load", code, errb)
}
}
func TestRunInlineTemplate(t *testing.T) {
code, out, errb := runOut("--seed", "1", "name: {/sv_SE.person.last}")
if code != 0 || !strings.HasPrefix(out, "name: ") || strings.Contains(out, "{") {
t.Fatalf("inline format string = %d, %q, stderr %q", code, out, errb)
}
code, out, errb = runOut("--seed", "1", `{"format":"name: {x}","x":["bosse","lina"]}`)
if code != 0 || (out != "name: bosse\n" && out != "name: lina\n") {
t.Fatalf("inline JSON template = %d, %q, want one name, stderr %q", code, out, errb)
}
code, out, errb = runOut("--seed", "1", `"name: {/sv_SE.person.last}"`)
if code != 0 || !strings.HasPrefix(out, "name: ") || strings.Contains(out, "{") {
t.Fatalf("inline JSON string = %d, %q, stderr %q", code, out, errb)
}
code, out, errb = runOut("--seed", "1", "-n", "2", `{digits(1)}`)
if code != 0 || len(strings.Split(strings.TrimRight(out, "\n"), "\n")) != 2 {
t.Fatalf("inline template with --repeat = %d, %q, stderr %q", code, out, errb)
}
}
func TestRunTemplateMisuse(t *testing.T) {
for arg, want := range map[string]string{
"{bad": "unterminated",
"[red,green]": "names no template either",
`{"format":"x"}`: "is a string",
"{/no.such.path}": "no entry",
"x[1]": "names no template either",
` ["a","b"] `: "may not be padded",
} {
code, out, errb := runOut("--seed", "1", arg)
if code != 2 || out != "" || !strings.Contains(errb, "try 'fejkdata --help'") || !strings.Contains(errb, want) {
t.Errorf("run(%q) = %d, %q, %q; want misuse naming %q and --help", arg, code, out, errb, want)
}
if strings.Contains(errb, "fejkdata: fejkdata:") {
t.Errorf("run(%q) doubled the program prefix: %q", arg, errb)
}
}
}
func TestRunNoShippedData(t *testing.T) {
code, list, errb := runOut("--no-shipped-data", "-d", svSE, "--list")
if code != 0 {
@@ -405,55 +316,3 @@ func TestRunNoShippedData(t *testing.T) {
t.Errorf("--no-shipped-data alone = %d, %q, want misuse naming --data-path", code, errb)
}
}
func TestRunRepeatIsBounded(t *testing.T) {
for _, args := range [][]string{
{"--repeat", "9223372036854775807", "sv_SE.word"},
{"--repeat", "1048577", "sv_SE.word"},
{"-n", "99999999999", "sv_SE.word"},
} {
code, out, errb := runOut(args...)
if code != 2 || out != "" || !strings.Contains(errb, "1..1048576") {
t.Errorf("run(%v) = %d, %q, %q, want misuse naming the range", args, code, out, errb)
}
}
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "x.json"), []byte(`"x"`), 0o644); err != nil {
t.Fatal(err)
}
code, out, errb := runOut("--no-shipped-data", "-d", dir, "--repeat", "1048576", "--separator", "", "x")
if code != 0 || len(out) != 1048576+1 {
t.Errorf("repeat at the cap = %d, %d bytes, stderr %q; want every render streamed", code, len(out), errb)
}
}
func TestRunListTakesNoRepeatOrSeparator(t *testing.T) {
for _, args := range [][]string{{"--list", "-n", "3"}, {"--list", "--separator", ","}} {
code, _, errb := runOut(args...)
if code != 2 || !strings.Contains(errb, "--list takes no") {
t.Errorf("run(%v) = %d, %q, want misuse", args, code, errb)
}
}
}
func TestRunUnknownFlagIsNamedByRune(t *testing.T) {
code, _, errb := runOut("-ä", "sv_SE.word")
if code != 2 || !strings.Contains(errb, "unknown flag -ä") {
t.Errorf("run(-ä) = %d, %q, want the flag named whole", code, errb)
}
}
func TestRunEmptyDataPathIsNamed(t *testing.T) {
code, _, errb := runOut("--data-path=", "sv_SE.word")
if code != 1 || !strings.Contains(errb, "empty") {
t.Errorf("run(--data-path=) = %d, %q, want the empty path named", code, errb)
}
}
func TestRunNumbersFollowTheShell(t *testing.T) {
_, want, _ := runOut("--seed", "7", "--repeat", "3", "sv_SE.word")
code, got, errb := runOut("--seed", "007", "--repeat", "+3", "sv_SE.word")
if code != 0 || got != want {
t.Errorf("run(--seed 007 --repeat +3) = %d, %q, stderr %q; want the same as --seed 7 --repeat 3 %q", code, got, errb, want)
}
}
compose.yaml
-4
View File
@@ -42,10 +42,6 @@ services:
<<: *go
command: go vet ./...
cyclo:
<<: *go
command: go run github.com/fzipp/gocyclo/cmd/gocyclo@v0.6.0 -over 14 -ignore _test .
build:
<<: *go
command: go build ./...
data.go
+24 -43
View File
@@ -10,12 +10,11 @@ import (
)
// dataSource is one tree to load: an fs.FS and the directory in it to start from.
// label prefixes file names in errors; onDisk marks path as a directory that must
// exist.
// label prefixes file names in errors; path, when set, is a directory on disk that
// must exist.
type dataSource struct {
fsys fs.FS
label string
onDisk bool
path string
root string
}
@@ -32,18 +31,15 @@ func (s dataSource) name(p string) string {
// keyed by its base name (address.json -> "address"); each subdirectory becomes a
// nested group, so folders turn into dot-path segments. Sources merge left to right:
// matching groups merge by their children, and any other clash is won by the last
// source loaded. Once merged, linkRefs binds every reference against the
// source loaded. Once merged, linkRefs binds every {..path} reference against the
// final tree.
func loadData(sources []dataSource) (map[string]node, error) {
root := map[string]node{}
for _, src := range sources {
if src.onDisk {
if src.path == "" {
return nil, fmt.Errorf("a data path is empty")
}
if src.path != "" {
info, err := os.Stat(src.path)
if err != nil {
return nil, err
return nil, fmt.Errorf("%s: no such directory", src.path)
}
if !info.IsDir() {
return nil, fmt.Errorf("%s is not a directory", src.path)
@@ -68,7 +64,7 @@ func loadData(sources []dataSource) (map[string]node, error) {
if err := checkNoCycles(root); err != nil {
return nil, err
}
if err := checkScope(treeScope(root)); err != nil {
if err := checkBoundLevelsHeld(root); err != nil {
return nil, err
}
return root, nil
@@ -88,57 +84,42 @@ func loadDir(src dataSource, dir string) (*group, error) {
continue
}
full := path.Join(dir, e.Name())
load := loadFile
if e.IsDir() {
load = loadFolder
}
if err := load(src, g, full, e.Name()); err != nil {
return nil, err
}
}
return g, nil
}
// loadFolder adds a subdirectory as a nested group, unless nothing under it is data.
func loadFolder(src dataSource, g *group, full, name string) error {
child, err := loadDir(src, full)
if err != nil {
return err
return nil, err
}
if len(child.children) == 0 {
return nil
continue
}
if err := checkName(e.Name()); err != nil {
return nil, fmt.Errorf("%s: folder %w", src.name(full), err)
}
g.children[e.Name()] = child
continue
}
if !strings.HasSuffix(e.Name(), ".json") {
continue
}
name := strings.TrimSuffix(e.Name(), ".json")
if err := checkName(name); err != nil {
return fmt.Errorf("%s: folder %w", src.name(full), err)
}
g.children[name] = child
return nil
}
// loadFile compiles a *.json file into a category named after it; any other file
// is skipped.
func loadFile(src dataSource, g *group, full, file string) error {
if !strings.HasSuffix(file, ".json") {
return nil
}
name := strings.TrimSuffix(file, ".json")
if err := checkName(name); err != nil {
return fmt.Errorf("%s: category %w", src.name(full), err)
return nil, fmt.Errorf("%s: category %w", src.name(full), err)
}
b, err := fs.ReadFile(src.fsys, full)
if err != nil {
return fmt.Errorf("%s: %w", src.name(full), err)
return nil, fmt.Errorf("%s: %w", src.name(full), err)
}
var raw any
if err := json.Unmarshal(b, &raw); err != nil {
return fmt.Errorf("%s: %w", src.name(full), err)
return nil, fmt.Errorf("%s: %w", src.name(full), err)
}
n, err := compile(raw)
if err != nil {
return fmt.Errorf("%s: %w", src.name(full), err)
return nil, fmt.Errorf("%s: %w", src.name(full), err)
}
g.children[name] = n
return nil
}
return g, nil
}
// mergeChildren overlays src onto dst. Two groups under the same key merge
data_test.go
-37
View File
@@ -264,40 +264,3 @@ func luhnValid(s string) bool {
}
return sum%10 == 0
}
// swedishName matches one or more letter-words, optionally space/hyphen joined
// ("Storgatan", "Norra Promenaden", "von Flemming"). Used by the composition
// tests so shipped name lists can grow without re-enumerating them here.
var swedishName = regexp.MustCompile(`^\p{L}+([ -]\p{L}+)*$`)
func TestShippedStreetComposition(t *testing.T) {
// street is a choice of composed {first}{last} templates and literal names.
f := newGenerator(t, "data/sv_SE", WithSeed(5))
for i := 0; i < 300; i++ {
if s := fake(t, f, "address.street"); !swedishName.MatchString(s) {
t.Fatalf("street %q is not a Swedish street name", s)
}
}
}
func TestShippedLastNameComposition(t *testing.T) {
// last is a choice of patronymic {first}sson templates, compound
// {first}{last} templates and literal surnames.
f := newGenerator(t, "data/sv_SE", WithSeed(6))
for i := 0; i < 300; i++ {
if s := fake(t, f, "person.last"); !swedishName.MatchString(s) {
t.Fatalf("last name %q is not a Swedish surname", s)
}
}
}
func TestShippedStreetNumberFormats(t *testing.T) {
// Reachable via a hyphenated path; covers all five weighted number variants.
f := newGenerator(t, "data/sv_SE", WithSeed(8))
re := regexp.MustCompile(`^[1-9]\d{0,2}[A-Z]?$`)
for i := 0; i < 300; i++ {
if n := fake(t, f, "address.street-number"); !re.MatchString(n) {
t.Fatalf("street-number %q does not match %s", n, re)
}
}
}
edge_test.go
+385
View File
@@ -0,0 +1,385 @@
package fejkdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"regexp"
"slices"
"strings"
"testing"
)
// --- format string edge cases ---
func TestHashIsLiteral(t *testing.T) {
f := engine(1)
cases := map[string]string{
`{"format":"","x":"v"}`: "",
`{"format":"#","x":"v"}`: "#",
`{"format":"##","x":"v"}`: "##",
`{"format":"#0#1#A#a","x":"v"}`: "#0#1#A#a",
`{"format":"#{x}","x":"v"}`: "#v",
}
for tmpl, want := range cases {
if got := mustRender(t, f, tmpl); got != want {
t.Errorf("render(%s) = %q, want %q", tmpl, got, want)
}
}
}
func TestMultibyteFormat(t *testing.T) {
// Scanning is rune-aware: multibyte literals coexist with class chars and
// tokens without corrupting indices.
got := mustRender(t, engine(2), `{"format":"Öster{x}-{digits(1)}å","x":"väg"}`)
if !regexp.MustCompile(`^Österväg-[0-9]å$`).MatchString(got) {
t.Fatalf("multibyte format = %q", got)
}
}
func TestAlternationThreeWay(t *testing.T) {
f := engine(4)
seen := map[string]bool{}
for i := 0; i < 200; i++ {
seen[mustRender(t, f, `{"format":"{a|b|c}","a":"A","b":"B","c":"C"}`)] = true
}
if !seen["A"] || !seen["B"] || !seen["C"] || len(seen) != 3 {
t.Fatalf("3-way alternation produced %v, want A, B and C", seen)
}
}
func TestNewErrors(t *testing.T) {
// Pointing New at a file (not a directory) fails.
file := filepath.Join(t.TempDir(), "xx_XX")
if err := os.WriteFile(file, []byte("{}"), 0o644); err != nil {
t.Fatal(err)
}
if _, err := New(WithoutShippedData(), WithDataPath(file)); err == nil {
t.Error("New(file) = nil error, want not-a-directory error")
}
// Invalid JSON in a category file fails.
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{"broken": `{ not json`}))); err == nil {
t.Error("New(invalid JSON) = nil error")
}
// An option that cannot take effect, and a category or folder no dot path can
// reach, are mistakes New must name rather than accept and ignore.
rejected := map[string]struct {
files map[string]string
want string
}{
"separator without repeat": {
map[string]string{"a": `{"format":"{x}","x":"1","separator":","}`},
"has no effect without a repeat above 1",
},
"separator with an explicit repeat of 1": {
map[string]string{"a": `{"format":"{x}","x":"1","separator":","}`},
"has no effect without a repeat above 1",
},
"weight outside a choice": {
map[string]string{"a": `{"format":"x","weight":5}`},
"weight only skews a choice's items",
},
"non-numeric weight outside a choice": {
map[string]string{"a": `{"format":"x","weight":"bad"}`},
"weight only skews a choice's items",
},
"weight used as a field": {
map[string]string{"a": `{"format":"{name} {weight}kg","name":"Anvil","weight":["7"]}`},
"can never be a field",
},
"an option name used as a token": {
map[string]string{"a": `"{weight}"`},
`"weight" is an option and can never be a field`,
},
"category name with a dot": {
map[string]string{"a.b": `"1"`},
`category "a.b" contains "."`,
},
"folder name with a dot": {
map[string]string{"a.b/cat": `"1"`},
`/a.b: folder "a.b" contains "."`,
},
// The token grammar reserves three more characters. A name carrying one
// still resolves by dot path, but no format can name it, so it is rejected
// where it is authored rather than at the token that cannot reach it.
"field name with a pipe": {
map[string]string{"a": `{"format":"{x}","x":"1","b|c":"2"}`},
`field "b|c" contains "|"`,
},
"field name with a paren": {
map[string]string{"a": `{"format":"{x}","x":"1","b(c":"2"}`},
`field "b(c" contains "("`,
},
"field name with a closing brace": {
map[string]string{"a": `{"format":"{x}","x":"1","b}c":"2"}`},
`field "b}c" contains "}"`,
},
"category name with a pipe": {
map[string]string{"a|b": `"1"`},
`category "a|b" contains "|"`,
},
// An empty name is not a path segment, so List never offered it — while a
// bare {}, a trailing dot in Fake("a.") and a {..a.} reference all reached
// it. The engine accepted spellings it would never advertise.
"empty field name": {
map[string]string{"a": `{"format":"[{}]","":"VALUE"}`},
`field "" is empty`,
},
"folder name with a paren": {
map[string]string{"a(b/cat": `"1"`},
`folder "a(b" contains "("`,
},
// A repeated arm skews an alternation, which weight is the spelling for.
"repeated alternation arm": {
map[string]string{"a": `{"format":"{x|x}","x":"1"}`},
`arm "x" is repeated`,
},
"repeated arm among others": {
map[string]string{"a": `{"format":"{x|y|x}","x":"1","y":"2"}`},
`arm "x" is repeated`,
},
"repeated path arm": {
map[string]string{"a": `{"format":"{p.v|p.v}","p":{"format":"{v}","v":"1"}}`},
`arm "p.v" is repeated`,
},
// A reference arm is the only kind that reaches the repeat check by passing
// the per-arm checks rather than falling through them.
"repeated reference arm": {
map[string]string{"a": `"x"`, "b": `"{..a|..a}"`},
`arm "..a" is repeated`,
},
// An arm that is broken on its own terms is reported as that, not as a
// repeat: the repeat is a consequence of the real mistake.
"repeated arm with no path": {
map[string]string{"a": `"{..|..}"`},
"reference has no path",
},
// No field can be named "", so the token is told that rather than sent to
// name one — the fix "no field" points at is itself a load error.
"repeated empty arm": {
map[string]string{"a": `"{|}"`},
"a name is never empty",
},
"bare empty token": {
map[string]string{"a": `{"format":"[{}]","x":"1"}`},
"a name is never empty",
},
}
for name, c := range rejected {
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, c.files)))
if err == nil {
t.Errorf("%s: New = nil error, want it rejected at load", name)
continue
}
if !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: New = %v, want it to mention %q", name, err, c.want)
}
}
// Data that works today must keep working, and stay reachable.
accepted := map[string]struct {
files map[string]string
path string
want string
}{
"option name as a field": {map[string]string{"a": `{"format":"{name} {Weight}kg","name":"Anvil","Weight":"7"}`}, "a", "Anvil 7kg"},
"format spelling as a field": {map[string]string{"a": `{"format":"{Format}","Format":"PDF"}`}, "a", "PDF"},
"hyphenated field": {map[string]string{"a": `{"format":"{x-y}","x-y":"1"}`}, "a.x-y", "1"},
"category named Format": {map[string]string{"Format": `"1"`}, "Format", "1"},
"folder named Repeat": {map[string]string{"Repeat/cat": `"1"`}, "Repeat.cat", "1"},
"field with a closing paren": {map[string]string{"a": `{"format":"{b)c}","b)c":"2"}`}, "a", "2"},
"repeat without a separator": {map[string]string{"a": `{"format":"{x}","repeat":3,"x":"1"}`}, "a", "111"},
// One name in two separate tokens is two independent draws, not a repeated
// arm; only a repeat within one alternation is rejected.
"one name in two tokens": {map[string]string{"a": `{"format":"{x}{x}","x":"1"}`}, "a", "11"},
}
for name, c := range accepted {
f, err := New(WithoutShippedData(), WithDataPath(writeData(t, c.files)))
if err != nil {
t.Errorf("%s: New = %v, want it accepted", name, err)
continue
}
if got, err := f.Fake(c.path); err != nil || got != c.want {
t.Errorf("%s: Fake(%q) = %q, %v, want %q", name, c.path, got, err, c.want)
}
}
}
// --- deep path navigation ---
func TestDeepDottedPath(t *testing.T) {
// A 5-segment path descends through alternating object/array nodes; choices
// on the path are single-variant, so it resolves deterministically.
f := engine(1)
f.categories = map[string]node{
"deep": compiled(t, `{"format":"{a}","a":{"format":"{b}","b":{"format":"{c}","c":{"format":"{d}","d":"leaf"}}}}`),
}
if got, err := f.Fake("deep.a.b.c.d"); err != nil || got != "leaf" {
t.Fatalf("Fake(deep.a.b.c.d) = %q, %v, want leaf", got, err)
}
// Rendering the whole tree resolves the same chain.
if got, err := f.Fake("deep"); err != nil || got != "leaf" {
t.Fatalf("Fake(deep) = %q, %v, want leaf", got, err)
}
}
func TestDescendIntoStringErrors(t *testing.T) {
f := engine(1)
f.categories = map[string]node{"greeting": compiled(t, `"hej"`)}
if _, err := f.Fake("greeting.extra"); err == nil || !strings.Contains(err.Error(), `no field "extra"`) {
t.Fatalf("Fake(greeting.extra) = %v, want a no-field error", err)
}
}
// TestPathThroughChoice pins the rule that keeps a dotted path from rendering on
// one call and failing on the next: every variant must carry the rest of the path.
func TestPathThroughChoice(t *testing.T) {
dir := writeData(t, map[string]string{
"every": `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2"}]`,
"notall": `[{"format":"{f}","f":"1"},"plain"]`,
"some": `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2","extra":"x"}]`,
})
f := newGenerator(t, dir, WithSeed(1))
for i := 0; i < 200; i++ {
if got := fake(t, f, "every.f"); got != "1" && got != "2" {
t.Fatalf("every.f = %q, want 1 or 2", got)
}
}
// A path only some variants carry is reported against what all of them carry.
if _, err := f.Fake("some.extra"); err == nil || !strings.Contains(err.Error(), "all carry [f]") {
t.Errorf("Fake(some.extra) = %v, want it to name what every variant carries", err)
}
var first string
for i := 0; i < 200; i++ {
_, err := f.Fake("notall.f")
if err == nil {
t.Fatal("notall.f = nil error, want the same failure every call")
}
if i == 0 {
first = err.Error()
} else if err.Error() != first {
t.Fatalf("notall.f error varies between calls:\n %s\n %s", first, err.Error())
}
}
}
// TestPathKeyIsUnambiguous pins that the one shape which could collide cannot be
// written: a field literally named "a.b" and a field "a" holding "b" would both
// spell "a.b", so a dotted field name is rejected at New and a dot means a path
// wherever it appears.
func TestPathKeyIsUnambiguous(t *testing.T) {
dir := writeData(t, map[string]string{
"cat": `[{"format":"{a.b}","a.b":"1"},{"format":"{a}","a":{"format":"{b}","b":"2"}}]`,
})
_, err := New(WithoutShippedData(), WithDataPath(dir))
if err == nil || !strings.Contains(err.Error(), `field "a.b" contains "."`) {
t.Fatalf("New = %v, want the dotted field name rejected", err)
}
// The same data without the dotted key is fine, and the path resolves.
f := newGenerator(t, writeData(t, map[string]string{
"cat": `{"format":"{a.b}","a":{"format":"{b}","b":"2"}}`,
}), WithSeed(1))
if !slices.Contains(f.List(), "cat.a.b") {
t.Error("List() omits cat.a.b, which the data carries")
}
if got := fake(t, f, "cat"); got != "2" {
t.Fatalf("cat = %q, want 2", got)
}
}
// TestMissingFieldNamesItself keeps the precise diagnosis for the ordinary typo: a
// single-variant choice always picks the same item, so it needs no every-variant
// guard and the error can name the field that is missing.
func TestMissingFieldNamesItself(t *testing.T) {
f := newGenerator(t, "data/sv_SE", WithSeed(1))
_, err := f.Fake("person.typo")
if err == nil || !strings.Contains(err.Error(), `no field "typo"`) {
t.Errorf("Fake(person.typo) = %v, want it to name the missing field", err)
}
}
// --- category root shapes ---
func TestCategoryRootShapes(t *testing.T) {
dir := writeData(t, map[string]string{
"obj": `"{digits(2)}"`, // object root
"lit": `"hello"`, // bare-string root
})
f := newGenerator(t, dir, WithSeed(1))
if got := fake(t, f, "obj"); !regexp.MustCompile(`^\d\d$`).MatchString(got) {
t.Errorf("object-root category = %q, want two digits", got)
}
if got := fake(t, f, "lit"); got != "hello" {
t.Errorf("string-root category = %q, want hello", got)
}
}
// --- very long lists ---
func TestLongStringList(t *testing.T) {
const n = 2000
names := make([]string, n)
for i := range names {
names[i] = fmt.Sprintf("name-%04d", i)
}
list, err := json.Marshal(names)
if err != nil {
t.Fatal(err)
}
f := newGenerator(t, writeData(t, map[string]string{"name": string(list)}), WithSeed(1))
valid := map[string]bool{}
for _, v := range names {
valid[v] = true
}
seen := map[string]bool{}
for i := 0; i < 20000; i++ {
v := fake(t, f, "name")
if !valid[v] {
t.Fatalf("got %q, not in the list", v)
}
seen[v] = true
}
if len(seen) < n*8/10 {
t.Fatalf("only %d/%d distinct values seen; selection looks skewed", len(seen), n)
}
}
// --- composition against the shipped sv_SE data ---
// swedishName matches one or more letter-words, optionally space/hyphen joined
// ("Storgatan", "Norra Promenaden", "von Flemming"). Used by the composition
// tests so shipped name lists can grow without re-enumerating them here.
var swedishName = regexp.MustCompile(`^\p{L}+([ -]\p{L}+)*$`)
func TestShippedStreetComposition(t *testing.T) {
// street is a choice of composed {first}{last} templates and literal names.
f := newGenerator(t, "data/sv_SE", WithSeed(5))
for i := 0; i < 300; i++ {
if s := fake(t, f, "address.street"); !swedishName.MatchString(s) {
t.Fatalf("street %q is not a Swedish street name", s)
}
}
}
func TestShippedLastNameComposition(t *testing.T) {
// last is a choice of patronymic {first}sson templates, compound
// {first}{last} templates and literal surnames.
f := newGenerator(t, "data/sv_SE", WithSeed(6))
for i := 0; i < 300; i++ {
if s := fake(t, f, "person.last"); !swedishName.MatchString(s) {
t.Fatalf("last name %q is not a Swedish surname", s)
}
}
}
func TestShippedStreetNumberFormats(t *testing.T) {
// Reachable via a hyphenated path; covers all five weighted number variants.
f := newGenerator(t, "data/sv_SE", WithSeed(8))
re := regexp.MustCompile(`^[1-9]\d{0,2}[A-Z]?$`)
for i := 0; i < 300; i++ {
if n := fake(t, f, "address.street-number"); !re.MatchString(n) {
t.Fatalf("street-number %q does not match %s", n, re)
}
}
}
fejkdata.go
+9 -31
View File
@@ -16,7 +16,6 @@ import (
crand "crypto/rand"
"embed"
"encoding/binary"
"errors"
"fmt"
"io/fs"
"math/rand/v2"
@@ -28,19 +27,9 @@ import (
//go:embed data
var shippedFS embed.FS
// MaxRepeat caps a repeat, and the renders nested repeats multiply to along any
// path; the CLI's --repeat shares it.
const MaxRepeat = 1 << 20
var _ [^uint(0)>>63 - 1]struct{} // 64-bit only, per the README's Decisions
// ErrNoData is returned by New when no source is loaded at all.
var ErrNoData = errors.New("no data: WithoutShippedData needs at least one WithDataPath or WithDataFS")
// Generator generates fake data from a loaded namespace tree. Create one with [New].
// It is safe for concurrent use; a seeded sequence is reproducible only when drawn
// from one goroutine. The compiled tree is immutable after [New], and Fake,
// NewTemplate and List read it concurrently without a lock.
// from one goroutine.
type Generator struct {
mu sync.Mutex
rand *session
@@ -79,7 +68,7 @@ func WithSeed(seed uint64) Option {
// layer several; the last wins a name clash.
func WithDataPath(dir string) Option {
return func(c *config) {
c.sources = append(c.sources, dataSource{fsys: os.DirFS(dir), label: dir, onDisk: true, path: dir})
c.sources = append(c.sources, dataSource{fsys: os.DirFS(dir), label: dir, path: dir})
}
}
@@ -110,17 +99,13 @@ func New(opts ...Option) (*Generator, error) {
}
sources = append(sources, c.sources...)
if len(sources) == 0 {
return nil, fmt.Errorf("fejkdata: %w", ErrNoData)
return nil, fmt.Errorf("fejkdata: no data: WithoutShippedData needs at least one WithDataPath or WithDataFS")
}
cats, err := loadData(sources)
if err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
rng, err := newRand(c.seed, c.seeded)
if err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
return &Generator{rand: rng, categories: cats}, nil
return &Generator{rand: newRand(c.seed, c.seeded), categories: cats}, nil
}
// List returns the sorted dotted paths Fake can render: every category, the dotted
@@ -211,19 +196,12 @@ func join(prefix, name string) string {
return prefix + "." + name
}
// randomBytes seeds an unseeded generator.
var randomBytes = crand.Read
func newRand(seed uint64, seeded bool) (*session, error) {
var r *rand.Rand
if seeded {
r = rand.New(rand.NewPCG(seed, seed^0x9e3779b97f4a7c15))
} else {
func newRand(seed uint64, seeded bool) *session {
r := rand.New(rand.NewPCG(seed, seed^0x9e3779b97f4a7c15))
if !seeded {
var b [16]byte
if _, err := randomBytes(b[:]); err != nil {
return nil, fmt.Errorf("seeding from crypto/rand: %w", err)
}
_, _ = crand.Read(b[:])
r = rand.New(rand.NewPCG(binary.LittleEndian.Uint64(b[:8]), binary.LittleEndian.Uint64(b[8:])))
}
return &session{Rand: r, counters: map[string]uint64{}}, nil
return &session{Rand: r, counters: map[string]uint64{}}
}
fejkdata_test.go
+2 -22
View File
@@ -1,11 +1,8 @@
package fejkdata
import (
"errors"
"io/fs"
"strings"
"testing"
"testing/fstest"
)
// newGenerator creates a generator over a single data directory, failing on
@@ -41,25 +38,8 @@ func fake(t *testing.T, f *Generator, path string) string {
func TestNewMissingDirectory(t *testing.T) {
_, err := New(WithoutShippedData(), WithDataPath("data/de_DE"))
if err == nil || !strings.Contains(err.Error(), "de_DE") || !errors.Is(err, fs.ErrNotExist) {
t.Fatalf("New(missing) error = %v, want the real error, naming the path", err)
}
}
func TestNewRejectsAnEmptyDataPath(t *testing.T) {
_, err := New(WithoutShippedData(), WithDataPath(""))
if err == nil || !strings.Contains(err.Error(), "empty") {
t.Fatalf("New(WithDataPath(\"\")) = %v, want the empty path named", err)
}
}
func TestNewReportsAnEntropyFailure(t *testing.T) {
saved := randomBytes
randomBytes = func([]byte) (int, error) { return 0, errors.New("no entropy") }
defer func() { randomBytes = saved }()
_, err := New(WithoutShippedData(), WithDataFS(fstest.MapFS{"w.json": {Data: []byte(`"x"`)}}))
if err == nil || !strings.Contains(err.Error(), "no entropy") {
t.Fatalf("New() without entropy = %v, want the failure reported", err)
if err == nil || !strings.Contains(err.Error(), "de_DE") {
t.Fatalf("New(missing) error = %v, want it to name the path", err)
}
}
graph.go
-251
View File
@@ -1,251 +0,0 @@
package fejkdata
import (
"fmt"
"sort"
"strings"
)
// walkNodes calls fn once per contained node, passing the dot path that reaches it,
// visiting keys in sorted order so which of several broken nodes gets reported does
// not depend on map iteration.
func walkNodes(root map[string]node, fn func(path string, n node) error) error {
for _, name := range sortedNames(root) {
if err := eachNode(root[name], name, fn); err != nil {
return err
}
}
return nil
}
// eachNode visits n and every node contained within it once, passing the dot path
// that reaches each. It never crosses a reference edge — a bound {/path} field is
// skipped — so a single inline node is walked on its own.
func eachNode(n node, path string, fn func(path string, n node) error) error {
seen := map[node]bool{}
var visit func(string, node) error
visit = func(path string, m node) error {
if m == nil || seen[m] {
return nil
}
seen[m] = true
if err := fn(path, m); err != nil {
return err
}
for _, c := range contained(m) {
if err := visit(join(path, c.name), c.node); err != nil {
return err
}
}
return nil
}
return visit(path, n)
}
// namedNode is a contained child and the segment reaching it; a choice's items carry
// no segment, matching how a dot path steps over a choice.
type namedNode struct {
name string
node node
}
func contained(n node) []namedNode {
switch n := n.(type) {
case *group:
return named(n.children)
case *choice:
out := make([]namedNode, len(n.items))
for i, it := range n.items {
out[i] = namedNode{node: it}
}
return out
case *template:
return named(n.fields)
default:
return nil
}
}
// named skips a bound {/path} key: it is a render edge, not containment, so using
// it as a path segment would report a node under a path that does not reach it. Only
// a template's fields hold bindings — loadDir skips a dot-prefixed entry, so a
// group's children never carry the prefix — so this one skip serves both.
func named(m map[string]node) []namedNode {
out := make([]namedNode, 0, len(m))
for _, name := range sortedNames(m) {
if isRef(name) {
continue
}
out = append(out, namedNode{name: name, node: m[name]})
}
return out
}
func sortedNames(m map[string]node) []string {
names := make([]string, 0, len(m))
for name := range m {
names = append(names, name)
}
sort.Strings(names)
return names
}
// renderEdge is a child a node renders into, labelled by what reaches it (a field
// name, reference, or choice index) for a readable cycle report. operand names
// the builtin when the label is its operand rather than a token, so an error can
// name it the way the author wrote it.
type renderEdge struct {
to node
label string
operand string
}
// reached names an edge as the author spelled it, the vocabulary boundReaders uses
// for the sibling fence.
func (e renderEdge) reached() string {
if e.operand != "" {
return fmt.Sprintf("%s operand %q", e.operand, e.label)
}
return "{" + e.label + "}"
}
// renderEdges lists the children rendering n recurses into, mirroring expand: a
// choice's items, and a template's field/reference tokens plus its operands. A
// group renders nothing, so it has no edges.
func renderEdges(n node) []renderEdge {
switch n := n.(type) {
case *choice:
es := make([]renderEdge, len(n.items))
for i, it := range n.items {
es[i] = renderEdge{to: it, label: fmt.Sprintf("[%d]", i)}
}
return es
case *template:
var es []renderEdge
add := func(name, operand string) {
a := splitArm(name, n.refs)
c, ok := n.fields[a.key]
if !ok {
return
}
for _, leaf := range pathLeaves(c, a.tail) {
es = append(es, renderEdge{leaf, name, operand})
}
}
_ = eachToken(n.format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if fn, _, isFunc := funcCall(t.body); isFunc {
for _, operand := range tokenOperands(t.body) {
add(operand, fn)
}
return nil
}
for _, name := range strings.Split(t.body, "|") {
add(name, "")
}
return nil
})
return es
default:
return nil
}
}
// pathLeaves lists what a token's dotted tail renders: a choice on the way
// contributes every variant, since any of them may be the one drawn. checkPath has
// already proved the tail resolves in every variant.
func pathLeaves(n node, tail []string) []node {
var out []node
_ = walkPath(n, tail, pathWalk{
choice: func(c *choice, _ []string) ([]node, error) { return c.items, nil },
leaf: func(n node) error { out = append(out, n); return nil },
})
return out
}
// nodeScope is the set of nodes one validation pass covers: a whole loaded tree,
// or a single inline node.
type nodeScope func(fn func(path string, n node) error) error
func treeScope(root map[string]node) nodeScope {
return func(fn func(path string, n node) error) error { return walkNodes(root, fn) }
}
func inlineScope(n node) nodeScope {
return func(fn func(path string, m node) error) error { return eachNode(n, "template", fn) }
}
// checkScope runs the per-node fences over a scope, each over the whole scope
// before the next, so which of several broken nodes is reported does not depend on
// the walk. It runs after checkNoCycles, whose guarantee is what lets the walks
// terminate.
func checkScope(s nodeScope) error {
mem := reachMemo{}
if err := s(func(path string, n node) error { return repeatCheck(path, n, mem) }); err != nil {
return err
}
return s(heldCheck)
}
type reachMemo map[node]int
func (m reachMemo) of(n node) int {
if r, done := m[n]; done {
return r
}
r := 1
for _, e := range renderEdges(n) {
if c := m.of(e.to); c > r {
r = c
}
}
if t, ok := n.(*template); ok {
r *= t.repeat
}
m[n] = r
return r
}
// repeatCheck bounds the renders a repeat multiplies to along any root-to-leaf
// path, so nested repeats cannot build what one repeat may not.
func repeatCheck(path string, n node, mem reachMemo) error {
if t, ok := n.(*template); ok && t.repeat > 1 && mem.of(n) > MaxRepeat {
return fmt.Errorf("%s: repeat %d multiplies to %d renders along one path, above the maximum %d", path, t.repeat, mem.of(n), MaxRepeat)
}
return nil
}
// checkNoCycles rejects a reference cycle: a node whose rendering can reach itself
// — directly, mutually, or through a chain — never terminates, so it must fail at
// New rather than stack-overflow at render. It is a depth-first walk of the render
// graph (renderEdges); grey marks nodes on the current path so a back-edge to one
// is the cycle, while black lets a shared node (a DAG, not a cycle) be skipped.
// Every node is a root: a field its parent's format never renders is still reachable
// by dot path, so a cycle in one would otherwise reach render and be fatal there.
func checkNoCycles(root map[string]node) error {
const (
grey = 1
black = 2
)
color := map[node]int{}
var visit func(n node, path string) error
visit = func(n node, path string) error {
switch color[n] {
case grey:
return fmt.Errorf("reference cycle: %s", path)
case black:
return nil
}
color[n] = grey
for _, e := range renderEdges(n) {
if err := visit(e.to, path+" -> "+e.label); err != nil {
return err
}
}
color[n] = black
return nil
}
return walkNodes(root, func(path string, n node) error { return visit(n, path) })
}
hold.go
-310
View File
@@ -1,310 +0,0 @@
package fejkdata
import (
"fmt"
"sort"
"strings"
)
// heldCheck rejects every route to a held name except the ones that read its draw.
// An expansion holds one draw of that name; anything else that renders it draws
// again, and the two disagree. checkNoOverlap settles the spellings within one
// format (a token, an operand); this settles the rest — a reference, whether it
// sits in that format or in anything the format renders, however deep.
func heldCheck(path string, n node) error {
t, ok := n.(*template)
if !ok || len(t.held) == 0 {
return nil
}
readers := boundReaders(t.format, t.bound, t.refs)
for _, head := range heldHeads(t) {
if err := checkHeadHeld(t, head, readers); err != nil {
return fmt.Errorf("%s: %w", path, err)
}
}
return nil
}
// heldHeads lists a template's held names, operand heads first, then paths, each
// in name order: a level read both ways is reported by the operand's fence, and
// which overlap is reported does not vary.
func heldHeads(t *template) []string {
heads := make([]string, 0, len(t.held))
for head := range t.held {
heads = append(heads, head)
}
sort.Slice(heads, func(i, j int) bool {
_, pi := t.bound[heads[i]]
_, pj := t.bound[heads[j]]
if pi != pj {
return !pi
}
return heads[i] < heads[j]
})
return heads
}
// pinned collects what the hold of head answers for: a path pins the levels it
// passes through and the leaf it lands on, an operand exactly the value its render
// produces.
func pinned(t *template, head string, readers []reader) map[node]bool {
held := map[node]bool{}
if _, isPath := t.bound[head]; !isPath {
operandDraw(t.fields[head], held)
return held
}
for _, r := range readers {
if a := splitArm(r.name, t.refs); a.key == head {
coverPath(t.fields[head], a.tail, held)
}
}
return held
}
// checkHeadHeld rejects every route to what head's hold pins except the readers
// holding it. One seen set across the edges: a node that cannot reach the level
// cannot reach it by another route either, so it is walked once.
func checkHeadHeld(t *template, head string, readers []reader) error {
held := pinned(t, head, readers)
if len(held) == 0 {
return nil // a fixed head holds nothing to reach
}
reader, isPath := t.bound[head]
seen := map[node]bool{}
for _, e := range renderEdges(t) {
if splitArm(e.label, t.refs).key == head || !renders(e.to, held, seen) {
continue
}
if isPath {
return fmt.Errorf("%s renders %q, which {%s} reads a path into; name the fields you want instead", e.reached(), head, reader)
}
return fmt.Errorf("%s renders %q, which a {%s()} also reads; reach it one way so it is drawn once", e.reached(), head, operandReader(t, head))
}
return nil
}
// operandReader names the builtin whose operand holds head.
func operandReader(t *template, head string) string {
fn := ""
_ = eachToken(t.format, func(tok ftoken) error {
if tok.kind != 'b' || fn != "" {
return nil
}
if name, _, isFunc := funcCall(tok.body); isFunc {
for _, operand := range tokenOperands(tok.body) {
if splitArm(operand, t.refs).key == head {
fn = name
}
}
}
return nil
})
return fn
}
// coverPath collects what holding one path pins: every choice level the path
// passes through, whole, and the leaf it renders.
func coverPath(n node, tail []string, into map[node]bool) {
_ = walkPath(n, tail, pathWalk{
choice: func(c *choice, _ []string) ([]node, error) { cover(c, into, false); return nil, nil },
leaf: func(n node) error { cover(n, into, false); return nil },
})
}
// cover collects a level and everything contained in it. A fixed string outside a
// choice is left out — it cannot disagree with itself — but inside one each
// variant carries its own, so there it counts.
func cover(n node, into map[node]bool, inChoice bool) {
if isFixed(n) && !inChoice {
return
}
into[n] = true
_, isChoice := n.(*choice)
for _, c := range contained(n) {
cover(c.node, into, inChoice || isChoice)
}
}
// operandDraw collects what one held draw of an operand answers for: the operand
// and what rendering it settles inside itself. The builtin renders its operand
// whole, so that draw fixes every value the render produced, and a second route to
// any of them disagrees with it.
//
// The walk stops at a reference edge, which is where the operand's own value ends
// and a shared source begins: two names referencing one category are two draws, the
// same rule {word} {word} follows.
func operandDraw(n node, into map[node]bool) {
if isFixed(n) {
return
}
if into[n] {
return
}
into[n] = true
for _, e := range renderEdges(n) {
if isRef(e.label) {
continue
}
operandDraw(e.to, into)
}
}
// isFixed is a string that varies nothing: fixed text with no fields to read into.
func isFixed(n node) bool {
t, ok := n.(*template)
return ok && t.fixed && len(t.fields) == 0
}
// renders reports whether rendering n can reach anything in want, following the
// same edges expand does. seen keeps a node shared by several routes from being
// walked twice; checkNoCycles has already proved the graph is a DAG, so the walk
// ends.
func renders(n node, want, seen map[node]bool) bool {
if want[n] {
return true
}
if seen[n] {
return false
}
seen[n] = true
for _, e := range renderEdges(n) {
if renders(e.to, want, seen) {
return true
}
}
return false
}
// checkNoOverlap rejects a format that both renders a level and reads a path into
// it — {p} beside {p.first}, {p.addr} beside {p.addr.city}, {.p} beside
// {/sv_SE.p.first}. The path reads the level's held draw while rendering the level
// expands it afresh, so their values would disagree. Reads are compared by their
// one spelling, in sorted order, so which pair is reported depends neither on how
// a reference was written nor on where the tokens sit.
func checkNoOverlap(format string, bound map[string]string, refs map[string]refBinding) error {
names := boundReaders(format, bound, refs)
// Stable over one format-order scan, so two readers of one name (a token and a
// calc operand both naming "p") are reported as the format writes them.
sort.SliceStable(names, func(i, j int) bool { return names[i].path < names[j].path })
for i, level := range names {
for _, path := range names[i+1:] {
if strings.HasPrefix(path.path, level.path+".") {
return fmt.Errorf("%s renders a level that {%s} reads a path into; name the fields you want instead", level.label, path.name)
}
}
}
return nil
}
// reader is one way a format reaches a bound field: as written, by its one
// spelling, and how to name it.
type reader struct{ name, path, label string }
// boundReaders lists every way a format reaches a bound field, in the order the
// format writes them. An operand renders its field, so it names a level exactly
// as a token does; one scan finds both, which is what puts them in one order.
func boundReaders(format string, bound map[string]string, refs map[string]refBinding) []reader {
var names []reader
_ = eachToken(format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if fn, _, isFunc := funcCall(t.body); isFunc {
for _, operand := range tokenOperands(t.body) {
a := splitArm(operand, refs)
if _, isBound := bound[a.key]; isBound {
names = append(names, reader{a.name, a.path, fmt.Sprintf("%s operand %q", fn, operand)})
}
}
return nil
}
for _, a := range splitArms(t.body, refs) {
if _, isBound := bound[a.key]; isBound {
names = append(names, reader{a.name, a.path, "token {" + a.name + "}"})
}
}
return nil
})
return names
}
// checkNoRepeatedRead rejects a bare token repeated on a held name: {w} {w} beside
// {uppercase(w)} would read one draw twice, where {w} {w} alone draws twice. The
// error names the single-token spelling.
func checkNoRepeatedRead(format string, c formatOps, refs map[string]refBinding) error {
count := map[string]int{}
return eachToken(format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if _, _, isFunc := funcCall(t.body); isFunc {
return nil
}
for _, a := range splitArms(t.body, refs) {
if len(a.tail) > 0 || !c.held[a.key] {
continue
}
if count[a.key]++; count[a.key] > 1 {
return fmt.Errorf("token {%s} is repeated, and %s holds %q to one draw per expansion; write {%s} once", a.name, c.holder[a.key], a.key, a.name)
}
}
return nil
})
}
// draws is what an expansion has already drawn for its held names: the variant each
// was drawn as, so every path under it reads one row, and the value each read, by
// its one spelling, so the same read written twice reads one value.
type draws struct {
variant map[string]node
value map[string]string
}
// readField renders one arm of a token. An arm's key is a sibling field or a
// reference linkRefs bound into fields. A name the expansion holds — a level some
// token addresses by dotted path, or a field an operand reads — is drawn once and
// kept, so {place.postal-code} and {place.locality} read one row, either read twice
// gives one value, and a shown operand is the operand computed. Every other name is
// drawn afresh, so {word} {word} still draws twice. checkTokens, checkPath and
// linkRefs prove every step, so the walk cannot fail.
func readField(s *session, t *template, held *draws, a arm) string {
if !t.held[a.key] {
if len(a.tail) > 0 {
panic(fmt.Sprintf("fejkdata: %q reads a path into %q, which the expansion does not hold", a.name, a.key))
}
return render(s, t.fields[a.key])
}
if v, read := held.value[a.path]; read {
return v
}
var v string
_ = walkPath(t.fields[a.key], a.tail, pathWalk{
// Hold the draw at every level passed through, so two paths sharing a
// prefix share it.
choice: func(c *choice, rest []string) ([]node, error) {
key := a.key
if consumed := len(a.tail) - len(rest); consumed > 0 {
key = a.steps[consumed-1]
}
n, drew := held.variant[key]
if !drew {
n = drawn(s, c)
held.variant[key] = n
}
return []node{n}, nil
},
leaf: func(n node) error { v = render(s, n); return nil },
})
held.value[a.path] = v
return v
}
// drawn resolves a choice to one variant, so a bound head is a concrete node the
// rest of the expansion shares. Nested choices unwrap too: a draw is one value, not
// another set to pick from.
func drawn(s *session, n node) node {
for c, ok := n.(*choice); ok; c, ok = n.(*choice) {
n = pick(s, c)
}
return n
}
inline.go
-88
View File
@@ -1,88 +0,0 @@
package fejkdata
import (
"encoding/json"
"fmt"
"strings"
)
// Template is an inline template compiled, referenced and validated against a
// generator's loaded data once, ready to render many times with [Template.Fake].
// It is safe for concurrent use: Fake serializes on its generator's lock, so a
// seeded sequence is reproducible only when a generator — and its templates — are
// drawn from one goroutine.
type Template struct {
g *Generator
n node
}
// Fake renders the template with one draw.
func (t *Template) Fake() string {
t.g.mu.Lock()
defer t.g.mu.Unlock()
return render(t.g.rand, t.n)
}
// NewTemplate compiles an inline template — a format string or a JSON value — and
// binds its references against the loaded tree, so repeated renders pay the
// compile and validation once. It shares [New]'s guarantees: a bad template errors
// here, and rendering cannot fail.
func (f *Generator) NewTemplate(input string) (*Template, error) {
n, err := compileInput(input)
if err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
scope := inlineScope(n)
if err := linkNodeRefs(scope, f.categories); err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
if err := checkScope(scope); err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
return &Template{g: f, n: n}, nil
}
// FakeTemplate compiles and renders an inline template in one call. It is
// [NewTemplate] then [Template.Fake]; to render the same template many times, hold
// the *Template and call its Fake.
func (f *Generator) FakeTemplate(input string) (string, error) {
t, err := f.NewTemplate(input)
if err != nil {
return "", err
}
return t.Fake(), nil
}
// compileInput compiles an inline template: a JSON value, or a bare format string
// when the input is not JSON.
func compileInput(input string) (node, error) {
var raw any
if err := json.Unmarshal([]byte(input), &raw); err != nil {
return compile(input)
}
if trimmed := strings.TrimSpace(input); trimmed != input {
return nil, fmt.Errorf("a JSON template may not be padded with spaces, which a format string would render; write %s", trimmed)
}
return compile(raw)
}
// linkNodeRefs binds the references in an inline node's templates against the
// loaded tree.
func linkNodeRefs(scope nodeScope, root map[string]node) error {
return scope(func(path string, m node) error {
t, ok := m.(*template)
if !ok {
return nil
}
for _, name := range refTokens(t.format) {
sigil, rest, err := refShape(name)
if err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
}
if sigil != "/" {
return fmt.Errorf("%s: reference {%s}: an inline template has no folder; write {/%s}", path, name, rest)
}
}
return linkTemplateRefs(nil, path, t, root)
})
}
inline_test.go
-159
View File
@@ -1,159 +0,0 @@
package fejkdata
import (
"strings"
"testing"
)
func tmpl(t *testing.T, f *Generator, input string) string {
t.Helper()
s, err := f.FakeTemplate(input)
if err != nil {
t.Fatalf("FakeTemplate(%q): %v", input, err)
}
return s
}
func shipped(t *testing.T, opts ...Option) *Generator {
t.Helper()
f, err := New(append([]Option{WithSeed(1)}, opts...)...)
if err != nil {
t.Fatal(err)
}
return f
}
func TestFakeTemplateFormatString(t *testing.T) {
f := shipped(t)
got := tmpl(t, f, "name: {/sv_SE.person.last}")
if !strings.HasPrefix(got, "name: ") || strings.HasSuffix(got, " ") || strings.Contains(got, "{") {
t.Fatalf("FakeTemplate = %q, want a rendered last name after the prefix", got)
}
}
func TestFakeTemplateJSONObject(t *testing.T) {
f := shipped(t)
seen := map[string]bool{}
for i := 0; i < 50; i++ {
seen[tmpl(t, f, `{"format":"name: {x}","x":["bosse","lina"]}`)] = true
}
if !seen["name: bosse"] || !seen["name: lina"] || len(seen) != 2 {
t.Fatalf("JSON template produced %v, want both names", seen)
}
}
func TestFakeTemplateJSONArray(t *testing.T) {
f := shipped(t)
seen := map[string]bool{}
for i := 0; i < 50; i++ {
seen[tmpl(t, f, `["foo","bar","baz"]`)] = true
}
if len(seen) != 3 {
t.Fatalf("JSON array choice produced %v, want three items", seen)
}
}
func TestFakeTemplateCorrelatedReferences(t *testing.T) {
dir := writeData(t, map[string]string{
"person": `[{"format":"{first} {last}","first":"Ada","last":"Lovelace"},{"format":"{first} {last}","first":"Bo","last":"Ek"}]`,
})
f := newGenerator(t, dir, WithSeed(1))
for i := 0; i < 100; i++ {
got := tmpl(t, f, "{/person.first} {/person.last}")
if got != "Ada Lovelace" && got != "Bo Ek" {
t.Fatalf("correlated references = %q, want one person's first and last", got)
}
}
}
func TestFakeTemplateDeterministic(t *testing.T) {
a, b := shipped(t), shipped(t)
for i := 0; i < 20; i++ {
in := "row: {/misc.uuid} {digits(3)}"
if x, y := tmpl(t, a, in), tmpl(t, b, in); x != y {
t.Fatalf("same seed diverged: %q != %q", x, y)
}
}
}
func TestFieldlessTokenHint(t *testing.T) {
f := shipped(t)
_, err := f.FakeTemplate(`{sv_SE.person.last}`)
if err == nil || !strings.Contains(err.Error(), "write {/sv_SE.person.last}") {
t.Fatalf("FakeTemplate(bare token) = %v, want a hint naming {/sv_SE.person.last}", err)
}
_, err = f.FakeTemplate(`{"format":"{x}","repeat":2}`)
if err == nil || !strings.Contains(err.Error(), `this template has none — write {/x}`) {
t.Errorf("FakeTemplate(fieldless object) = %v, want the hint without calling it a bare string", err)
}
// A hint is only a drop-in where the name is the whole token: {/x} inside a
// transform or an alternation renders a different value, so none is offered.
for _, input := range []string{`{ /sv_SE.person.last }`, "{lowercase(x)}", "{x|y}"} {
_, err := f.FakeTemplate(input)
if err == nil || strings.Contains(err.Error(), "write {") {
t.Errorf("FakeTemplate(%q) = %v, want no hint naming a spelling that means something else", input, err)
}
}
}
func TestFakeTemplateErrors(t *testing.T) {
f := shipped(t)
for _, c := range []struct {
input string
want string
}{
{`{"x":"Q"}`, "missing string \"format\""},
{`"{x}"`, `no field "x"`},
{`"{digits(0)}"`, "must be positive"},
{`name: {/no.such.path}`, "no entry"},
{`name: {..nope}`, "write {/nope}"},
{`{"format":"x"}`, "is a string"},
{`{/misc.country} {/misc.country.alpha2}`, "renders a level"},
{`{"format":"{/misc.country.alpha2} {x}","x":"{/misc.country}"}`, "reads a path into"},
} {
_, err := f.FakeTemplate(c.input)
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("FakeTemplate(%q) = %v, want an error containing %q", c.input, err, c.want)
}
}
}
func TestFakeTemplateJSONString(t *testing.T) {
f := shipped(t)
got := tmpl(t, f, `"name: {/sv_SE.person.last}"`)
if !strings.HasPrefix(got, "name: ") || strings.Contains(got, "{") {
t.Fatalf("FakeTemplate(JSON string) = %q, want a rendered last name after the prefix", got)
}
}
func TestPaddedJSONIsRejected(t *testing.T) {
f := shipped(t)
in := `{"format":"{x}","x":["a","b"]}`
_, err := f.NewTemplate(" " + in + " ")
if err == nil || !strings.Contains(err.Error(), "write "+in) {
t.Fatalf("NewTemplate(padded JSON) = %v, want an error naming the unpadded spelling", err)
}
}
func TestNewTemplateReusable(t *testing.T) {
f := shipped(t)
reusable, err := f.NewTemplate(`{digits(2)}`)
if err != nil {
t.Fatalf("NewTemplate: %v", err)
}
seen := map[string]bool{}
for i := 0; i < 50; i++ {
seen[reusable.Fake()] = true
}
if len(seen) < 2 {
t.Fatalf("Template.Fake() repeated %v, want varied draws from one compile", seen)
}
}
func TestFakeTemplateRepeatBound(t *testing.T) {
f := shipped(t)
_, err := f.FakeTemplate(`{"format":"{x}","repeat":200,"x":{"format":"{y}","repeat":200,"y":{"format":"z","repeat":200}}}`)
if err == nil || !strings.Contains(err.Error(), "maximum") {
t.Errorf("nested repeat over the cap = %v, want it rejected naming the maximum", err)
}
}
loading_test.go
+3 -217
View File
@@ -1,12 +1,9 @@
package fejkdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"regexp"
"slices"
"strings"
"testing"
@@ -19,8 +16,8 @@ func TestList(t *testing.T) {
"person": `{"format":"{first} {last}","first":"A","last":"B"}`,
"word": `["x", "y"]`,
"geo/city": `"Z"`,
// A bound {/path} reference is a render edge, not an addressable field.
"greeting": `{"format":"hej {/person.first} and {own}","own":"x"}`,
// A bound {..path} reference is a render edge, not an addressable field.
"greeting": `{"format":"hej {..person.first} and {own}","own":"x"}`,
// Only the fields every variant carries are addressable, so "extra" is not.
"coin": `[{"format":"{code}","code":"A","name":"Aa"},{"format":"{code}","code":"B","name":"Bb","extra":"x"}]`,
})
@@ -189,7 +186,7 @@ func TestRepeatProductAlongAPathIsCapped(t *testing.T) {
for name, files := range map[string]map[string]string{
"nested": {"cat": `{"format":"{a}","repeat":2048,"a":{"format":"{b}","repeat":2048,"b":"x"}}`},
"through a reference": {
"a": `{"format":"{/b}","repeat":2048}`,
"a": `{"format":"{..b}","repeat":2048}`,
"b": `{"format":"x","repeat":2048}`,
},
} {
@@ -204,214 +201,3 @@ func TestRepeatProductAlongAPathIsCapped(t *testing.T) {
t.Errorf("New = %v, want 1024 x 1024 along one path accepted", err)
}
}
func TestNewErrors(t *testing.T) {
// Pointing New at a file (not a directory) fails.
file := filepath.Join(t.TempDir(), "xx_XX")
if err := os.WriteFile(file, []byte("{}"), 0o644); err != nil {
t.Fatal(err)
}
if _, err := New(WithoutShippedData(), WithDataPath(file)); err == nil {
t.Error("New(file) = nil error, want not-a-directory error")
}
// Invalid JSON in a category file fails.
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{"broken": `{ not json`}))); err == nil {
t.Error("New(invalid JSON) = nil error")
}
// An option that cannot take effect, and a category or folder no dot path can
// reach, are mistakes New must name rather than accept and ignore.
rejected := map[string]struct {
files map[string]string
want string
}{
"separator without repeat": {
map[string]string{"a": `{"format":"{x}","x":"1","separator":","}`},
"has no effect without a repeat above 1",
},
"separator with an explicit repeat of 1": {
map[string]string{"a": `{"format":"{x}","x":"1","separator":","}`},
"has no effect without a repeat above 1",
},
"weight outside a choice": {
map[string]string{"a": `{"format":"x","weight":5}`},
"weight only skews a choice's items",
},
"non-numeric weight outside a choice": {
map[string]string{"a": `{"format":"x","weight":"bad"}`},
"weight only skews a choice's items",
},
"weight used as a field": {
map[string]string{"a": `{"format":"{name} {weight}kg","name":"Anvil","weight":["7"]}`},
"can never be a field",
},
"an option name used as a token": {
map[string]string{"a": `"{weight}"`},
`"weight" is an option and can never be a field`,
},
"category name with a dot": {
map[string]string{"a.b": `"1"`},
`category "a.b" contains "."`,
},
"folder name with a dot": {
map[string]string{"a.b/cat": `"1"`},
`/a.b: folder "a.b" contains "."`,
},
// The token grammar reserves three more characters. A name carrying one
// still resolves by dot path, but no format can name it, so it is rejected
// where it is authored rather than at the token that cannot reach it.
"field name with a pipe": {
map[string]string{"a": `{"format":"{x}","x":"1","b|c":"2"}`},
`field "b|c" contains "|"`,
},
"field name with a paren": {
map[string]string{"a": `{"format":"{x}","x":"1","b(c":"2"}`},
`field "b(c" contains "("`,
},
"field name with a closing brace": {
map[string]string{"a": `{"format":"{x}","x":"1","b}c":"2"}`},
`field "b}c" contains "}"`,
},
"category name with a pipe": {
map[string]string{"a|b": `"1"`},
`category "a|b" contains "|"`,
},
"field name with a bracket": {
map[string]string{"a": `{"format":"{x}","x":"1","b[c":"2"}`},
`field "b[c" contains "["`,
},
"category name with a bracket": {
map[string]string{"[abc]": `"1"`},
`category "[abc]" contains "["`,
},
"field name with a quote": {
map[string]string{"a": `{"format":"{x}","x":"1","b\"c":"2"}`},
`field "b\"c" contains "\""`,
},
// An empty name is not a path segment, so List never offered it — while a
// bare {}, a trailing dot in Fake("a.") and a {/a.} reference all reached
// it. The engine accepted spellings it would never advertise.
"empty field name": {
map[string]string{"a": `{"format":"[{}]","":"VALUE"}`},
`field "" is empty`,
},
"folder name with a paren": {
map[string]string{"a(b/cat": `"1"`},
`folder "a(b" contains "("`,
},
// A repeated arm skews an alternation, which weight is the spelling for.
"repeated alternation arm": {
map[string]string{"a": `{"format":"{x|x}","x":"1"}`},
`arm "x" is repeated`,
},
"repeated arm among others": {
map[string]string{"a": `{"format":"{x|y|x}","x":"1","y":"2"}`},
`arm "x" is repeated`,
},
"repeated path arm": {
map[string]string{"a": `{"format":"{p.v|p.v}","p":{"format":"{v}","v":"1"}}`},
`arm "p.v" is repeated`,
},
// A reference arm is the only kind that reaches the repeat check by passing
// the per-arm checks rather than falling through them.
"repeated reference arm": {
map[string]string{"a": `"x"`, "b": `"{/a|/a}"`},
`arm "/a" is repeated`,
},
// An arm that is broken on its own terms is reported as that, not as a
// repeat: the repeat is a consequence of the real mistake.
"repeated arm with no path": {
map[string]string{"a": `"{/|/}"`},
"reference has no path",
},
// No field can be named "", so the token is told that rather than sent to
// name one — the fix "no field" points at is itself a load error.
"repeated empty arm": {
map[string]string{"a": `"{|}"`},
"a name is never empty",
},
"bare empty token": {
map[string]string{"a": `{"format":"[{}]","x":"1"}`},
"a name is never empty",
},
}
for name, c := range rejected {
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, c.files)))
if err == nil {
t.Errorf("%s: New = nil error, want it rejected at load", name)
continue
}
if !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: New = %v, want it to mention %q", name, err, c.want)
}
}
// Data that works today must keep working, and stay reachable.
accepted := map[string]struct {
files map[string]string
path string
want string
}{
"option name as a field": {map[string]string{"a": `{"format":"{name} {Weight}kg","name":"Anvil","Weight":"7"}`}, "a", "Anvil 7kg"},
"format spelling as a field": {map[string]string{"a": `{"format":"{Format}","Format":"PDF"}`}, "a", "PDF"},
"hyphenated field": {map[string]string{"a": `{"format":"{x-y}","x-y":"1"}`}, "a.x-y", "1"},
"category named Format": {map[string]string{"Format": `"1"`}, "Format", "1"},
"folder named Repeat": {map[string]string{"Repeat/cat": `"1"`}, "Repeat.cat", "1"},
"field with a closing paren": {map[string]string{"a": `{"format":"{b)c}","b)c":"2"}`}, "a", "2"},
"repeat without a separator": {map[string]string{"a": `{"format":"{x}","repeat":3,"x":"1"}`}, "a", "111"},
// One name in two separate tokens is two independent draws, not a repeated
// arm; only a repeat within one alternation is rejected.
"one name in two tokens": {map[string]string{"a": `{"format":"{x}{x}","x":"1"}`}, "a", "11"},
}
for name, c := range accepted {
f, err := New(WithoutShippedData(), WithDataPath(writeData(t, c.files)))
if err != nil {
t.Errorf("%s: New = %v, want it accepted", name, err)
continue
}
if got, err := f.Fake(c.path); err != nil || got != c.want {
t.Errorf("%s: Fake(%q) = %q, %v, want %q", name, c.path, got, err, c.want)
}
}
}
func TestCategoryRootShapes(t *testing.T) {
dir := writeData(t, map[string]string{
"obj": `"{digits(2)}"`, // object root
"lit": `"hello"`, // bare-string root
})
f := newGenerator(t, dir, WithSeed(1))
if got := fake(t, f, "obj"); !regexp.MustCompile(`^\d\d$`).MatchString(got) {
t.Errorf("object-root category = %q, want two digits", got)
}
if got := fake(t, f, "lit"); got != "hello" {
t.Errorf("string-root category = %q, want hello", got)
}
}
func TestLongStringList(t *testing.T) {
const n = 2000
names := make([]string, n)
for i := range names {
names[i] = fmt.Sprintf("name-%04d", i)
}
list, err := json.Marshal(names)
if err != nil {
t.Fatal(err)
}
f := newGenerator(t, writeData(t, map[string]string{"name": string(list)}), WithSeed(1))
valid := map[string]bool{}
for _, v := range names {
valid[v] = true
}
seen := map[string]bool{}
for i := 0; i < 20000; i++ {
v := fake(t, f, "name")
if !valid[v] {
t.Fatalf("got %q, not in the list", v)
}
seen[v] = true
}
if len(seen) < n*8/10 {
t.Fatalf("only %d/%d distinct values seen; selection looks skewed", len(seen), n)
}
}
node.go
+77 -96
View File
@@ -45,7 +45,7 @@ type template struct {
grow int // minimum output size, to size the render buffer
fixed bool // no op varies, so every render is lit
lit string // the whole output when fixed
refs map[string]refBinding // each reference the format reads -> what it is bound to
refs map[string]string // each {..path} the format reads -> the head it is bound under
// bound maps each field the format addresses by dotted path to one path token
// reading it, which is the half of an overlap the fences name. nil when the
// format takes no path.
@@ -57,15 +57,6 @@ type template struct {
func (*template) isNode() {}
// field is the node a path segment names; a binding is a render edge, not a field.
func (t *template) field(seg string) (node, bool) {
if isRef(seg) {
return nil, false
}
n, ok := t.fields[seg]
return n, ok
}
// compile converts parsed JSON into a node tree, validating structure up front.
// Only a choice's items carry a weight, so one here would be inert whatever its type.
func compile(v any) (node, error) {
@@ -87,24 +78,10 @@ func compileItem(v any) (node, error) {
case map[string]any:
return compileTemplate(v)
default:
return nil, fmt.Errorf("a template value must be a string, a list or an object, not %s", jsonKind(v))
return nil, fmt.Errorf("unsupported node type %T", v)
}
}
// jsonKind names a JSON value a template cannot hold, in the data format's own
// terms rather than the decoding library's.
func jsonKind(v any) string {
switch v.(type) {
case float64:
return "a number"
case bool:
return "a boolean"
case nil:
return "null"
}
return fmt.Sprintf("%T", v)
}
func compileString(s string) (node, error) {
if err := checkTokens(s, nil); err != nil {
return nil, err
@@ -167,13 +144,13 @@ func compileChoice(items []any) (node, error) {
c.items[i] = n
}
if weighted { // uniform choices skip the weight table and pick in O(1)
if math.IsInf(total, 1) {
return nil, fmt.Errorf("choice weights must sum to a finite number, got %v", total)
if total <= 0 || math.IsInf(total, 1) {
return nil, fmt.Errorf("choice weights must sum to a finite positive number, got %v", total)
}
c.cum = cum
}
// Computed before linkRefs binds references into the items: a binding is keyed
// by a reference sigil, which paths skips, so the set is the same after.
// Safe to precompute: a choice's items come from one file, so no group can
// appear inside one, and neither mergeChildren nor linkRefs can reach in.
c.shared = sharedPaths(c.items)
return c, nil
}
@@ -204,75 +181,36 @@ func checkNoRepeatedItem(items []any) error {
}
func compileTemplate(m map[string]any) (node, error) {
o, err := readOptions(m)
if err != nil {
return nil, err
}
fields, err := compileFields(m)
if err != nil {
return nil, err
}
if len(fields) == 0 && o.repeat == 1 && !o.weighted {
return nil, fmt.Errorf("an object holding only a format is a string; write %q", o.format)
}
if err := checkTokens(o.format, fields); err != nil {
return nil, err
}
t := &template{format: o.format, fields: fields, repeat: o.repeat, separator: o.separator}
if err := t.compileFormat(); err != nil {
return nil, err
}
return t, nil
}
// templateOptions is what a template object's option keys say.
type templateOptions struct {
format string
repeat int
separator string
weighted bool
}
func readOptions(m map[string]any) (templateOptions, error) {
var o templateOptions
format, ok := m["format"].(string)
if !ok {
return o, fmt.Errorf("template object missing string \"format\"")
return nil, fmt.Errorf("template object missing string \"format\"")
}
o.format = format
repeat, err := repeatOf(m)
if err != nil {
return o, err
return nil, err
}
o.repeat = repeat
sep := ""
if sv, ok := m["separator"]; ok {
if o.separator, ok = sv.(string); !ok {
return o, fmt.Errorf("separator must be a string, got %T", sv)
if sep, ok = sv.(string); !ok {
return nil, fmt.Errorf("separator must be a string, got %T", sv)
}
if repeat == 1 {
return o, fmt.Errorf("separator joins repeated renders, so it has no effect without a repeat above 1")
}
if o.separator == "" {
return o, fmt.Errorf("separator \"\" is the default, so it has no effect; drop it")
return nil, fmt.Errorf("separator joins repeated renders, so it has no effect without a repeat above 1")
}
}
_, o.weighted = m["weight"]
return o, nil
}
// compileFields compiles every non-option key of a template object, in name order
// so which of several bad fields is reported does not vary.
func compileFields(m map[string]any) (map[string]node, error) {
fields := make(map[string]node, len(m))
t := &template{format: format, fields: make(map[string]node, len(m)), repeat: repeat, separator: sep}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
sort.Strings(keys) // so which of several bad fields is reported does not vary
for _, k := range keys {
if isOption(k) {
continue
}
if isRef(k) {
return nil, fmt.Errorf("field %q starts with %q, which is reserved for {..path} bindings", k, refPrefix)
}
if err := checkName(k); err != nil {
return nil, fmt.Errorf("field %w", err)
}
@@ -280,9 +218,55 @@ func compileFields(m map[string]any) (map[string]node, error) {
if err != nil {
return nil, fmt.Errorf("field %q: %w", k, err)
}
fields[k] = n
t.fields[k] = n
}
if _, weighted := m["weight"]; len(t.fields) == 0 && repeat == 1 && !weighted {
return nil, fmt.Errorf("an object holding only a format is a string; write %q", format)
}
if err := checkTokens(format, t.fields); err != nil {
return nil, err
}
if err := t.compileFormat(); err != nil {
return nil, err
}
return t, nil
}
// checkPath reports whether a token's dotted tail can address a node whichever way
// the draw goes, by the reachability rule descend applies — a choice must carry
// the whole remaining path in the set every variant shares — plus the
// rules a held draw adds, which descend has no need of: a level a path reads may
// not carry a repeat, and each variant answers for that itself. So a path that
// validates here resolves on every render, and a typo is a New-time error.
func checkPath(n node, tail []string, level string) error {
if len(tail) == 0 {
return nil
}
switch n := n.(type) {
case *template:
if n.repeat > 1 {
return fmt.Errorf("the level %q carries a repeat, which a path reading one draw of it cannot apply", level)
}
child, ok := n.fields[tail[0]]
if !ok {
return fmt.Errorf("no field %q", tail[0])
}
return checkPath(child, tail[1:], level+"."+tail[0])
case *choice:
if want := strings.Join(tail, "."); !n.shared[want] {
return unreachableInChoice(n, want)
}
// Reachability is settled; each variant still answers for itself, so a
// rule about the level (its repeat) holds behind a choice as in front.
for _, item := range n.items {
if err := checkPath(item, tail, level); err != nil {
return err
}
}
return nil
default:
return fmt.Errorf("cannot descend into %T at %q", n, tail[0])
}
return fields, nil
}
// repeatOf reads a template's "repeat" (default 1): how many times its format
@@ -302,14 +286,14 @@ func repeatOf(m map[string]any) (int, error) {
if r == 1 {
return 0, fmt.Errorf("repeat 1 is the default, so it has no effect; drop it")
}
if r > MaxRepeat { // caps the renders one repeat asks for; repeatCheck bounds what nested ones multiply to
return 0, fmt.Errorf("repeat %v exceeds the maximum %d", rv, MaxRepeat)
if r > maxLen { // cap so a fat-fingered repeat can't build a multi-GB string
return 0, fmt.Errorf("repeat %v exceeds the maximum %d", rv, maxLen)
}
return int(r), nil
}
// weightOf reads a node's "weight" (default 1) from its raw JSON form. Only
// template objects carry weight; a present one must be finite and positive.
// template objects carry weight; a present one must be finite and non-negative.
func weightOf(raw any) (float64, error) {
m, ok := raw.(map[string]any)
if !ok {
@@ -324,10 +308,7 @@ func weightOf(raw any) (float64, error) {
return 0, fmt.Errorf("weight must be a number, got %T", wv)
}
if w < 0 || math.IsNaN(w) || math.IsInf(w, 0) {
return 0, fmt.Errorf("weight must be finite and positive, got %v", w)
}
if w == 0 {
return 0, fmt.Errorf("weight 0 means the item is never drawn; remove the item instead")
return 0, fmt.Errorf("weight must be finite and non-negative, got %v", w)
}
if w == 1 {
return 0, fmt.Errorf("weight 1 is the default, so it has no effect; drop it")
@@ -337,23 +318,23 @@ func weightOf(raw any) (float64, error) {
// reservedInName is what a category, folder or field name may not contain: a dot
// separates the segments of a path, '|' the arms of a token, '(' opens a function
// call, braces delimit the token, '/' starts a reference, and brackets and a quote
// open a JSON value. A name carrying one is rejected where it is authored rather
// than where it would be unreachable.
const reservedInName = ".|({}/[]\""
// call and braces delimit the token. A name carrying one is reachable by no format,
// so it is rejected where it is authored rather than at the token that cannot
// reach it.
const reservedInName = ".|({}"
// reservedList spells reservedInName for an error message, so the two cannot drift.
var reservedList = strings.Join(strings.Split(reservedInName, ""), " ")
// checkName rejects a name the dot path, {token} and JSON grammars cannot spell.
// Both a category or folder and a field go through it, so there is one answer to
// what a name may contain.
// checkName rejects a name the dot path and {token} grammars cannot spell. Both a
// category or folder and a field go through it, so there is one answer to what a
// name may contain.
func checkName(name string) error {
if name == "" {
return fmt.Errorf("%q is empty, which is not a path segment, so List never offers it", name)
}
if i := strings.IndexAny(name, reservedInName); i >= 0 {
return fmt.Errorf("%q contains %q; a name may not use %s, which the dot path, {token} and JSON grammars reserve",
return fmt.Errorf("%q contains %q; a name may not use %s, which the dot path and {token} grammars reserve",
name, name[i:i+1], reservedList)
}
return nil
one_spelling_test.go
-11
View File
@@ -35,7 +35,6 @@ func TestInertObjectIsRejected(t *testing.T) {
`[{"format":"a","weight":1},"b"]`: "weight 1",
`{"format":"{x}","x":"v","repeat":1}`: "repeat 1",
`{"format":"{x}","x":"v","separator":","}`: "separator",
`{"format":"{x}","x":"v","repeat":2,"separator":""}`: "default",
} {
if _, err := compile(parse(t, src)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("compile(%s) = %v, want an error mentioning %s", src, err, want)
@@ -47,13 +46,3 @@ func TestInertObjectIsRejected(t *testing.T) {
}
}
}
func TestInlineFolderSigilsAreRejected(t *testing.T) {
f := shipped(t)
for _, input := range []string{"{.sv_SE.person.last}", "{..sv_SE.person.last}"} {
_, err := f.NewTemplate(input)
if err == nil || !strings.Contains(err.Error(), "write {/sv_SE.person.last}") {
t.Errorf("NewTemplate(%q) = %v, want an error naming the root spelling", input, err)
}
}
}
path.go
-110
View File
@@ -1,110 +0,0 @@
package fejkdata
import (
"fmt"
"sort"
"strings"
)
// pathWalk is what one walk of a dotted path does at each kind of level: choice
// returns the variants to continue into (none stops the walk); level runs at each
// template a segment descends into; leaf runs where the tail ends. A nil action
// is skipped.
type pathWalk struct {
choice func(c *choice, rest []string) ([]node, error)
level func(t *template, rest []string) error
leaf func(n node) error
}
// walkPath descends tail from n: a group or template by its next segment, a
// choice by w.choice, which consumes no segment. A missing segment is an error,
// so no walk reaches past what the data holds. A table-shaped dispatch, one case
// per node kind, kept whole on purpose.
func walkPath(n node, tail []string, w pathWalk) error {
if len(tail) == 0 {
if w.leaf != nil {
return w.leaf(n)
}
return nil
}
switch n := n.(type) {
case *group:
child, ok := n.children[tail[0]]
if !ok {
return fmt.Errorf("no entry %q", tail[0])
}
return walkPath(child, tail[1:], w)
case *template:
if w.level != nil {
if err := w.level(n, tail); err != nil {
return err
}
}
child, ok := n.field(tail[0])
if !ok {
return fmt.Errorf("no field %q", tail[0])
}
return walkPath(child, tail[1:], w)
case *choice:
if w.choice == nil {
return nil
}
next, err := w.choice(n, tail)
if err != nil {
return err
}
for _, item := range next {
if err := walkPath(item, tail, w); err != nil {
return err
}
}
return nil
}
return fmt.Errorf("cannot descend into %T at %q", n, tail[0])
}
// carriedByAll is the choice rule a path that must resolve on every call obeys:
// the rest of the tail must be one every variant carries.
func carriedByAll(c *choice, rest []string) error {
if want := strings.Join(rest, "."); !c.shared[want] {
return unreachableInChoice(c, want)
}
return nil
}
// unreachableInChoice reports that a path cannot step through this choice, listing
// what every variant does carry. It reads the precomputed set, so a failing path
// costs no more than a rendering one.
func unreachableInChoice(c *choice, want string) error {
if len(c.shared) == 0 {
return fmt.Errorf("no variant of this %d-way choice carries %q", len(c.items), want)
}
offered := make([]string, 0, len(c.shared))
for p := range c.shared {
offered = append(offered, p)
}
sort.Strings(offered)
return fmt.Errorf("not every variant of this %d-way choice carries %q; all carry %v", len(c.items), want, offered)
}
// checkPath proves a dotted tail resolves whichever way the draws go — a choice
// must carry the rest of the path in the set every variant shares — and that no
// level a path reads carries a repeat, which one draw of it could not apply. So a
// path that validates here resolves on every render, and a typo is a New-time
// error.
func checkPath(n node, tail []string, level string) error {
return walkPath(n, tail, pathWalk{
choice: func(c *choice, rest []string) ([]node, error) {
if err := carriedByAll(c, rest); err != nil {
return nil, err
}
return c.items, nil
},
level: func(t *template, rest []string) error {
if t.repeat > 1 {
return fmt.Errorf("the level %q carries a repeat, which a path reading one draw of it cannot apply", join(level, strings.Join(tail[:len(tail)-len(rest)], ".")))
}
return nil
},
})
}
path_test.go
-152
View File
@@ -1,152 +0,0 @@
package fejkdata
import (
"slices"
"strings"
"testing"
)
func TestWalkPathStopsAtAMissingSegment(t *testing.T) {
n := compiled(t, `{"format":"{a}","a":{"format":"{b}","b":"leaf"}}`)
var seen []string
walk := pathWalk{
level: func(tm *template, rest []string) error { seen = append(seen, "level:"+rest[0]); return nil },
leaf: func(n node) error { seen = append(seen, "leaf"); return nil },
}
if err := walkPath(n, []string{"a", "b"}, walk); err != nil {
t.Fatalf("walkPath(a.b) = %v", err)
}
if want := []string{"level:a", "level:b", "leaf"}; !slices.Equal(seen, want) {
t.Errorf("walk visited %v, want %v", seen, want)
}
seen = nil
err := walkPath(n, []string{"a", "nope", "deeper"}, walk)
if err == nil || !strings.Contains(err.Error(), `no field "nope"`) {
t.Errorf("walkPath(a.nope.deeper) = %v, want the missing segment named", err)
}
if slices.Contains(seen, "leaf") {
t.Errorf("walk reached a leaf past a missing segment: %v", seen)
}
}
func TestWalkPathChoiceConsumesNoSegment(t *testing.T) {
n := compiled(t, `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2"}]`)
var leaves []node
err := walkPath(n, []string{"f"}, pathWalk{
choice: func(c *choice, rest []string) ([]node, error) {
if len(rest) != 1 || rest[0] != "f" {
t.Errorf("choice saw rest %v, want [f]", rest)
}
return c.items, nil
},
leaf: func(n node) error { leaves = append(leaves, n); return nil },
})
if err != nil || len(leaves) != 2 {
t.Fatalf("walkPath through a choice = %v, %d leaves, want both variants' f", err, len(leaves))
}
}
func TestDeepDottedPath(t *testing.T) {
// A 5-segment path descends through alternating object/array nodes; choices
// on the path are single-variant, so it resolves deterministically.
f := engine(1)
f.categories = map[string]node{
"deep": compiled(t, `{"format":"{a}","a":{"format":"{b}","b":{"format":"{c}","c":{"format":"{d}","d":"leaf"}}}}`),
}
if got, err := f.Fake("deep.a.b.c.d"); err != nil || got != "leaf" {
t.Fatalf("Fake(deep.a.b.c.d) = %q, %v, want leaf", got, err)
}
// Rendering the whole tree resolves the same chain.
if got, err := f.Fake("deep"); err != nil || got != "leaf" {
t.Fatalf("Fake(deep) = %q, %v, want leaf", got, err)
}
}
func TestDescendIntoStringErrors(t *testing.T) {
f := engine(1)
f.categories = map[string]node{"greeting": compiled(t, `"hej"`)}
if _, err := f.Fake("greeting.extra"); err == nil || !strings.Contains(err.Error(), `no field "extra"`) {
t.Fatalf("Fake(greeting.extra) = %v, want a no-field error", err)
}
}
// TestPathThroughChoice pins the rule that keeps a dotted path from rendering on
// one call and failing on the next: every variant must carry the rest of the path.
func TestPathThroughChoice(t *testing.T) {
dir := writeData(t, map[string]string{
"every": `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2"}]`,
"notall": `[{"format":"{f}","f":"1"},"plain"]`,
"some": `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2","extra":"x"}]`,
})
f := newGenerator(t, dir, WithSeed(1))
for i := 0; i < 200; i++ {
if got := fake(t, f, "every.f"); got != "1" && got != "2" {
t.Fatalf("every.f = %q, want 1 or 2", got)
}
}
// A path only some variants carry is reported against what all of them carry.
if _, err := f.Fake("some.extra"); err == nil || !strings.Contains(err.Error(), "all carry [f]") {
t.Errorf("Fake(some.extra) = %v, want it to name what every variant carries", err)
}
var first string
for i := 0; i < 200; i++ {
_, err := f.Fake("notall.f")
if err == nil {
t.Fatal("notall.f = nil error, want the same failure every call")
}
if i == 0 {
first = err.Error()
} else if err.Error() != first {
t.Fatalf("notall.f error varies between calls:\n %s\n %s", first, err.Error())
}
}
}
// TestPathKeyIsUnambiguous pins that the one shape which could collide cannot be
// written: a field literally named "a.b" and a field "a" holding "b" would both
// spell "a.b", so a dotted field name is rejected at New and a dot means a path
// wherever it appears.
func TestPathKeyIsUnambiguous(t *testing.T) {
dir := writeData(t, map[string]string{
"cat": `[{"format":"{a.b}","a.b":"1"},{"format":"{a}","a":{"format":"{b}","b":"2"}}]`,
})
_, err := New(WithoutShippedData(), WithDataPath(dir))
if err == nil || !strings.Contains(err.Error(), `field "a.b" contains "."`) {
t.Fatalf("New = %v, want the dotted field name rejected", err)
}
// The same data without the dotted key is fine, and the path resolves.
f := newGenerator(t, writeData(t, map[string]string{
"cat": `{"format":"{a.b}","a":{"format":"{b}","b":"2"}}`,
}), WithSeed(1))
if !slices.Contains(f.List(), "cat.a.b") {
t.Error("List() omits cat.a.b, which the data carries")
}
if got := fake(t, f, "cat"); got != "2" {
t.Fatalf("cat = %q, want 2", got)
}
}
// TestMissingFieldNamesItself keeps the precise diagnosis for the ordinary typo: a
// single-variant choice always picks the same item, so it needs no every-variant
// guard and the error can name the field that is missing.
func TestMissingFieldNamesItself(t *testing.T) {
f := newGenerator(t, "data/sv_SE", WithSeed(1))
_, err := f.Fake("person.typo")
if err == nil || !strings.Contains(err.Error(), `no field "typo"`) {
t.Errorf("Fake(person.typo) = %v, want it to name the missing field", err)
}
}
func TestBindingKeyIsNotAPathSegment(t *testing.T) {
dir := writeData(t, map[string]string{
"color": `["red","blue"]`,
"name": `{"format":"{/color} {w}","w":"x"}`,
})
f := newGenerator(t, dir, WithSeed(1))
if slices.Contains(f.List(), "name./color") {
t.Error("List() advertises a binding key")
}
if _, err := f.Fake("name./color"); err == nil || !strings.Contains(err.Error(), `no field "/color"`) {
t.Errorf("Fake(name./color) = %v, want a no-field error", err)
}
}
perf_test.go
-61
View File
@@ -1,61 +0,0 @@
package fejkdata
import (
"fmt"
"strings"
"testing"
"testing/fstest"
)
// One Fake call descends depth levels: each level's "a" is the next template down.
func nestedJSON(depth int) string {
s := `"leaf"`
for i := 0; i < depth; i++ {
s = fmt.Sprintf(`{"format":"{a}","a":%s}`, s)
}
return s
}
// One Fake call expands n sibling tokens.
func wideTokenJSON(n int) string {
var toks, fields strings.Builder
for i := 0; i < n; i++ {
fmt.Fprintf(&toks, "{f%d}", i)
if i > 0 {
fields.WriteByte(',')
}
fmt.Fprintf(&fields, `"f%d":"x"`, i)
}
return fmt.Sprintf(`{"format":"%s",%s}`, toks.String(), fields.String())
}
func TestNoRenderAllocRegression(t *testing.T) {
shapes := []struct {
name string
json string
base float64
}{
{"nested depth 25", nestedJSON(25), 27},
{"nested depth 100", nestedJSON(100), 102},
{"wide 500 tokens", wideTokenJSON(500), 9},
}
for _, s := range shapes {
f, err := New(WithoutShippedData(), WithDataFS(fstest.MapFS{"x.json": {Data: []byte(s.json)}}))
if err != nil {
t.Fatalf("New(%s): %v", s.name, err)
}
allocs := testing.AllocsPerRun(10000, func() { f.Fake("x") })
if allocs > s.base*1.10 {
t.Errorf("%s: %.1f allocs/op regressed past %.1f (baseline %.1f + 10%%); bump the baseline only as a deliberate change", s.name, allocs, s.base*1.10, s.base)
}
}
}
func BenchmarkNestedDepth25(b *testing.B) { benchPath(b, tmpData(b, "deep", nestedJSON(25)), "deep") }
func BenchmarkNestedDepth100(b *testing.B) { benchPath(b, tmpData(b, "deep", nestedJSON(100)), "deep") }
func BenchmarkWideTokens100(b *testing.B) {
benchPath(b, tmpData(b, "wide", wideTokenJSON(100)), "wide")
}
func BenchmarkWideTokens500(b *testing.B) {
benchPath(b, tmpData(b, "wide", wideTokenJSON(500)), "wide")
}
reference.go
+365 -103
View File
@@ -2,84 +2,30 @@ package fejkdata
import (
"fmt"
"sort"
"strings"
)
// A reference names a node by path rather than as a sibling field: {/a.b} from the
// data root, {.a} from the folder this file sits in, {..a} from the folder above.
func isRef(name string) bool { return strings.HasPrefix(name, ".") || strings.HasPrefix(name, "/") }
// refPrefix marks a {..path} token: a reference to a node elsewhere in the data
// root rather than a sibling field. The path is resolved across every loaded
// directory (see linkRefs).
const refPrefix = ".."
// refBinding is what a reference was bound to: the head key its category is held
// under, and the tail read into it.
type refBinding struct {
key string
tail []string
}
func isRef(name string) bool { return strings.HasPrefix(name, refPrefix) }
// refShape splits a reference into its sigil and the dotted path after it.
func refShape(name string) (sigil, rest string, err error) {
switch {
case strings.HasPrefix(name, "/"):
sigil, rest = "/", name[1:]
case strings.HasPrefix(name, ".."):
sigil, rest = "..", name[2:]
default:
sigil, rest = ".", name[1:]
}
if rest == "" {
return "", "", fmt.Errorf("reference has no path")
}
if strings.HasPrefix(rest, "/") {
return "", "", fmt.Errorf("the path after %s starts at a name, not a /; write {%s%s}", sigil, sigil, rest[1:])
}
if strings.HasPrefix(rest, ".") {
return "", "", fmt.Errorf("a reference starts with / (the root), . (this folder) or .. (the folder above)")
}
for _, seg := range strings.Split(rest, ".") {
if seg == "" {
return "", "", fmt.Errorf("path has an empty segment")
}
}
return sigil, rest, nil
}
// refSegments resolves a reference written in folder to a path from the root.
func refSegments(name string, folder []string) ([]string, error) {
sigil, rest, err := refShape(name)
if err != nil {
return nil, err
}
var base []string
switch sigil {
case ".":
base = folder
case "..":
if len(folder) == 0 {
return nil, fmt.Errorf("no folder above the root")
}
base = folder[:len(folder)-1]
}
return append(append([]string{}, base...), strings.Split(rest, ".")...), nil
}
// linkRefs resolves every reference in the assembled tree. The head of the path —
// up to the category it names — is bound into the referring template's fields
// under its root path, and the rest reads into it the way a sibling path does, so
// a reference is held like a sibling and two spellings of one target are one
// draw. It runs once, after all data is merged, so a reference sees the final
// (override-resolved) tree. A path that is unknown, names a folder, or reads a
// field not every variant carries fails here, keeping a bad reference a New-time
// error, never a random render-time one.
// linkRefs resolves every {..path} reference in the assembled tree. The head of the
// path — up to the category it names — is bound into the referring template's
// fields, and the rest reads into it the way a sibling path does, so a reference
// is held like a sibling. It runs once, after all data is merged, so a reference
// sees the final (override-resolved) tree. A path that is unknown, names a folder,
// or reads a field not every variant carries fails here, keeping a bad reference a
// New-time error, never a random render-time one.
func linkRefs(root map[string]node) error {
return eachTemplate(root, func(folder []string, path string, t *template) error {
return linkTemplateRefs(folder, path, t, root)
})
return walkNodes(root, func(path string, n node) error {
t, ok := n.(*template)
if !ok {
return nil
}
// linkTemplateRefs binds one template's references against root. A template with
// none is left untouched, so an inline format that references nothing costs only
// the refTokens scan.
func linkTemplateRefs(folder []string, path string, t *template, root map[string]node) error {
names := refTokens(t.format)
if len(names) == 0 {
return nil
@@ -87,69 +33,270 @@ func linkTemplateRefs(folder []string, path string, t *template, root map[string
if t.fields == nil {
t.fields = map[string]node{}
}
t.refs = make(map[string]refBinding, len(names))
t.refs = make(map[string]string, len(names))
for _, name := range names {
segments, err := refSegments(name, folder)
head, target, tail, err := resolveRef(root, strings.Split(name[len(refPrefix):], "."))
if err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
}
head, target, tail, err := resolveRef(root, segments)
if err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
}
key := "/" + strings.Join(head, ".")
key := refPrefix + strings.Join(head, ".")
if err := checkPath(target, tail, key); err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
}
t.fields[key] = target
t.refs[name] = refBinding{key, tail}
t.refs[name] = key
}
if err := t.compileFormat(); err != nil {
return fmt.Errorf("%s: %w", path, err)
}
return nil
})
}
// eachTemplate calls fn once per template, with the folder its category sits in
// and the dot path reaching it, folders and names in sorted order.
func eachTemplate(root map[string]node, fn func(folder []string, path string, t *template) error) error {
var inCategory func(folder []string, path string, n node) error
inCategory = func(folder []string, path string, n node) error {
if t, ok := n.(*template); ok {
if err := fn(folder, path, t); err != nil {
return err
// checkBoundLevelsHeld rejects every route to a held name except the ones that read
// its draw. An expansion holds one draw of that name; anything else that renders it
// draws again, and the two disagree. checkNoOverlap settles the spellings within one
// format (a token, an operand); this settles the rest — a reference, whether it
// sits in that format or in anything the format renders, however deep.
//
// It runs after checkNoCycles, whose guarantee is what lets the walk terminate.
func checkBoundLevelsHeld(root map[string]node) error {
return walkNodes(root, func(path string, n node) error {
t, ok := n.(*template)
if !ok || len(t.held) == 0 {
return nil
}
heads := make([]string, 0, len(t.held))
for head := range t.held {
heads = append(heads, head)
}
// Operand heads first, then paths, each in name order: a level read both
// ways is reported by the operand's fence, and which overlap is reported
// does not vary.
sort.Slice(heads, func(i, j int) bool {
_, pi := t.bound[heads[i]]
_, pj := t.bound[heads[j]]
if pi != pj {
return !pi
}
return heads[i] < heads[j]
})
readers := boundReaders(t.format, t.bound, t.refs)
for _, head := range heads {
// What one draw answers for depends on how the draw is read: a path pins
// the levels it passes through and the leaf it lands on, an operand
// exactly the value its render produces.
held := map[node]bool{}
reader, isPath := t.bound[head]
if isPath {
for _, r := range readers {
if a := splitArm(r.name, t.refs); a.key == head {
coverPath(t.fields[head], a.tail, held)
}
}
for _, c := range contained(n) {
if err := inCategory(folder, join(path, c.name), c.node); err != nil {
return err
} else {
operandDraw(t.fields[head], held)
}
if len(held) == 0 {
continue // an early out: a fixed head holds nothing to reach
}
// One seen set across the edges: a node that cannot reach the level
// cannot reach it by another route either, so it is walked once here.
seen := map[node]bool{}
for _, e := range renderEdges(t) {
if splitArm(e.label, t.refs).key == head {
continue // a token or operand reading this draw, the routes allowed
}
if renders(e.to, held, seen) {
if isPath {
return fmt.Errorf("%s: %s renders %q, which {%s} reads a path into; name the fields you want instead", path, e.reached(), head, reader)
}
return fmt.Errorf("%s: %s renders %q, which a {%s()} also reads; reach it one way so it is drawn once", path, e.reached(), head, operandReader(t, head))
}
}
}
return nil
})
}
var inFolder func(folder []string, children map[string]node) error
inFolder = func(folder []string, children map[string]node) error {
for _, name := range sortedNames(children) {
path := join(strings.Join(folder, "."), name)
if g, ok := children[name].(*group); ok {
if err := inFolder(append(folder[:len(folder):len(folder)], name), g.children); err != nil {
return err
// operandReader names the builtin whose operand holds head.
func operandReader(t *template, head string) string {
fn := ""
_ = eachToken(t.format, func(tok ftoken) error {
if tok.kind != 'b' || fn != "" {
return nil
}
if name, _, isFunc := funcCall(tok.body); isFunc {
for _, operand := range tokenOperands(tok.body) {
if splitArm(operand, t.refs).key == head {
fn = name
}
}
}
return nil
})
return fn
}
// coverPath collects what holding one path pins: every choice level the path
// passes through, whole, and the leaf it renders.
func coverPath(n node, tail []string, into map[node]bool) {
if _, isChoice := n.(*choice); isChoice || len(tail) == 0 {
cover(n, into, false)
return
}
t, ok := n.(*template)
if !ok {
return
}
if child, ok := t.fields[tail[0]]; ok {
coverPath(child, tail[1:], into)
}
}
// cover collects a level and everything contained in it. A fixed string outside a
// choice is left out — it cannot disagree with itself — but inside one each
// variant carries its own, so there it counts.
func cover(n node, into map[node]bool, inChoice bool) {
if isFixed(n) && !inChoice {
return
}
into[n] = true
_, isChoice := n.(*choice)
for _, c := range contained(n) {
cover(c.node, into, inChoice || isChoice)
}
}
// operandDraw collects what one held draw of an operand answers for: the operand
// and what rendering it settles inside itself. The builtin renders its operand
// whole, so that draw fixes every value the render produced, and a second route to
// any of them disagrees with it.
//
// The walk stops at a {..path} edge, which is where the operand's own value ends
// and a shared source begins: two names referencing one category are two draws, the
// same rule {word} {word} follows.
func operandDraw(n node, into map[node]bool) {
if isFixed(n) {
return
}
if into[n] {
return
}
into[n] = true
for _, e := range renderEdges(n) {
if isRef(e.label) {
continue
}
if err := inCategory(folder, path, children[name]); err != nil {
operandDraw(e.to, into)
}
}
// isFixed is a string that varies nothing: fixed text with no fields to read into.
func isFixed(n node) bool {
t, ok := n.(*template)
return ok && t.fixed && len(t.fields) == 0
}
// renders reports whether rendering n can reach anything in want, following the
// same edges expand does. seen keeps a node shared by several routes from being
// walked twice; checkNoCycles has already proved the graph is a DAG, so the walk
// ends.
func renders(n node, want, seen map[node]bool) bool {
if want[n] {
return true
}
if seen[n] {
return false
}
seen[n] = true
for _, e := range renderEdges(n) {
if renders(e.to, want, seen) {
return true
}
}
return false
}
// walkNodes calls fn once per contained node, passing the dot path that reaches it,
// visiting keys in sorted order so which of several broken nodes gets reported does
// not depend on map iteration.
func walkNodes(root map[string]node, fn func(path string, n node) error) error {
seen := map[node]bool{}
var visit func(string, node) error
visit = func(path string, n node) error {
if n == nil || seen[n] {
return nil
}
seen[n] = true
if err := fn(path, n); err != nil {
return err
}
for _, c := range contained(n) {
if err := visit(join(path, c.name), c.node); err != nil {
return err
}
}
return nil
}
return inFolder(nil, root)
for _, name := range sortedNames(root) {
if err := visit(name, root[name]); err != nil {
return err
}
}
return nil
}
// namedNode is a contained child and the segment reaching it; a choice's items carry
// no segment, matching how a dot path steps over a choice.
type namedNode struct {
name string
node node
}
func contained(n node) []namedNode {
switch n := n.(type) {
case *group:
return named(n.children)
case *choice:
out := make([]namedNode, len(n.items))
for i, it := range n.items {
out[i] = namedNode{node: it}
}
return out
case *template:
return named(n.fields)
default:
return nil
}
}
// named skips a bound {..path} key: it is a render edge, not containment, so using
// it as a path segment would report a node under a path that does not reach it. Only
// a template's fields hold bindings — loadDir skips a dot-prefixed entry, so a
// group's children never carry the prefix — so this one skip serves both.
func named(m map[string]node) []namedNode {
out := make([]namedNode, 0, len(m))
for _, name := range sortedNames(m) {
if isRef(name) {
continue
}
out = append(out, namedNode{name: name, node: m[name]})
}
return out
}
func sortedNames(m map[string]node) []string {
names := make([]string, 0, len(m))
for name := range m {
names = append(names, name)
}
sort.Strings(names)
return names
}
// resolveRef walks a reference path through the folders to the category it names,
// returning that head, the node, and the tail left to read into it. A descent of
// its own rather than a walkPath: it walks groups only and returns where they end,
// not a leaf.
// returning that head, the node, and the tail left to read into it.
func resolveRef(root map[string]node, segments []string) (head []string, target node, tail []string, err error) {
var n node = &group{children: root}
i := 0
@@ -170,7 +317,7 @@ func resolveRef(root map[string]node, segments []string) (head []string, target
return segments[:i], n, segments[i:], nil
}
// refTokens returns the reference names a format reads, as tokens or as operands.
// refTokens returns the {..path} names a format reads, as tokens or as operands.
func refTokens(format string) []string {
var refs []string
seen := map[string]bool{}
@@ -182,3 +329,118 @@ func refTokens(format string) []string {
}
return refs
}
// renderEdge is a child a node renders into, labelled by what reaches it (a field
// name, reference, or choice index) for a readable cycle report. operand names
// the builtin when the label is its operand rather than a token, so an error can
// name it the way the author wrote it.
type renderEdge struct {
to node
label string
operand string
}
// reached names an edge as the author spelled it, the vocabulary boundReaders uses
// for the sibling fence.
func (e renderEdge) reached() string {
if e.operand != "" {
return fmt.Sprintf("%s operand %q", e.operand, e.label)
}
return "{" + e.label + "}"
}
// renderEdges lists the children rendering n recurses into, mirroring expand: a
// choice's items, and a template's field/reference tokens plus its operands. A
// group renders nothing, so it has no edges.
func renderEdges(n node) []renderEdge {
switch n := n.(type) {
case *choice:
es := make([]renderEdge, len(n.items))
for i, it := range n.items {
es[i] = renderEdge{to: it, label: fmt.Sprintf("[%d]", i)}
}
return es
case *template:
var es []renderEdge
add := func(name, operand string) {
a := splitArm(name, n.refs)
c, ok := n.fields[a.key]
if !ok {
return
}
for _, leaf := range pathLeaves(c, a.tail) {
es = append(es, renderEdge{leaf, name, operand})
}
}
_ = eachToken(n.format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if fn, _, isFunc := funcCall(t.body); isFunc {
for _, operand := range tokenOperands(t.body) {
add(operand, fn)
}
return nil
}
for _, name := range strings.Split(t.body, "|") {
add(name, "")
}
return nil
})
return es
default:
return nil
}
}
// pathLeaves lists what a token's dotted tail renders. A path draws the levels it
// passes through but renders only what it lands on, so the leaf is the edge — a
// bare token, whose tail is empty, lands on the field itself. A choice on the way
// contributes every variant, since any of them may be the one drawn. checkPath has
// already proved the tail resolves in every variant, so the walk drops nothing.
func pathLeaves(n node, tail []string) []node {
if len(tail) == 0 {
return []node{n}
}
if c, ok := n.(*choice); ok {
var out []node
for _, it := range c.items {
out = append(out, pathLeaves(it, tail)...)
}
return out
}
return pathLeaves(child(n, tail[0]), tail[1:])
}
// checkNoCycles rejects a reference cycle: a node whose rendering can reach itself
// — directly, mutually, or through a chain — never terminates, so it must fail at
// New rather than stack-overflow at render. It is a depth-first walk of the render
// graph (renderEdges); grey marks nodes on the current path so a back-edge to one
// is the cycle, while black lets a shared node (a DAG, not a cycle) be skipped.
// Every node is a root: a field its parent's format never renders is still reachable
// by dot path, so a cycle in one would otherwise reach render and be fatal there.
func checkNoCycles(root map[string]node) error {
const (
grey = 1
black = 2
)
color := map[node]int{}
var visit func(n node, path string) error
visit = func(n node, path string) error {
switch color[n] {
case grey:
return fmt.Errorf("reference cycle: %s", path)
case black:
return nil
}
color[n] = grey
for _, e := range renderEdges(n) {
if err := visit(e.to, path+" -> "+e.label); err != nil {
return err
}
}
color[n] = black
return nil
}
return walkNodes(root, func(path string, n node) error { return visit(n, path) })
}
reference_test.go
+39 -120
View File
@@ -6,11 +6,11 @@ import (
)
// TestRootReferenceAcrossFolders is the headline case: a category in one folder
// pulls a value from another via a {/path} reference resolved from the data root.
// pulls a value from another via a {..path} reference resolved from the data root.
func TestRootReferenceAcrossFolders(t *testing.T) {
dir := writeData(t, map[string]string{
"en_US/person": `"Pat Smith"`,
"sv_SE/greeting": `"Hej, {/en_US.person}!"`,
"sv_SE/greeting": `"Hej, {..en_US.person}!"`,
})
f := newGenerator(t, dir, WithSeed(1))
if got := fake(t, f, "sv_SE.greeting"); got != "Hej, Pat Smith!" {
@@ -22,7 +22,7 @@ func TestRootReferenceAcrossFolders(t *testing.T) {
func TestReferenceIntoAField(t *testing.T) {
dir := writeData(t, map[string]string{
"who": `{"format":"{first} {last}","first":"Ada","last":"Byron"}`,
"card": `"signed {/who.last}"`,
"card": `"signed {..who.last}"`,
})
f := newGenerator(t, dir, WithSeed(1))
if got := fake(t, f, "card"); got != "signed Byron" {
@@ -30,12 +30,12 @@ func TestReferenceIntoAField(t *testing.T) {
}
}
// TestReferenceInAlternation lets a reference stand as one arm of a {a|/b}
// TestReferenceInAlternation lets a reference stand as one arm of a {a|..b}
// alternation, so a field and a cross-file value share one slot.
func TestReferenceInAlternation(t *testing.T) {
dir := writeData(t, map[string]string{
"far": `"X"`,
"near": `{"format":"{here|/far}","here":"H"}`,
"near": `{"format":"{here|..far}","here":"H"}`,
})
f := newGenerator(t, dir, WithSeed(2))
seen := map[string]bool{}
@@ -51,7 +51,7 @@ func TestReferenceInAlternation(t *testing.T) {
// model: data layered from two dirs can point at each other through the root.
func TestReferenceCombinesLoadedPaths(t *testing.T) {
a := writeData(t, map[string]string{"en_US/word": `"river"`})
b := writeData(t, map[string]string{"mine/slug": `"the-{/en_US.word}"`})
b := writeData(t, map[string]string{"mine/slug": `"the-{..en_US.word}"`})
f := newGeneratorN(t, []string{a, b}, WithSeed(1))
if got := fake(t, f, "mine.slug"); got != "the-river" {
t.Fatalf("slug = %q, want the-river", got)
@@ -62,8 +62,8 @@ func TestReferenceCombinesLoadedPaths(t *testing.T) {
// linking order cannot matter.
func TestReferenceChain(t *testing.T) {
dir := writeData(t, map[string]string{
"a": `"{/b}"`,
"b": `"{/c}"`,
"a": `"{..b}"`,
"b": `"{..c}"`,
"c": `"deep"`,
})
f := newGenerator(t, dir, WithSeed(1))
@@ -76,37 +76,37 @@ func TestReferenceChain(t *testing.T) {
// fails at load, never at a random render.
func TestReferenceErrors(t *testing.T) {
cases := map[string]map[string]string{
"missing target": {"card": `"{/nope.gone}"`},
"folder target": {"en_US/word": `"w"`, "card": `"{/en_US}"`},
"missing target": {"card": `"{..nope.gone}"`},
"folder target": {"en_US/word": `"w"`, "card": `"{..en_US}"`},
"a variant on the path lacks the field": {
"who": `[{"format":"{f}","f":"1"},{"format":"{g}","g":"2"}]`,
"card": `"{/who.f}"`,
"card": `"{..who.f}"`,
},
"empty reference path": {"card": `"{/}"`},
"empty reference path": {"card": `"{..}"`},
// A reference that leads back to its own value never terminates at render,
// so New must reject the cycle up front (direct, mutual, or chained).
"direct cycle": {"a": `"x{/a}"`},
"mutual cycle": {"a": `"{/b}"`, "b": `"{/a}"`},
"chain cycle": {"a": `"{/b}"`, "b": `"{/c}"`, "c": `"{/a}"`},
"direct cycle": {"a": `"x{..a}"`},
"mutual cycle": {"a": `"{..b}"`, "b": `"{..a}"`},
"chain cycle": {"a": `"{..b}"`, "b": `"{..c}"`, "c": `"{..a}"`},
// calc renders its operands, so a cycle through one must be caught too.
"calc operand cycle": {"x": `{"format":"{calc(y)}","y":"{/x}"}`},
"calc operand cycle": {"x": `{"format":"{calc(y)}","y":"{..x}"}`},
// A field its parent's format never renders is still reachable by dot path,
// so a cycle hiding in one must fail at New rather than at render.
"cycle in an unrendered field": {"cat": `{"format":"hi","x":"{/cat.x}"}`},
"cycle in an unrendered field": {"cat": `{"format":"hi","x":"{..cat.x}"}`},
"mutual cycle between unrendered fields": {
"cat": `{"format":"hi","x":"{/cat.y}","y":"{/cat.x}"}`,
"cat": `{"format":"hi","x":"{..cat.y}","y":"{..cat.x}"}`,
},
"cycle in an unrendered field of a choice arm": {
"cat": `{"format":"hi","x":"{/cat.x}"}`,
"cat": `{"format":"hi","x":"{..cat.x}"}`,
},
// The shipped layout puts categories in folders, so a cycle one level down
// is the common case, not an edge case.
"cycle in a subfolder": {"sv_SE/a": `"x{/sv_SE.a}"`},
"mutual cycle within a subfolder": {"sv_SE/a": `"{/sv_SE.b}"`, "sv_SE/b": `"{/sv_SE.a}"`},
"mutual cycle across two folders": {"en_US/a": `"{/sv_SE.b}"`, "sv_SE/b": `"{/en_US.a}"`},
"cycle in a subfolder": {"sv_SE/a": `"x{..sv_SE.a}"`},
"mutual cycle within a subfolder": {"sv_SE/a": `"{..sv_SE.b}"`, "sv_SE/b": `"{..sv_SE.a}"`},
"mutual cycle across two folders": {"en_US/a": `"{..sv_SE.b}"`, "sv_SE/b": `"{..en_US.a}"`},
// ".." is reserved for bound references, so an authored key using it would
// name a node nothing can reach and nothing would validate.
"field key using the reference prefix": {"cat": `{"format":"hi","..x":"{/nope}"}`},
"field key using the reference prefix": {"cat": `{"format":"hi","..x":"{..nope}"}`},
}
for name, files := range cases {
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, files))); err == nil {
@@ -122,8 +122,8 @@ func TestReferenceErrors(t *testing.T) {
func TestDotPrefixedDataEntriesAreSkipped(t *testing.T) {
f := newGenerator(t, writeData(t, map[string]string{
"sv_SE/ok": `"fine"`,
"sv_SE/..bad": `"{/nope}"`,
"sv_SE/..y/ct": `"{/nope}"`,
"sv_SE/..bad": `"{..nope}"`,
"sv_SE/..y/ct": `"{..nope}"`,
".git/config": `"not data"`,
}), WithSeed(1))
if got := f.List(); len(got) != 1 || got[0] != "sv_SE.ok" {
@@ -135,7 +135,7 @@ func TestDotPrefixedDataEntriesAreSkipped(t *testing.T) {
// over-rejecting: a field the format never renders may point back at its own
// category, which terminates, and stays renderable by path.
func TestReferenceFromUnrenderedFieldTerminates(t *testing.T) {
dir := writeData(t, map[string]string{"cat": `{"format":"hi","x":"see {/cat}"}`})
dir := writeData(t, map[string]string{"cat": `{"format":"hi","x":"see {..cat}"}`})
f := newGenerator(t, dir, WithSeed(1))
if got := fake(t, f, "cat"); got != "hi" {
t.Fatalf("cat = %q, want hi", got)
@@ -151,9 +151,9 @@ func TestReferenceFromUnrenderedFieldTerminates(t *testing.T) {
func TestNewErrorIsDeterministic(t *testing.T) {
cases := map[string]map[string]string{
"three bad references": {
"a": `"{/nope.one}"`,
"b": `"{/nope.two}"`,
"c": `"{/nope.three}"`,
"a": `"{..nope.one}"`,
"b": `"{..nope.two}"`,
"c": `"{..nope.three}"`,
},
"two bad fields in one template": {
"cat": `{"format":"hi","aaa":{"no":1},"zzz":{"no":2}}`,
@@ -180,7 +180,7 @@ func TestNewErrorIsDeterministic(t *testing.T) {
}
// TestNewErrorPathIsCanonical pins the node path a load error names: a choice arm
// adds no segment, and a bound {/path} reference is not a containment segment at
// adds no segment, and a bound {..path} reference is not a containment segment at
// all, so a bad reference is reported against the node that holds it.
func TestNewErrorPathIsCanonical(t *testing.T) {
cases := []struct {
@@ -190,13 +190,13 @@ func TestNewErrorPathIsCanonical(t *testing.T) {
}{
{
"cycle inside a choice arm",
map[string]string{"cat": `{"format":"hi","x":"{/cat.x}"}`},
"fejkdata: reference cycle: cat.x -> /cat.x",
map[string]string{"cat": `{"format":"hi","x":"{..cat.x}"}`},
"fejkdata: reference cycle: cat.x -> ..cat.x",
},
{
"bad reference reached through another reference",
map[string]string{"a": `"{/b}"`, "b": `"{/nope}"`},
`fejkdata: b: reference {/nope}: no entry "nope"`,
map[string]string{"a": `"{..b}"`, "b": `"{..nope}"`},
`fejkdata: b: reference {..nope}: no entry "nope"`,
},
}
for _, c := range cases {
@@ -214,7 +214,7 @@ func TestNewErrorPathIsCanonical(t *testing.T) {
func TestReferencePathIsHeld(t *testing.T) {
dir := writeData(t, map[string]string{
"person": `[{"format":"{first} {last}","first":"Anna","last":"Andersson"},{"format":"{first} {last}","first":"Bo","last":"Berg"}]`,
"card": `"{/person.first} {/person.last}"`,
"card": `"{..person.first} {..person.last}"`,
})
f := newGenerator(t, dir, WithSeed(3))
seen := map[string]bool{}
@@ -233,7 +233,7 @@ func TestReferencePathIsHeld(t *testing.T) {
func TestReferenceThroughChoiceNeedsEveryVariant(t *testing.T) {
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{
"who": `[{"format":"{f}{h}","f":"1","h":"x"},{"format":"{g}{h}","g":"2","h":"y"}]`,
"card": `"{/who.f}"`,
"card": `"{..who.f}"`,
})))
if err == nil || !strings.Contains(err.Error(), "not every variant") {
t.Fatalf("New = %v, want the missing variant named", err)
@@ -243,7 +243,7 @@ func TestReferenceThroughChoiceNeedsEveryVariant(t *testing.T) {
func TestBareReferenceDrawsEachTime(t *testing.T) {
dir := writeData(t, map[string]string{
"die": `["1","2","3","4","5","6"]`,
"roll": `"{/die} {/die}"`,
"roll": `"{..die} {..die}"`,
})
f := newGenerator(t, dir, WithSeed(1))
for i := 0; i < 50; i++ {
@@ -256,8 +256,8 @@ func TestBareReferenceDrawsEachTime(t *testing.T) {
func TestReferenceOverlapIsRejected(t *testing.T) {
for name, file := range map[string]string{
"head beside a path": `{"format":"{/cat.p} {/cat.p.first}","p":[{"format":"{first}","first":"A"},{"format":"{first}","first":"B"}]}`,
"sibling path beside a reference path": `{"format":"{p.first} {/cat.p.last}","p":[{"format":"{first}","first":"A","last":"1"},{"format":"{first}","first":"B","last":"2"}]}`,
"head beside a path": `{"format":"{..cat.p} {..cat.p.first}","p":[{"format":"{first}","first":"A"},{"format":"{first}","first":"B"}]}`,
"sibling path beside a reference path": `{"format":"{p.first} {..cat.p.last}","p":[{"format":"{first}","first":"A","last":"1"},{"format":"{first}","first":"B","last":"2"}]}`,
} {
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{"cat": file})))
if err == nil || !strings.Contains(err.Error(), "reads a path into") {
@@ -265,84 +265,3 @@ func TestReferenceOverlapIsRejected(t *testing.T) {
}
}
}
func TestRelativeReferences(t *testing.T) {
dir := writeData(t, map[string]string{
"sv_SE/username": `"bob"`,
"sv_SE/email": `"{.username}@example.com"`,
"sv_SE/deep/card": `"{..username} via {/sv_SE.username}"`,
"top": `"{.sv_SE.username}"`,
})
f := newGenerator(t, dir, WithSeed(1))
for path, want := range map[string]string{
"sv_SE.email": "bob@example.com",
"sv_SE.deep.card": "bob via bob",
"top": "bob",
} {
if got := fake(t, f, path); got != want {
t.Errorf("%s = %q, want %q", path, got, want)
}
}
}
func TestRelativeAndRootSpellingsBindOneDraw(t *testing.T) {
dir := writeData(t, map[string]string{
"sv_SE/person": `[{"format":"{first} {last}","first":"Anna","last":"Andersson"},{"format":"{first} {last}","first":"Bo","last":"Berg"}]`,
"sv_SE/card": `"{.person.first} {/sv_SE.person.last}"`,
})
f := newGenerator(t, dir, WithSeed(3))
for i := 0; i < 50; i++ {
if got := fake(t, f, "sv_SE.card"); got != "Anna Andersson" && got != "Bo Berg" {
t.Fatalf("card = %q, want both spellings to read one person", got)
}
}
}
func TestReferenceSigilErrors(t *testing.T) {
for name, files := range map[string]map[string]string{
"no folder above the root": {"a": `"{..b}"`, "b": `"x"`},
"three dots": {"a": `"{...b}"`, "b": `"x"`},
"root sigil alone": {"a": `"{/}"`},
"dot alone": {"a": `"{.}"`},
"missing sibling": {"sv_SE/a": `"{.nope}"`},
"slash in a field name": {"a": `{"format":"{x}","x":"1","a/b":"2"}`},
} {
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, files))); err == nil {
t.Errorf("%s: New = nil error, want a reference error", name)
}
}
}
func TestSpellingsOfOneReferenceAreOneLevel(t *testing.T) {
person := `[{"format":"{first} {last}","first":"Ada","last":"Byron"},{"format":"{first} {last}","first":"Bo","last":"Ek"}]`
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{
"sv_SE/person": person,
"sv_SE/mail": `"{.person} <{/sv_SE.person.first}>"`,
})))
if err == nil || !strings.Contains(err.Error(), "reads a path into") || !strings.Contains(err.Error(), "{.person}") {
t.Errorf("New = %v, want the bare spelling rejected beside the path spelling", err)
}
dir := writeData(t, map[string]string{
"sv_SE/word": `["alpha","beta","gamma"]`,
"sv_SE/loud": `"{/sv_SE.word} {uppercase(.word)}"`,
})
f := newGenerator(t, dir, WithSeed(2))
for i := 0; i < 50; i++ {
got := strings.Fields(fake(t, f, "sv_SE.loud"))
if len(got) != 2 || strings.ToUpper(got[0]) != got[1] {
t.Fatalf("loud = %q, want one draw under both spellings", got)
}
}
}
func TestSlashAfterAReferenceSigilIsRejected(t *testing.T) {
for name, files := range map[string]map[string]string{
"after ..": {"sv_SE/person": `"Ada"`, "sv_SE/deep/a": `"{../person}"`},
"after .": {"sv_SE/person": `"Ada"`, "sv_SE/a": `"{./person}"`},
} {
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, files)))
if err == nil || !strings.Contains(err.Error(), "person}") || !strings.Contains(err.Error(), "write {") {
t.Errorf("%s: New = %v, want the slash rejected naming the spelling", name, err)
}
}
}
render.go
+114 -13
View File
@@ -33,20 +33,48 @@ func (f *Generator) Fake(path string) (string, error) {
// descend walks named fields to the node a path names. It is the one render-side
// step that can fail, because the path comes from the caller and may name a field
// that does not exist. A choice consumes no segment, so the rest of the path must
// be one every variant carries before a variant is picked — a path that resolves
// at all resolves on every call.
func descend(s *session, root node, segments []string) (node, error) {
var found node
err := walkPath(root, segments, pathWalk{
choice: func(c *choice, rest []string) ([]node, error) {
if err := carriedByAll(c, rest); err != nil {
return nil, err
// be one every variant carries (the set compile stored) before a variant is picked
// — a path that resolves at all resolves on every call.
func descend(s *session, n node, segments []string) (node, error) {
if len(segments) == 0 {
return n, nil
}
return []node{pick(s, c)}, nil
},
leaf: func(n node) error { found = n; return nil },
})
return found, err
switch n := n.(type) {
case *group:
child, ok := n.children[segments[0]]
if !ok {
return nil, fmt.Errorf("no entry %q", segments[0])
}
return descend(s, child, segments[1:])
case *template:
child, ok := n.fields[segments[0]]
if !ok {
return nil, fmt.Errorf("no field %q", segments[0])
}
return descend(s, child, segments[1:])
case *choice:
if want := strings.Join(segments, "."); !n.shared[want] {
return nil, unreachableInChoice(n, want)
}
return descend(s, pick(s, n), segments)
default:
return nil, fmt.Errorf("cannot descend into %T at %q", n, segments[0])
}
}
// unreachableInChoice reports that a path cannot step through this choice, listing
// what every variant does carry. It reads the precomputed set, so a failing path
// costs no more than a rendering one.
func unreachableInChoice(c *choice, want string) error {
if len(c.shared) == 0 {
return fmt.Errorf("no variant of this %d-way choice carries %q", len(c.items), want)
}
offered := make([]string, 0, len(c.shared))
for p := range c.shared {
offered = append(offered, p)
}
sort.Strings(offered)
return fmt.Errorf("not every variant of this %d-way choice carries %q; all carry %v", len(c.items), want, offered)
}
// render evaluates a compiled node to a string. compile validates every node up
@@ -124,3 +152,76 @@ func expand(s *session, t *template) string {
}
return b.String()
}
// draws is what an expansion has already drawn for its held names: the variant each
// was drawn as, so every path under it reads one row, and the value each read, so
// the same name read twice reads one value.
type draws struct {
variant map[string]node
value map[string]string
}
// readField renders one arm of a token. An arm's key is a sibling field or a
// {..path} reference, which linkRefs bound into fields too. A name the expansion
// holds — a level some token addresses by dotted path, or a sibling a {calc()}
// reads — is drawn once and kept, so {place.postal-code} and {place.locality} read
// one row, either read twice gives one value, and a shown operand is the operand
// computed. Every other name is drawn afresh, so {word} {word} still draws twice.
// checkTokens, checkPath and linkRefs prove every step, so this cannot fail.
func readField(s *session, t *template, held *draws, a arm) string {
if !t.held[a.key] {
return render(s, t.fields[a.key])
}
if v, read := held.value[a.name]; read {
return v
}
n, drew := held.variant[a.key]
if !drew {
n = drawn(s, t.fields[a.key])
held.variant[a.key] = n
}
// Hold the draw at every level passed through, so two paths sharing a prefix
// share it.
for i, seg := range a.tail {
if i < len(a.steps) {
step, drew := held.variant[a.steps[i]]
if !drew {
step = drawn(s, child(n, seg))
held.variant[a.steps[i]] = step
}
n = step
continue
}
n = child(n, seg)
}
v := render(s, n)
held.value[a.name] = v
return v
}
// child is the node one path segment names below an already-drawn node. It holds
// while checkPath and the set a choice shares (see sharedPaths) agree with this
// walk: both prove the segment exists and that drawn leaves a template here. Each
// way that can break panics naming the segment, so a slip in that agreement
// reports where it happened rather than surfacing a nil node a level later.
func child(n node, seg string) node {
t, ok := n.(*template)
if !ok {
panic(fmt.Sprintf("fejkdata: %q under %T, which carries no fields", seg, n))
}
c, ok := t.fields[seg]
if !ok {
panic(fmt.Sprintf("fejkdata: no field %q under a drawn level", seg))
}
return c
}
// drawn resolves a choice to one variant, so a bound head is a concrete node the
// rest of the expansion shares. Nested choices unwrap too: a draw is one value, not
// another set to pick from.
func drawn(s *session, n node) node {
for c, ok := n.(*choice); ok; c, ok = n.(*choice) {
n = pick(s, c)
}
return n
}
shipped_data_test.go
+1 -7
View File
@@ -44,7 +44,7 @@ func TestWithDataPathLayersOverShipped(t *testing.T) {
}
func TestUserDataMayReferenceShipped(t *testing.T) {
dir := writeData(t, map[string]string{"greeting": `"Hej {/sv_SE.person}!"`})
dir := writeData(t, map[string]string{"greeting": `"Hej {..sv_SE.person}!"`})
f, err := New(WithDataPath(dir), WithSeed(1))
if err != nil {
t.Fatalf("New = %v", err)
@@ -108,12 +108,6 @@ func TestFakeIsSafeForConcurrentUse(t *testing.T) {
return
}
f.List()
tmpl, err := f.NewTemplate("{/sv_SE.person.last}")
if err != nil {
t.Error(err)
return
}
tmpl.Fake()
}
}()
}
template.go
+105 -55
View File
@@ -2,6 +2,7 @@ package fejkdata
import (
"fmt"
"sort"
"strings"
)
@@ -15,8 +16,7 @@ type ftoken struct {
// eachToken scans a format string once and calls fn for each unit, the single
// source of truth for how braces are read: "{{" and "}}" are literal braces, a "{"
// opens a token that must reach its "}", and a lone "}" is an error. A table-shaped
// scanner, one case per rune kind, kept whole on purpose.
// opens a token that must reach its "}", and a lone "}" is an error.
func eachToken(format string, fn func(ftoken) error) error {
rs := []rune(format)
for i := 0; i < len(rs); i++ {
@@ -135,12 +135,12 @@ func checkTokens(format string, fields map[string]node) error {
names := strings.Split(t.body, "|")
for _, name := range names {
if isRef(name) {
if _, _, err := refShape(name); err != nil {
return fmt.Errorf("token {%s}: %w", t.body, err)
if name == refPrefix {
return fmt.Errorf("token {%s}: reference has no path", t.body)
}
continue // its target is checked at New (see linkRefs)
continue // a root reference; its target is checked at New (see linkRefs)
}
if err := checkArm(name, fields, len(names) == 1); err != nil {
if err := checkArm(name, fields); err != nil {
return fmt.Errorf("token {%s}: %w", t.body, err)
}
}
@@ -151,9 +151,7 @@ func checkTokens(format string, fields map[string]node) error {
}
// checkArm validates one sibling name or path against a template's fields.
// wholeToken says the name is the token's entire body, so {/name} would render
// the same value and can be offered as the reference spelling.
func checkArm(name string, fields map[string]node, wholeToken bool) error {
func checkArm(name string, fields map[string]node) error {
a := splitArm(name, nil)
if err := checkSegments(a); err != nil {
return err
@@ -166,13 +164,6 @@ func checkArm(name string, fields map[string]node, wholeToken bool) error {
if isOption(a.key) {
return fmt.Errorf("%q is an option and can never be a field", a.key)
}
if len(fields) == 0 {
hint := ""
if wholeToken && hintableRef(name) {
hint = fmt.Sprintf(" — write {/%s} to reference the data", name)
}
return fmt.Errorf("no field %q; a token names a sibling field, and this template has none%s", a.key, hint)
}
return fmt.Errorf("no field %q", a.key)
}
if err := checkPath(head, a.tail, a.key); err != nil {
@@ -181,17 +172,6 @@ func checkArm(name string, fields map[string]node, wholeToken bool) error {
return nil
}
// hintableRef reports whether {/name} is a reference the grammar accepts, so the
// hint never names a spelling that fails too.
func hintableRef(name string) bool {
for _, seg := range strings.Split(name, ".") {
if checkName(seg) != nil {
return false
}
}
return true
}
// tokenOperands lists the fields one {token} body reads as operands, empty for a
// field token or a builtin that reads none.
func tokenOperands(body string) []string {
@@ -250,35 +230,30 @@ func fieldTokens(format string) []string {
}
// arm is one alternative of a {a|b} token or one operand, split into the key
// naming the node in a template's fields (a sibling field, or the head a
// naming the node in a template's fields (a sibling field, or the "..path" head a
// reference is bound under) and the tail of a dotted path into it. A non-empty
// tail is what makes the arm a bound draw: its head is drawn once per expansion
// (see compileOps).
type arm struct {
name string // as written, for messages
name string // as written, and the key a bound draw's value is held under
key string
tail []string
steps []string // key per level passed through; the head and leaf hold their own
path string // key and tail, the one spelling every way of writing this read shares
}
// splitArm splits one name into key and tail. refs maps a reference to what
// linkRefs bound it to; before linking, a reference is whole.
func splitArm(name string, refs map[string]refBinding) arm {
// splitArm splits one name into key and tail. refs maps a reference to the head
// linkRefs bound it under; before linking, a reference is whole.
func splitArm(name string, refs map[string]string) arm {
if isRef(name) {
b, bound := refs[name]
if !bound || len(b.tail) == 0 {
key := name
if bound {
key = b.key
key, bound := refs[name]
if !bound || key == name {
return arm{name: name, key: name}
}
return arm{name: name, key: key, path: key}
}
return pathArm(name, b.key, b.tail)
return pathArm(name, key, strings.Split(name[len(key)+1:], "."))
}
head, tail, dotted := strings.Cut(name, ".")
if !dotted {
return arm{name: name, key: name, path: name}
return arm{name: name, key: name}
}
return pathArm(name, head, strings.Split(tail, "."))
}
@@ -288,21 +263,62 @@ func pathArm(name, key string, segs []string) arm {
for i := 0; i < len(segs)-1; i++ { // every level except the leaf's own
steps = append(steps, key+"."+strings.Join(segs[:i+1], "."))
}
return arm{name: name, key: key, tail: segs, steps: steps, path: key + "." + strings.Join(segs, ".")}
return arm{name: name, key: key, tail: segs, steps: steps}
}
// splitArms splits a token body's '|' alternatives.
func splitArms(body string, refs map[string]refBinding) []arm {
parts := strings.Split(body, "|")
arms := make([]arm, len(parts))
for i, p := range parts {
arms[i] = splitArm(p, refs)
// checkNoOverlap rejects a format that both renders a level and reads a path into
// it — {p} beside {p.first}, or {p.addr} beside {p.addr.city}. The path reads the
// level's held draw while rendering the level expands it afresh, so their values
// would disagree. Names are compared in sorted order, so which pair is reported
// does not depend on where the tokens sit.
func checkNoOverlap(format string, bound map[string]string, refs map[string]string) error {
names := boundReaders(format, bound, refs)
// Stable over one format-order scan, so two readers of one name (a token and a
// calc operand both naming "p") are reported as the format writes them.
sort.SliceStable(names, func(i, j int) bool { return names[i].name < names[j].name })
for i, level := range names {
for _, path := range names[i+1:] {
if strings.HasPrefix(path.name, level.name+".") {
return fmt.Errorf("%s renders a level that {%s} reads a path into; name the fields you want instead", level.label, path.name)
}
return arms
}
}
return nil
}
// checkSegments rejects an unfinished path: "{a.}" and "{a..b}" each have a
// segment naming nothing. A field really named "" would otherwise make them
// reader is one way a format reaches a bound field, and how to name that spelling.
type reader struct{ name, label string }
// boundReaders lists every way a format reaches a bound field, in the order the
// format writes them. An operand renders its field, so it names a level exactly
// as a token does; one scan finds both, which is what puts them in one order.
func boundReaders(format string, bound map[string]string, refs map[string]string) []reader {
var names []reader
_ = eachToken(format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if fn, _, isFunc := funcCall(t.body); isFunc {
for _, operand := range tokenOperands(t.body) {
a := splitArm(operand, refs)
if _, isBound := bound[a.key]; isBound {
names = append(names, reader{a.name, fmt.Sprintf("%s operand %q", fn, operand)})
}
}
return nil
}
for _, a := range splitArms(t.body, refs) {
if _, isBound := bound[a.key]; isBound {
names = append(names, reader{a.name, "token {" + a.name + "}"})
}
}
return nil
})
return names
}
// checkSegments rejects an unfinished path: "{a.}", "{.b}" and "{a..b}" each have
// a segment naming nothing. A field really named "" would otherwise make them
// resolve, so a typo would read as a path that worked.
func checkSegments(a arm) error {
if len(a.tail) == 0 {
@@ -319,6 +335,16 @@ func checkSegments(a arm) error {
return nil
}
// splitArms splits a token body's '|' alternatives.
func splitArms(body string, refs map[string]string) []arm {
parts := strings.Split(body, "|")
arms := make([]arm, len(parts))
for i, p := range parts {
arms[i] = splitArm(p, refs)
}
return arms
}
// callFn is a builtin bound to one call site: its args already parsed. It reads the
// output emitted so far in the current expansion (a derivation's payload) and the
// values of the operands it named, which expand read for it.
@@ -370,7 +396,7 @@ func (c *formatOps) hold(a arm, label string) {
}
}
func (c *formatOps) function(body string, refs map[string]refBinding) {
func (c *formatOps) function(body string, refs map[string]string) {
name, args, _ := funcCall(body)
var operands []arm
for _, operand := range tokenOperands(body) {
@@ -381,7 +407,7 @@ func (c *formatOps) function(body string, refs map[string]refBinding) {
c.ops = append(c.ops, op{kind: 'b', call: builtins[name].prep(args), operands: operands})
}
func (c *formatOps) field(body string, refs map[string]refBinding) {
func (c *formatOps) field(body string, refs map[string]string) {
arms := splitArms(body, refs)
for _, a := range arms {
if len(a.tail) > 0 {
@@ -393,7 +419,7 @@ func (c *formatOps) field(body string, refs map[string]refBinding) {
// compileOps compiles a format string. Call checkTokens first: it is what proves
// the scan and every token are valid.
func compileOps(format string, refs map[string]refBinding) formatOps {
func compileOps(format string, refs map[string]string) formatOps {
var c formatOps
var lit strings.Builder
flush := func() {
@@ -420,3 +446,27 @@ func compileOps(format string, refs map[string]refBinding) formatOps {
flush()
return c
}
// checkNoRepeatedRead rejects a bare token repeated on a held name: {w} {w} beside
// {uppercase(w)} would read one draw twice, where {w} {w} alone draws twice. The
// error names the single-token spelling.
func checkNoRepeatedRead(format string, c formatOps, refs map[string]string) error {
count := map[string]int{}
return eachToken(format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if _, _, isFunc := funcCall(t.body); isFunc {
return nil
}
for _, a := range splitArms(t.body, refs) {
if len(a.tail) > 0 || !c.held[a.key] {
continue
}
if count[a.key]++; count[a.key] > 1 {
return fmt.Errorf("token {%s} is repeated, and %s holds %q to one draw per expansion; write {%s} once", a.name, c.holder[a.key], a.key, a.name)
}
}
return nil
})
}
template_stability_test.go
+1 -1
View File
@@ -13,7 +13,7 @@ func TestSeededOutputIsStable(t *testing.T) {
"escapes": `{"format":"01Aa#{x}","x":"!"}`,
"funcs": `"{hex(6)} {int(10,99)} {float(0,1,3)} {nanoid(5)} {seq()}"`,
"nested": `{"format":"{outer}","outer":{"format":"{inner}-{digits(2)}","inner":"i"}}`,
"ref": `"see {/alt}"`,
"ref": `"see {..alt}"`,
"repeat": `{"format":"{w}","repeat":4,"separator":",","w":["x","y","z"]}`,
"sums": `{"format":"9{d}{luhn()} {e}{ean()} {m}{mod11()}","d":"012345678901234","e":"123456789012","m":"12345678"}`,
"weights": `[{"format":"big","weight":9},"tiny"]`,
template_test.go
+1 -69
View File
@@ -8,13 +8,7 @@ import (
)
// engine builds a seeded generator with no loaded categories, for rendering tests.
func engine(seed uint64) *Generator {
s, err := newRand(seed, true)
if err != nil {
panic(err)
}
return &Generator{rand: s}
}
func engine(seed uint64) *Generator { return &Generator{rand: newRand(seed, true)} }
// parse unmarshals a JSON template fragment into its dynamic form.
func parse(t *testing.T, s string) any {
@@ -244,18 +238,6 @@ func TestCompileErrors(t *testing.T) {
`"{base64(0)}"`, // count must be positive
`"{float(1,2)}"`, // wrong arity
`"{float(1,2,-1)}"`, // negative decimals
`"{float(NaN,NaN,2)}"`, // bounds must be finite
`"{float(Inf,Inf,2)}"`, // same-sign infinities
`"{float(1,NaN,2)}"`, // one NaN bound
`"{float(-Inf,1,2)}"`, // one infinite bound
`"{digits(+5)}"`, // a count is a plain integer
`"{digits(05)}"`, // no leading zero
`"{int(+1,5)}"`, // a bound is a plain integer
`"{int(5,5)}"`, // a constant is written as text
`"{float(1,1,2)}"`, // a constant is written as text
`{"format":"{x}","x":"v","repeat":2,"separator":""}`, // separator "" is the default
`"{calc(1/0)}"`, // a constant zero divisor
`{"format":"{calc(x/y)}","x":"1","y":"0"}`, // a fixed zero divisor
`"{iban(US)}"`, // unsupported country
`"{seq(a,b)}"`, // seq takes at most one name
`"{calc()}"`, // calc needs an expression
@@ -329,53 +311,3 @@ func quote(s string) string {
}
return string(b)
}
func TestHashIsLiteral(t *testing.T) {
f := engine(1)
cases := map[string]string{
`{"format":"","x":"v"}`: "",
`{"format":"#","x":"v"}`: "#",
`{"format":"##","x":"v"}`: "##",
`{"format":"#0#1#A#a","x":"v"}`: "#0#1#A#a",
`{"format":"#{x}","x":"v"}`: "#v",
}
for tmpl, want := range cases {
if got := mustRender(t, f, tmpl); got != want {
t.Errorf("render(%s) = %q, want %q", tmpl, got, want)
}
}
}
func TestMultibyteFormat(t *testing.T) {
// Scanning is rune-aware: multibyte literals coexist with class chars and
// tokens without corrupting indices.
got := mustRender(t, engine(2), `{"format":"Öster{x}-{digits(1)}å","x":"väg"}`)
if !regexp.MustCompile(`^Österväg-[0-9]å$`).MatchString(got) {
t.Fatalf("multibyte format = %q", got)
}
}
func TestAlternationThreeWay(t *testing.T) {
f := engine(4)
seen := map[string]bool{}
for i := 0; i < 200; i++ {
seen[mustRender(t, f, `{"format":"{a|b|c}","a":"A","b":"B","c":"C"}`)] = true
}
if !seen["A"] || !seen["B"] || !seen["C"] || len(seen) != 3 {
t.Fatalf("3-way alternation produced %v, want A, B and C", seen)
}
}
func TestArgErrorsNameTheSpelling(t *testing.T) {
for src, want := range map[string]string{
`"{float(1,2,02)}"`: "write 2",
`{"format":"{calc(a,02)}","a":"1"}`: "write 2",
`"{digits(+5)}"`: "write 5",
`"{hex(99999999999999999999)}"`: "exceeds the maximum",
`"{int(007,9)}"`: "write 7",
} {
if _, err := compile(parse(t, src)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("compile(%s) = %v, want an error saying %q", src, err, want)
}
}
}
todo.md
-28
View File
@@ -1,28 +0,0 @@
# Release checklist
What to settle before the first tag, then the work that follows in a later,
data-heavy release.
## Before the first release — settle the record flag and API contract
- `--format` vocabulary — confirm `text`, `json`, `ndjson`, `csv`, `sql`; the
`json`-as-array vs `ndjson`-as-lines split; `--table` (the SQL INSERT target);
the `--separator` rejection on record formats; and the exit codes (misuse 2,
runtime 1).
- Library surface — confirm `Record`, `FakeRecord`, `NewRecordTemplate`,
`RecordTemplate`, `Column`/`Columns()`, and the `JSON()`, `CSVHeader()`,
`CSVLine()`, `SQLInsert()` serializers.
- Typed scalars — columns are strings today (`"42"`, quoted SQL). Confirm that
stays out of scope, or add a per-column `kind` before the tag.
- Struct-filling — `fake:"..."` tags (reflection over an arbitrary struct) stay
out of scope; `Columns()` hands the caller the values to map themselves.
Confirm.
- Independent reference draw — within one record every tailed reference to a
category is one draw, with no spelling for "these columns should disagree".
Confirm the per-record contract, or add the spelling.
## Later, in a data-heavy release
- Shipped-data de-duplication — `email.json`'s `local` is a drifted copy of
`username.json`; fold it in when the shipped set grows and we add lots more
data.
transform_test.go
+1 -7
View File
@@ -28,7 +28,7 @@ func TestTransformReadsTheHeldDraw(t *testing.T) {
func TestTransformOverAReference(t *testing.T) {
dir := writeData(t, map[string]string{
"person": `[{"format":"{first} {last}","first":"Åsa","last":"Öberg"},{"format":"{first} {last}","first":"Bo","last":"Ek"}]`,
"email": `"{/person.first} {/person.last} <{lowercase(ascii(/person.first))}.{lowercase(ascii(/person.last))}@example.com>"`,
"email": `"{..person.first} {..person.last} <{lowercase(ascii(..person.first))}.{lowercase(ascii(..person.last))}@example.com>"`,
})
f := newGenerator(t, dir, WithSeed(5))
for i := 0; i < 50; i++ {
@@ -51,9 +51,3 @@ func TestTransformArgs(t *testing.T) {
}
}
}
func TestAsciiKeepsDEL(t *testing.T) {
if got := mustRender(t, engine(1), `{"format":"{ascii(x)}","x":"a\u007fb"}`); got != "a\u007fb" {
t.Errorf("ascii over DEL = %q, want it kept: DEL is ASCII", got)
}
}