larvit/fejkdata
3
0
Fork 0
Code Pull Requests 1 Actions Activity

Compare commits

base: larvit/fejkdata:record-output
Branches Tags
larvit/fejkdata:main larvit/fejkdata:record-output
..
compare: larvit/fejkdata:fcc58a74ab294d29a1cafaa8d8b0d83ab2895c80
Branches Tags
larvit/fejkdata:main larvit/fejkdata:record-output

1 Commits
record-output .. fcc58a74ab

Author SHA1 Message Date
lilleman fcc58a74ab Tests for reference sigils: / the root, . this folder, .. the folder above
Tests / vet + fmt + tests (pull_request) Failing after 37s
Details
2026-09-02 18:24:58 +02:00
35 changed files with 1327 additions and 3680 deletions
Expand all files Collapse all files
AGENTS.md
-1
View File
@@ -7,4 +7,3 @@
- One spelling per result: reject the other at `New`, and let the error name the spelling to use.
- A standing choice a reader would relitigate goes under Decisions in the README, not in a comment.
- A README example is a `json` block that loads and renders as a category; `readme_test.go` runs every one.
- Cyclomatic complexity is gated at 14: the table-shaped dispatches (`eachToken`, `calc.factor`, `walkPath`) sit at 13–14 and stay whole; anything else that reaches 14 is decomposed.
Dockerfile
-1
View File
@@ -13,7 +13,6 @@ RUN go mod download
COPY . .
RUN go vet ./... && \
go run github.com/fzipp/gocyclo/cmd/gocyclo@v0.6.0 -over 14 -ignore _test . && \
{ unformatted="$(gofmt -l .)"; test -z "$unformatted" || \
{ echo "unformatted files:"; echo "$unformatted"; exit 1; }; } && \
go test -race ./...
README.md
+63 -273
View File
@@ -1,17 +1,32 @@
# fejkdata
Locale-aware fake data for tests and fixtures, generated from JSON templates. Use
it as a Go library or the CLI — no data on disk, no dependencies, and a seed makes output
reproducible.
A Go library and CLI for generating locale-aware fake data from JSON templates.
Forked from [github.com/Timewave-AB/fakes](https://github.com/Timewave-AB/fakes).
```sh
go install gitea.larvit.se/larvit/fejkdata/cmd/fejkdata@latest
fejkdata sv_SE.person # Sara Eriksson
```
## Goals
1. **Valid by construction** — every value passes the check its real consumer
applies; facts that belong together come from one draw, within a value and
across categories.
2. **Text means what it says** — a format renders as written; only `{…}` varies,
random characters included (`{digits(3)}`). One spelling per result; the wrong
one is a load error naming the right one.
3. **Every mistake is a load error** — `New` rejects; `Fake` on a loaded generator
fails only for an unknown path.
4. **Zero to a value in one command** — `go install`, then `fejkdata sv_SE.person`:
no checkout, no flag. Flags are GNU-form (`--seed 42`, `-n 3`) in any position;
the first custom template needs no escape and no option.
5. **Data lives in JSON** — a builtin only for what data can't express.
6. **Reproducible** — seed in, same stream out; no builtin reads a clock.
7. **Zero dependencies** — standard library only.
8. **Docs index the grammar** — every syntax feature is a heading; every example
runs under test and shows its output; a rule is stated once.
## CLI
```sh
go install gitea.larvit.se/larvit/fejkdata/cmd/fejkdata@latest
fejkdata sv_SE.person # Sara Eriksson
fejkdata sv_SE.person.last # Eriksson
fejkdata --seed 42 sv_SE.address # the same address every run
@@ -19,39 +34,25 @@ fejkdata -n 3 --separator ', ' sv_SE.word # nät, barn, sol
fejkdata --list # every path the data offers
fejkdata --data-path ./mydata sv_SE.word # layer a directory over the shipped data
fejkdata --no-shipped-data -d ./mydata --list # only your data
fejkdata 'name: {/sv_SE.person.last}' # name: <a surname> — an inline template
fejkdata '{"format":"name: {x}","x":["bosse","lina"]}' # name: bosse or name: lina
```
A path names a category, or a field inside one: each dot segment descends one
level — folders, then the category (a JSON file), then fields. An argument that is
a JSON object, array or string, or that carries a `{` token, is instead an
**inline template**: a format string or a JSON value compiled and rendered on the
spot. Its tokens reach the data by reference from the root —
`{/sv_SE.person.last}`, so shipped and `--data-path` categories are alike
available. An inline template sits in no folder, so the folder-relative `{.name}`
and `{..name}` are rejected naming the root spelling. A path never contains a
brace, a bracket or a quote, so the two cannot collide (see
[Decisions](#decisions)).
level — folders, then the category (a JSON file), then fields.
| Flag | |
|------|--|
| `-d`, `--data-path D` | a directory to layer over the shipped data; repeatable, the last wins a name clash |
| `--no-shipped-data` | load only the `--data-path` directories |
| `-s`, `--seed N` | reproducible output |
| `-n`, `--repeat N` | render the value N times (up to 1048576), each an independent draw, streamed |
| `-n`, `--repeat N` | render the path N times, each an independent draw |
| `--separator S` | between repeated values (default a newline) |
| `--format F` | `text` (default), `json`, `ndjson`, `csv` or `sql` — a record's columns, one record per row (json frames them as an array) |
| `--table T` | the INSERT target for `--format sql` (default: the path's last segment, or `records` for an inline template) |
| `--list` | print every path, then exit |
| `--version`, `-h`, `--help` | print, then exit |
`--name value` and `--name=value` both work, a short flag's value attaches or
follows (`-n3`, `-n 3`) and short flags bundle (`-hn 3`) — see
[Decisions](#decisions); flags go anywhere, `--` ends them. Exit codes: `0` success, `1` runtime error (missing
dir, unknown path), `2` misuse — a bad flag, an argument that names neither a
template nor a path, or an inline template that does not compile. From a checkout:
`go run ./cmd/fejkdata …`.
dir, unknown path), `2` misuse. From a checkout: `go run ./cmd/fejkdata …`.
### Your own data
@@ -75,67 +76,6 @@ fejkdata --seed 1 --data-path ./mydata sql
fejkdata --repeat 100 --data-path ./mydata sql > seed.sql
```
That `format` string is the free-form spelling — you hand-write the whole row.
For structured output a record writes the row for you.
### Records
A record is a template seen as columns: its fields are the columns, its `format`
the whole. `--format json|ndjson|csv|sql` writes the records; the library's
`Record` (below) hands back the columns. Every column is a string — typed scalars
are on the release checklist, see [`todo.md`](todo.md). Save
`mydata/users.json`:
```json
{
"format": "{first} {last}",
"first": ["Ada", "Bo"],
"last": ["Lovelace", "Ek"]
}
```
```sh
fejkdata --seed 1 --data-path ./mydata --format json users # [{"first":"Bo","last":"Lovelace"}]
fejkdata --seed 1 --data-path ./mydata --format ndjson users # {"first":"Bo","last":"Lovelace"}
fejkdata --seed 1 --data-path ./mydata --format csv users # first,last → Bo,Lovelace
fejkdata --seed 1 --data-path ./mydata --format sql users # INSERT INTO "users" ("first", "last") VALUES ('Bo', 'Lovelace');
fejkdata --seed 1 --data-path ./mydata --format sql --table people users # INSERT into another table
```
`--repeat` streams that many records — `json` frames them as one array document,
`ndjson` writes one object per line, `csv` a row after a header, `sql` one INSERT
per line. Only a category-level template is a record; a field, choice or folder
errors, and so does a `repeat` on
the template itself, which composes the format into one string rather than
projecting columns — ask for more records with `--repeat`. A `repeat` on a column
is fine.
A column carrying a newline keeps it inside the quoted CSV field or the SQL string
literal, so a row can span physical lines: read the stream with a CSV or SQL
parser rather than splitting it on newlines.
The SQL is ANSI — identifiers in double quotes, a literal quote doubled (`''`),
backslashes passed through — so a hyphenated field like `postal-code` stays a
valid identifier. PostgreSQL and SQLite take it as written; MySQL and MariaDB need
`ANSI_QUOTES` and `NO_BACKSLASH_ESCAPES` set first, or they read `"users"` as a
string and a backslash as an escape.
A record written only to emit columns still needs a `format` — the grammar's one
required key — so `"format": ""` carries the fields with an inert format: it
renders nothing by `Fake`, and is compiled only so the tree's fences still run.
The columns are the point, and their facts stay together: two columns that read a
path into one category — `{/currency.code}` and `{/currency.symbol}` — share one
draw of it, so the record is internally consistent. A bare `{/currency}` names no
field, so it keeps drawing on its own. That one draw is also why two
columns may not read overlapping reference *paths* — `{/cat.a}` beside
`{/cat.a.b}` is refused, naming the fields to write instead, as
[One draw, one spelling](#one-draw-one-spelling) refuses that pair inside a single
format. A column may not reference the record it belongs to by any spelling: `{/users.first}`
or a bare `{/users}` inside `users` describes a draw other than the columns beside
it, so put a value two columns share in its own category and reference that. A field hold, transform or
operand ties fields together within one column as always (see
[Correlated fields](#correlated-fields) and [Decisions](#decisions)).
## Library
```sh
@@ -149,12 +89,6 @@ if err != nil {
}
v, err := f.Fake("sv_SE.address") // "Kungsvägen 68\n379 17 Stockholm"
paths := f.List() // every path Fake accepts, sorted
v, err = f.FakeTemplate("name: {/sv_SE.person.last}") // compile + render in one call
t, err := f.NewTemplate(`{"format":"name: {x}","x":["bosse","lina"]}`) // compile once
v = t.Fake() // render many times, no re-parse
r, err := f.Record("users") // one record: each field a column
s := r.JSON() // {"first":"Ada","last":"Lovelace"}
r, err = f.FakeRecord(`{"format":"{x}","x":["a","b"]}`) // compile + render inline
```
| Option | |
@@ -164,11 +98,6 @@ r, err = f.FakeRecord(`{"format":"{x}","x":["a","b"]}`) // compile + render inli
| `WithDataFS(fsys)` | layer an `fs.FS`, such as your own `embed.FS` |
| `WithoutShippedData()` | load only what you give |
A `*Record` carries its columns via `Columns()`, and serializes them with `JSON()`
(one object), `CSVHeader()`/`CSVLine()`, or `SQLInsert(table)` — the shapes the
CLI's `--format` writes. `Record` and `FakeRecord` take a record; a path or
template that is not one — a bare string, a choice, or a folder — errors.
A `*Generator` is safe for concurrent use; a seeded sequence is reproducible only
when drawn from one goroutine. Changing how a value is composed shifts the seeded
stream for that value and everything drawn after it.
@@ -181,8 +110,8 @@ work with no data on disk. A directory is a namespace: each JSON file is a
category named after the file, each subdirectory a dot-path segment, so
`mydata/sv_SE/person.json` is `sv_SE.person` and replaces the shipped one.
Sources merge in order; matching folders combine, any other clash is won by the
last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"` or `/`;
dot-prefixed entries are skipped, so a data directory can also be a checkout.
last loaded. Names may not use `.`, `|`, `(`, `{` or `}`; dot-prefixed entries
are skipped, so a data directory can also be a checkout.
Each locale carries `address`, `color`, `company`, `date`, `email`, `ip`,
`person`, `phone`, `price`, `sentence`, `ssn`, `time`, `url`, `username`,
@@ -212,7 +141,7 @@ Every character is literal except a `{…}` token:
| `{name.field}` | `field` of one draw of `name` ([Correlated fields](#correlated-fields)) |
| `{a\|b}` | one of the named fields, even odds |
| `{fn(args)}` | a builtin ([Functions](#functions)) |
| `{/path}`, `{.path}`, `{..path}` | a node reached from the data root, this file's folder, or the folder above ([References](#references)) |
| `{..path}` | a node reached from the data root ([References](#references)) |
| `{{`, `}}` | a literal `{` or `}` |
```json
@@ -251,9 +180,9 @@ many times — each an independent draw — joined by `separator` (default `""`)
{ "format": "{word}", "repeat": 3, "separator": " ", "word": ["foo", "bar", "baz"] }
```
Renders e.g. `bar foo baz`. Rejected at load: a `separator` without a `repeat`,
a `separator` of `""` (the default), and a `repeat` that multiplies to more than
1 048 576 renders along any path of nested repeats.
Renders e.g. `bar foo baz`. A `separator` without a `repeat` is rejected, and so
is a `repeat` that multiplies to more than 1 048 576 renders along any path of
nested repeats.
### Options and fields
@@ -265,11 +194,9 @@ choice naming its item.
### Functions
A `{name(args)}` token calls a builtin. Arguments are checked at `New`: a bad
count, range, country or expression fails fast; an integer is written plain
(`5`, not `+5` or `05`); bounds are finite; a sample that could only ever emit one
value (`int(5,5)`, `float(1,1,2)`) is rejected naming the text to write instead;
and a length, count or decimal place beyond a sane maximum is rejected, so a
fat-fingered `hex(2000000000)` never tries to allocate gigabytes. Every builtin draws only from the seed — a
count, range, country or expression fails fast, and a length, count or decimal
place beyond a sane maximum is rejected, so a fat-fingered `hex(2000000000)`
never tries to allocate gigabytes. Every builtin draws only from the seed — a
time-based id takes its timestamp from the rng, not the clock — so seeded output
stays reproducible.
@@ -316,10 +243,8 @@ hyphenated field can't be an operand.
```
Renders e.g. `19.99 x 3 = 59.97`. An operand that can never be a number (`"abc"`,
or a choice of such) is rejected at load, as is a division by a constant zero
(`1/0`, or a fixed `"0"` field); an operand that sometimes is not a number yields
`NaN`, and a division by one that is not constant `Inf` — both print rather than
fail.
or a choice of such) is rejected at load; one that sometimes is not yields `NaN`,
and a division by zero `Inf` — both print rather than fail.
### Transforms
@@ -342,25 +267,21 @@ a mix.
### References
A reference renders a node from elsewhere in the data — the path `Fake` takes,
across every loaded source — so one category borrows another. The sigil says
where the path starts, as in a filesystem: `{/en_US.person}` from the data root,
`{.username}` from the folder this file sits in, `{..username}` from the folder
above. `data/sv_SE/email.json` can therefore read its own locale's `username`
without naming `sv_SE`:
`{..path}` renders a node from the **data root** — the path `Fake` takes, across
every loaded source — so one category borrows another, even across folders or
layered directories:
```json
"Hej, {/en_US.person}!"
"Hej, {..en_US.person}!"
```
Renders e.g. `Hej, Pat Smith!`. A reference into a category is held like a
[correlated](#correlated-fields) path — `{.person.first} {.person.last}` name one
person, `{lowercase(.person.first)}` reads that same draw, and `{.person.first}`
beside `{/sv_SE.person.last}` in `sv_SE` is one person too — while a bare
`{/misc.uuid} {/misc.uuid}` is two draws. Rejected at `New`: a path that is
unknown, names a folder, has no folder above, or reads a field not every variant
of a choice carries, and a reference that leads back to its own value, directly,
mutually or through a chain.
[correlated](#correlated-fields) path — `{..sv_SE.person.first} {..sv_SE.person.last}`
name one person, `{lowercase(..sv_SE.person.first)}` reads that same draw — while
a bare `{..misc.uuid} {..misc.uuid}` is two draws. Rejected at `New`: a path that
is unknown, names a folder, or reads a field not every variant of a choice
carries, and a reference that leads back to its own value, directly, mutually or
through a chain.
### Correlated fields
@@ -394,8 +315,8 @@ The sub-fields stay addressable — `Fake("address.place.locality")` renders, an
A name any token reads as a path (`{p.first}`) or as an operand (`{calc(net * 2)}`,
`{uppercase(w)}`) is drawn **once per expansion**, and every other route to it —
a bare `{p}`, a second bare `{w}`, `{/cat.net}`, a nested template rendering
`{/cat.p.last}`, at any depth — is a load error naming the spelling to use. A
a bare `{p}`, a second bare `{w}`, `{..cat.net}`, a nested template rendering
`{..cat.p.last}`, at any depth — is a load error naming the spelling to use. A
name nothing reads that way is drawn each time: `{word} {word}` differs. An
expansion is one render of one format, so each `repeat` iteration and each nested
template draws again.
@@ -412,29 +333,6 @@ then costs about what its output costs: an unweighted pick is O(1) whatever the
list's length, a weighted one O(log n), and long formats, deep nesting and many
tokens add cost in proportion to the output.
## Goals
1. **Valid by construction** — every value passes the check its real consumer
applies; facts that belong together come from one draw, within a value and
across categories.
2. **Text means what it says** — a format renders as written; only `{…}` varies,
random characters included (`{digits(3)}`). One spelling per result; the wrong
one is a load error naming the right one.
3. **Every mistake is a load error** — `New` rejects the data and `NewTemplate`
the inline template; on a loaded generator `Fake` fails only for an unknown
path, and `Template.Fake` cannot fail at all.
4. **Zero to a value in one command** — `go install`, then `fejkdata sv_SE.person`:
no checkout, no flag. Flags are GNU-form (`--seed 42`, `-n 3`) in any position;
the first custom template needs no escape and no option.
5. **Data lives in JSON** — a builtin only for what data can't express.
6. **Reproducible** — seed in, same stream out; no builtin reads a clock.
7. **Zero dependencies** — standard library only.
8. **Docs index the grammar** — every syntax feature is a heading; every example
runs under test and shows its output; a rule is stated once.
9. **Fast enough to be free** — a value renders in about a microsecond and `New`
parses and validates the whole set once upfront, so generating fixtures stays
noise against a test's own runtime.
## Decisions
- **Options and fields share one namespace.** `format`, `weight`, `repeat` and
@@ -450,119 +348,17 @@ tokens add cost in proportion to the output.
naming the double-dash spelling, and `-s=42` is rejected naming both short
spellings: `=` belongs to the long form, and reading `=42` as the value would
make `-d=./x` a directory named `=./x`.
- **An argument is a template by its shape, not by a flag.** A JSON object, array
or string, or a string carrying a `{` token, is an inline template; anything else
is a path. A name may not contain a brace, a bracket or a quote, so a path can
never collide with any of those spellings, and the leading `[` or `"` is gated on
valid JSON so a stray copied bracket never swallows an argument — it names
nothing, and says so. No `--template` flag is needed. Reserving the characters
whole — though only a leading one could collide — keeps one simple name rule
instead of a leading-position special case. The JSON string is what makes the
library's own advice reachable: the error for an object holding only a format
names `"…"`, and that spelling has to work where it is printed.
- **An inline template skips the cycle fence, and only that one.** `New` proves the
loaded tree acyclic, an inline node is a finite tree of its own, and nothing in
the tree can reference it, so no render of it reaches itself. Every other fence
runs over both, from one `checkScope`.
- **An inline template that does not compile is misuse (exit 2), including a
reference that resolves to nothing** — the whole argument is the spelling under
test, and `NewTemplate` compiles, links and validates as one step. An unknown
*path* stays a runtime error (exit 1): there the argument is well-formed and only
the data is absent.
- **A padded JSON argument is rejected, not trimmed.** Padding is the one place the
two readings disagree — a format string renders it, JSON drops it — so the
spelling that renders is named rather than silently chosen.
- **`FakeTemplate` and `NewTemplate` both stay.** They reach the same value but not
at the same cost: `NewTemplate` pays the compile and validation once and renders
many times, `FakeTemplate` is the one-shot call, and `--repeat` is exactly the
case that needs the first. The pair is `regexp.MustCompile` and `regexp.Match`,
not two spellings of one result.
- **The shipped data is embedded, not discovered.** A directory a machine happens
to have would make `--seed 42` machine-dependent. Data still lives in `data/`
as JSON; `--data-path` layers over it.
- **A bare reference draws each time; a reference path is held.** `{/p} {/p}`
is two draws, as `{word} {word}` is, while `{/p.first}` beside a nested template
rendering `{/p.first}` is a load error: a bare token is by contract an
- **A bare reference draws each time; a reference path is held.** `{..p} {..p}`
is two draws, as `{word} {word}` is, while `{..p.first}` beside a nested template
rendering `{..p.first}` is a load error: a bare token is by contract an
independent draw, a path pins its level, and any route into a pinned level from
another expansion could show another row.
- **Reference sigils follow the filesystem.** `/` is the root, `.` this file's
folder, `..` the folder above — what those spellings already mean to anyone who
has typed a path. A locale's files reach each other without naming the locale,
so a folder renames and copies without editing its references.
- **After the first tag, a new fence is a major version.** Data files are the
public API, and one spelling per result grows by tightening, so every fence
invalidates some file. Each such release names the rejected spelling and its
replacement in the changelog and in the load error, and that is the whole
migration: a fence rejects one spelling with one replacement, so the fix is
local to each site. A fence that would need a non-local rewrite ships a
converter with its release instead. Before the first tag there is no
compatibility promise.
- **A `--data-path` override rebinds every reference to the category it
replaces.** References bind against the merged tree, so once shipped data uses
`{.person}`, a consumer's `sv_SE/person.json` is what every shipped reference
into `person` reads, and `New` fails on shipped data the consumer never wrote
when that file lacks a field those references read. Accepted: overriding is the
point of layering, the error names the reference and the field, and the fix is
the consumer's file carrying the fields the shipped tree reads.
- **The repeat cap bounds renders, not bytes.** A repeat, alone or nested, may
ask for at most 1 048 576 renders; how large each render is stays what the data
asked for, so `{hex(1048576)}` repeated to the cap is a terabyte, loaded without
complaint. A byte estimate would need every builtin to declare a width to fence
a shape no data comes near, and the harm lands on the author who wrote it.
- **64-bit targets only.** The gate builds amd64, and the buffer sizing a render
pre-computes (renders × bytes) assumes a 64-bit int; on a 32-bit target it could
overflow and panic.
- **A constant zero divisor is a load error; a divisor that is not constant prints
`Inf`.** `1/0` and a fixed `"0"` field are decidable, so they join the
never-numeric operand as a load error; the fold stops where an operand varies,
so `a/(b*c)` with `b` fixed at `0` and `c` varying loads and prints `Inf` every
draw — catching it needs zero-absorbing algebra for a shape nobody writes.
- **In data, a default written out and a constant spelled as a sample are load
errors.** `weight: 1`, `repeat: 1`, `separator: ""`, `int(5,5)`, `float(1,1,2)`,
`+5` and `05` each spell what a shorter form already spells, so each is rejected
naming that form. The CLI's numbers follow the shell instead: `--seed 007` and
`--repeat +3` are 7 and 3, as every command line reads them.
- **Samples say what they emit, transforms what they do.** `{upper(2)}` is two
letters, `{uppercase(x)}` is `x` upper-cased; one name for both would turn on
whether the argument looks like a number.
- **A record is a template seen as columns, not a second schema format.** A
template's `format` composes its fields into one string; `Record` and
`--format` project the same fields as columns. Two views of one dataset, so a
record author writes the same JSON they already know, and a column is the same
field `Fake` renders by dotted path. The `format` is inert to a record — a
record-only template writes `"format": ""` — but it is compiled and fenced, so
a template that loads renders as whichever shape is asked for.
- **A record shares one reference draw per category.** Two columns that reference
one category — `{/currency.code}` beside `{/currency.symbol}` — read one draw of
it, so a record's facts agree the way a template's [correlated
fields](#correlated-fields) do. The draw is one per record, so it spans a
column's `repeat` and nested templates too (one record is one coherent unit);
a bare reference — `{/currency}`, no field — stays an independent draw every
time, the rule a format string already follows. Only references share: a sibling
field is local to its own column, so a `first` column does not silently bind to
a `first` in the column next to it.
The string view of that same template does not share. `Fake` renders each
sibling field as its own expansion, so a `{/currency.code}` field beside a
`{/currency.symbol}` field is two draws and may render `EUR $`; writing both
references in one `format` holds them together, as
[One draw, one spelling](#one-draw-one-spelling) says. The scope is what makes a
row coherent when the columns *are* the output, and there the caller cannot fall
back on one format string. Widening it to every render would change what `Fake`
has emitted since the start, for a correlation a single format already reaches.
- **A record's column set is fixed before the first draw.** Only a category-level
template is a record: a path descending into a field, or naming a folder or a
choice, errors. A tail may pass through a choice whose variants carry different
fields, so the columns — and with them the CSV header written once ahead of every
row — would vary per draw. A fixed column set is what the CSV and `INSERT`
contracts rest on, so the restriction holds even where a particular choice would
happen to agree.
- **The performance gate asserts allocations, not wall-clock time.** `AllocsPerRun`
is deterministic across machines, so a ±10% ceiling does not flake under CI load,
while time varies with the machine and its neighbours. A rendering slowdown
almost always costs an allocation too (a lost pre-size, a per-item map, an extra
copy). The benchmark suite (see Development) reports time for a human, not as a
pass/fail gate.
## Development
@@ -575,7 +371,6 @@ docker compose run --rm cover # tests with coverage
docker compose run --rm bench # benchmarks
docker compose run --rm build # compile the library
docker compose run --rm vet # go vet
docker compose run --rm cyclo # cyclomatic complexity over 14 (test files excluded)
docker compose run --rm dev # interactive shell
```
@@ -588,7 +383,7 @@ docker compose run --rm --user "$(id -u):$(id -g)" tidy # go mod tidy
Every pull request runs `docker build .` against both the latest and the lowest
supported Go, and must pass before it can be merged. That build is the whole
gate — vet, complexity, format check and tests — so run it locally before pushing:
gate — vet, format check and tests — so run it locally before pushing:
```sh
docker build . # latest
@@ -599,23 +394,18 @@ GO_VERSION=1.22.12 docker compose run --rm test # the same tests, without the im
## Layout
```
fejkdata.go Generator, New, options, the embedded data set, List
node.go the node model and JSON -> node compilation
path.go the dotted-path walk, and proving a path resolves
render.go Fake and the recursive renderer (choices, format strings, expansions)
record.go records: Record, the JSON/CSV/SQL serializers, and their entry points
inline.go inline templates: Template, NewTemplate, FakeTemplate, and their compile and link
template.go the {token} grammar: scanning, tokens, operands, validation, compiling a format
hold.go the hold: one draw per expansion for paths and operands, and its fences
reference.go reference sigils, and binding references across the tree
graph.go the render graph: edges, cycles, the repeat bound, tree walks
builtins.go the {name()} function registry and its implementations
calc.go the {calc()} arithmetic evaluator: parser, eval, validation
data.go data loading: fs.FS folders/files -> namespace tree, multi-source merge
cmd/fejkdata/ the fejkdata CLI
data/ shipped data (JSON), embedded at build: locale folders + a misc folder
fejkdata.go Generator, New, options, the embedded data set, List
node.go the node model and JSON -> node compilation
render.go Fake and the recursive renderer (choices, format strings, paths, held draws)
template.go the {token} grammar: scanning, arms, operands, validation
reference.go {..path} binding across the tree, the render graph, and the walks over it
builtins.go the {name()} function registry and its implementations
calc.go the {calc()} arithmetic evaluator: parser, eval, validation
data.go data loading: fs.FS folders/files -> namespace tree, multi-source merge
cmd/fejkdata/ the fejkdata CLI
data/ shipped data (JSON), embedded at build: locale folders + a misc folder
```
## License
MIT — see [LICENSE](LICENSE). Forked from [github.com/Timewave-AB/fakes](https://github.com/Timewave-AB/fakes).
MIT — see [LICENSE](LICENSE).
hold_test.go → bound_test.go
+4 -13
View File
@@ -642,11 +642,12 @@ func TestDeepPathsUnderOneHeadStayIndependentWhereTheyDiverge(t *testing.T) {
}
func TestEmptyPathSegmentIsRejected(t *testing.T) {
// "{a.}" and "{a..b}" are unfinished paths, and the segment naming nothing is
// reported as that rather than as a missing field. An empty name is rejected
// where it is authored, so no data can make these resolve.
// "{a.}", "{.b}" and "{a..b}" are unfinished paths, and the segment naming
// nothing is reported as that rather than as a missing field. An empty name is
// rejected where it is authored, so no data can make these resolve.
rejected := map[string]string{
"trailing dot": `{"format":"[{a.}]","a":"x"}`,
"leading dot": `{"format":"[{.b}]","a":{"format":"{b}","b":"V"}}`,
"double dot": `{"format":"[{a..b}]","a":"x"}`,
}
for name, file := range rejected {
@@ -711,13 +712,3 @@ func TestRepeatedBareTokenOfAHeldNameIsRejected(t *testing.T) {
}
}
}
func TestReadFieldPanicsOnAnUnheldPath(t *testing.T) {
tm, ok := compiled(t, `{"format":"{w}","w":{"format":"{x}","x":"1"}}`).(*template)
if !ok {
t.Fatal("not a template")
}
mustPanic(t, "unheld arm with a path", func() {
readField(engine(1).rand, tm, nil, nil, arm{name: "w.x", key: "w", tail: []string{"x"}, path: "w.x"})
})
}
builtins.go
+24 -59
View File
@@ -2,7 +2,6 @@ package fejkdata
import (
"encoding/base64"
"errors"
"fmt"
"math"
"strconv"
@@ -10,12 +9,12 @@ import (
"unicode"
)
// maxLen caps sample output lengths (hex, nanoid, base64, digits, upper, lower)
// and maxDecimals float/calc decimal places, so a fat-fingered or overflowing
// argument fails at New instead of trying to allocate gigabytes — or panicking —
// at render.
// maxLen caps sample output lengths (hex, nanoid, base64) and the renders a repeat
// multiplies to along any path; maxDecimals caps float/calc decimal places. So a
// fat-fingered or overflowing argument fails at New instead of trying to allocate
// gigabytes — or panicking — at render.
const (
maxLen = 1 << 20
maxLen = 1 << 20 // 1,048,576 chars/bytes
maxDecimals = 1024
)
@@ -130,10 +129,12 @@ func transformArg(fields map[string]node, a []string) error {
return err
}
if isRef(leaf) {
_, _, err := refShape(leaf)
return err
if leaf == refPrefix {
return fmt.Errorf("reference has no path")
}
return nil
}
return checkArm(leaf, fields, false)
return checkArm(leaf, fields)
}
func transformOperand(a []string) []string {
@@ -177,7 +178,7 @@ var asciiFolds = map[rune]string{
func asciiFold(s string) string {
var b strings.Builder
for _, r := range s {
if r <= unicode.MaxASCII {
if r < unicode.MaxASCII {
b.WriteRune(r)
} else {
b.WriteString(asciiFolds[r])
@@ -222,31 +223,10 @@ func randChars(r rng, n int, alphabet string) string {
return string(b)
}
// plainInt parses an integer arg written the one way: no sign, no leading zero.
func plainInt(s string) (int, error) {
n, err := strconv.Atoi(s)
if errors.Is(err, strconv.ErrRange) {
return 0, fmt.Errorf("%q is past the integer range: %w", s, err)
}
if err != nil {
return 0, fmt.Errorf("%q is not an integer", s)
}
if strconv.Itoa(n) != s {
return 0, fmt.Errorf("%q is not a plain integer; write %d", s, n)
}
return n, nil
}
func posIntArg(_ map[string]node, a []string) error {
n, err := plainInt(a[0])
if errors.Is(err, strconv.ErrRange) {
return fmt.Errorf("count %q exceeds the maximum %d", a[0], maxLen)
}
if err != nil {
return fmt.Errorf("count %w", err)
}
if n < 1 {
return fmt.Errorf("count %q must be positive", a[0])
n, err := strconv.Atoi(a[0])
if err != nil || n < 1 {
return fmt.Errorf("count %q must be a positive integer", a[0])
}
if n > maxLen {
return fmt.Errorf("count %d exceeds the maximum %d", n, maxLen)
@@ -255,20 +235,14 @@ func posIntArg(_ map[string]node, a []string) error {
}
func intRangeArgs(_ map[string]node, a []string) error {
lo, err := plainInt(a[0])
if err != nil {
return fmt.Errorf("int(min,max): min %w", err)
}
hi, err := plainInt(a[1])
if err != nil {
return fmt.Errorf("int(min,max): max %w", err)
lo, e1 := strconv.Atoi(a[0])
hi, e2 := strconv.Atoi(a[1])
if e1 != nil || e2 != nil {
return fmt.Errorf("int(min,max) needs integer args, got %q,%q", a[0], a[1])
}
if lo > hi {
return fmt.Errorf("int(min,max): min %d > max %d", lo, hi)
}
if lo == hi {
return fmt.Errorf("int(%d,%d) is the constant %d; write it as text", lo, hi, lo)
}
if uint64(hi)-uint64(lo) >= uint64(math.MaxInt64) { // span hi-lo+1 would overflow int -> IntN panic
return fmt.Errorf("int(min,max): range %d..%d is too wide", lo, hi)
}
@@ -278,28 +252,19 @@ func intRangeArgs(_ map[string]node, a []string) error {
func floatArgs(_ map[string]node, a []string) error {
lo, e1 := strconv.ParseFloat(a[0], 64)
hi, e2 := strconv.ParseFloat(a[1], 64)
if e1 != nil || e2 != nil {
return fmt.Errorf("float(min,max,dp) needs numeric bounds, got %q,%q", a[0], a[1])
}
dp, err := plainInt(a[2])
if err != nil {
return fmt.Errorf("float(min,max,dp): decimals %w", err)
}
if math.IsNaN(lo) || math.IsNaN(hi) || math.IsInf(lo, 0) || math.IsInf(hi, 0) {
return fmt.Errorf("float(min,max,dp) needs finite bounds, got %q,%q", a[0], a[1])
dp, e3 := strconv.Atoi(a[2])
if e1 != nil || e2 != nil || e3 != nil {
return fmt.Errorf("float(min,max,dp) needs numeric args, got %q,%q,%q", a[0], a[1], a[2])
}
if lo > hi {
return fmt.Errorf("float(min,max,dp): min %v > max %v", lo, hi)
}
if dp < 0 || dp > maxDecimals {
return fmt.Errorf("float(min,max,dp): decimals %d out of range 0..%d", dp, maxDecimals)
}
if lo == hi {
return fmt.Errorf("float(%s,%s,%d) is the constant %q; write it as text", a[0], a[1], dp, strconv.FormatFloat(lo, 'f', dp, 64))
}
if math.IsInf(hi-lo, 0) { // an overflowing span would render as "+Inf"
return fmt.Errorf("float(min,max,dp): range %v..%v is too wide", lo, hi)
}
if dp < 0 || dp > maxDecimals {
return fmt.Errorf("float(min,max,dp): decimals %d out of range 0..%d", dp, maxDecimals)
}
return nil
}
calc.go
+2 -71
View File
@@ -78,82 +78,14 @@ func checkCalc(fields map[string]node, args []string) error {
return fmt.Errorf("calc(%q): operand %q is never a number: it renders %q", args[0], name, text)
}
}
if divisor, zero := constantZeroDivisor(expr, fields); zero {
return fmt.Errorf("calc(%q) divides by %s, which is always zero", args[0], divisor)
}
if len(args) == 2 {
dp, err := plainInt(args[1])
if err != nil {
return fmt.Errorf("calc decimals %w", err)
}
if dp < 0 || dp > maxDecimals {
return fmt.Errorf("calc decimals %d must be in 0..%d", dp, maxDecimals)
if dp, err := strconv.Atoi(args[1]); err != nil || dp < 0 || dp > maxDecimals {
return fmt.Errorf("calc decimals %q must be an integer in 0..%d", args[1], maxDecimals)
}
}
return nil
}
// constantZeroDivisor finds a division whose right side is a constant zero: number
// literals and fixed operands folded, anything that varies left unknown.
func constantZeroDivisor(n calcNode, fields map[string]node) (string, bool) {
switch n := n.(type) {
case calcNeg:
return constantZeroDivisor(n.x, fields)
case calcBin:
if n.op == '/' {
if v, known := constantValue(n.r, fields); known && v == 0 {
return calcText(n.r), true
}
}
if d, zero := constantZeroDivisor(n.l, fields); zero {
return d, true
}
return constantZeroDivisor(n.r, fields)
}
return "", false
}
// constantValue evaluates an expression whose every operand is fixed.
func constantValue(n calcNode, fields map[string]node) (float64, bool) {
switch n := n.(type) {
case calcNum:
return float64(n), true
case calcVar:
t, ok := fields[string(n)].(*template)
if !ok || !t.fixed || t.repeat > 1 {
return 0, false
}
v, err := strconv.ParseFloat(strings.TrimSpace(t.lit), 64)
return v, err == nil
case calcNeg:
v, ok := constantValue(n.x, fields)
return -v, ok
case calcBin:
l, lok := constantValue(n.l, fields)
r, rok := constantValue(n.r, fields)
if !lok || !rok {
return 0, false
}
return calcBin{n.op, calcNum(l), calcNum(r)}.eval(nil), true
}
return 0, false
}
// calcText spells an expression node the way an author would read it.
func calcText(n calcNode) string {
switch n := n.(type) {
case calcNum:
return strconv.FormatFloat(float64(n), 'f', -1, 64)
case calcVar:
return string(n)
case calcNeg:
return "-" + calcText(n.x)
case calcBin:
return "(" + calcText(n.l) + " " + string(n.op) + " " + calcText(n.r) + ")"
}
return "?"
}
// neverNumeric reports a node no render of which is a number: fixed text that does
// not parse, or a choice of only such items. text is one such render.
func neverNumeric(n node) (text string, never bool) {
@@ -310,7 +242,6 @@ func (p *calcParser) binary(next func() (calcNode, error), ops ...byte) (calcNod
}
}
// factor is a table-shaped scanner, one case per token kind, kept whole on purpose.
func (p *calcParser) factor() (calcNode, error) {
p.space()
if p.pos >= len(p.rs) {
calc_test.go
-25
View File
@@ -245,28 +245,3 @@ func TestCalcOverANeverNumericOperandIsRejected(t *testing.T) {
}
}
}
func TestCalcConstantZeroDivisorIsRejected(t *testing.T) {
for src, want := range map[string]string{
`"{calc(1/0)}"`: "divides by 0",
`"{calc(2/(1-1))}"`: "divides by (1 - 1)",
`{"format":"{calc(x/y)}","x":"1","y":"0"}`: "divides by y",
`{"format":"{calc(x/(y*2))}","x":"1","y":" 0 "}`: "divides by (y * 2)",
`{"format":"{calc((a/0)+b)}","a":"1","b":"2"}`: "divides by 0",
`{"format":"{calc(a/-0)}","a":"1"}`: "divides by -0",
} {
if _, err := compile(parse(t, src)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("compile(%s) = %v, want the constant zero divisor rejected naming %q", src, err, want)
}
}
if _, err := compile(parse(t, `{"format":"{calc(a/(b*c))}","a":"1","b":"0","c":["1","2"]}`)); err != nil {
t.Errorf("compile(a/(b*c)) = %v, want a divisor that varies accepted", err)
}
f := engine(1)
for i := 0; i < 50; i++ {
if got := mustRender(t, f, `{"format":"{calc(x/y)}","x":"1","y":["0","1"]}`); got == "+Inf" {
return
}
}
t.Fatal("a sometimes-zero divisor never printed +Inf in 50 draws")
}
cmd/fejkdata/main.go
+41 -255
View File
@@ -8,69 +8,45 @@
package main
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"runtime/debug"
"sort"
"strconv"
"strings"
"gitea.larvit.se/larvit/fejkdata"
)
const usage = `Usage: fejkdata [flags] <path|template>
const usage = `Usage: fejkdata [flags] <path>
<path> a category, or a dotted path into one (person, person.last)
<template> a format string or JSON value to render inline, e.g.
'name: {/sv_SE.person.last}' or '{"format":"{x}","x":["bosse","lina"]}'
An argument containing a { token, or a JSON object, array or string, is a
template; any other argument is a path (a path never contains a brace, a bracket
or a quote). Templates reach the data by reference from the root —
{/sv_SE.person.last} — whether the data is shipped or layered with --data-path. An
argument carrying a bracket, a closing brace or a quote but no valid JSON names
neither.
With --format json, ndjson, csv or sql the argument must name a record — a
template whose fields are its columns — and the rows are written as one JSON
array, one JSON object per line, one CSV row (after a header), or one INSERT.
-d, --data-path D a data directory to layer over the shipped data (repeatable; last wins on a clash)
--format F output form: text (default), json, ndjson, csv or sql
-h, --help print this help, then exit
--list list the paths the data offers, then exit
--no-shipped-data load only the --data-path directories
-n, --repeat N render the value N times, 1..1048576 (default 1)
-n, --repeat N render the path N times (default 1)
-s, --seed N seed for reproducible output
--separator S string between repeated values (default newline)
--table T the INSERT target for --format sql (default: the path's last segment, or records for an inline template)
--version print the version, then exit
Flags may come before or after <path|template>; -- ends the flags. A short flag's
value attaches or follows (-n3, -n 3); short flags bundle (-hn 3).
Flags may come before or after <path>; -- ends the flags. A short flag's value
attaches or follows (-n3, -n 3); short flags bundle (-hn 3).
`
type invocation struct {
dirs []string
format string
formatSet bool
help bool
list bool
noShipped bool
paths []string
repeat int
repeatSet bool
seed uint64
seeded bool
separator string
separatorSet bool
table string
tableSet bool
version bool
dirs []string
help bool
list bool
noShipped bool
paths []string
repeat int
seed uint64
seeded bool
separator string
version bool
}
type flagDef struct {
@@ -82,16 +58,15 @@ type flagDef struct {
var flagDefs = []flagDef{
{"data-path", "d", true, func(in *invocation, v string) error { in.dirs = append(in.dirs, v); return nil }},
{"format", "", true, func(in *invocation, v string) error { in.format, in.formatSet = v, true; return nil }},
{"help", "h", false, func(in *invocation, _ string) error { in.help = true; return nil }},
{"list", "", false, func(in *invocation, _ string) error { in.list = true; return nil }},
{"no-shipped-data", "", false, func(in *invocation, _ string) error { in.noShipped = true; return nil }},
{"repeat", "n", true, func(in *invocation, v string) error {
n, err := strconv.Atoi(v)
if err != nil || n < 1 || n > fejkdata.MaxRepeat {
return fmt.Errorf("--repeat needs an integer in 1..%d, got %q", fejkdata.MaxRepeat, v)
if err != nil || n < 1 {
return fmt.Errorf("--repeat needs a positive integer, got %q", v)
}
in.repeat, in.repeatSet = n, true
in.repeat = n
return nil
}},
{"seed", "s", true, func(in *invocation, v string) error {
@@ -102,8 +77,7 @@ var flagDefs = []flagDef{
in.seed, in.seeded = n, true
return nil
}},
{"separator", "", true, func(in *invocation, v string) error { in.separator, in.separatorSet = v, true; return nil }},
{"table", "", true, func(in *invocation, v string) error { in.table, in.tableSet = v, true; return nil }},
{"separator", "", true, func(in *invocation, v string) error { in.separator = v; return nil }},
{"version", "", false, func(in *invocation, _ string) error { in.version = true; return nil }},
}
@@ -150,8 +124,8 @@ func splitFlags(arg string) ([]flagArg, error) {
}
var flags []flagArg
letters := arg[1:]
for i, r := range letters {
letter := string(r)
for i := 0; i < len(letters); i++ {
letter := letters[i : i+1]
def := flagByShort(letter)
if def == nil {
return nil, fmt.Errorf("unknown flag -%s", letter)
@@ -160,7 +134,7 @@ func splitFlags(arg string) ([]flagArg, error) {
flags = append(flags, flagArg{def: def})
continue
}
rest := letters[i+len(letter):]
rest := letters[i+1:]
if strings.HasPrefix(rest, "=") {
return nil, fmt.Errorf("-%s takes its value attached (-%s%s) or next (-%s %s); = belongs to --%s=%s",
letter, letter, rest[1:], letter, rest[1:], def.long, rest[1:])
@@ -171,7 +145,7 @@ func splitFlags(arg string) ([]flagArg, error) {
}
func parseArgs(argv []string) (invocation, error) {
in := invocation{repeat: 1, separator: "\n", format: "text"}
in := invocation{repeat: 1, separator: "\n"}
for i := 0; i < len(argv); i++ {
arg := argv[i]
if arg == "--" {
@@ -209,81 +183,17 @@ func parseArgs(argv []string) (invocation, error) {
return in, nil
}
// recordFormat is one way to write a record out: the line each record renders,
// the header that precedes the first one, and the open/close frame plus the
// between-record separator a document form needs.
type recordFormat struct {
header func(*fejkdata.Record) string
line func(r *fejkdata.Record, table string) string
open string
close string
sep string
}
// recordFormats is every --format that writes records. json frames the records
// as one array document; ndjson is the same column, one object per line.
var recordFormats = map[string]recordFormat{
"csv": {header: (*fejkdata.Record).CSVHeader, line: func(r *fejkdata.Record, _ string) string { return r.CSVLine() }, sep: "\n"},
"json": {line: jsonLine, open: "[", close: "]", sep: ",\n"},
"ndjson": {line: jsonLine, sep: "\n"},
"sql": {line: func(r *fejkdata.Record, table string) string { return r.SQLInsert(table) }, sep: "\n"},
}
func jsonLine(r *fejkdata.Record, _ string) string { return r.JSON() }
// writesRecords reports whether the format writes records rather than plain text.
func (in invocation) writesRecords() bool {
return in.format != "text"
}
// formatNames lists the --format values, text included, for the misuse error.
func formatNames() string {
names := []string{"text"}
for name := range recordFormats {
names = append(names, name)
// check rejects a flag combination that cannot run.
func (in invocation) check() error {
if in.list && len(in.paths) > 0 {
return errors.New("--list takes no path")
}
sort.Strings(names)
return strings.Join(names[:len(names)-1], ", ") + " or " + names[len(names)-1]
}
// checkFlags rejects a flag value or combination that cannot run.
func (in invocation) checkFlags() error {
if _, ok := recordFormats[in.format]; !ok && in.format != "text" {
return fmt.Errorf("--format takes %s, got %q", formatNames(), in.format)
}
if in.tableSet && in.table == "" {
return errors.New("--table names the INSERT target, so it cannot be empty")
}
if in.tableSet && in.format != "sql" {
return errors.New("--table names the INSERT target, so it needs --format sql")
}
if in.writesRecords() && in.separatorSet {
return errors.New("--separator joins text values, so it has no effect with --format " + in.format)
if !in.list && len(in.paths) != 1 {
return fmt.Errorf("expected one path, got %d", len(in.paths))
}
return nil
}
// check rejects a flag combination or an argument that cannot run, and reports
// what the argument names, so its shape is settled before any data is read.
func (in invocation) check() (argKind, error) {
if err := in.checkFlags(); err != nil {
return argPath, err
}
if in.list && len(in.paths) > 0 {
return argPath, errors.New("--list takes no path")
}
if in.list && (in.repeatSet || in.separatorSet || in.formatSet || in.tableSet) {
return argPath, errors.New("--list takes no --repeat, --separator, --format or --table")
}
if in.list {
return argPath, nil
}
if len(in.paths) != 1 {
return argPath, fmt.Errorf("expected one path or template, got %d", len(in.paths))
}
return classify(in.paths[0])
}
func (in invocation) options() []fejkdata.Option {
var opts []fejkdata.Option
if in.noShipped {
@@ -298,137 +208,17 @@ func (in invocation) options() []fejkdata.Option {
return opts
}
// write streams the argument's renders to w, repeat of them joined by the
// separator and ended by a newline. A value that renders once renders every time,
// so a render failure comes before anything is written; a write failure surfaces
// from Flush, bufio keeping the first one.
func (in invocation) write(f *fejkdata.Generator, kind argKind, w io.Writer) error {
if in.writesRecords() {
return in.writeRecords(f, kind, w)
}
draw, err := in.textDraw(f, kind, in.paths[0])
if err != nil {
return err
}
out := bufio.NewWriter(w)
for i := 0; i < in.repeat; i++ {
v, err := draw()
// values renders the path repeat times, joined by the separator.
func (in invocation) values(f *fejkdata.Generator) (string, error) {
vals := make([]string, in.repeat)
for i := range vals {
v, err := f.Fake(in.paths[0])
if err != nil {
return err
return "", err
}
if i > 0 {
out.WriteString(in.separator)
}
out.WriteString(v)
vals[i] = v
}
out.WriteString("\n")
return out.Flush()
}
// textDraw builds what one text render yields: the value, from a path or an
// inline template.
func (in invocation) textDraw(f *fejkdata.Generator, kind argKind, arg string) (func() (string, error), error) {
if kind != argTemplate {
return func() (string, error) { return f.Fake(arg) }, nil
}
t, err := f.NewTemplate(arg)
if err != nil {
return nil, templateError{err}
}
return func() (string, error) { return t.Fake(), nil }, nil
}
// recordStream builds the record drawer for the argument, plus the INSERT table
// a sql format names.
func (in invocation) recordStream(f *fejkdata.Generator, kind argKind, arg string) (func() (*fejkdata.Record, error), string, error) {
record := func() (*fejkdata.Record, error) { return f.Record(arg) }
if kind == argTemplate {
t, err := f.NewRecordTemplate(arg)
if err != nil {
return nil, "", templateError{err}
}
record = func() (*fejkdata.Record, error) { return t.Fake(), nil }
}
table := in.table
if table == "" {
table = defaultTable(arg, kind)
}
return record, table, nil
}
// writeRecords streams a record per line in the chosen format, framing a document
// form with its open/close brackets and a header preceding the first record.
func (in invocation) writeRecords(f *fejkdata.Generator, kind argKind, w io.Writer) error {
record, table, err := in.recordStream(f, kind, in.paths[0])
if err != nil {
return err
}
format := recordFormats[in.format]
out := bufio.NewWriter(w)
if format.open != "" {
out.WriteString(format.open)
out.WriteByte('\n')
}
for i := 0; i < in.repeat; i++ {
r, err := record()
if err != nil {
return err
}
if i == 0 && format.header != nil {
out.WriteString(format.header(r))
out.WriteByte('\n')
}
if i > 0 {
out.WriteString(format.sep)
}
out.WriteString(format.line(r, table))
}
if format.close != "" {
out.WriteByte('\n')
out.WriteString(format.close)
}
out.WriteByte('\n')
return out.Flush()
}
// defaultTable names the INSERT target when --table is absent: the path's last
// segment, or "records" for an inline template that sits in no folder.
func defaultTable(arg string, kind argKind) string {
if kind == argTemplate {
return "records"
}
segments := strings.Split(arg, ".")
return segments[len(segments)-1]
}
// templateError marks a render failure that is the argument's own fault — an
// inline template that does not compile. run reports it as misuse (exit 2, with a
// pointer to --help), unlike an unknown path, which is a runtime error (exit 1).
type templateError struct{ error }
func (e templateError) Unwrap() error { return e.error }
type argKind int
const (
argPath argKind = iota
argTemplate
)
// classify reads what a positional argument names by its shape: a { token, or a
// JSON object, array or string, is an inline template; anything else is a path.
func classify(arg string) (argKind, error) {
if strings.ContainsRune(arg, '{') || (isJSONStart(strings.TrimSpace(arg)) && json.Valid([]byte(arg))) {
return argTemplate, nil
}
if i := strings.IndexAny(arg, `[]}"`); i >= 0 {
return argPath, fmt.Errorf("%q holds a %q, which no path may, and it is not valid JSON, so it names no template either", arg, arg[i:i+1])
}
return argPath, nil
}
func isJSONStart(arg string) bool {
return strings.HasPrefix(arg, "[") || strings.HasPrefix(arg, `"`)
return strings.Join(vals, in.separator), nil
}
func main() { os.Exit(run(os.Args[1:], os.Stdout, os.Stderr)) }
@@ -447,8 +237,7 @@ func run(args []string, stdout, stderr io.Writer) int {
fmt.Fprintln(stdout, "fejkdata "+buildVersion())
return 0
}
kind, err := in.check()
if err != nil {
if err := in.check(); err != nil {
return misuse(stderr, err)
}
f, err := fejkdata.New(in.options()...)
@@ -465,20 +254,17 @@ func run(args []string, stdout, stderr io.Writer) int {
}
return 0
}
if err := in.write(f, kind, stdout); err != nil {
var te templateError
if errors.As(err, &te) {
return misuse(stderr, te.error)
}
out, err := in.values(f)
if err != nil {
fmt.Fprintln(stderr, err)
return 1
}
fmt.Fprintln(stdout, out)
return 0
}
func misuse(stderr io.Writer, err error) int {
// A library error already names the program, so the prefix is not doubled.
fmt.Fprintf(stderr, "fejkdata: %s\ntry 'fejkdata --help'\n", strings.TrimPrefix(err.Error(), "fejkdata: "))
fmt.Fprintf(stderr, "fejkdata: %v\ntry 'fejkdata --help'\n", err)
return 2
}
cmd/fejkdata/main_test.go
+1 -283
View File
@@ -2,11 +2,6 @@ package main
import (
"bytes"
"encoding/csv"
"encoding/json"
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
@@ -95,7 +90,7 @@ func TestRunShortFlagValues(t *testing.T) {
{[]string{"-s=42", "-d", svSE, "address"}, "-s42"},
{[]string{"-s=42", "-d", svSE, "address"}, "--seed=42"},
{[]string{"-d=" + svSE, "address"}, "--data-path="},
{[]string{"-nd", "3", "-d", svSE, "word"}, `--repeat needs an integer in 1..1048576, got "d"`},
{[]string{"-nd", "3", "-d", svSE, "word"}, `--repeat needs a positive integer, got "d"`},
{[]string{"--seed=", "-d", svSE, "word"}, `--seed needs an unsigned integer, got ""`},
} {
code, out, errb := runOut(c.args...)
@@ -304,92 +299,6 @@ func TestRunShippedDataByDefault(t *testing.T) {
}
}
func TestClassify(t *testing.T) {
for arg, want := range map[string]argKind{
"sv_SE.person": argPath,
"person.last": argPath,
"name: {x}": argTemplate, // a { token: a path can never carry a brace
`{"format":"x"}`: argTemplate,
`["a","b"]`: argTemplate, // a JSON array carries no brace
`[1, 2]`: argTemplate,
` ["a","b"]`: argTemplate, // padding is the template's own error, not a shape verdict
`"hello"`: argTemplate, // a JSON string, the spelling a format-only object names
} {
got, err := classify(arg)
if err != nil || got != want {
t.Errorf("classify(%q) = %v, %v; want %v", arg, got, err, want)
}
}
for arg, want := range map[string]string{
"[abc]": `holds a "["`,
"[abc].field": `holds a "["`,
"x[1]": `holds a "["`,
"a]b": `holds a "]"`,
"a}b": `holds a "}"`,
`"abc`: `holds a "\""`,
`"a]b`: `holds a "\""`, // the opener the reader typed, not the bracket behind it
} {
_, err := classify(arg)
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("classify(%q) = %v; want it rejected naming %s", arg, err, want)
}
}
}
func TestUsageReferencesResolve(t *testing.T) {
for _, token := range regexp.MustCompile(`\{/[^}]+\}`).FindAllString(usage, -1) {
code, out, errb := runOut("--seed", "1", token)
if code != 0 || strings.TrimSpace(out) == "" {
t.Errorf("usage advertises %s: run = %d, %q, stderr %q", token, code, out, errb)
}
}
}
func TestRunShapeMisuseBeforeLoad(t *testing.T) {
code, _, errb := runOut("--no-shipped-data", "[abc]")
if code != 2 || !strings.Contains(errb, "[abc]") || strings.Contains(errb, "--data-path") {
t.Fatalf("shape misuse with no data = %d, %q; want the shape error before any load", code, errb)
}
}
func TestRunInlineTemplate(t *testing.T) {
code, out, errb := runOut("--seed", "1", "name: {/sv_SE.person.last}")
if code != 0 || !strings.HasPrefix(out, "name: ") || strings.Contains(out, "{") {
t.Fatalf("inline format string = %d, %q, stderr %q", code, out, errb)
}
code, out, errb = runOut("--seed", "1", `{"format":"name: {x}","x":["bosse","lina"]}`)
if code != 0 || (out != "name: bosse\n" && out != "name: lina\n") {
t.Fatalf("inline JSON template = %d, %q, want one name, stderr %q", code, out, errb)
}
code, out, errb = runOut("--seed", "1", `"name: {/sv_SE.person.last}"`)
if code != 0 || !strings.HasPrefix(out, "name: ") || strings.Contains(out, "{") {
t.Fatalf("inline JSON string = %d, %q, stderr %q", code, out, errb)
}
code, out, errb = runOut("--seed", "1", "-n", "2", `{digits(1)}`)
if code != 0 || len(strings.Split(strings.TrimRight(out, "\n"), "\n")) != 2 {
t.Fatalf("inline template with --repeat = %d, %q, stderr %q", code, out, errb)
}
}
func TestRunTemplateMisuse(t *testing.T) {
for arg, want := range map[string]string{
"{bad": "unterminated",
"[red,green]": "names no template either",
`{"format":"x"}`: "is a string",
"{/no.such.path}": "no entry",
"x[1]": "names no template either",
` ["a","b"] `: "may not be padded",
} {
code, out, errb := runOut("--seed", "1", arg)
if code != 2 || out != "" || !strings.Contains(errb, "try 'fejkdata --help'") || !strings.Contains(errb, want) {
t.Errorf("run(%q) = %d, %q, %q; want misuse naming %q and --help", arg, code, out, errb, want)
}
if strings.Contains(errb, "fejkdata: fejkdata:") {
t.Errorf("run(%q) doubled the program prefix: %q", arg, errb)
}
}
}
func TestRunNoShippedData(t *testing.T) {
code, list, errb := runOut("--no-shipped-data", "-d", svSE, "--list")
if code != 0 {
@@ -407,194 +316,3 @@ func TestRunNoShippedData(t *testing.T) {
t.Errorf("--no-shipped-data alone = %d, %q, want misuse naming --data-path", code, errb)
}
}
func TestRunRepeatIsBounded(t *testing.T) {
for _, args := range [][]string{
{"--repeat", "9223372036854775807", "sv_SE.word"},
{"--repeat", "1048577", "sv_SE.word"},
{"-n", "99999999999", "sv_SE.word"},
} {
code, out, errb := runOut(args...)
if code != 2 || out != "" || !strings.Contains(errb, "1..1048576") {
t.Errorf("run(%v) = %d, %q, %q, want misuse naming the range", args, code, out, errb)
}
}
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "x.json"), []byte(`"x"`), 0o644); err != nil {
t.Fatal(err)
}
code, out, errb := runOut("--no-shipped-data", "-d", dir, "--repeat", "1048576", "--separator", "", "x")
if code != 0 || len(out) != 1048576+1 {
t.Errorf("repeat at the cap = %d, %d bytes, stderr %q; want every render streamed", code, len(out), errb)
}
}
func TestRunListTakesNoRepeatOrSeparator(t *testing.T) {
for _, args := range [][]string{{"--list", "-n", "3"}, {"--list", "--separator", ","}} {
code, _, errb := runOut(args...)
if code != 2 || !strings.Contains(errb, "--list takes no") {
t.Errorf("run(%v) = %d, %q, want misuse", args, code, errb)
}
}
}
func TestRunUnknownFlagIsNamedByRune(t *testing.T) {
code, _, errb := runOut("-ä", "sv_SE.word")
if code != 2 || !strings.Contains(errb, "unknown flag -ä") {
t.Errorf("run(-ä) = %d, %q, want the flag named whole", code, errb)
}
}
func TestRunEmptyDataPathIsNamed(t *testing.T) {
code, _, errb := runOut("--data-path=", "sv_SE.word")
if code != 1 || !strings.Contains(errb, "empty") {
t.Errorf("run(--data-path=) = %d, %q, want the empty path named", code, errb)
}
}
func recordDir(t *testing.T) string {
t.Helper()
dir := t.TempDir()
content := `{"format":"{first} {last}","first":["Ada","Bo"],"last":["Lovelace","Ek"]}`
if err := os.WriteFile(filepath.Join(dir, "users.json"), []byte(content), 0o644); err != nil {
t.Fatal(err)
}
return dir
}
func TestRunRecordJSON(t *testing.T) {
code, out, errb := runOut("--seed", "1", "--format", "json", "--repeat", "2", "--data-path", recordDir(t), "users")
if code != 0 {
t.Fatalf("run = %d, stderr=%q", code, errb)
}
var arr []map[string]string
if err := json.Unmarshal([]byte(out), &arr); err != nil {
t.Fatalf("json output is not one JSON array: %v\n%q", err, out)
}
if len(arr) != 2 || len(arr[0]) != 2 || arr[0]["first"] == "" || arr[0]["last"] == "" {
t.Fatalf("json output = %q, want an array of two records with first and last columns", out)
}
}
func TestRunRecordNDJSON(t *testing.T) {
code, out, errb := runOut("--seed", "1", "--format", "ndjson", "--repeat", "2", "--data-path", recordDir(t), "users")
if code != 0 {
t.Fatalf("run = %d, stderr=%q", code, errb)
}
lines := strings.Split(strings.TrimRight(out, "\n"), "\n")
if len(lines) != 2 {
t.Fatalf("ndjson output = %d lines %q, want one object per line", len(lines), out)
}
for _, line := range lines {
var m map[string]string
if err := json.Unmarshal([]byte(line), &m); err != nil || len(m) != 2 {
t.Fatalf("ndjson line %q is not one object: %v", line, err)
}
}
}
func TestRunRecordCSVRoundTrips(t *testing.T) {
code, out, errb := runOut("--seed", "1", "--format", "csv", "--repeat", "3", "--data-path", recordDir(t), "users")
if code != 0 {
t.Fatalf("run = %d, stderr=%q", code, errb)
}
lines := strings.Split(strings.TrimRight(out, "\n"), "\n")
if len(lines) != 4 {
t.Fatalf("csv output = %d lines %q, want a header and 3 rows", len(lines), out)
}
if lines[0] != "first,last" {
t.Fatalf("csv header = %q, want first,last", lines[0])
}
r, err := csv.NewReader(strings.NewReader(out)).ReadAll()
if err != nil {
t.Fatalf("csv output is not valid CSV: %v", err)
}
if len(r) != 4 || len(r[0]) != 2 {
t.Fatalf("csv parsed to %v, want 4 records of 2 fields", r)
}
}
func TestRunRecordSQL(t *testing.T) {
code, out, errb := runOut("--seed", "1", "--format", "sql", "--repeat", "2", "--table", "people", "--data-path", recordDir(t), "users")
if code != 0 {
t.Fatalf("run = %d, stderr=%q", code, errb)
}
lines := strings.Split(strings.TrimRight(out, "\n"), "\n")
if len(lines) != 2 || !strings.HasPrefix(lines[0], `INSERT INTO "people" ("first", "last") VALUES (`) {
t.Fatalf("sql output = %q, want two INSERT INTO people statements", out)
}
}
func TestRunRecordDefaultTable(t *testing.T) {
code, out, errb := runOut("--seed", "1", "--format", "sql", "--data-path", recordDir(t), "users")
if code != 0 || !strings.HasPrefix(out, `INSERT INTO "users" (`) {
t.Fatalf("default table = %d, %q, want the path's last segment; stderr %q", code, out, errb)
}
}
func TestRunRecordInlineTemplate(t *testing.T) {
code, out, errb := runOut("--seed", "1", "--format", "json", `{"format":"{x}","x":["a","b"]}`)
if code != 0 {
t.Fatalf("inline record = %d, stderr=%q", code, errb)
}
var arr []map[string]string
if err := json.Unmarshal([]byte(out), &arr); err != nil || len(arr) != 1 || (arr[0]["x"] != "a" && arr[0]["x"] != "b") {
t.Fatalf("inline record json = %q, want one record with a column x (err %v)", out, err)
}
}
func TestRunRecordMisuse(t *testing.T) {
for _, c := range []struct {
args []string
want string
}{
{[]string{"--format", "yaml", "users"}, "--format takes csv, json, ndjson, sql or text"},
{[]string{"--format", "json", "--separator", ",", "users"}, "--separator joins text values"},
{[]string{"--table", "t", "users"}, "--table names the INSERT target"},
{[]string{"--format", "json", "--table", "t", "users"}, "--table names the INSERT target"},
{[]string{"--format", "sql", "--table", "", "users"}, "--table names the INSERT target"},
} {
code, out, errb := runOut(c.args...)
if code != 2 || out != "" || !strings.Contains(errb, c.want) {
t.Errorf("run(%v) = %d, %q, %q; want misuse naming %q", c.args, code, out, errb, c.want)
}
}
}
func TestRunRecordOnABareValueIsRuntimeError(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "word.json"), []byte(`["a","b"]`), 0o644); err != nil {
t.Fatal(err)
}
code, _, errb := runOut("--format", "json", "--data-path", dir, "word")
if code != 1 {
t.Fatalf("record on a choice = %d, want exit 1 (runtime error), stderr %q", code, errb)
}
}
func TestRunRecordRejectsATopLevelRepeat(t *testing.T) {
dir := t.TempDir()
content := `{"format":"{a}-","repeat":3,"separator":"|","a":["x","y"]}`
if err := os.WriteFile(filepath.Join(dir, "rep.json"), []byte(content), 0o644); err != nil {
t.Fatal(err)
}
code, _, errb := runOut("--format", "json", "--data-path", dir, "rep")
if code != 1 || !strings.Contains(errb, "carries repeat 3") {
t.Errorf("record on a repeating path = %d, %q; want exit 1 naming the repeat", code, errb)
}
code, _, errb = runOut("--format", "json", content)
if code != 2 || !strings.Contains(errb, "carries repeat 3") {
t.Errorf("record on a repeating inline template = %d, %q; want exit 2 naming the repeat", code, errb)
}
if code, out, _ := runOut("--seed", "1", "--data-path", dir, "rep"); code != 0 || strings.TrimSpace(out) != "x-|x-|x-" {
t.Errorf("text view = %d, %q; want the repeat still composed", code, out)
}
}
func TestRunNumbersFollowTheShell(t *testing.T) {
_, want, _ := runOut("--seed", "7", "--repeat", "3", "sv_SE.word")
code, got, errb := runOut("--seed", "007", "--repeat", "+3", "sv_SE.word")
if code != 0 || got != want {
t.Errorf("run(--seed 007 --repeat +3) = %d, %q, stderr %q; want the same as --seed 7 --repeat 3 %q", code, got, errb, want)
}
}
compose.yaml
-4
View File
@@ -42,10 +42,6 @@ services:
<<: *go
command: go vet ./...
cyclo:
<<: *go
command: go run github.com/fzipp/gocyclo/cmd/gocyclo@v0.6.0 -over 14 -ignore _test .
build:
<<: *go
command: go build ./...
data.go
+44 -60
View File
@@ -10,14 +10,13 @@ import (
)
// dataSource is one tree to load: an fs.FS and the directory in it to start from.
// label prefixes file names in errors; onDisk marks path as a directory that must
// exist.
// label prefixes file names in errors; path, when set, is a directory on disk that
// must exist.
type dataSource struct {
fsys fs.FS
label string
onDisk bool
path string
root string
fsys fs.FS
label string
path string
root string
}
func (s dataSource) name(p string) string {
@@ -32,18 +31,15 @@ func (s dataSource) name(p string) string {
// keyed by its base name (address.json -> "address"); each subdirectory becomes a
// nested group, so folders turn into dot-path segments. Sources merge left to right:
// matching groups merge by their children, and any other clash is won by the last
// source loaded. Once merged, linkRefs binds every reference against the
// source loaded. Once merged, linkRefs binds every {..path} reference against the
// final tree.
func loadData(sources []dataSource) (map[string]node, error) {
root := map[string]node{}
for _, src := range sources {
if src.onDisk {
if src.path == "" {
return nil, fmt.Errorf("a data path is empty")
}
if src.path != "" {
info, err := os.Stat(src.path)
if err != nil {
return nil, err
return nil, fmt.Errorf("%s: no such directory", src.path)
}
if !info.IsDir() {
return nil, fmt.Errorf("%s is not a directory", src.path)
@@ -68,7 +64,10 @@ func loadData(sources []dataSource) (map[string]node, error) {
if err := checkNoCycles(root); err != nil {
return nil, err
}
if err := checkScope(treeScope(root)); err != nil {
if err := checkRepeatReach(root); err != nil {
return nil, err
}
if err := checkBoundLevelsHeld(root); err != nil {
return nil, err
}
return root, nil
@@ -88,59 +87,44 @@ func loadDir(src dataSource, dir string) (*group, error) {
continue
}
full := path.Join(dir, e.Name())
load := loadFile
if e.IsDir() {
load = loadFolder
child, err := loadDir(src, full)
if err != nil {
return nil, err
}
if len(child.children) == 0 {
continue
}
if err := checkName(e.Name()); err != nil {
return nil, fmt.Errorf("%s: folder %w", src.name(full), err)
}
g.children[e.Name()] = child
continue
}
if err := load(src, g, full, e.Name()); err != nil {
return nil, err
if !strings.HasSuffix(e.Name(), ".json") {
continue
}
name := strings.TrimSuffix(e.Name(), ".json")
if err := checkName(name); err != nil {
return nil, fmt.Errorf("%s: category %w", src.name(full), err)
}
b, err := fs.ReadFile(src.fsys, full)
if err != nil {
return nil, fmt.Errorf("%s: %w", src.name(full), err)
}
var raw any
if err := json.Unmarshal(b, &raw); err != nil {
return nil, fmt.Errorf("%s: %w", src.name(full), err)
}
n, err := compile(raw)
if err != nil {
return nil, fmt.Errorf("%s: %w", src.name(full), err)
}
g.children[name] = n
}
return g, nil
}
// loadFolder adds a subdirectory as a nested group, unless nothing under it is data.
func loadFolder(src dataSource, g *group, full, name string) error {
child, err := loadDir(src, full)
if err != nil {
return err
}
if len(child.children) == 0 {
return nil
}
if err := checkName(name); err != nil {
return fmt.Errorf("%s: folder %w", src.name(full), err)
}
g.children[name] = child
return nil
}
// loadFile compiles a *.json file into a category named after it; any other file
// is skipped.
func loadFile(src dataSource, g *group, full, file string) error {
if !strings.HasSuffix(file, ".json") {
return nil
}
name := strings.TrimSuffix(file, ".json")
if err := checkName(name); err != nil {
return fmt.Errorf("%s: category %w", src.name(full), err)
}
b, err := fs.ReadFile(src.fsys, full)
if err != nil {
return fmt.Errorf("%s: %w", src.name(full), err)
}
var raw any
if err := json.Unmarshal(b, &raw); err != nil {
return fmt.Errorf("%s: %w", src.name(full), err)
}
n, err := compile(raw)
if err != nil {
return fmt.Errorf("%s: %w", src.name(full), err)
}
g.children[name] = n
return nil
}
// mergeChildren overlays src onto dst. Two groups under the same key merge
// recursively (so locales/categories from several paths combine); every other
// key is replaced, making the last-loaded directory win on a conflict.
data_test.go
-37
View File
@@ -264,40 +264,3 @@ func luhnValid(s string) bool {
}
return sum%10 == 0
}
// swedishName matches one or more letter-words, optionally space/hyphen joined
// ("Storgatan", "Norra Promenaden", "von Flemming"). Used by the composition
// tests so shipped name lists can grow without re-enumerating them here.
var swedishName = regexp.MustCompile(`^\p{L}+([ -]\p{L}+)*$`)
func TestShippedStreetComposition(t *testing.T) {
// street is a choice of composed {first}{last} templates and literal names.
f := newGenerator(t, "data/sv_SE", WithSeed(5))
for i := 0; i < 300; i++ {
if s := fake(t, f, "address.street"); !swedishName.MatchString(s) {
t.Fatalf("street %q is not a Swedish street name", s)
}
}
}
func TestShippedLastNameComposition(t *testing.T) {
// last is a choice of patronymic {first}sson templates, compound
// {first}{last} templates and literal surnames.
f := newGenerator(t, "data/sv_SE", WithSeed(6))
for i := 0; i < 300; i++ {
if s := fake(t, f, "person.last"); !swedishName.MatchString(s) {
t.Fatalf("last name %q is not a Swedish surname", s)
}
}
}
func TestShippedStreetNumberFormats(t *testing.T) {
// Reachable via a hyphenated path; covers all five weighted number variants.
f := newGenerator(t, "data/sv_SE", WithSeed(8))
re := regexp.MustCompile(`^[1-9]\d{0,2}[A-Z]?$`)
for i := 0; i < 300; i++ {
if n := fake(t, f, "address.street-number"); !re.MatchString(n) {
t.Fatalf("street-number %q does not match %s", n, re)
}
}
}
edge_test.go
+385
View File
@@ -0,0 +1,385 @@
package fejkdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"regexp"
"slices"
"strings"
"testing"
)
// --- format string edge cases ---
func TestHashIsLiteral(t *testing.T) {
f := engine(1)
cases := map[string]string{
`{"format":"","x":"v"}`: "",
`{"format":"#","x":"v"}`: "#",
`{"format":"##","x":"v"}`: "##",
`{"format":"#0#1#A#a","x":"v"}`: "#0#1#A#a",
`{"format":"#{x}","x":"v"}`: "#v",
}
for tmpl, want := range cases {
if got := mustRender(t, f, tmpl); got != want {
t.Errorf("render(%s) = %q, want %q", tmpl, got, want)
}
}
}
func TestMultibyteFormat(t *testing.T) {
// Scanning is rune-aware: multibyte literals coexist with class chars and
// tokens without corrupting indices.
got := mustRender(t, engine(2), `{"format":"Öster{x}-{digits(1)}å","x":"väg"}`)
if !regexp.MustCompile(`^Österväg-[0-9]å$`).MatchString(got) {
t.Fatalf("multibyte format = %q", got)
}
}
func TestAlternationThreeWay(t *testing.T) {
f := engine(4)
seen := map[string]bool{}
for i := 0; i < 200; i++ {
seen[mustRender(t, f, `{"format":"{a|b|c}","a":"A","b":"B","c":"C"}`)] = true
}
if !seen["A"] || !seen["B"] || !seen["C"] || len(seen) != 3 {
t.Fatalf("3-way alternation produced %v, want A, B and C", seen)
}
}
func TestNewErrors(t *testing.T) {
// Pointing New at a file (not a directory) fails.
file := filepath.Join(t.TempDir(), "xx_XX")
if err := os.WriteFile(file, []byte("{}"), 0o644); err != nil {
t.Fatal(err)
}
if _, err := New(WithoutShippedData(), WithDataPath(file)); err == nil {
t.Error("New(file) = nil error, want not-a-directory error")
}
// Invalid JSON in a category file fails.
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{"broken": `{ not json`}))); err == nil {
t.Error("New(invalid JSON) = nil error")
}
// An option that cannot take effect, and a category or folder no dot path can
// reach, are mistakes New must name rather than accept and ignore.
rejected := map[string]struct {
files map[string]string
want string
}{
"separator without repeat": {
map[string]string{"a": `{"format":"{x}","x":"1","separator":","}`},
"has no effect without a repeat above 1",
},
"separator with an explicit repeat of 1": {
map[string]string{"a": `{"format":"{x}","x":"1","separator":","}`},
"has no effect without a repeat above 1",
},
"weight outside a choice": {
map[string]string{"a": `{"format":"x","weight":5}`},
"weight only skews a choice's items",
},
"non-numeric weight outside a choice": {
map[string]string{"a": `{"format":"x","weight":"bad"}`},
"weight only skews a choice's items",
},
"weight used as a field": {
map[string]string{"a": `{"format":"{name} {weight}kg","name":"Anvil","weight":["7"]}`},
"can never be a field",
},
"an option name used as a token": {
map[string]string{"a": `"{weight}"`},
`"weight" is an option and can never be a field`,
},
"category name with a dot": {
map[string]string{"a.b": `"1"`},
`category "a.b" contains "."`,
},
"folder name with a dot": {
map[string]string{"a.b/cat": `"1"`},
`/a.b: folder "a.b" contains "."`,
},
// The token grammar reserves three more characters. A name carrying one
// still resolves by dot path, but no format can name it, so it is rejected
// where it is authored rather than at the token that cannot reach it.
"field name with a pipe": {
map[string]string{"a": `{"format":"{x}","x":"1","b|c":"2"}`},
`field "b|c" contains "|"`,
},
"field name with a paren": {
map[string]string{"a": `{"format":"{x}","x":"1","b(c":"2"}`},
`field "b(c" contains "("`,
},
"field name with a closing brace": {
map[string]string{"a": `{"format":"{x}","x":"1","b}c":"2"}`},
`field "b}c" contains "}"`,
},
"category name with a pipe": {
map[string]string{"a|b": `"1"`},
`category "a|b" contains "|"`,
},
// An empty name is not a path segment, so List never offered it — while a
// bare {}, a trailing dot in Fake("a.") and a {/a.} reference all reached
// it. The engine accepted spellings it would never advertise.
"empty field name": {
map[string]string{"a": `{"format":"[{}]","":"VALUE"}`},
`field "" is empty`,
},
"folder name with a paren": {
map[string]string{"a(b/cat": `"1"`},
`folder "a(b" contains "("`,
},
// A repeated arm skews an alternation, which weight is the spelling for.
"repeated alternation arm": {
map[string]string{"a": `{"format":"{x|x}","x":"1"}`},
`arm "x" is repeated`,
},
"repeated arm among others": {
map[string]string{"a": `{"format":"{x|y|x}","x":"1","y":"2"}`},
`arm "x" is repeated`,
},
"repeated path arm": {
map[string]string{"a": `{"format":"{p.v|p.v}","p":{"format":"{v}","v":"1"}}`},
`arm "p.v" is repeated`,
},
// A reference arm is the only kind that reaches the repeat check by passing
// the per-arm checks rather than falling through them.
"repeated reference arm": {
map[string]string{"a": `"x"`, "b": `"{/a|..a}"`},
`arm "..a" is repeated`,
},
// An arm that is broken on its own terms is reported as that, not as a
// repeat: the repeat is a consequence of the real mistake.
"repeated arm with no path": {
map[string]string{"a": `"{/|..}"`},
"reference has no path",
},
// No field can be named "", so the token is told that rather than sent to
// name one — the fix "no field" points at is itself a load error.
"repeated empty arm": {
map[string]string{"a": `"{|}"`},
"a name is never empty",
},
"bare empty token": {
map[string]string{"a": `{"format":"[{}]","x":"1"}`},
"a name is never empty",
},
}
for name, c := range rejected {
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, c.files)))
if err == nil {
t.Errorf("%s: New = nil error, want it rejected at load", name)
continue
}
if !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: New = %v, want it to mention %q", name, err, c.want)
}
}
// Data that works today must keep working, and stay reachable.
accepted := map[string]struct {
files map[string]string
path string
want string
}{
"option name as a field": {map[string]string{"a": `{"format":"{name} {Weight}kg","name":"Anvil","Weight":"7"}`}, "a", "Anvil 7kg"},
"format spelling as a field": {map[string]string{"a": `{"format":"{Format}","Format":"PDF"}`}, "a", "PDF"},
"hyphenated field": {map[string]string{"a": `{"format":"{x-y}","x-y":"1"}`}, "a.x-y", "1"},
"category named Format": {map[string]string{"Format": `"1"`}, "Format", "1"},
"folder named Repeat": {map[string]string{"Repeat/cat": `"1"`}, "Repeat.cat", "1"},
"field with a closing paren": {map[string]string{"a": `{"format":"{b)c}","b)c":"2"}`}, "a", "2"},
"repeat without a separator": {map[string]string{"a": `{"format":"{x}","repeat":3,"x":"1"}`}, "a", "111"},
// One name in two separate tokens is two independent draws, not a repeated
// arm; only a repeat within one alternation is rejected.
"one name in two tokens": {map[string]string{"a": `{"format":"{x}{x}","x":"1"}`}, "a", "11"},
}
for name, c := range accepted {
f, err := New(WithoutShippedData(), WithDataPath(writeData(t, c.files)))
if err != nil {
t.Errorf("%s: New = %v, want it accepted", name, err)
continue
}
if got, err := f.Fake(c.path); err != nil || got != c.want {
t.Errorf("%s: Fake(%q) = %q, %v, want %q", name, c.path, got, err, c.want)
}
}
}
// --- deep path navigation ---
func TestDeepDottedPath(t *testing.T) {
// A 5-segment path descends through alternating object/array nodes; choices
// on the path are single-variant, so it resolves deterministically.
f := engine(1)
f.categories = map[string]node{
"deep": compiled(t, `{"format":"{a}","a":{"format":"{b}","b":{"format":"{c}","c":{"format":"{d}","d":"leaf"}}}}`),
}
if got, err := f.Fake("deep.a.b.c.d"); err != nil || got != "leaf" {
t.Fatalf("Fake(deep.a.b.c.d) = %q, %v, want leaf", got, err)
}
// Rendering the whole tree resolves the same chain.
if got, err := f.Fake("deep"); err != nil || got != "leaf" {
t.Fatalf("Fake(deep) = %q, %v, want leaf", got, err)
}
}
func TestDescendIntoStringErrors(t *testing.T) {
f := engine(1)
f.categories = map[string]node{"greeting": compiled(t, `"hej"`)}
if _, err := f.Fake("greeting.extra"); err == nil || !strings.Contains(err.Error(), `no field "extra"`) {
t.Fatalf("Fake(greeting.extra) = %v, want a no-field error", err)
}
}
// TestPathThroughChoice pins the rule that keeps a dotted path from rendering on
// one call and failing on the next: every variant must carry the rest of the path.
func TestPathThroughChoice(t *testing.T) {
dir := writeData(t, map[string]string{
"every": `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2"}]`,
"notall": `[{"format":"{f}","f":"1"},"plain"]`,
"some": `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2","extra":"x"}]`,
})
f := newGenerator(t, dir, WithSeed(1))
for i := 0; i < 200; i++ {
if got := fake(t, f, "every.f"); got != "1" && got != "2" {
t.Fatalf("every.f = %q, want 1 or 2", got)
}
}
// A path only some variants carry is reported against what all of them carry.
if _, err := f.Fake("some.extra"); err == nil || !strings.Contains(err.Error(), "all carry [f]") {
t.Errorf("Fake(some.extra) = %v, want it to name what every variant carries", err)
}
var first string
for i := 0; i < 200; i++ {
_, err := f.Fake("notall.f")
if err == nil {
t.Fatal("notall.f = nil error, want the same failure every call")
}
if i == 0 {
first = err.Error()
} else if err.Error() != first {
t.Fatalf("notall.f error varies between calls:\n %s\n %s", first, err.Error())
}
}
}
// TestPathKeyIsUnambiguous pins that the one shape which could collide cannot be
// written: a field literally named "a.b" and a field "a" holding "b" would both
// spell "a.b", so a dotted field name is rejected at New and a dot means a path
// wherever it appears.
func TestPathKeyIsUnambiguous(t *testing.T) {
dir := writeData(t, map[string]string{
"cat": `[{"format":"{a.b}","a.b":"1"},{"format":"{a}","a":{"format":"{b}","b":"2"}}]`,
})
_, err := New(WithoutShippedData(), WithDataPath(dir))
if err == nil || !strings.Contains(err.Error(), `field "a.b" contains "."`) {
t.Fatalf("New = %v, want the dotted field name rejected", err)
}
// The same data without the dotted key is fine, and the path resolves.
f := newGenerator(t, writeData(t, map[string]string{
"cat": `{"format":"{a.b}","a":{"format":"{b}","b":"2"}}`,
}), WithSeed(1))
if !slices.Contains(f.List(), "cat.a.b") {
t.Error("List() omits cat.a.b, which the data carries")
}
if got := fake(t, f, "cat"); got != "2" {
t.Fatalf("cat = %q, want 2", got)
}
}
// TestMissingFieldNamesItself keeps the precise diagnosis for the ordinary typo: a
// single-variant choice always picks the same item, so it needs no every-variant
// guard and the error can name the field that is missing.
func TestMissingFieldNamesItself(t *testing.T) {
f := newGenerator(t, "data/sv_SE", WithSeed(1))
_, err := f.Fake("person.typo")
if err == nil || !strings.Contains(err.Error(), `no field "typo"`) {
t.Errorf("Fake(person.typo) = %v, want it to name the missing field", err)
}
}
// --- category root shapes ---
func TestCategoryRootShapes(t *testing.T) {
dir := writeData(t, map[string]string{
"obj": `"{digits(2)}"`, // object root
"lit": `"hello"`, // bare-string root
})
f := newGenerator(t, dir, WithSeed(1))
if got := fake(t, f, "obj"); !regexp.MustCompile(`^\d\d$`).MatchString(got) {
t.Errorf("object-root category = %q, want two digits", got)
}
if got := fake(t, f, "lit"); got != "hello" {
t.Errorf("string-root category = %q, want hello", got)
}
}
// --- very long lists ---
func TestLongStringList(t *testing.T) {
const n = 2000
names := make([]string, n)
for i := range names {
names[i] = fmt.Sprintf("name-%04d", i)
}
list, err := json.Marshal(names)
if err != nil {
t.Fatal(err)
}
f := newGenerator(t, writeData(t, map[string]string{"name": string(list)}), WithSeed(1))
valid := map[string]bool{}
for _, v := range names {
valid[v] = true
}
seen := map[string]bool{}
for i := 0; i < 20000; i++ {
v := fake(t, f, "name")
if !valid[v] {
t.Fatalf("got %q, not in the list", v)
}
seen[v] = true
}
if len(seen) < n*8/10 {
t.Fatalf("only %d/%d distinct values seen; selection looks skewed", len(seen), n)
}
}
// --- composition against the shipped sv_SE data ---
// swedishName matches one or more letter-words, optionally space/hyphen joined
// ("Storgatan", "Norra Promenaden", "von Flemming"). Used by the composition
// tests so shipped name lists can grow without re-enumerating them here.
var swedishName = regexp.MustCompile(`^\p{L}+([ -]\p{L}+)*$`)
func TestShippedStreetComposition(t *testing.T) {
// street is a choice of composed {first}{last} templates and literal names.
f := newGenerator(t, "data/sv_SE", WithSeed(5))
for i := 0; i < 300; i++ {
if s := fake(t, f, "address.street"); !swedishName.MatchString(s) {
t.Fatalf("street %q is not a Swedish street name", s)
}
}
}
func TestShippedLastNameComposition(t *testing.T) {
// last is a choice of patronymic {first}sson templates, compound
// {first}{last} templates and literal surnames.
f := newGenerator(t, "data/sv_SE", WithSeed(6))
for i := 0; i < 300; i++ {
if s := fake(t, f, "person.last"); !swedishName.MatchString(s) {
t.Fatalf("last name %q is not a Swedish surname", s)
}
}
}
func TestShippedStreetNumberFormats(t *testing.T) {
// Reachable via a hyphenated path; covers all five weighted number variants.
f := newGenerator(t, "data/sv_SE", WithSeed(8))
re := regexp.MustCompile(`^[1-9]\d{0,2}[A-Z]?$`)
for i := 0; i < 300; i++ {
if n := fake(t, f, "address.street-number"); !re.MatchString(n) {
t.Fatalf("street-number %q does not match %s", n, re)
}
}
}
fejkdata.go
+8 -27
View File
@@ -28,24 +28,16 @@ import (
//go:embed data
var shippedFS embed.FS
// MaxRepeat caps a repeat, and the renders nested repeats multiply to along any
// path; the CLI's --repeat shares it.
const MaxRepeat = 1 << 20
var _ [^uint(0)>>63 - 1]struct{} // 64-bit only, per the README's Decisions
// ErrNoData is returned by New when no source is loaded at all.
var ErrNoData = errors.New("no data: WithoutShippedData needs at least one WithDataPath or WithDataFS")
// Generator generates fake data from a loaded namespace tree. Create one with [New].
// It is safe for concurrent use; a seeded sequence is reproducible only when drawn
// from one goroutine. The compiled tree is immutable after [New], and Fake,
// NewTemplate and List read it concurrently without a lock.
// from one goroutine.
type Generator struct {
mu sync.Mutex
rand *session
categories map[string]node
records map[node]recordShape
}
// session is one generator's mutable render state: the seeded rng plus the {seq()}
@@ -80,7 +72,7 @@ func WithSeed(seed uint64) Option {
// layer several; the last wins a name clash.
func WithDataPath(dir string) Option {
return func(c *config) {
c.sources = append(c.sources, dataSource{fsys: os.DirFS(dir), label: dir, onDisk: true, path: dir})
c.sources = append(c.sources, dataSource{fsys: os.DirFS(dir), label: dir, path: dir})
}
}
@@ -117,11 +109,7 @@ func New(opts ...Option) (*Generator, error) {
if err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
rng, err := newRand(c.seed, c.seeded)
if err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
return &Generator{rand: rng, categories: cats}, nil
return &Generator{rand: newRand(c.seed, c.seeded), categories: cats}, nil
}
// List returns the sorted dotted paths Fake can render: every category, the dotted
@@ -212,19 +200,12 @@ func join(prefix, name string) string {
return prefix + "." + name
}
// randomBytes seeds an unseeded generator.
var randomBytes = crand.Read
func newRand(seed uint64, seeded bool) (*session, error) {
var r *rand.Rand
if seeded {
r = rand.New(rand.NewPCG(seed, seed^0x9e3779b97f4a7c15))
} else {
func newRand(seed uint64, seeded bool) *session {
r := rand.New(rand.NewPCG(seed, seed^0x9e3779b97f4a7c15))
if !seeded {
var b [16]byte
if _, err := randomBytes(b[:]); err != nil {
return nil, fmt.Errorf("seeding from crypto/rand: %w", err)
}
_, _ = crand.Read(b[:])
r = rand.New(rand.NewPCG(binary.LittleEndian.Uint64(b[:8]), binary.LittleEndian.Uint64(b[8:])))
}
return &session{Rand: r, counters: map[string]uint64{}}, nil
return &session{Rand: r, counters: map[string]uint64{}}
}
fejkdata_test.go
+2 -22
View File
@@ -1,11 +1,8 @@
package fejkdata
import (
"errors"
"io/fs"
"strings"
"testing"
"testing/fstest"
)
// newGenerator creates a generator over a single data directory, failing on
@@ -41,25 +38,8 @@ func fake(t *testing.T, f *Generator, path string) string {
func TestNewMissingDirectory(t *testing.T) {
_, err := New(WithoutShippedData(), WithDataPath("data/de_DE"))
if err == nil || !strings.Contains(err.Error(), "de_DE") || !errors.Is(err, fs.ErrNotExist) {
t.Fatalf("New(missing) error = %v, want the real error, naming the path", err)
}
}
func TestNewRejectsAnEmptyDataPath(t *testing.T) {
_, err := New(WithoutShippedData(), WithDataPath(""))
if err == nil || !strings.Contains(err.Error(), "empty") {
t.Fatalf("New(WithDataPath(\"\")) = %v, want the empty path named", err)
}
}
func TestNewReportsAnEntropyFailure(t *testing.T) {
saved := randomBytes
randomBytes = func([]byte) (int, error) { return 0, errors.New("no entropy") }
defer func() { randomBytes = saved }()
_, err := New(WithoutShippedData(), WithDataFS(fstest.MapFS{"w.json": {Data: []byte(`"x"`)}}))
if err == nil || !strings.Contains(err.Error(), "no entropy") {
t.Fatalf("New() without entropy = %v, want the failure reported", err)
if err == nil || !strings.Contains(err.Error(), "de_DE") {
t.Fatalf("New(missing) error = %v, want it to name the path", err)
}
}
graph.go
-251
View File
@@ -1,251 +0,0 @@
package fejkdata
import (
"fmt"
"sort"
"strings"
)
// walkNodes calls fn once per contained node, passing the dot path that reaches it,
// visiting keys in sorted order so which of several broken nodes gets reported does
// not depend on map iteration.
func walkNodes(root map[string]node, fn func(path string, n node) error) error {
for _, name := range sortedNames(root) {
if err := eachNode(root[name], name, fn); err != nil {
return err
}
}
return nil
}
// eachNode visits n and every node contained within it once, passing the dot path
// that reaches each. It never crosses a reference edge — a bound {/path} field is
// skipped — so a single inline node is walked on its own.
func eachNode(n node, path string, fn func(path string, n node) error) error {
seen := map[node]bool{}
var visit func(string, node) error
visit = func(path string, m node) error {
if m == nil || seen[m] {
return nil
}
seen[m] = true
if err := fn(path, m); err != nil {
return err
}
for _, c := range contained(m) {
if err := visit(join(path, c.name), c.node); err != nil {
return err
}
}
return nil
}
return visit(path, n)
}
// namedNode is a contained child and the segment reaching it; a choice's items carry
// no segment, matching how a dot path steps over a choice.
type namedNode struct {
name string
node node
}
func contained(n node) []namedNode {
switch n := n.(type) {
case *group:
return named(n.children)
case *choice:
out := make([]namedNode, len(n.items))
for i, it := range n.items {
out[i] = namedNode{node: it}
}
return out
case *template:
return named(n.fields)
default:
return nil
}
}
// named skips a bound {/path} key: it is a render edge, not containment, so using
// it as a path segment would report a node under a path that does not reach it. Only
// a template's fields hold bindings — loadDir skips a dot-prefixed entry, so a
// group's children never carry the prefix — so this one skip serves both.
func named(m map[string]node) []namedNode {
out := make([]namedNode, 0, len(m))
for _, name := range sortedNames(m) {
if isRef(name) {
continue
}
out = append(out, namedNode{name: name, node: m[name]})
}
return out
}
func sortedNames(m map[string]node) []string {
names := make([]string, 0, len(m))
for name := range m {
names = append(names, name)
}
sort.Strings(names)
return names
}
// renderEdge is a child a node renders into, labelled by what reaches it (a field
// name, reference, or choice index) for a readable cycle report. operand names
// the builtin when the label is its operand rather than a token, so an error can
// name it the way the author wrote it.
type renderEdge struct {
to node
label string
operand string
}
// reached names an edge as the author spelled it, the vocabulary boundReaders uses
// for the sibling fence.
func (e renderEdge) reached() string {
if e.operand != "" {
return fmt.Sprintf("%s operand %q", e.operand, e.label)
}
return "{" + e.label + "}"
}
// renderEdges lists the children rendering n recurses into, mirroring expand: a
// choice's items, and a template's field/reference tokens plus its operands. A
// group renders nothing, so it has no edges.
func renderEdges(n node) []renderEdge {
switch n := n.(type) {
case *choice:
es := make([]renderEdge, len(n.items))
for i, it := range n.items {
es[i] = renderEdge{to: it, label: fmt.Sprintf("[%d]", i)}
}
return es
case *template:
var es []renderEdge
add := func(name, operand string) {
a := splitArm(name, n.refs)
c, ok := n.fields[a.key]
if !ok {
return
}
for _, leaf := range pathLeaves(c, a.tail) {
es = append(es, renderEdge{leaf, name, operand})
}
}
_ = eachToken(n.format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if fn, _, isFunc := funcCall(t.body); isFunc {
for _, operand := range tokenOperands(t.body) {
add(operand, fn)
}
return nil
}
for _, name := range strings.Split(t.body, "|") {
add(name, "")
}
return nil
})
return es
default:
return nil
}
}
// pathLeaves lists what a token's dotted tail renders: a choice on the way
// contributes every variant, since any of them may be the one drawn. checkPath has
// already proved the tail resolves in every variant.
func pathLeaves(n node, tail []string) []node {
var out []node
_ = walkPath(n, tail, pathWalk{
choice: func(c *choice, _ []string) ([]node, error) { return c.items, nil },
leaf: func(n node) error { out = append(out, n); return nil },
})
return out
}
// nodeScope is the set of nodes one validation pass covers: a whole loaded tree,
// or a single inline node.
type nodeScope func(fn func(path string, n node) error) error
func treeScope(root map[string]node) nodeScope {
return func(fn func(path string, n node) error) error { return walkNodes(root, fn) }
}
func inlineScope(n node) nodeScope {
return func(fn func(path string, m node) error) error { return eachNode(n, "template", fn) }
}
// checkScope runs the per-node fences over a scope, each over the whole scope
// before the next, so which of several broken nodes is reported does not depend on
// the walk. It runs after checkNoCycles, whose guarantee is what lets the walks
// terminate.
func checkScope(s nodeScope) error {
mem := reachMemo{}
if err := s(func(path string, n node) error { return repeatCheck(path, n, mem) }); err != nil {
return err
}
return s(heldCheck)
}
type reachMemo map[node]int
func (m reachMemo) of(n node) int {
if r, done := m[n]; done {
return r
}
r := 1
for _, e := range renderEdges(n) {
if c := m.of(e.to); c > r {
r = c
}
}
if t, ok := n.(*template); ok {
r *= t.repeat
}
m[n] = r
return r
}
// repeatCheck bounds the renders a repeat multiplies to along any root-to-leaf
// path, so nested repeats cannot build what one repeat may not.
func repeatCheck(path string, n node, mem reachMemo) error {
if t, ok := n.(*template); ok && t.repeat > 1 && mem.of(n) > MaxRepeat {
return fmt.Errorf("%s: repeat %d multiplies to %d renders along one path, above the maximum %d", path, t.repeat, mem.of(n), MaxRepeat)
}
return nil
}
// checkNoCycles rejects a reference cycle: a node whose rendering can reach itself
// — directly, mutually, or through a chain — never terminates, so it must fail at
// New rather than stack-overflow at render. It is a depth-first walk of the render
// graph (renderEdges); grey marks nodes on the current path so a back-edge to one
// is the cycle, while black lets a shared node (a DAG, not a cycle) be skipped.
// Every node is a root: a field its parent's format never renders is still reachable
// by dot path, so a cycle in one would otherwise reach render and be fatal there.
func checkNoCycles(root map[string]node) error {
const (
grey = 1
black = 2
)
color := map[node]int{}
var visit func(n node, path string) error
visit = func(n node, path string) error {
switch color[n] {
case grey:
return fmt.Errorf("reference cycle: %s", path)
case black:
return nil
}
color[n] = grey
for _, e := range renderEdges(n) {
if err := visit(e.to, path+" -> "+e.label); err != nil {
return err
}
}
color[n] = black
return nil
}
return walkNodes(root, func(path string, n node) error { return visit(n, path) })
}
hold.go
-316
View File
@@ -1,316 +0,0 @@
package fejkdata
import (
"fmt"
"sort"
"strings"
)
// heldCheck rejects every route to a held name except the ones that read its draw.
// An expansion holds one draw of that name; anything else that renders it draws
// again, and the two disagree. checkNoOverlap settles the spellings within one
// format (a token, an operand); this settles the rest — a reference, whether it
// sits in that format or in anything the format renders, however deep.
func heldCheck(path string, n node) error {
t, ok := n.(*template)
if !ok || len(t.held) == 0 {
return nil
}
readers := boundReaders(t.format, t.bound, t.refs)
for _, head := range heldHeads(t) {
if err := checkHeadHeld(t, head, readers); err != nil {
return fmt.Errorf("%s: %w", path, err)
}
}
return nil
}
// heldHeads lists a template's held names, operand heads first, then paths, each
// in name order: a level read both ways is reported by the operand's fence, and
// which overlap is reported does not vary.
func heldHeads(t *template) []string {
heads := make([]string, 0, len(t.held))
for head := range t.held {
heads = append(heads, head)
}
sort.Slice(heads, func(i, j int) bool {
_, pi := t.bound[heads[i]]
_, pj := t.bound[heads[j]]
if pi != pj {
return !pi
}
return heads[i] < heads[j]
})
return heads
}
// pinned collects what the hold of head answers for: a path pins the levels it
// passes through and the leaf it lands on, an operand exactly the value its render
// produces.
func pinned(t *template, head string, readers []reader) map[node]bool {
held := map[node]bool{}
if _, isPath := t.bound[head]; !isPath {
operandDraw(t.fields[head], held)
return held
}
for _, r := range readers {
if a := splitArm(r.name, t.refs); a.key == head {
coverPath(t.fields[head], a.tail, held)
}
}
return held
}
// checkHeadHeld rejects every route to what head's hold pins except the readers
// holding it. One seen set across the edges: a node that cannot reach the level
// cannot reach it by another route either, so it is walked once.
func checkHeadHeld(t *template, head string, readers []reader) error {
held := pinned(t, head, readers)
if len(held) == 0 {
return nil // a fixed head holds nothing to reach
}
reader, isPath := t.bound[head]
seen := map[node]bool{}
for _, e := range renderEdges(t) {
if splitArm(e.label, t.refs).key == head || !renders(e.to, held, seen) {
continue
}
if isPath {
return fmt.Errorf("%s renders %q, which {%s} reads a path into; name the fields you want instead", e.reached(), head, reader)
}
return fmt.Errorf("%s renders %q, which a {%s()} also reads; reach it one way so it is drawn once", e.reached(), head, operandReader(t, head))
}
return nil
}
// operandReader names the builtin whose operand holds head.
func operandReader(t *template, head string) string {
fn := ""
_ = eachToken(t.format, func(tok ftoken) error {
if tok.kind != 'b' || fn != "" {
return nil
}
if name, _, isFunc := funcCall(tok.body); isFunc {
for _, operand := range tokenOperands(tok.body) {
if splitArm(operand, t.refs).key == head {
fn = name
}
}
}
return nil
})
return fn
}
// coverPath collects what holding one path pins: every choice level the path
// passes through, whole, and the leaf it renders.
func coverPath(n node, tail []string, into map[node]bool) {
_ = walkPath(n, tail, pathWalk{
choice: func(c *choice, _ []string) ([]node, error) { cover(c, into, false); return nil, nil },
leaf: func(n node) error { cover(n, into, false); return nil },
})
}
// cover collects a level and everything contained in it. A fixed string outside a
// choice is left out — it cannot disagree with itself — but inside one each
// variant carries its own, so there it counts.
func cover(n node, into map[node]bool, inChoice bool) {
if isFixed(n) && !inChoice {
return
}
into[n] = true
_, isChoice := n.(*choice)
for _, c := range contained(n) {
cover(c.node, into, inChoice || isChoice)
}
}
// operandDraw collects what one held draw of an operand answers for: the operand
// and what rendering it settles inside itself. The builtin renders its operand
// whole, so that draw fixes every value the render produced, and a second route to
// any of them disagrees with it.
//
// The walk stops at a reference edge, which is where the operand's own value ends
// and a shared source begins: two names referencing one category are two draws, the
// same rule {word} {word} follows.
func operandDraw(n node, into map[node]bool) {
if isFixed(n) {
return
}
if into[n] {
return
}
into[n] = true
for _, e := range renderEdges(n) {
if isRef(e.label) {
continue
}
operandDraw(e.to, into)
}
}
// isFixed is a string that varies nothing: fixed text with no fields to read into.
func isFixed(n node) bool {
t, ok := n.(*template)
return ok && t.fixed && len(t.fields) == 0
}
// renders reports whether rendering n can reach anything in want, following the
// same edges expand does. seen keeps a node shared by several routes from being
// walked twice; checkNoCycles has already proved the graph is a DAG, so the walk
// ends.
func renders(n node, want, seen map[node]bool) bool {
if want[n] {
return true
}
if seen[n] {
return false
}
seen[n] = true
for _, e := range renderEdges(n) {
if renders(e.to, want, seen) {
return true
}
}
return false
}
// checkNoOverlap rejects a format that both renders a level and reads a path into
// it — {p} beside {p.first}, {p.addr} beside {p.addr.city}, {.p} beside
// {/sv_SE.p.first}. The path reads the level's held draw while rendering the level
// expands it afresh, so their values would disagree. Reads are compared by their
// one spelling, in sorted order, so which pair is reported depends neither on how
// a reference was written nor on where the tokens sit.
func checkNoOverlap(format string, bound map[string]string, refs map[string]refBinding) error {
names := boundReaders(format, bound, refs)
// Stable over one format-order scan, so two readers of one name (a token and a
// calc operand both naming "p") are reported as the format writes them.
sort.SliceStable(names, func(i, j int) bool { return names[i].path < names[j].path })
for i, level := range names {
for _, path := range names[i+1:] {
if strings.HasPrefix(path.path, level.path+".") {
return fmt.Errorf("%s renders a level that {%s} reads a path into; name the fields you want instead", level.label, path.name)
}
}
}
return nil
}
// reader is one way a format reaches a bound field: as written, by its one
// spelling, and how to name it.
type reader struct{ name, path, label string }
// boundReaders lists every way a format reaches a bound field, in the order the
// format writes them. An operand renders its field, so it names a level exactly
// as a token does; one scan finds both, which is what puts them in one order.
func boundReaders(format string, bound map[string]string, refs map[string]refBinding) []reader {
var names []reader
_ = eachToken(format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if fn, _, isFunc := funcCall(t.body); isFunc {
for _, operand := range tokenOperands(t.body) {
a := splitArm(operand, refs)
if _, isBound := bound[a.key]; isBound {
names = append(names, reader{a.name, a.path, fmt.Sprintf("%s operand %q", fn, operand)})
}
}
return nil
}
for _, a := range splitArms(t.body, refs) {
if _, isBound := bound[a.key]; isBound {
names = append(names, reader{a.name, a.path, "token {" + a.name + "}"})
}
}
return nil
})
return names
}
// checkNoRepeatedRead rejects a bare token repeated on a held name: {w} {w} beside
// {uppercase(w)} would read one draw twice, where {w} {w} alone draws twice. The
// error names the single-token spelling.
func checkNoRepeatedRead(format string, c formatOps, refs map[string]refBinding) error {
count := map[string]int{}
return eachToken(format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if _, _, isFunc := funcCall(t.body); isFunc {
return nil
}
for _, a := range splitArms(t.body, refs) {
if len(a.tail) > 0 || !c.held[a.key] {
continue
}
if count[a.key]++; count[a.key] > 1 {
return fmt.Errorf("token {%s} is repeated, and %s holds %q to one draw per expansion; write {%s} once", a.name, c.holder[a.key], a.key, a.name)
}
}
return nil
})
}
// draws is what an expansion has already drawn for its held names: the variant each
// was drawn as, so every path under it reads one row, and the value each read, by
// its one spelling, so the same read written twice reads one value.
type draws struct {
variant map[string]node
value map[string]string
}
// readField renders one arm of a token. An arm's key is a sibling field or a
// reference linkRefs bound into fields. A name the expansion holds — a level some
// token addresses by dotted path, or a field an operand reads — is drawn once and
// kept, so {place.postal-code} and {place.locality} read one row, either read twice
// gives one value, and a shown operand is the operand computed. Every other name is
// drawn afresh, so {word} {word} still draws twice. checkTokens, checkPath and
// linkRefs prove every step, so the walk cannot fail.
func readField(s *session, t *template, held, refScope *draws, a arm) string {
if !t.held[a.key] {
if len(a.tail) > 0 {
panic(fmt.Sprintf("fejkdata: %q reads a path into %q, which the expansion does not hold", a.name, a.key))
}
return render(s, t.fields[a.key], refScope)
}
// A reference that reads a path reads the caller's scope, so its draw outlives
// this expansion; a sibling, and a reference read whole, stay local to it.
d := held
if isRef(a.key) && refScope != nil && len(a.tail) > 0 {
d = refScope
}
if v, read := d.value[a.path]; read {
return v
}
var v string
_ = walkPath(t.fields[a.key], a.tail, pathWalk{
// Hold the draw at every level passed through, so two paths sharing a
// prefix share it.
choice: func(c *choice, rest []string) ([]node, error) {
key := a.key
if consumed := len(a.tail) - len(rest); consumed > 0 {
key = a.steps[consumed-1]
}
n, drew := d.variant[key]
if !drew {
n = drawn(s, c)
d.variant[key] = n
}
return []node{n}, nil
},
leaf: func(n node) error { v = render(s, n, refScope); return nil },
})
d.value[a.path] = v
return v
}
// drawn resolves a choice to one variant, so a bound head is a concrete node the
// rest of the expansion shares. Nested choices unwrap too: a draw is one value, not
// another set to pick from.
func drawn(s *session, n node) node {
for c, ok := n.(*choice); ok; c, ok = n.(*choice) {
n = pick(s, c)
}
return n
}
inline.go
-88
View File
@@ -1,88 +0,0 @@
package fejkdata
import (
"encoding/json"
"fmt"
"strings"
)
// Template is an inline template compiled, referenced and validated against a
// generator's loaded data once, ready to render many times with [Template.Fake].
// It is safe for concurrent use: Fake serializes on its generator's lock, so a
// seeded sequence is reproducible only when a generator — and its templates — are
// drawn from one goroutine.
type Template struct {
g *Generator
n node
}
// Fake renders the template with one draw.
func (t *Template) Fake() string {
t.g.mu.Lock()
defer t.g.mu.Unlock()
return render(t.g.rand, t.n, nil)
}
// NewTemplate compiles an inline template — a format string or a JSON value — and
// binds its references against the loaded tree, so repeated renders pay the
// compile and validation once. It shares [New]'s guarantees: a bad template errors
// here, and rendering cannot fail.
func (f *Generator) NewTemplate(input string) (*Template, error) {
n, err := compileInput(input)
if err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
scope := inlineScope(n)
if err := linkNodeRefs(scope, f.categories); err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
if err := checkScope(scope); err != nil {
return nil, fmt.Errorf("fejkdata: %w", err)
}
return &Template{g: f, n: n}, nil
}
// FakeTemplate compiles and renders an inline template in one call. It is
// [NewTemplate] then [Template.Fake]; to render the same template many times, hold
// the *Template and call its Fake.
func (f *Generator) FakeTemplate(input string) (string, error) {
t, err := f.NewTemplate(input)
if err != nil {
return "", err
}
return t.Fake(), nil
}
// compileInput compiles an inline template: a JSON value, or a bare format string
// when the input is not JSON.
func compileInput(input string) (node, error) {
var raw any
if err := json.Unmarshal([]byte(input), &raw); err != nil {
return compile(input)
}
if trimmed := strings.TrimSpace(input); trimmed != input {
return nil, fmt.Errorf("a JSON template may not be padded with spaces, which a format string would render; write %s", trimmed)
}
return compile(raw)
}
// linkNodeRefs binds the references in an inline node's templates against the
// loaded tree.
func linkNodeRefs(scope nodeScope, root map[string]node) error {
return scope(func(path string, m node) error {
t, ok := m.(*template)
if !ok {
return nil
}
for _, name := range refTokens(t.format) {
sigil, rest, err := refShape(name)
if err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
}
if sigil != "/" {
return fmt.Errorf("%s: reference {%s}: an inline template has no folder; write {/%s}", path, name, rest)
}
}
return linkTemplateRefs(nil, path, t, root)
})
}
inline_test.go
-159
View File
@@ -1,159 +0,0 @@
package fejkdata
import (
"strings"
"testing"
)
func tmpl(t *testing.T, f *Generator, input string) string {
t.Helper()
s, err := f.FakeTemplate(input)
if err != nil {
t.Fatalf("FakeTemplate(%q): %v", input, err)
}
return s
}
func shipped(t *testing.T, opts ...Option) *Generator {
t.Helper()
f, err := New(append([]Option{WithSeed(1)}, opts...)...)
if err != nil {
t.Fatal(err)
}
return f
}
func TestFakeTemplateFormatString(t *testing.T) {
f := shipped(t)
got := tmpl(t, f, "name: {/sv_SE.person.last}")
if !strings.HasPrefix(got, "name: ") || strings.HasSuffix(got, " ") || strings.Contains(got, "{") {
t.Fatalf("FakeTemplate = %q, want a rendered last name after the prefix", got)
}
}
func TestFakeTemplateJSONObject(t *testing.T) {
f := shipped(t)
seen := map[string]bool{}
for i := 0; i < 50; i++ {
seen[tmpl(t, f, `{"format":"name: {x}","x":["bosse","lina"]}`)] = true
}
if !seen["name: bosse"] || !seen["name: lina"] || len(seen) != 2 {
t.Fatalf("JSON template produced %v, want both names", seen)
}
}
func TestFakeTemplateJSONArray(t *testing.T) {
f := shipped(t)
seen := map[string]bool{}
for i := 0; i < 50; i++ {
seen[tmpl(t, f, `["foo","bar","baz"]`)] = true
}
if len(seen) != 3 {
t.Fatalf("JSON array choice produced %v, want three items", seen)
}
}
func TestFakeTemplateCorrelatedReferences(t *testing.T) {
dir := writeData(t, map[string]string{
"person": `[{"format":"{first} {last}","first":"Ada","last":"Lovelace"},{"format":"{first} {last}","first":"Bo","last":"Ek"}]`,
})
f := newGenerator(t, dir, WithSeed(1))
for i := 0; i < 100; i++ {
got := tmpl(t, f, "{/person.first} {/person.last}")
if got != "Ada Lovelace" && got != "Bo Ek" {
t.Fatalf("correlated references = %q, want one person's first and last", got)
}
}
}
func TestFakeTemplateDeterministic(t *testing.T) {
a, b := shipped(t), shipped(t)
for i := 0; i < 20; i++ {
in := "row: {/misc.uuid} {digits(3)}"
if x, y := tmpl(t, a, in), tmpl(t, b, in); x != y {
t.Fatalf("same seed diverged: %q != %q", x, y)
}
}
}
func TestFieldlessTokenHint(t *testing.T) {
f := shipped(t)
_, err := f.FakeTemplate(`{sv_SE.person.last}`)
if err == nil || !strings.Contains(err.Error(), "write {/sv_SE.person.last}") {
t.Fatalf("FakeTemplate(bare token) = %v, want a hint naming {/sv_SE.person.last}", err)
}
_, err = f.FakeTemplate(`{"format":"{x}","repeat":2}`)
if err == nil || !strings.Contains(err.Error(), `this template has none — write {/x}`) {
t.Errorf("FakeTemplate(fieldless object) = %v, want the hint without calling it a bare string", err)
}
// A hint is only a drop-in where the name is the whole token: {/x} inside a
// transform or an alternation renders a different value, so none is offered.
for _, input := range []string{`{ /sv_SE.person.last }`, "{lowercase(x)}", "{x|y}"} {
_, err := f.FakeTemplate(input)
if err == nil || strings.Contains(err.Error(), "write {") {
t.Errorf("FakeTemplate(%q) = %v, want no hint naming a spelling that means something else", input, err)
}
}
}
func TestFakeTemplateErrors(t *testing.T) {
f := shipped(t)
for _, c := range []struct {
input string
want string
}{
{`{"x":"Q"}`, "missing string \"format\""},
{`"{x}"`, `no field "x"`},
{`"{digits(0)}"`, "must be positive"},
{`name: {/no.such.path}`, "no entry"},
{`name: {..nope}`, "write {/nope}"},
{`{"format":"x"}`, "is a string"},
{`{/misc.country} {/misc.country.alpha2}`, "renders a level"},
{`{"format":"{/misc.country.alpha2} {x}","x":"{/misc.country}"}`, "reads a path into"},
} {
_, err := f.FakeTemplate(c.input)
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("FakeTemplate(%q) = %v, want an error containing %q", c.input, err, c.want)
}
}
}
func TestFakeTemplateJSONString(t *testing.T) {
f := shipped(t)
got := tmpl(t, f, `"name: {/sv_SE.person.last}"`)
if !strings.HasPrefix(got, "name: ") || strings.Contains(got, "{") {
t.Fatalf("FakeTemplate(JSON string) = %q, want a rendered last name after the prefix", got)
}
}
func TestPaddedJSONIsRejected(t *testing.T) {
f := shipped(t)
in := `{"format":"{x}","x":["a","b"]}`
_, err := f.NewTemplate(" " + in + " ")
if err == nil || !strings.Contains(err.Error(), "write "+in) {
t.Fatalf("NewTemplate(padded JSON) = %v, want an error naming the unpadded spelling", err)
}
}
func TestNewTemplateReusable(t *testing.T) {
f := shipped(t)
reusable, err := f.NewTemplate(`{digits(2)}`)
if err != nil {
t.Fatalf("NewTemplate: %v", err)
}
seen := map[string]bool{}
for i := 0; i < 50; i++ {
seen[reusable.Fake()] = true
}
if len(seen) < 2 {
t.Fatalf("Template.Fake() repeated %v, want varied draws from one compile", seen)
}
}
func TestFakeTemplateRepeatBound(t *testing.T) {
f := shipped(t)
_, err := f.FakeTemplate(`{"format":"{x}","repeat":200,"x":{"format":"{y}","repeat":200,"y":{"format":"z","repeat":200}}}`)
if err == nil || !strings.Contains(err.Error(), "maximum") {
t.Errorf("nested repeat over the cap = %v, want it rejected naming the maximum", err)
}
}
loading_test.go
-214
View File
@@ -1,12 +1,9 @@
package fejkdata
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"regexp"
"slices"
"strings"
"testing"
@@ -204,214 +201,3 @@ func TestRepeatProductAlongAPathIsCapped(t *testing.T) {
t.Errorf("New = %v, want 1024 x 1024 along one path accepted", err)
}
}
func TestNewErrors(t *testing.T) {
// Pointing New at a file (not a directory) fails.
file := filepath.Join(t.TempDir(), "xx_XX")
if err := os.WriteFile(file, []byte("{}"), 0o644); err != nil {
t.Fatal(err)
}
if _, err := New(WithoutShippedData(), WithDataPath(file)); err == nil {
t.Error("New(file) = nil error, want not-a-directory error")
}
// Invalid JSON in a category file fails.
if _, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{"broken": `{ not json`}))); err == nil {
t.Error("New(invalid JSON) = nil error")
}
// An option that cannot take effect, and a category or folder no dot path can
// reach, are mistakes New must name rather than accept and ignore.
rejected := map[string]struct {
files map[string]string
want string
}{
"separator without repeat": {
map[string]string{"a": `{"format":"{x}","x":"1","separator":","}`},
"has no effect without a repeat above 1",
},
"separator with an explicit repeat of 1": {
map[string]string{"a": `{"format":"{x}","x":"1","separator":","}`},
"has no effect without a repeat above 1",
},
"weight outside a choice": {
map[string]string{"a": `{"format":"x","weight":5}`},
"weight only skews a choice's items",
},
"non-numeric weight outside a choice": {
map[string]string{"a": `{"format":"x","weight":"bad"}`},
"weight only skews a choice's items",
},
"weight used as a field": {
map[string]string{"a": `{"format":"{name} {weight}kg","name":"Anvil","weight":["7"]}`},
"can never be a field",
},
"an option name used as a token": {
map[string]string{"a": `"{weight}"`},
`"weight" is an option and can never be a field`,
},
"category name with a dot": {
map[string]string{"a.b": `"1"`},
`category "a.b" contains "."`,
},
"folder name with a dot": {
map[string]string{"a.b/cat": `"1"`},
`/a.b: folder "a.b" contains "."`,
},
// The token grammar reserves three more characters. A name carrying one
// still resolves by dot path, but no format can name it, so it is rejected
// where it is authored rather than at the token that cannot reach it.
"field name with a pipe": {
map[string]string{"a": `{"format":"{x}","x":"1","b|c":"2"}`},
`field "b|c" contains "|"`,
},
"field name with a paren": {
map[string]string{"a": `{"format":"{x}","x":"1","b(c":"2"}`},
`field "b(c" contains "("`,
},
"field name with a closing brace": {
map[string]string{"a": `{"format":"{x}","x":"1","b}c":"2"}`},
`field "b}c" contains "}"`,
},
"category name with a pipe": {
map[string]string{"a|b": `"1"`},
`category "a|b" contains "|"`,
},
"field name with a bracket": {
map[string]string{"a": `{"format":"{x}","x":"1","b[c":"2"}`},
`field "b[c" contains "["`,
},
"category name with a bracket": {
map[string]string{"[abc]": `"1"`},
`category "[abc]" contains "["`,
},
"field name with a quote": {
map[string]string{"a": `{"format":"{x}","x":"1","b\"c":"2"}`},
`field "b\"c" contains "\""`,
},
// An empty name is not a path segment, so List never offered it — while a
// bare {}, a trailing dot in Fake("a.") and a {/a.} reference all reached
// it. The engine accepted spellings it would never advertise.
"empty field name": {
map[string]string{"a": `{"format":"[{}]","":"VALUE"}`},
`field "" is empty`,
},
"folder name with a paren": {
map[string]string{"a(b/cat": `"1"`},
`folder "a(b" contains "("`,
},
// A repeated arm skews an alternation, which weight is the spelling for.
"repeated alternation arm": {
map[string]string{"a": `{"format":"{x|x}","x":"1"}`},
`arm "x" is repeated`,
},
"repeated arm among others": {
map[string]string{"a": `{"format":"{x|y|x}","x":"1","y":"2"}`},
`arm "x" is repeated`,
},
"repeated path arm": {
map[string]string{"a": `{"format":"{p.v|p.v}","p":{"format":"{v}","v":"1"}}`},
`arm "p.v" is repeated`,
},
// A reference arm is the only kind that reaches the repeat check by passing
// the per-arm checks rather than falling through them.
"repeated reference arm": {
map[string]string{"a": `"x"`, "b": `"{/a|/a}"`},
`arm "/a" is repeated`,
},
// An arm that is broken on its own terms is reported as that, not as a
// repeat: the repeat is a consequence of the real mistake.
"repeated arm with no path": {
map[string]string{"a": `"{/|/}"`},
"reference has no path",
},
// No field can be named "", so the token is told that rather than sent to
// name one — the fix "no field" points at is itself a load error.
"repeated empty arm": {
map[string]string{"a": `"{|}"`},
"a name is never empty",
},
"bare empty token": {
map[string]string{"a": `{"format":"[{}]","x":"1"}`},
"a name is never empty",
},
}
for name, c := range rejected {
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, c.files)))
if err == nil {
t.Errorf("%s: New = nil error, want it rejected at load", name)
continue
}
if !strings.Contains(err.Error(), c.want) {
t.Errorf("%s: New = %v, want it to mention %q", name, err, c.want)
}
}
// Data that works today must keep working, and stay reachable.
accepted := map[string]struct {
files map[string]string
path string
want string
}{
"option name as a field": {map[string]string{"a": `{"format":"{name} {Weight}kg","name":"Anvil","Weight":"7"}`}, "a", "Anvil 7kg"},
"format spelling as a field": {map[string]string{"a": `{"format":"{Format}","Format":"PDF"}`}, "a", "PDF"},
"hyphenated field": {map[string]string{"a": `{"format":"{x-y}","x-y":"1"}`}, "a.x-y", "1"},
"category named Format": {map[string]string{"Format": `"1"`}, "Format", "1"},
"folder named Repeat": {map[string]string{"Repeat/cat": `"1"`}, "Repeat.cat", "1"},
"field with a closing paren": {map[string]string{"a": `{"format":"{b)c}","b)c":"2"}`}, "a", "2"},
"repeat without a separator": {map[string]string{"a": `{"format":"{x}","repeat":3,"x":"1"}`}, "a", "111"},
// One name in two separate tokens is two independent draws, not a repeated
// arm; only a repeat within one alternation is rejected.
"one name in two tokens": {map[string]string{"a": `{"format":"{x}{x}","x":"1"}`}, "a", "11"},
}
for name, c := range accepted {
f, err := New(WithoutShippedData(), WithDataPath(writeData(t, c.files)))
if err != nil {
t.Errorf("%s: New = %v, want it accepted", name, err)
continue
}
if got, err := f.Fake(c.path); err != nil || got != c.want {
t.Errorf("%s: Fake(%q) = %q, %v, want %q", name, c.path, got, err, c.want)
}
}
}
func TestCategoryRootShapes(t *testing.T) {
dir := writeData(t, map[string]string{
"obj": `"{digits(2)}"`, // object root
"lit": `"hello"`, // bare-string root
})
f := newGenerator(t, dir, WithSeed(1))
if got := fake(t, f, "obj"); !regexp.MustCompile(`^\d\d$`).MatchString(got) {
t.Errorf("object-root category = %q, want two digits", got)
}
if got := fake(t, f, "lit"); got != "hello" {
t.Errorf("string-root category = %q, want hello", got)
}
}
func TestLongStringList(t *testing.T) {
const n = 2000
names := make([]string, n)
for i := range names {
names[i] = fmt.Sprintf("name-%04d", i)
}
list, err := json.Marshal(names)
if err != nil {
t.Fatal(err)
}
f := newGenerator(t, writeData(t, map[string]string{"name": string(list)}), WithSeed(1))
valid := map[string]bool{}
for _, v := range names {
valid[v] = true
}
seen := map[string]bool{}
for i := 0; i < 20000; i++ {
v := fake(t, f, "name")
if !valid[v] {
t.Fatalf("got %q, not in the list", v)
}
seen[v] = true
}
if len(seen) < n*8/10 {
t.Fatalf("only %d/%d distinct values seen; selection looks skewed", len(seen), n)
}
}
node.go
+77 -93
View File
@@ -41,11 +41,11 @@ type template struct {
fields map[string]node
repeat int
separator string
ops []op // format compiled once (see compileOps); what expand walks
grow int // minimum output size, to size the render buffer
fixed bool // no op varies, so every render is lit
lit string // the whole output when fixed
refs map[string]refBinding // each reference the format reads -> what it is bound to
ops []op // format compiled once (see compileOps); what expand walks
grow int // minimum output size, to size the render buffer
fixed bool // no op varies, so every render is lit
lit string // the whole output when fixed
refs map[string]string // each {..path} the format reads -> the head it is bound under
// bound maps each field the format addresses by dotted path to one path token
// reading it, which is the half of an overlap the fences name. nil when the
// format takes no path.
@@ -57,15 +57,6 @@ type template struct {
func (*template) isNode() {}
// field is the node a path segment names; a binding is a render edge, not a field.
func (t *template) field(seg string) (node, bool) {
if isRef(seg) {
return nil, false
}
n, ok := t.fields[seg]
return n, ok
}
// compile converts parsed JSON into a node tree, validating structure up front.
// Only a choice's items carry a weight, so one here would be inert whatever its type.
func compile(v any) (node, error) {
@@ -87,24 +78,10 @@ func compileItem(v any) (node, error) {
case map[string]any:
return compileTemplate(v)
default:
return nil, fmt.Errorf("a template value must be a string, a list or an object, not %s", jsonKind(v))
return nil, fmt.Errorf("unsupported node type %T", v)
}
}
// jsonKind names a JSON value a template cannot hold, in the data format's own
// terms rather than the decoding library's.
func jsonKind(v any) string {
switch v.(type) {
case float64:
return "a number"
case bool:
return "a boolean"
case nil:
return "null"
}
return fmt.Sprintf("%T", v)
}
func compileString(s string) (node, error) {
if err := checkTokens(s, nil); err != nil {
return nil, err
@@ -172,8 +149,8 @@ func compileChoice(items []any) (node, error) {
}
c.cum = cum
}
// Computed before linkRefs binds references into the items: a binding is keyed
// by a reference sigil, which paths skips, so the set is the same after.
// Safe to precompute: a choice's items come from one file, so no group can
// appear inside one, and neither mergeChildren nor linkRefs can reach in.
c.shared = sharedPaths(c.items)
return c, nil
}
@@ -204,75 +181,36 @@ func checkNoRepeatedItem(items []any) error {
}
func compileTemplate(m map[string]any) (node, error) {
o, err := readOptions(m)
if err != nil {
return nil, err
}
fields, err := compileFields(m)
if err != nil {
return nil, err
}
if len(fields) == 0 && o.repeat == 1 && !o.weighted {
return nil, fmt.Errorf("an object holding only a format is a string; write %q", o.format)
}
if err := checkTokens(o.format, fields); err != nil {
return nil, err
}
t := &template{format: o.format, fields: fields, repeat: o.repeat, separator: o.separator}
if err := t.compileFormat(); err != nil {
return nil, err
}
return t, nil
}
// templateOptions is what a template object's option keys say.
type templateOptions struct {
format string
repeat int
separator string
weighted bool
}
func readOptions(m map[string]any) (templateOptions, error) {
var o templateOptions
format, ok := m["format"].(string)
if !ok {
return o, fmt.Errorf("template object missing string \"format\"")
return nil, fmt.Errorf("template object missing string \"format\"")
}
o.format = format
repeat, err := repeatOf(m)
if err != nil {
return o, err
return nil, err
}
o.repeat = repeat
sep := ""
if sv, ok := m["separator"]; ok {
if o.separator, ok = sv.(string); !ok {
return o, fmt.Errorf("separator must be a string, got %T", sv)
if sep, ok = sv.(string); !ok {
return nil, fmt.Errorf("separator must be a string, got %T", sv)
}
if repeat == 1 {
return o, fmt.Errorf("separator joins repeated renders, so it has no effect without a repeat above 1")
}
if o.separator == "" {
return o, fmt.Errorf("separator \"\" is the default, so it has no effect; drop it")
return nil, fmt.Errorf("separator joins repeated renders, so it has no effect without a repeat above 1")
}
}
_, o.weighted = m["weight"]
return o, nil
}
// compileFields compiles every non-option key of a template object, in name order
// so which of several bad fields is reported does not vary.
func compileFields(m map[string]any) (map[string]node, error) {
fields := make(map[string]node, len(m))
t := &template{format: format, fields: make(map[string]node, len(m)), repeat: repeat, separator: sep}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
sort.Strings(keys) // so which of several bad fields is reported does not vary
for _, k := range keys {
if isOption(k) {
continue
}
if isRef(k) {
return nil, fmt.Errorf("field %q starts with %q, which is reserved for {..path} bindings", k, refPrefix)
}
if err := checkName(k); err != nil {
return nil, fmt.Errorf("field %w", err)
}
@@ -280,9 +218,55 @@ func compileFields(m map[string]any) (map[string]node, error) {
if err != nil {
return nil, fmt.Errorf("field %q: %w", k, err)
}
fields[k] = n
t.fields[k] = n
}
if _, weighted := m["weight"]; len(t.fields) == 0 && repeat == 1 && !weighted {
return nil, fmt.Errorf("an object holding only a format is a string; write %q", format)
}
if err := checkTokens(format, t.fields); err != nil {
return nil, err
}
if err := t.compileFormat(); err != nil {
return nil, err
}
return t, nil
}
// checkPath reports whether a token's dotted tail can address a node whichever way
// the draw goes, by the reachability rule descend applies — a choice must carry
// the whole remaining path in the set every variant shares — plus the
// rules a held draw adds, which descend has no need of: a level a path reads may
// not carry a repeat, and each variant answers for that itself. So a path that
// validates here resolves on every render, and a typo is a New-time error.
func checkPath(n node, tail []string, level string) error {
if len(tail) == 0 {
return nil
}
switch n := n.(type) {
case *template:
if n.repeat > 1 {
return fmt.Errorf("the level %q carries a repeat, which a path reading one draw of it cannot apply", level)
}
child, ok := n.fields[tail[0]]
if !ok {
return fmt.Errorf("no field %q", tail[0])
}
return checkPath(child, tail[1:], level+"."+tail[0])
case *choice:
if want := strings.Join(tail, "."); !n.shared[want] {
return unreachableInChoice(n, want)
}
// Reachability is settled; each variant still answers for itself, so a
// rule about the level (its repeat) holds behind a choice as in front.
for _, item := range n.items {
if err := checkPath(item, tail, level); err != nil {
return err
}
}
return nil
default:
return fmt.Errorf("cannot descend into %T at %q", n, tail[0])
}
return fields, nil
}
// repeatOf reads a template's "repeat" (default 1): how many times its format
@@ -302,8 +286,8 @@ func repeatOf(m map[string]any) (int, error) {
if r == 1 {
return 0, fmt.Errorf("repeat 1 is the default, so it has no effect; drop it")
}
if r > MaxRepeat { // caps the renders one repeat asks for; repeatCheck bounds what nested ones multiply to
return 0, fmt.Errorf("repeat %v exceeds the maximum %d", rv, MaxRepeat)
if r > maxLen { // cap so a fat-fingered repeat can't build a multi-GB string
return 0, fmt.Errorf("repeat %v exceeds the maximum %d", rv, maxLen)
}
return int(r), nil
}
@@ -337,23 +321,23 @@ func weightOf(raw any) (float64, error) {
// reservedInName is what a category, folder or field name may not contain: a dot
// separates the segments of a path, '|' the arms of a token, '(' opens a function
// call, braces delimit the token, '/' starts a reference, and brackets and a quote
// open a JSON value. A name carrying one is rejected where it is authored rather
// than where it would be unreachable.
const reservedInName = ".|({}/[]\""
// call and braces delimit the token. A name carrying one is reachable by no format,
// so it is rejected where it is authored rather than at the token that cannot
// reach it.
const reservedInName = ".|({}"
// reservedList spells reservedInName for an error message, so the two cannot drift.
var reservedList = strings.Join(strings.Split(reservedInName, ""), " ")
// checkName rejects a name the dot path, {token} and JSON grammars cannot spell.
// Both a category or folder and a field go through it, so there is one answer to
// what a name may contain.
// checkName rejects a name the dot path and {token} grammars cannot spell. Both a
// category or folder and a field go through it, so there is one answer to what a
// name may contain.
func checkName(name string) error {
if name == "" {
return fmt.Errorf("%q is empty, which is not a path segment, so List never offers it", name)
}
if i := strings.IndexAny(name, reservedInName); i >= 0 {
return fmt.Errorf("%q contains %q; a name may not use %s, which the dot path, {token} and JSON grammars reserve",
return fmt.Errorf("%q contains %q; a name may not use %s, which the dot path and {token} grammars reserve",
name, name[i:i+1], reservedList)
}
return nil
one_spelling_test.go
+5 -16
View File
@@ -30,12 +30,11 @@ func TestRepeatedChoiceItemIsRejected(t *testing.T) {
func TestInertObjectIsRejected(t *testing.T) {
for src, want := range map[string]string{
`{"format":"Malmö"}`: `write "Malmö"`,
`{"format":"{digits(3)}"}`: `write "{digits(3)}"`,
`[{"format":"a","weight":1},"b"]`: "weight 1",
`{"format":"{x}","x":"v","repeat":1}`: "repeat 1",
`{"format":"{x}","x":"v","separator":","}`: "separator",
`{"format":"{x}","x":"v","repeat":2,"separator":""}`: "default",
`{"format":"Malmö"}`: `write "Malmö"`,
`{"format":"{digits(3)}"}`: `write "{digits(3)}"`,
`[{"format":"a","weight":1},"b"]`: "weight 1",
`{"format":"{x}","x":"v","repeat":1}`: "repeat 1",
`{"format":"{x}","x":"v","separator":","}`: "separator",
} {
if _, err := compile(parse(t, src)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("compile(%s) = %v, want an error mentioning %s", src, err, want)
@@ -47,13 +46,3 @@ func TestInertObjectIsRejected(t *testing.T) {
}
}
}
func TestInlineFolderSigilsAreRejected(t *testing.T) {
f := shipped(t)
for _, input := range []string{"{.sv_SE.person.last}", "{..sv_SE.person.last}"} {
_, err := f.NewTemplate(input)
if err == nil || !strings.Contains(err.Error(), "write {/sv_SE.person.last}") {
t.Errorf("NewTemplate(%q) = %v, want an error naming the root spelling", input, err)
}
}
}
path.go
-110
View File
@@ -1,110 +0,0 @@
package fejkdata
import (
"fmt"
"sort"
"strings"
)
// pathWalk is what one walk of a dotted path does at each kind of level: choice
// returns the variants to continue into (none stops the walk); level runs at each
// template a segment descends into; leaf runs where the tail ends. A nil action
// is skipped.
type pathWalk struct {
choice func(c *choice, rest []string) ([]node, error)
level func(t *template, rest []string) error
leaf func(n node) error
}
// walkPath descends tail from n: a group or template by its next segment, a
// choice by w.choice, which consumes no segment. A missing segment is an error,
// so no walk reaches past what the data holds. A table-shaped dispatch, one case
// per node kind, kept whole on purpose.
func walkPath(n node, tail []string, w pathWalk) error {
if len(tail) == 0 {
if w.leaf != nil {
return w.leaf(n)
}
return nil
}
switch n := n.(type) {
case *group:
child, ok := n.children[tail[0]]
if !ok {
return fmt.Errorf("no entry %q", tail[0])
}
return walkPath(child, tail[1:], w)
case *template:
if w.level != nil {
if err := w.level(n, tail); err != nil {
return err
}
}
child, ok := n.field(tail[0])
if !ok {
return fmt.Errorf("no field %q", tail[0])
}
return walkPath(child, tail[1:], w)
case *choice:
if w.choice == nil {
return nil
}
next, err := w.choice(n, tail)
if err != nil {
return err
}
for _, item := range next {
if err := walkPath(item, tail, w); err != nil {
return err
}
}
return nil
}
return fmt.Errorf("cannot descend into %T at %q", n, tail[0])
}
// carriedByAll is the choice rule a path that must resolve on every call obeys:
// the rest of the tail must be one every variant carries.
func carriedByAll(c *choice, rest []string) error {
if want := strings.Join(rest, "."); !c.shared[want] {
return unreachableInChoice(c, want)
}
return nil
}
// unreachableInChoice reports that a path cannot step through this choice, listing
// what every variant does carry. It reads the precomputed set, so a failing path
// costs no more than a rendering one.
func unreachableInChoice(c *choice, want string) error {
if len(c.shared) == 0 {
return fmt.Errorf("no variant of this %d-way choice carries %q", len(c.items), want)
}
offered := make([]string, 0, len(c.shared))
for p := range c.shared {
offered = append(offered, p)
}
sort.Strings(offered)
return fmt.Errorf("not every variant of this %d-way choice carries %q; all carry %v", len(c.items), want, offered)
}
// checkPath proves a dotted tail resolves whichever way the draws go — a choice
// must carry the rest of the path in the set every variant shares — and that no
// level a path reads carries a repeat, which one draw of it could not apply. So a
// path that validates here resolves on every render, and a typo is a New-time
// error.
func checkPath(n node, tail []string, level string) error {
return walkPath(n, tail, pathWalk{
choice: func(c *choice, rest []string) ([]node, error) {
if err := carriedByAll(c, rest); err != nil {
return nil, err
}
return c.items, nil
},
level: func(t *template, rest []string) error {
if t.repeat > 1 {
return fmt.Errorf("the level %q carries a repeat, which a path reading one draw of it cannot apply", join(level, strings.Join(tail[:len(tail)-len(rest)], ".")))
}
return nil
},
})
}
path_test.go
-152
View File
@@ -1,152 +0,0 @@
package fejkdata
import (
"slices"
"strings"
"testing"
)
func TestWalkPathStopsAtAMissingSegment(t *testing.T) {
n := compiled(t, `{"format":"{a}","a":{"format":"{b}","b":"leaf"}}`)
var seen []string
walk := pathWalk{
level: func(tm *template, rest []string) error { seen = append(seen, "level:"+rest[0]); return nil },
leaf: func(n node) error { seen = append(seen, "leaf"); return nil },
}
if err := walkPath(n, []string{"a", "b"}, walk); err != nil {
t.Fatalf("walkPath(a.b) = %v", err)
}
if want := []string{"level:a", "level:b", "leaf"}; !slices.Equal(seen, want) {
t.Errorf("walk visited %v, want %v", seen, want)
}
seen = nil
err := walkPath(n, []string{"a", "nope", "deeper"}, walk)
if err == nil || !strings.Contains(err.Error(), `no field "nope"`) {
t.Errorf("walkPath(a.nope.deeper) = %v, want the missing segment named", err)
}
if slices.Contains(seen, "leaf") {
t.Errorf("walk reached a leaf past a missing segment: %v", seen)
}
}
func TestWalkPathChoiceConsumesNoSegment(t *testing.T) {
n := compiled(t, `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2"}]`)
var leaves []node
err := walkPath(n, []string{"f"}, pathWalk{
choice: func(c *choice, rest []string) ([]node, error) {
if len(rest) != 1 || rest[0] != "f" {
t.Errorf("choice saw rest %v, want [f]", rest)
}
return c.items, nil
},
leaf: func(n node) error { leaves = append(leaves, n); return nil },
})
if err != nil || len(leaves) != 2 {
t.Fatalf("walkPath through a choice = %v, %d leaves, want both variants' f", err, len(leaves))
}
}
func TestDeepDottedPath(t *testing.T) {
// A 5-segment path descends through alternating object/array nodes; choices
// on the path are single-variant, so it resolves deterministically.
f := engine(1)
f.categories = map[string]node{
"deep": compiled(t, `{"format":"{a}","a":{"format":"{b}","b":{"format":"{c}","c":{"format":"{d}","d":"leaf"}}}}`),
}
if got, err := f.Fake("deep.a.b.c.d"); err != nil || got != "leaf" {
t.Fatalf("Fake(deep.a.b.c.d) = %q, %v, want leaf", got, err)
}
// Rendering the whole tree resolves the same chain.
if got, err := f.Fake("deep"); err != nil || got != "leaf" {
t.Fatalf("Fake(deep) = %q, %v, want leaf", got, err)
}
}
func TestDescendIntoStringErrors(t *testing.T) {
f := engine(1)
f.categories = map[string]node{"greeting": compiled(t, `"hej"`)}
if _, err := f.Fake("greeting.extra"); err == nil || !strings.Contains(err.Error(), `no field "extra"`) {
t.Fatalf("Fake(greeting.extra) = %v, want a no-field error", err)
}
}
// TestPathThroughChoice pins the rule that keeps a dotted path from rendering on
// one call and failing on the next: every variant must carry the rest of the path.
func TestPathThroughChoice(t *testing.T) {
dir := writeData(t, map[string]string{
"every": `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2"}]`,
"notall": `[{"format":"{f}","f":"1"},"plain"]`,
"some": `[{"format":"{f}","f":"1"},{"format":"{f}","f":"2","extra":"x"}]`,
})
f := newGenerator(t, dir, WithSeed(1))
for i := 0; i < 200; i++ {
if got := fake(t, f, "every.f"); got != "1" && got != "2" {
t.Fatalf("every.f = %q, want 1 or 2", got)
}
}
// A path only some variants carry is reported against what all of them carry.
if _, err := f.Fake("some.extra"); err == nil || !strings.Contains(err.Error(), "all carry [f]") {
t.Errorf("Fake(some.extra) = %v, want it to name what every variant carries", err)
}
var first string
for i := 0; i < 200; i++ {
_, err := f.Fake("notall.f")
if err == nil {
t.Fatal("notall.f = nil error, want the same failure every call")
}
if i == 0 {
first = err.Error()
} else if err.Error() != first {
t.Fatalf("notall.f error varies between calls:\n %s\n %s", first, err.Error())
}
}
}
// TestPathKeyIsUnambiguous pins that the one shape which could collide cannot be
// written: a field literally named "a.b" and a field "a" holding "b" would both
// spell "a.b", so a dotted field name is rejected at New and a dot means a path
// wherever it appears.
func TestPathKeyIsUnambiguous(t *testing.T) {
dir := writeData(t, map[string]string{
"cat": `[{"format":"{a.b}","a.b":"1"},{"format":"{a}","a":{"format":"{b}","b":"2"}}]`,
})
_, err := New(WithoutShippedData(), WithDataPath(dir))
if err == nil || !strings.Contains(err.Error(), `field "a.b" contains "."`) {
t.Fatalf("New = %v, want the dotted field name rejected", err)
}
// The same data without the dotted key is fine, and the path resolves.
f := newGenerator(t, writeData(t, map[string]string{
"cat": `{"format":"{a.b}","a":{"format":"{b}","b":"2"}}`,
}), WithSeed(1))
if !slices.Contains(f.List(), "cat.a.b") {
t.Error("List() omits cat.a.b, which the data carries")
}
if got := fake(t, f, "cat"); got != "2" {
t.Fatalf("cat = %q, want 2", got)
}
}
// TestMissingFieldNamesItself keeps the precise diagnosis for the ordinary typo: a
// single-variant choice always picks the same item, so it needs no every-variant
// guard and the error can name the field that is missing.
func TestMissingFieldNamesItself(t *testing.T) {
f := newGenerator(t, "data/sv_SE", WithSeed(1))
_, err := f.Fake("person.typo")
if err == nil || !strings.Contains(err.Error(), `no field "typo"`) {
t.Errorf("Fake(person.typo) = %v, want it to name the missing field", err)
}
}
func TestBindingKeyIsNotAPathSegment(t *testing.T) {
dir := writeData(t, map[string]string{
"color": `["red","blue"]`,
"name": `{"format":"{/color} {w}","w":"x"}`,
})
f := newGenerator(t, dir, WithSeed(1))
if slices.Contains(f.List(), "name./color") {
t.Error("List() advertises a binding key")
}
if _, err := f.Fake("name./color"); err == nil || !strings.Contains(err.Error(), `no field "/color"`) {
t.Errorf("Fake(name./color) = %v, want a no-field error", err)
}
}
perf_test.go
-81
View File
@@ -1,81 +0,0 @@
package fejkdata
import (
"fmt"
"strings"
"testing"
"testing/fstest"
)
// One Fake call descends depth levels: each level's "a" is the next template down.
func nestedJSON(depth int) string {
s := `"leaf"`
for i := 0; i < depth; i++ {
s = fmt.Sprintf(`{"format":"{a}","a":%s}`, s)
}
return s
}
// One Fake call expands n sibling tokens.
func wideTokenJSON(n int) string {
var toks, fields strings.Builder
for i := 0; i < n; i++ {
fmt.Fprintf(&toks, "{f%d}", i)
if i > 0 {
fields.WriteByte(',')
}
fmt.Fprintf(&fields, `"f%d":"x"`, i)
}
return fmt.Sprintf(`{"format":"%s",%s}`, toks.String(), fields.String())
}
func TestNoRenderAllocRegression(t *testing.T) {
shapes := []struct {
name string
json string
base float64
}{
{"nested depth 25", nestedJSON(25), 27},
{"nested depth 100", nestedJSON(100), 102},
{"wide 500 tokens", wideTokenJSON(500), 9},
}
for _, s := range shapes {
f, err := New(WithoutShippedData(), WithDataFS(fstest.MapFS{"x.json": {Data: []byte(s.json)}}))
if err != nil {
t.Fatalf("New(%s): %v", s.name, err)
}
allocs := testing.AllocsPerRun(10000, func() { f.Fake("x") })
if allocs > s.base*1.10 {
t.Errorf("%s: %.1f allocs/op regressed past %.1f (baseline %.1f + 10%%); bump the baseline only as a deliberate change", s.name, allocs, s.base*1.10, s.base)
}
}
}
// A record's fences read the compiled tree, so they belong to New, not to a draw.
func TestNoRecordAllocRegression(t *testing.T) {
for _, s := range []struct{ name, json string }{
{"record 3 columns", `{"format":"","a":"x","b":"y","c":"z"}`},
{"record 50 columns", wideTokenJSON(50)},
} {
f, err := New(WithoutShippedData(), WithDataFS(fstest.MapFS{"x.json": {Data: []byte(s.json)}}))
if err != nil {
t.Fatalf("New(%s): %v", s.name, err)
}
if _, err := f.Record("x"); err != nil {
t.Fatalf("Record(%s): %v", s.name, err) // else the gate would measure the error path
}
const base = 4.0
if allocs := testing.AllocsPerRun(10000, func() { f.Record("x") }); allocs > base*1.10 {
t.Errorf("%s: %.1f allocs/op regressed past %.1f (baseline %.1f + 10%%); a record fence running per draw is the usual cause", s.name, allocs, base*1.10, base)
}
}
}
func BenchmarkNestedDepth25(b *testing.B) { benchPath(b, tmpData(b, "deep", nestedJSON(25)), "deep") }
func BenchmarkNestedDepth100(b *testing.B) { benchPath(b, tmpData(b, "deep", nestedJSON(100)), "deep") }
func BenchmarkWideTokens100(b *testing.B) {
benchPath(b, tmpData(b, "wide", wideTokenJSON(100)), "wide")
}
func BenchmarkWideTokens500(b *testing.B) {
benchPath(b, tmpData(b, "wide", wideTokenJSON(500)), "wide")
}
readme_test.go
-28
View File
@@ -43,34 +43,6 @@ func TestReadmeExamplesLoadAndRender(t *testing.T) {
}
}
func TestReadmeRecordExample(t *testing.T) {
src := readme(t)
section := src[strings.Index(src, "### Records"):]
section = section[:strings.Index(section, "## Library")]
block := jsonBlock.FindStringSubmatch(section)
if block == nil {
t.Fatal("README lost the Records example")
}
f, err := New(WithDataPath(writeData(t, map[string]string{"users": block[1]})), WithSeed(1))
if err != nil {
t.Fatal(err)
}
r, err := f.Record("users")
if err != nil {
t.Fatal(err)
}
got := r.Columns()
if len(got) != 2 || got[0].Name != "first" || got[0].Value != "Bo" || got[1].Name != "last" || got[1].Value != "Lovelace" {
t.Fatalf("record columns = %v, want first=Bo, last=Lovelace with seed 1", got)
}
if r.CSVHeader() != "first,last" || r.CSVLine() != "Bo,Lovelace" {
t.Fatalf("csv = %q, %q, want first,last / Bo,Lovelace", r.CSVHeader(), r.CSVLine())
}
if r.SQLInsert("users") != `INSERT INTO "users" ("first", "last") VALUES ('Bo', 'Lovelace');` {
t.Fatalf("sql = %q, want the seeded INSERT", r.SQLInsert("users"))
}
}
func TestReadmeSQLExampleOutput(t *testing.T) {
src := readme(t)
src = src[strings.Index(src, "### Your own data"):]
record.go
-285
View File
@@ -1,285 +0,0 @@
package fejkdata
import (
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
)
// Column is one rendered column of a record.
type Column struct {
Name string
Value string
}
// Record is one record rendered from a template: every direct field is a column,
// listed in name order. Each column is its own expansion, so a sibling field is
// local to it, while a reference that reads a path is drawn once for the whole
// record.
type Record struct {
columns []Column
}
// Columns returns the record's columns in name order.
func (r *Record) Columns() []Column {
return append([]Column(nil), r.columns...)
}
// JSON renders the record as one JSON object, every column a string.
func (r *Record) JSON() string {
m := make(map[string]string, len(r.columns))
for _, c := range r.columns {
m[c.Name] = c.Value
}
b, _ := json.Marshal(m)
return string(b)
}
// CSVHeader renders the column names as one CSV header line.
func (r *Record) CSVHeader() string {
return csvLine(r.names())
}
// CSVLine renders the column values as one CSV row.
func (r *Record) CSVLine() string {
return csvLine(r.values())
}
func (r *Record) names() []string {
out := make([]string, len(r.columns))
for i, c := range r.columns {
out[i] = c.Name
}
return out
}
func (r *Record) values() []string {
out := make([]string, len(r.columns))
for i, c := range r.columns {
out[i] = c.Value
}
return out
}
func csvLine(cols []string) string {
var b strings.Builder
w := csv.NewWriter(&b)
_ = w.Write(cols)
w.Flush()
line := strings.TrimSuffix(b.String(), "\n")
if line == "" {
return `""` // a blank line is a row every CSV reader drops
}
return line
}
// SQLInsert renders the record as one INSERT statement into table: identifiers in
// ANSI double quotes, every value a single-quoted string literal.
func (r *Record) SQLInsert(table string) string {
cols := make([]string, len(r.columns))
vals := make([]string, len(r.columns))
for i, c := range r.columns {
cols[i] = quoteIdent(c.Name)
vals[i] = "'" + strings.ReplaceAll(c.Value, "'", "''") + "'"
}
return fmt.Sprintf("INSERT INTO %s (%s) VALUES (%s);", quoteIdent(table), strings.Join(cols, ", "), strings.Join(vals, ", "))
}
func quoteIdent(s string) string {
return `"` + strings.ReplaceAll(s, `"`, `""`) + `"`
}
// Record renders a path as one record: the template it names, with each direct
// field drawn as a column. Only a category-level template is a record — a path
// that descends into a field, or that names a folder or a choice, is an error.
func (f *Generator) Record(path string) (*Record, error) {
f.mu.Lock()
defer f.mu.Unlock()
_, n, tail, err := resolveCategory(f.categories, strings.Split(path, "."))
if err != nil {
return nil, fmt.Errorf("fejkdata: %s: %w", path, err)
}
if len(tail) > 0 {
return nil, fmt.Errorf("fejkdata: %s descends into %q, a field; only a category-level template is a record", path, tail[0])
}
shape := f.recordShapeOf(n)
if shape.err != nil {
return nil, fmt.Errorf("fejkdata: %s %w", path, shape.err)
}
return renderRecord(f.rand, shape.t, shape.columns), nil
}
// recordShape is what recordOf settled about a node: the template to project, its
// columns, or why it is not a record.
type recordShape struct {
t *template
columns []string
err error
}
// recordShapeOf fences a node once and remembers the answer. Callers hold the
// generator's lock.
func (f *Generator) recordShapeOf(n node) recordShape {
if shape, done := f.records[n]; done {
return shape
}
t, columns, err := recordOf(n)
shape := recordShape{t: t, columns: columns, err: err}
if f.records == nil {
f.records = map[node]recordShape{}
}
f.records[n] = shape
return shape
}
// RecordTemplate is an inline record compiled, referenced and validated once,
// ready to render many times with [RecordTemplate.Fake].
type RecordTemplate struct {
g *Generator
t *template
columns []string
}
// Fake renders the record with one draw.
func (t *RecordTemplate) Fake() *Record {
t.g.mu.Lock()
defer t.g.mu.Unlock()
return renderRecord(t.g.rand, t.t, t.columns)
}
// NewRecordTemplate compiles an inline record — a JSON object with a format and
// fields — and binds its references against the loaded tree.
func (f *Generator) NewRecordTemplate(input string) (*RecordTemplate, error) {
t, err := f.NewTemplate(input)
if err != nil {
return nil, err
}
tm, columns, err := recordOf(t.n)
if err != nil {
return nil, fmt.Errorf("fejkdata: an inline record %w", err)
}
return &RecordTemplate{g: f, t: tm, columns: columns}, nil
}
// FakeRecord compiles and renders an inline record in one call.
func (f *Generator) FakeRecord(input string) (*Record, error) {
t, err := f.NewRecordTemplate(input)
if err != nil {
return nil, err
}
return t.Fake(), nil
}
// recordOf is the fence both record entry points pass. The columns come back with
// the template, fixed for every draw the caller goes on to make.
func recordOf(n node) (*template, []string, error) {
t, ok := n.(*template)
if !ok {
return nil, nil, errors.New("names a choice, not a template; a record is a template whose fields are its columns")
}
if t.repeat != 1 {
return nil, nil, fmt.Errorf("carries repeat %d, which composes its format into one string; a record projects columns instead — drop the repeat and render the record again for more rows", t.repeat)
}
columns := recordColumns(t)
if len(columns) == 0 {
return nil, nil, errors.New("has no fields, so no columns")
}
if err := checkColumnRefs(t, columns); err != nil {
return nil, nil, err
}
return t, columns, nil
}
// checkColumnRefs rejects the reference reads a record's shared draw cannot answer
// for: one column rendering a level another reads a path into, and a column
// reading the record back through its own path.
func checkColumnRefs(t *template, columns []string) error {
reads, err := columnRefs(t, columns)
if err != nil {
return err
}
sort.Slice(reads, func(i, j int) bool {
if reads[i].a.path != reads[j].a.path {
return reads[i].a.path < reads[j].a.path
}
return reads[i].column < reads[j].column
})
for i, level := range reads {
for _, into := range reads[i+1:] {
if strings.HasPrefix(into.a.path, level.a.path+".") {
return fmt.Errorf("column %q renders {%s}, a level column %q reads a path into with {%s}; name the fields you want instead",
level.column, level.a.name, into.column, into.a.name)
}
}
}
return nil
}
// columnRef is one held reference read, and the column whose render reaches it.
type columnRef struct {
column string
a arm
}
// columnRefs lists every reference that reads a path, anywhere a column renders,
// following the same edges expand does. A reference landing back on the record
// itself is reported rather than collected, whether it reads a path or the record
// whole: either way the column describes a draw other than its neighbours'.
func columnRefs(t *template, columns []string) ([]columnRef, error) {
var out []columnRef
var err error
for _, name := range columns {
seen := map[node]bool{}
var walk func(n node)
walk = func(n node) {
if n == nil || seen[n] || err != nil {
return
}
seen[n] = true
if tm, ok := n.(*template); ok {
for _, ref := range refTokens(tm.format) {
a := splitArm(ref, tm.refs)
if tm.fields[a.key] == node(t) {
err = fmt.Errorf("column %q reads {%s}, which points back at this record; a column cannot read another column — move the shared value into its own category and reference that", name, a.name)
return
}
if len(a.tail) > 0 {
out = append(out, columnRef{name, a})
}
}
}
for _, e := range renderEdges(n) {
walk(e.to)
}
}
walk(t.fields[name])
}
return out, err
}
// renderRecord draws each column once, in the name order recordOf fixed, over one
// reference scope shared across them.
func renderRecord(s *session, t *template, columns []string) *Record {
scope := &draws{variant: map[string]node{}, value: map[string]string{}}
r := &Record{columns: make([]Column, len(columns))}
for i, name := range columns {
r.columns[i] = Column{Name: name, Value: render(s, t.fields[name], scope)}
}
return r
}
// recordColumns is the sorted non-reference field names — the columns a record
// projects. A {/path} binding is carried in fields under its root path, so only a
// name that is not a reference is a column.
func recordColumns(t *template) []string {
var names []string
for _, name := range sortedNames(t.fields) {
if !isRef(name) {
names = append(names, name)
}
}
return names
}
record_test.go
-408
View File
@@ -1,408 +0,0 @@
package fejkdata
import (
"encoding/csv"
"encoding/json"
"strings"
"testing"
)
func recordCat(t *testing.T) *Generator {
t.Helper()
dir := writeData(t, map[string]string{
"users": `{
"format": "{first} {last}",
"first": ["Ada", "Bo"],
"last": ["Lovelace", "Ek"]
}`,
})
return newGenerator(t, dir, WithSeed(1))
}
func TestRecordProjectsFieldsAsColumns(t *testing.T) {
f := recordCat(t)
r, err := f.Record("users")
if err != nil {
t.Fatal(err)
}
got := r.Columns()
if len(got) != 2 || got[0].Name != "first" || got[1].Name != "last" {
t.Fatalf("Record.Columns() = %v, want columns first, last in name order", got)
}
if (got[0].Value != "Ada" && got[0].Value != "Bo") || (got[1].Value != "Lovelace" && got[1].Value != "Ek") {
t.Fatalf("columns = %v, want the field values", got)
}
}
func TestRecordIsDeterministic(t *testing.T) {
a, b := recordCat(t), recordCat(t)
for i := 0; i < 20; i++ {
x, _ := a.Record("users")
y, _ := b.Record("users")
if x.JSON() != y.JSON() {
t.Fatalf("same seed diverged: %s != %s", x.JSON(), y.JSON())
}
}
}
func TestRecordSkipsReferenceBindings(t *testing.T) {
dir := writeData(t, map[string]string{
"name": `{"format": "{first} {last}", "first": "Ada", "last": "Lovelace"}`,
"user": `{"format": "they are {/name}", "id": "1"}`,
})
f := newGenerator(t, dir, WithSeed(1))
r, err := f.Record("user")
if err != nil {
t.Fatal(err)
}
got := r.Columns()
if len(got) != 1 || got[0].Name != "id" {
t.Fatalf("Record.Columns() = %v, want only the id column (a {/path} is not a column)", got)
}
}
func TestRecordErrors(t *testing.T) {
dir := writeData(t, map[string]string{
"bare": `"hello"`,
"pick": `["a", "b"]`,
"group/cat": `"x"`,
})
f := newGenerator(t, dir, WithSeed(1))
for _, c := range []struct {
path string
want string
}{
{"bare", "has no fields, so no columns"},
{"pick", "names a choice"},
{"group", "names a folder"},
{"nope", "no entry"},
} {
if _, err := f.Record(c.path); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("Record(%q) = %v, want an error containing %q", c.path, err, c.want)
}
}
}
func TestRecordJSON(t *testing.T) {
f := recordCat(t)
r, err := f.Record("users")
if err != nil {
t.Fatal(err)
}
var m map[string]string
if err := json.Unmarshal([]byte(r.JSON()), &m); err != nil {
t.Fatalf("Record.JSON() is not valid JSON: %v\n%s", err, r.JSON())
}
if len(m) != 2 || (m["first"] != "Ada" && m["first"] != "Bo") {
t.Fatalf("Record.JSON() = %s, want two addressable columns", r.JSON())
}
for _, f := range r.Columns() {
if m[f.Name] != f.Value {
t.Fatalf("JSON column %q = %q, want %q", f.Name, m[f.Name], f.Value)
}
}
}
func TestRecordCSV(t *testing.T) {
dir := writeData(t, map[string]string{
"note": `{"format": "{word}, {word}", "word": ["a", "b,c", "d\"e", "f\n"]}`,
})
f := newGenerator(t, dir, WithSeed(1))
seen := map[string]bool{}
for i := 0; i < 200 && len(seen) < 4; i++ {
r, err := f.Record("note")
if err != nil {
t.Fatal(err)
}
if got := r.CSVHeader(); got != "word" {
t.Fatalf("CSVHeader() = %q, want word", got)
}
rec, err := csv.NewReader(strings.NewReader(r.CSVLine() + "\n")).Read()
if err != nil {
t.Fatalf("CSVLine() is not valid CSV: %v\n%q", err, r.CSVLine())
}
want := r.Columns()[0].Value
if len(rec) != 1 || rec[0] != want {
t.Fatalf("CSVLine() = %v, want the field value %q round-tripped", rec, want)
}
seen[want] = true
}
if len(seen) != 4 {
t.Fatalf("round-tripped %d of the 4 values; the comma, quote and newline shapes must each survive", len(seen))
}
}
func TestRecordCSVEmptyValueStaysARow(t *testing.T) {
dir := writeData(t, map[string]string{"blank": `{"format": "", "note": ""}`})
f := newGenerator(t, dir, WithSeed(1))
r, err := f.Record("blank")
if err != nil {
t.Fatal(err)
}
rows, err := csv.NewReader(strings.NewReader(r.CSVHeader() + "\n" + r.CSVLine() + "\n")).ReadAll()
if err != nil {
t.Fatalf("csv: %v", err)
}
if len(rows) != 2 || len(rows[1]) != 1 || rows[1][0] != "" {
t.Fatalf("one empty column parsed to %v, want a header and one row of one empty field", rows)
}
}
func TestRecordRejectsOverlappingReferenceColumns(t *testing.T) {
cat := `{"format":"","a":[{"format":"A={b}","b":"1"},{"format":"A={b}","b":"2"}]}`
for _, c := range []struct{ name, row string }{
{"sibling columns", `{"format":"","whole":"{/cat.a}","inner":"{/cat.a.b}"}`},
{"through a nested template", `{"format":"","whole":"{/cat.a}","inner":{"format":"{/cat.a.b} {x}","x":"1"}}`},
{"through a column repeat", `{"format":"","whole":"{/cat.a}","inner":{"format":"{/cat.a.b}","repeat":2,"separator":"-"}}`},
{"through a choice variant", `{"format":"","whole":"{/cat.a}","inner":[{"format":"{/cat.a.b} {x}","x":"1"},{"format":"{/cat.a.b}! {x}","x":"2"}]}`},
{"as a builtin operand", `{"format":"","whole":"{uppercase(/cat.a)}","inner":"{/cat.a.b}"}`},
} {
f := newGenerator(t, writeData(t, map[string]string{"cat": cat, "row": c.row}), WithSeed(1))
_, err := f.Record("row")
if err == nil || !strings.Contains(err.Error(), "reads a path into") {
t.Errorf("%s: Record = %v, want the overlap rejected the way one format is", c.name, err)
continue
}
if !strings.Contains(err.Error(), `"whole"`) || !strings.Contains(err.Error(), `"inner"`) {
t.Errorf("%s: error %q names neither column; it must name both", c.name, err)
}
if _, err := f.Fake("row"); err != nil {
t.Errorf("%s: Fake(row) = %v, want the string view untouched", c.name, err)
}
}
}
func TestInlineRecordRejectsOverlappingColumns(t *testing.T) {
dir := writeData(t, map[string]string{
"cat": `{"format":"","a":[{"format":"A={b}","b":"1"},{"format":"A={b}","b":"2"}]}`,
})
f := newGenerator(t, dir, WithSeed(1))
_, err := f.FakeRecord(`{"format":"","whole":"{/cat.a}","inner":"{/cat.a.b}"}`)
if err == nil || !strings.Contains(err.Error(), "reads a path into") {
t.Fatalf("inline record over an overlapping pair = %v, want the inline entry point to refuse it too", err)
}
}
func TestRecordRejectsAColumnReadingItsOwnRecord(t *testing.T) {
for _, c := range []struct{ name, column string }{
{"a path into itself", `"full":"{/person.first} {/person.last}"`},
{"the record read whole", `"whole":"{/person}"`},
{"the record as an operand", `"up":"{uppercase(/person)}"`},
} {
person := `{"format":"{first} {last}","first":["Ada","Bo"],"last":["Lovelace","Ek"],` + c.column + `}`
f := newGenerator(t, writeData(t, map[string]string{"person": person}), WithSeed(1))
if _, err := f.Record("person"); err == nil || !strings.Contains(err.Error(), "points back at this record") {
t.Errorf("%s: Record = %v, want it refused; the column would contradict the columns beside it", c.name, err)
}
}
}
func TestRecordBareReferenceStaysIndependentAsAnOperand(t *testing.T) {
dir := writeData(t, map[string]string{
"cur": `[{"format":"{code}","code":"aud"},{"format":"{code}","code":"eur"}]`,
"row": `{"format":"","up":"{uppercase(/cur)}","low":"{lowercase(/cur)}"}`,
})
f := newGenerator(t, dir, WithSeed(1))
sawMismatch := false
for i := 0; i < 200 && !sawMismatch; i++ {
r, err := f.Record("row")
if err != nil {
t.Fatal(err)
}
m := map[string]string{}
for _, c := range r.Columns() {
m[c.Name] = c.Value
}
sawMismatch = !strings.EqualFold(m["up"], m["low"])
}
if !sawMismatch {
t.Error("two bare-reference operand columns never disagreed; a bare reference draws on its own, as the plain spelling does")
}
for i := 0; i < 50; i++ {
v, err := f.FakeTemplate("{/cur}|{uppercase(/cur)}")
if err != nil {
t.Fatal(err)
}
if parts := strings.Split(v, "|"); !strings.EqualFold(parts[0], parts[1]) {
t.Fatalf("one format rendered %q; within an expansion a bare reference is still one draw", v)
}
}
}
func TestRecordSQLInsert(t *testing.T) {
dir := writeData(t, map[string]string{
"person": `{"format": "{last}", "last": "O'Brien"}`,
})
f := newGenerator(t, dir, WithSeed(1))
r, err := f.Record("person")
if err != nil {
t.Fatal(err)
}
got := r.SQLInsert("people")
if got != `INSERT INTO "people" ("last") VALUES ('O''Brien');` {
t.Fatalf("SQLInsert() = %q, want quoted identifiers and the single quote doubled", got)
}
}
func TestRecordRejectsFieldDescent(t *testing.T) {
dir := writeData(t, map[string]string{
"cat": `{"format":"{sub}","sub":{"format":"{x}","x":"1"}}`,
"row": `[{"format":"{x}","x":"1"},{"format":"{x}","x":"2"}]`,
})
f := newGenerator(t, dir, WithSeed(1))
for _, path := range []string{"cat.sub", "row.x"} {
if _, err := f.Record(path); err == nil || !strings.Contains(err.Error(), "field") {
t.Errorf("Record(%q) = %v, want a 'descends into a field' error", path, err)
}
}
}
func TestRecordSharesAReferenceAcrossColumns(t *testing.T) {
dir := writeData(t, map[string]string{
"currency": `[{"format":"{code}","code":"AUD","symbol":"$"},{"format":"{code}","code":"EUR","symbol":"€"}]`,
"price": `{"format":"","code":"{/currency.code}","symbol":"{/currency.symbol}"}`,
})
f := newGenerator(t, dir, WithSeed(1))
for i := 0; i < 100; i++ {
r, err := f.Record("price")
if err != nil {
t.Fatal(err)
}
m := map[string]string{}
for _, c := range r.Columns() {
m[c.Name] = c.Value
}
switch m["code"] {
case "AUD":
if m["symbol"] != "$" {
t.Fatalf("record %q: code AUD but symbol %q, want one currency draw across columns", r.JSON(), m["symbol"])
}
case "EUR":
if m["symbol"] != "€" {
t.Fatalf("record %q: code EUR but symbol %q, want one currency draw across columns", r.JSON(), m["symbol"])
}
default:
t.Fatalf("record %q has unexpected code %q", r.JSON(), m["code"])
}
}
}
func TestRecordSharesAReferenceIntoAColumnRepeat(t *testing.T) {
dir := writeData(t, map[string]string{
"currency": `[{"format":"{code}","code":"AUD"},{"format":"{code}","code":"EUR"}]`,
"order": `{"format":"","codes":{"format":"{/currency.code}","repeat":3,"separator":"-"}}`,
})
f := newGenerator(t, dir, WithSeed(1))
for i := 0; i < 50; i++ {
r, err := f.Record("order")
if err != nil {
t.Fatal(err)
}
parts := strings.Split(r.Columns()[0].Value, "-")
if len(parts) != 3 || parts[0] != parts[1] || parts[1] != parts[2] {
t.Fatalf("codes column = %q, want one shared draw across its repeat", r.Columns()[0].Value)
}
}
}
func TestRecordBareReferenceStaysIndependent(t *testing.T) {
dir := writeData(t, map[string]string{
"currency": `[{"format":"{code}","code":"AUD"},{"format":"{code}","code":"EUR"}]`,
"order": `{"format":"","whole":"{/currency}","code":"{/currency.code}"}`,
})
f := newGenerator(t, dir, WithSeed(1))
sawMismatch := false
for i := 0; i < 100; i++ {
r, err := f.Record("order")
if err != nil {
t.Fatal(err)
}
m := map[string]string{}
for _, c := range r.Columns() {
m[c.Name] = c.Value
}
if m["whole"] != m["code"] {
sawMismatch = true
break
}
}
if !sawMismatch {
t.Fatal("a bare {/currency} column never disagreed with a tailed {/currency.code} column; a bare reference should draw independently")
}
}
func TestRecordSQLQuotesIdentifiers(t *testing.T) {
dir := writeData(t, map[string]string{
"row": `{"format": "", "postal-code": "1", "street-number": "2"}`,
})
f := newGenerator(t, dir, WithSeed(1))
r, err := f.Record("row")
if err != nil {
t.Fatal(err)
}
got := r.SQLInsert("my-table")
if got != `INSERT INTO "my-table" ("postal-code", "street-number") VALUES ('1', '2');` {
t.Fatalf("SQLInsert() = %q, want hyphenated identifiers and table quoted", got)
}
}
func TestFakeRecordAndTemplate(t *testing.T) {
f := recordCat(t)
in := `{"format":"{x} {y}","x":["1","2"],"y":["3","4"]}`
want, err := f.FakeRecord(in)
if err != nil {
t.Fatalf("FakeRecord: %v", err)
}
if len(want.Columns()) != 2 {
t.Fatalf("FakeRecord columns = %v, want two columns", want.Columns())
}
reusable, err := f.NewRecordTemplate(in)
if err != nil {
t.Fatalf("NewRecordTemplate: %v", err)
}
for i := 0; i < 20; i++ {
if got := reusable.Fake(); len(got.Columns()) != 2 {
t.Fatalf("RecordTemplate.Fake() = %v, want two columns", got.Columns())
}
}
}
func TestInlineRecordErrors(t *testing.T) {
f := recordCat(t)
for _, c := range []struct {
input string
want string
}{
{`"hello"`, "has no fields, so no columns"},
{`["a","b"]`, "a record is a template whose fields are its columns"},
{`{"format":"{a}-","repeat":3,"separator":"|","a":["x","y"]}`, "carries repeat 3"},
} {
if _, err := f.FakeRecord(c.input); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("FakeRecord(%q) = %v, want an error naming %q", c.input, err, c.want)
}
}
}
func TestRecordRejectsATopLevelRepeat(t *testing.T) {
dir := writeData(t, map[string]string{
"rep": `{"format":"{a}-","repeat":3,"separator":"|","a":["x","y"]}`,
})
f := newGenerator(t, dir, WithSeed(1))
if _, err := f.Record("rep"); err == nil || !strings.Contains(err.Error(), "carries repeat 3") {
t.Errorf("Record on a repeating template = %v, want an error naming the repeat", err)
}
if v, err := f.Fake("rep"); err != nil || v != "x-|x-|x-" {
t.Errorf("Fake(rep) = %q, %v, want the repeat still composed for the string view", v, err)
}
}
func TestRecordTemplateRejectsATopLevelRepeat(t *testing.T) {
f := recordCat(t)
in := `{"format":"{a}-","repeat":3,"separator":"|","a":["x","y"]}`
if _, err := f.NewRecordTemplate(in); err == nil || !strings.Contains(err.Error(), "carries repeat 3") {
t.Errorf("NewRecordTemplate on a repeating template = %v, want an error naming the repeat", err)
}
if _, err := f.NewTemplate(in); err != nil {
t.Errorf("NewTemplate on the same input = %v, want the string view to still compile", err)
}
}
reference.go
+424 -132
View File
@@ -2,155 +2,302 @@ package fejkdata
import (
"fmt"
"sort"
"strings"
)
// A reference names a node by path rather than as a sibling field: {/a.b} from the
// data root, {.a} from the folder this file sits in, {..a} from the folder above.
func isRef(name string) bool { return strings.HasPrefix(name, ".") || strings.HasPrefix(name, "/") }
// refPrefix marks a {..path} token: a reference to a node elsewhere in the data
// root rather than a sibling field. The path is resolved across every loaded
// directory (see linkRefs).
const refPrefix = ".."
// refBinding is what a reference was bound to: the head key its category is held
// under, and the tail read into it.
type refBinding struct {
key string
tail []string
}
func isRef(name string) bool { return strings.HasPrefix(name, refPrefix) }
// refShape splits a reference into its sigil and the dotted path after it.
func refShape(name string) (sigil, rest string, err error) {
switch {
case strings.HasPrefix(name, "/"):
sigil, rest = "/", name[1:]
case strings.HasPrefix(name, ".."):
sigil, rest = "..", name[2:]
default:
sigil, rest = ".", name[1:]
}
if rest == "" {
return "", "", fmt.Errorf("reference has no path")
}
if strings.HasPrefix(rest, "/") {
return "", "", fmt.Errorf("the path after %s starts at a name, not a /; write {%s%s}", sigil, sigil, rest[1:])
}
if strings.HasPrefix(rest, ".") {
return "", "", fmt.Errorf("a reference starts with / (the root), . (this folder) or .. (the folder above)")
}
for _, seg := range strings.Split(rest, ".") {
if seg == "" {
return "", "", fmt.Errorf("path has an empty segment")
}
}
return sigil, rest, nil
}
// refSegments resolves a reference written in folder to a path from the root.
func refSegments(name string, folder []string) ([]string, error) {
sigil, rest, err := refShape(name)
if err != nil {
return nil, err
}
var base []string
switch sigil {
case ".":
base = folder
case "..":
if len(folder) == 0 {
return nil, fmt.Errorf("no folder above the root")
}
base = folder[:len(folder)-1]
}
return append(append([]string{}, base...), strings.Split(rest, ".")...), nil
}
// linkRefs resolves every reference in the assembled tree. The head of the path —
// up to the category it names — is bound into the referring template's fields
// under its root path, and the rest reads into it the way a sibling path does, so
// a reference is held like a sibling and two spellings of one target are one
// draw. It runs once, after all data is merged, so a reference sees the final
// (override-resolved) tree. A path that is unknown, names a folder, or reads a
// field not every variant carries fails here, keeping a bad reference a New-time
// error, never a random render-time one.
// linkRefs resolves every {..path} reference in the assembled tree. The head of the
// path — up to the category it names — is bound into the referring template's
// fields, and the rest reads into it the way a sibling path does, so a reference
// is held like a sibling. It runs once, after all data is merged, so a reference
// sees the final (override-resolved) tree. A path that is unknown, names a folder,
// or reads a field not every variant carries fails here, keeping a bad reference a
// New-time error, never a random render-time one.
func linkRefs(root map[string]node) error {
return eachTemplate(root, func(folder []string, path string, t *template) error {
return linkTemplateRefs(folder, path, t, root)
return walkNodes(root, func(path string, n node) error {
t, ok := n.(*template)
if !ok {
return nil
}
names := refTokens(t.format)
if len(names) == 0 {
return nil
}
if t.fields == nil {
t.fields = map[string]node{}
}
t.refs = make(map[string]string, len(names))
for _, name := range names {
head, target, tail, err := resolveRef(root, strings.Split(name[len(refPrefix):], "."))
if err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
}
key := refPrefix + strings.Join(head, ".")
if err := checkPath(target, tail, key); err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
}
t.fields[key] = target
t.refs[name] = key
}
if err := t.compileFormat(); err != nil {
return fmt.Errorf("%s: %w", path, err)
}
return nil
})
}
// linkTemplateRefs binds one template's references against root. A template with
// none is left untouched, so an inline format that references nothing costs only
// the refTokens scan.
func linkTemplateRefs(folder []string, path string, t *template, root map[string]node) error {
names := refTokens(t.format)
if len(names) == 0 {
// checkBoundLevelsHeld rejects every route to a held name except the ones that read
// its draw. An expansion holds one draw of that name; anything else that renders it
// draws again, and the two disagree. checkNoOverlap settles the spellings within one
// format (a token, an operand); this settles the rest — a reference, whether it
// sits in that format or in anything the format renders, however deep.
//
// It runs after checkNoCycles, whose guarantee is what lets the walk terminate.
func checkBoundLevelsHeld(root map[string]node) error {
return walkNodes(root, func(path string, n node) error {
t, ok := n.(*template)
if !ok || len(t.held) == 0 {
return nil
}
heads := make([]string, 0, len(t.held))
for head := range t.held {
heads = append(heads, head)
}
// Operand heads first, then paths, each in name order: a level read both
// ways is reported by the operand's fence, and which overlap is reported
// does not vary.
sort.Slice(heads, func(i, j int) bool {
_, pi := t.bound[heads[i]]
_, pj := t.bound[heads[j]]
if pi != pj {
return !pi
}
return heads[i] < heads[j]
})
readers := boundReaders(t.format, t.bound, t.refs)
for _, head := range heads {
// What one draw answers for depends on how the draw is read: a path pins
// the levels it passes through and the leaf it lands on, an operand
// exactly the value its render produces.
held := map[node]bool{}
reader, isPath := t.bound[head]
if isPath {
for _, r := range readers {
if a := splitArm(r.name, t.refs); a.key == head {
coverPath(t.fields[head], a.tail, held)
}
}
} else {
operandDraw(t.fields[head], held)
}
if len(held) == 0 {
continue // an early out: a fixed head holds nothing to reach
}
// One seen set across the edges: a node that cannot reach the level
// cannot reach it by another route either, so it is walked once here.
seen := map[node]bool{}
for _, e := range renderEdges(t) {
if splitArm(e.label, t.refs).key == head {
continue // a token or operand reading this draw, the routes allowed
}
if renders(e.to, held, seen) {
if isPath {
return fmt.Errorf("%s: %s renders %q, which {%s} reads a path into; name the fields you want instead", path, e.reached(), head, reader)
}
return fmt.Errorf("%s: %s renders %q, which a {%s()} also reads; reach it one way so it is drawn once", path, e.reached(), head, operandReader(t, head))
}
}
}
return nil
})
}
// operandReader names the builtin whose operand holds head.
func operandReader(t *template, head string) string {
fn := ""
_ = eachToken(t.format, func(tok ftoken) error {
if tok.kind != 'b' || fn != "" {
return nil
}
if name, _, isFunc := funcCall(tok.body); isFunc {
for _, operand := range tokenOperands(tok.body) {
if splitArm(operand, t.refs).key == head {
fn = name
}
}
}
return nil
})
return fn
}
// coverPath collects what holding one path pins: every choice level the path
// passes through, whole, and the leaf it renders.
func coverPath(n node, tail []string, into map[node]bool) {
if _, isChoice := n.(*choice); isChoice || len(tail) == 0 {
cover(n, into, false)
return
}
t, ok := n.(*template)
if !ok {
return
}
if child, ok := t.fields[tail[0]]; ok {
coverPath(child, tail[1:], into)
}
}
// cover collects a level and everything contained in it. A fixed string outside a
// choice is left out — it cannot disagree with itself — but inside one each
// variant carries its own, so there it counts.
func cover(n node, into map[node]bool, inChoice bool) {
if isFixed(n) && !inChoice {
return
}
into[n] = true
_, isChoice := n.(*choice)
for _, c := range contained(n) {
cover(c.node, into, inChoice || isChoice)
}
}
// operandDraw collects what one held draw of an operand answers for: the operand
// and what rendering it settles inside itself. The builtin renders its operand
// whole, so that draw fixes every value the render produced, and a second route to
// any of them disagrees with it.
//
// The walk stops at a {..path} edge, which is where the operand's own value ends
// and a shared source begins: two names referencing one category are two draws, the
// same rule {word} {word} follows.
func operandDraw(n node, into map[node]bool) {
if isFixed(n) {
return
}
if into[n] {
return
}
into[n] = true
for _, e := range renderEdges(n) {
if isRef(e.label) {
continue
}
operandDraw(e.to, into)
}
}
// isFixed is a string that varies nothing: fixed text with no fields to read into.
func isFixed(n node) bool {
t, ok := n.(*template)
return ok && t.fixed && len(t.fields) == 0
}
// renders reports whether rendering n can reach anything in want, following the
// same edges expand does. seen keeps a node shared by several routes from being
// walked twice; checkNoCycles has already proved the graph is a DAG, so the walk
// ends.
func renders(n node, want, seen map[node]bool) bool {
if want[n] {
return true
}
if seen[n] {
return false
}
seen[n] = true
for _, e := range renderEdges(n) {
if renders(e.to, want, seen) {
return true
}
}
return false
}
// walkNodes calls fn once per contained node, passing the dot path that reaches it,
// visiting keys in sorted order so which of several broken nodes gets reported does
// not depend on map iteration.
func walkNodes(root map[string]node, fn func(path string, n node) error) error {
seen := map[node]bool{}
var visit func(string, node) error
visit = func(path string, n node) error {
if n == nil || seen[n] {
return nil
}
seen[n] = true
if err := fn(path, n); err != nil {
return err
}
for _, c := range contained(n) {
if err := visit(join(path, c.name), c.node); err != nil {
return err
}
}
return nil
}
if t.fields == nil {
t.fields = map[string]node{}
}
t.refs = make(map[string]refBinding, len(names))
for _, name := range names {
segments, err := refSegments(name, folder)
if err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
for _, name := range sortedNames(root) {
if err := visit(name, root[name]); err != nil {
return err
}
head, target, tail, err := resolveCategory(root, segments)
if err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
}
key := "/" + strings.Join(head, ".")
if err := checkPath(target, tail, key); err != nil {
return fmt.Errorf("%s: reference {%s}: %w", path, name, err)
}
t.fields[key] = target
t.refs[name] = refBinding{key, tail}
}
if err := t.compileFormat(); err != nil {
return fmt.Errorf("%s: %w", path, err)
}
return nil
}
// eachTemplate calls fn once per template, with the folder its category sits in
// and the dot path reaching it, folders and names in sorted order.
func eachTemplate(root map[string]node, fn func(folder []string, path string, t *template) error) error {
var inCategory func(folder []string, path string, n node) error
inCategory = func(folder []string, path string, n node) error {
if t, ok := n.(*template); ok {
if err := fn(folder, path, t); err != nil {
return err
}
}
for _, c := range contained(n) {
if err := inCategory(folder, join(path, c.name), c.node); err != nil {
return err
}
}
return nil
}
var inFolder func(folder []string, children map[string]node) error
inFolder = func(folder []string, children map[string]node) error {
for _, name := range sortedNames(children) {
path := join(strings.Join(folder, "."), name)
if g, ok := children[name].(*group); ok {
if err := inFolder(append(folder[:len(folder):len(folder)], name), g.children); err != nil {
return err
}
continue
}
if err := inCategory(folder, path, children[name]); err != nil {
return err
}
}
return nil
}
return inFolder(nil, root)
// namedNode is a contained child and the segment reaching it; a choice's items carry
// no segment, matching how a dot path steps over a choice.
type namedNode struct {
name string
node node
}
// resolveCategory walks a dotted path through the folders to the category it
// names, returning that head, the node, and the tail left to read into it. A
// descent of its own rather than a walkPath: it walks groups only and returns
// where they end, not a leaf.
func resolveCategory(root map[string]node, segments []string) (head []string, target node, tail []string, err error) {
func contained(n node) []namedNode {
switch n := n.(type) {
case *group:
return named(n.children)
case *choice:
out := make([]namedNode, len(n.items))
for i, it := range n.items {
out[i] = namedNode{node: it}
}
return out
case *template:
return named(n.fields)
default:
return nil
}
}
// named skips a bound {..path} key: it is a render edge, not containment, so using
// it as a path segment would report a node under a path that does not reach it. Only
// a template's fields hold bindings — loadDir skips a dot-prefixed entry, so a
// group's children never carry the prefix — so this one skip serves both.
func named(m map[string]node) []namedNode {
out := make([]namedNode, 0, len(m))
for _, name := range sortedNames(m) {
if isRef(name) {
continue
}
out = append(out, namedNode{name: name, node: m[name]})
}
return out
}
func sortedNames(m map[string]node) []string {
names := make([]string, 0, len(m))
for name := range m {
names = append(names, name)
}
sort.Strings(names)
return names
}
// resolveRef walks a reference path through the folders to the category it names,
// returning that head, the node, and the tail left to read into it.
func resolveRef(root map[string]node, segments []string) (head []string, target node, tail []string, err error) {
var n node = &group{children: root}
i := 0
for ; i < len(segments); i++ {
@@ -170,7 +317,7 @@ func resolveCategory(root map[string]node, segments []string) (head []string, ta
return segments[:i], n, segments[i:], nil
}
// refTokens returns the reference names a format reads, as tokens or as operands.
// refTokens returns the {..path} names a format reads, as tokens or as operands.
func refTokens(format string) []string {
var refs []string
seen := map[string]bool{}
@@ -182,3 +329,148 @@ func refTokens(format string) []string {
}
return refs
}
// renderEdge is a child a node renders into, labelled by what reaches it (a field
// name, reference, or choice index) for a readable cycle report. operand names
// the builtin when the label is its operand rather than a token, so an error can
// name it the way the author wrote it.
type renderEdge struct {
to node
label string
operand string
}
// reached names an edge as the author spelled it, the vocabulary boundReaders uses
// for the sibling fence.
func (e renderEdge) reached() string {
if e.operand != "" {
return fmt.Sprintf("%s operand %q", e.operand, e.label)
}
return "{" + e.label + "}"
}
// renderEdges lists the children rendering n recurses into, mirroring expand: a
// choice's items, and a template's field/reference tokens plus its operands. A
// group renders nothing, so it has no edges.
func renderEdges(n node) []renderEdge {
switch n := n.(type) {
case *choice:
es := make([]renderEdge, len(n.items))
for i, it := range n.items {
es[i] = renderEdge{to: it, label: fmt.Sprintf("[%d]", i)}
}
return es
case *template:
var es []renderEdge
add := func(name, operand string) {
a := splitArm(name, n.refs)
c, ok := n.fields[a.key]
if !ok {
return
}
for _, leaf := range pathLeaves(c, a.tail) {
es = append(es, renderEdge{leaf, name, operand})
}
}
_ = eachToken(n.format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if fn, _, isFunc := funcCall(t.body); isFunc {
for _, operand := range tokenOperands(t.body) {
add(operand, fn)
}
return nil
}
for _, name := range strings.Split(t.body, "|") {
add(name, "")
}
return nil
})
return es
default:
return nil
}
}
// pathLeaves lists what a token's dotted tail renders. A path draws the levels it
// passes through but renders only what it lands on, so the leaf is the edge — a
// bare token, whose tail is empty, lands on the field itself. A choice on the way
// contributes every variant, since any of them may be the one drawn. checkPath has
// already proved the tail resolves in every variant, so the walk drops nothing.
func pathLeaves(n node, tail []string) []node {
if len(tail) == 0 {
return []node{n}
}
if c, ok := n.(*choice); ok {
var out []node
for _, it := range c.items {
out = append(out, pathLeaves(it, tail)...)
}
return out
}
return pathLeaves(child(n, tail[0]), tail[1:])
}
// checkRepeatReach bounds the renders a repeat multiplies to along any root-to-leaf
// path, so nested repeats cannot build what one repeat may not. It runs after
// checkNoCycles, whose guarantee is what lets the walk terminate.
func checkRepeatReach(root map[string]node) error {
reach := map[node]int{}
var of func(n node) int
of = func(n node) int {
if r, done := reach[n]; done {
return r
}
r := 1
for _, e := range renderEdges(n) {
if c := of(e.to); c > r {
r = c
}
}
if t, ok := n.(*template); ok {
r *= t.repeat
}
reach[n] = r
return r
}
return walkNodes(root, func(path string, n node) error {
if t, ok := n.(*template); ok && t.repeat > 1 && of(n) > maxLen {
return fmt.Errorf("%s: repeat %d multiplies to %d renders along one path, above the maximum %d", path, t.repeat, of(n), maxLen)
}
return nil
})
}
// checkNoCycles rejects a reference cycle: a node whose rendering can reach itself
// — directly, mutually, or through a chain — never terminates, so it must fail at
// New rather than stack-overflow at render. It is a depth-first walk of the render
// graph (renderEdges); grey marks nodes on the current path so a back-edge to one
// is the cycle, while black lets a shared node (a DAG, not a cycle) be skipped.
// Every node is a root: a field its parent's format never renders is still reachable
// by dot path, so a cycle in one would otherwise reach render and be fatal there.
func checkNoCycles(root map[string]node) error {
const (
grey = 1
black = 2
)
color := map[node]int{}
var visit func(n node, path string) error
visit = func(n node, path string) error {
switch color[n] {
case grey:
return fmt.Errorf("reference cycle: %s", path)
case black:
return nil
}
color[n] = grey
for _, e := range renderEdges(n) {
if err := visit(e.to, path+" -> "+e.label); err != nil {
return err
}
}
color[n] = black
return nil
}
return walkNodes(root, func(path string, n node) error { return visit(n, path) })
}
reference_test.go
+3 -37
View File
@@ -30,12 +30,12 @@ func TestReferenceIntoAField(t *testing.T) {
}
}
// TestReferenceInAlternation lets a reference stand as one arm of a {a|/b}
// TestReferenceInAlternation lets a reference stand as one arm of a {a|..b}
// alternation, so a field and a cross-file value share one slot.
func TestReferenceInAlternation(t *testing.T) {
dir := writeData(t, map[string]string{
"far": `"X"`,
"near": `{"format":"{here|/far}","here":"H"}`,
"near": `{"format":"{here|..far}","here":"H"}`,
})
f := newGenerator(t, dir, WithSeed(2))
seen := map[string]bool{}
@@ -191,7 +191,7 @@ func TestNewErrorPathIsCanonical(t *testing.T) {
{
"cycle inside a choice arm",
map[string]string{"cat": `{"format":"hi","x":"{/cat.x}"}`},
"fejkdata: reference cycle: cat.x -> /cat.x",
"fejkdata: reference cycle: cat.x -> ..cat.x",
},
{
"bad reference reached through another reference",
@@ -312,37 +312,3 @@ func TestReferenceSigilErrors(t *testing.T) {
}
}
}
func TestSpellingsOfOneReferenceAreOneLevel(t *testing.T) {
person := `[{"format":"{first} {last}","first":"Ada","last":"Byron"},{"format":"{first} {last}","first":"Bo","last":"Ek"}]`
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, map[string]string{
"sv_SE/person": person,
"sv_SE/mail": `"{.person} <{/sv_SE.person.first}>"`,
})))
if err == nil || !strings.Contains(err.Error(), "reads a path into") || !strings.Contains(err.Error(), "{.person}") {
t.Errorf("New = %v, want the bare spelling rejected beside the path spelling", err)
}
dir := writeData(t, map[string]string{
"sv_SE/word": `["alpha","beta","gamma"]`,
"sv_SE/loud": `"{/sv_SE.word} {uppercase(.word)}"`,
})
f := newGenerator(t, dir, WithSeed(2))
for i := 0; i < 50; i++ {
got := strings.Fields(fake(t, f, "sv_SE.loud"))
if len(got) != 2 || strings.ToUpper(got[0]) != got[1] {
t.Fatalf("loud = %q, want one draw under both spellings", got)
}
}
}
func TestSlashAfterAReferenceSigilIsRejected(t *testing.T) {
for name, files := range map[string]map[string]string{
"after ..": {"sv_SE/person": `"Ada"`, "sv_SE/deep/a": `"{../person}"`},
"after .": {"sv_SE/person": `"Ada"`, "sv_SE/a": `"{./person}"`},
} {
_, err := New(WithoutShippedData(), WithDataPath(writeData(t, files)))
if err == nil || !strings.Contains(err.Error(), "person}") || !strings.Contains(err.Error(), "write {") {
t.Errorf("%s: New = %v, want the slash rejected naming the spelling", name, err)
}
}
}
render.go
+125 -26
View File
@@ -27,41 +27,68 @@ func (f *Generator) Fake(path string) (string, error) {
if _, ok := n.(*group); ok {
return "", fmt.Errorf("fejkdata: %s names a folder, not a value", path)
}
return render(f.rand, n, nil), nil
return render(f.rand, n), nil
}
// descend walks named fields to the node a path names. It is the one render-side
// step that can fail, because the path comes from the caller and may name a field
// that does not exist. A choice consumes no segment, so the rest of the path must
// be one every variant carries before a variant is picked — a path that resolves
// at all resolves on every call.
func descend(s *session, root node, segments []string) (node, error) {
var found node
err := walkPath(root, segments, pathWalk{
choice: func(c *choice, rest []string) ([]node, error) {
if err := carriedByAll(c, rest); err != nil {
return nil, err
}
return []node{pick(s, c)}, nil
},
leaf: func(n node) error { found = n; return nil },
})
return found, err
// be one every variant carries (the set compile stored) before a variant is picked
// — a path that resolves at all resolves on every call.
func descend(s *session, n node, segments []string) (node, error) {
if len(segments) == 0 {
return n, nil
}
switch n := n.(type) {
case *group:
child, ok := n.children[segments[0]]
if !ok {
return nil, fmt.Errorf("no entry %q", segments[0])
}
return descend(s, child, segments[1:])
case *template:
child, ok := n.fields[segments[0]]
if !ok {
return nil, fmt.Errorf("no field %q", segments[0])
}
return descend(s, child, segments[1:])
case *choice:
if want := strings.Join(segments, "."); !n.shared[want] {
return nil, unreachableInChoice(n, want)
}
return descend(s, pick(s, n), segments)
default:
return nil, fmt.Errorf("cannot descend into %T at %q", n, segments[0])
}
}
// unreachableInChoice reports that a path cannot step through this choice, listing
// what every variant does carry. It reads the precomputed set, so a failing path
// costs no more than a rendering one.
func unreachableInChoice(c *choice, want string) error {
if len(c.shared) == 0 {
return fmt.Errorf("no variant of this %d-way choice carries %q", len(c.items), want)
}
offered := make([]string, 0, len(c.shared))
for p := range c.shared {
offered = append(offered, p)
}
sort.Strings(offered)
return fmt.Errorf("not every variant of this %d-way choice carries %q; all carry %v", len(c.items), want, offered)
}
// render evaluates a compiled node to a string. compile validates every node up
// front, so rendering a compiled tree cannot fail. refScope carries the draws a
// reference shares beyond its own expansion; nil keeps every reference local.
func render(s *session, n node, refScope *draws) string {
// front, so rendering a compiled tree cannot fail.
func render(s *session, n node) string {
switch n := n.(type) {
case *choice:
return render(s, pick(s, n), refScope)
return render(s, pick(s, n))
case *template:
if n.repeat == 1 {
if n.fixed {
return n.lit
}
return expand(s, n, refScope)
return expand(s, n)
}
var b strings.Builder
b.Grow(n.repeat * (n.grow + len(n.separator)))
@@ -69,7 +96,7 @@ func render(s *session, n node, refScope *draws) string {
if i > 0 {
b.WriteString(n.separator)
}
b.WriteString(expand(s, n, refScope))
b.WriteString(expand(s, n))
}
return b.String()
default:
@@ -91,12 +118,11 @@ func pick(r rng, c *choice) node {
// expand renders a template's compiled ops. compile validated every token, so this
// cannot fail.
func expand(s *session, t *template, refScope *draws) string {
func expand(s *session, t *template) string {
var b strings.Builder
b.Grow(t.grow)
// One draw per held name, for this expansion only: a nested template and each
// repeat iteration get their own, since each is its own expansion. A reference
// reads the caller's scope instead, whenever one was supplied.
// repeat iteration get their own, since each is its own expansion.
var held *draws
if len(t.held) > 0 {
held = &draws{
@@ -110,7 +136,7 @@ func expand(s *session, t *template, refScope *draws) string {
case 'l':
b.WriteString(o.lit)
case 'f':
b.WriteString(readField(s, t, held, refScope, o.arms[s.IntN(len(o.arms))]))
b.WriteString(readField(s, t, held, o.arms[s.IntN(len(o.arms))]))
case 'b':
// Read before the call, so the value a calc computes is the value the
// format showed. calcVars fixed the order op.operands holds.
@@ -118,7 +144,7 @@ func expand(s *session, t *template, refScope *draws) string {
if len(o.operands) > 0 {
operands = make([]string, len(o.operands))
for j, a := range o.operands {
operands[j] = readField(s, t, held, refScope, a)
operands[j] = readField(s, t, held, a)
}
}
b.WriteString(o.call(s, b.String(), operands)) // b.String() is the output so far
@@ -126,3 +152,76 @@ func expand(s *session, t *template, refScope *draws) string {
}
return b.String()
}
// draws is what an expansion has already drawn for its held names: the variant each
// was drawn as, so every path under it reads one row, and the value each read, so
// the same name read twice reads one value.
type draws struct {
variant map[string]node
value map[string]string
}
// readField renders one arm of a token. An arm's key is a sibling field or a
// {..path} reference, which linkRefs bound into fields too. A name the expansion
// holds — a level some token addresses by dotted path, or a sibling a {calc()}
// reads — is drawn once and kept, so {place.postal-code} and {place.locality} read
// one row, either read twice gives one value, and a shown operand is the operand
// computed. Every other name is drawn afresh, so {word} {word} still draws twice.
// checkTokens, checkPath and linkRefs prove every step, so this cannot fail.
func readField(s *session, t *template, held *draws, a arm) string {
if !t.held[a.key] {
return render(s, t.fields[a.key])
}
if v, read := held.value[a.name]; read {
return v
}
n, drew := held.variant[a.key]
if !drew {
n = drawn(s, t.fields[a.key])
held.variant[a.key] = n
}
// Hold the draw at every level passed through, so two paths sharing a prefix
// share it.
for i, seg := range a.tail {
if i < len(a.steps) {
step, drew := held.variant[a.steps[i]]
if !drew {
step = drawn(s, child(n, seg))
held.variant[a.steps[i]] = step
}
n = step
continue
}
n = child(n, seg)
}
v := render(s, n)
held.value[a.name] = v
return v
}
// child is the node one path segment names below an already-drawn node. It holds
// while checkPath and the set a choice shares (see sharedPaths) agree with this
// walk: both prove the segment exists and that drawn leaves a template here. Each
// way that can break panics naming the segment, so a slip in that agreement
// reports where it happened rather than surfacing a nil node a level later.
func child(n node, seg string) node {
t, ok := n.(*template)
if !ok {
panic(fmt.Sprintf("fejkdata: %q under %T, which carries no fields", seg, n))
}
c, ok := t.fields[seg]
if !ok {
panic(fmt.Sprintf("fejkdata: no field %q under a drawn level", seg))
}
return c
}
// drawn resolves a choice to one variant, so a bound head is a concrete node the
// rest of the expansion shares. Nested choices unwrap too: a draw is one value, not
// another set to pick from.
func drawn(s *session, n node) node {
for c, ok := n.(*choice); ok; c, ok = n.(*choice) {
n = pick(s, c)
}
return n
}
shipped_data_test.go
-10
View File
@@ -108,16 +108,6 @@ func TestFakeIsSafeForConcurrentUse(t *testing.T) {
return
}
f.List()
if _, err := f.Record("sv_SE.person"); err != nil {
t.Error(err)
return
}
tmpl, err := f.NewTemplate("{/sv_SE.person.last}")
if err != nil {
t.Error(err)
return
}
tmpl.Fake()
}
}()
}
template.go
+105 -55
View File
@@ -2,6 +2,7 @@ package fejkdata
import (
"fmt"
"sort"
"strings"
)
@@ -15,8 +16,7 @@ type ftoken struct {
// eachToken scans a format string once and calls fn for each unit, the single
// source of truth for how braces are read: "{{" and "}}" are literal braces, a "{"
// opens a token that must reach its "}", and a lone "}" is an error. A table-shaped
// scanner, one case per rune kind, kept whole on purpose.
// opens a token that must reach its "}", and a lone "}" is an error.
func eachToken(format string, fn func(ftoken) error) error {
rs := []rune(format)
for i := 0; i < len(rs); i++ {
@@ -135,12 +135,12 @@ func checkTokens(format string, fields map[string]node) error {
names := strings.Split(t.body, "|")
for _, name := range names {
if isRef(name) {
if _, _, err := refShape(name); err != nil {
return fmt.Errorf("token {%s}: %w", t.body, err)
if name == refPrefix {
return fmt.Errorf("token {%s}: reference has no path", t.body)
}
continue // its target is checked at New (see linkRefs)
continue // a root reference; its target is checked at New (see linkRefs)
}
if err := checkArm(name, fields, len(names) == 1); err != nil {
if err := checkArm(name, fields); err != nil {
return fmt.Errorf("token {%s}: %w", t.body, err)
}
}
@@ -151,9 +151,7 @@ func checkTokens(format string, fields map[string]node) error {
}
// checkArm validates one sibling name or path against a template's fields.
// wholeToken says the name is the token's entire body, so {/name} would render
// the same value and can be offered as the reference spelling.
func checkArm(name string, fields map[string]node, wholeToken bool) error {
func checkArm(name string, fields map[string]node) error {
a := splitArm(name, nil)
if err := checkSegments(a); err != nil {
return err
@@ -166,13 +164,6 @@ func checkArm(name string, fields map[string]node, wholeToken bool) error {
if isOption(a.key) {
return fmt.Errorf("%q is an option and can never be a field", a.key)
}
if len(fields) == 0 {
hint := ""
if wholeToken && hintableRef(name) {
hint = fmt.Sprintf(" — write {/%s} to reference the data", name)
}
return fmt.Errorf("no field %q; a token names a sibling field, and this template has none%s", a.key, hint)
}
return fmt.Errorf("no field %q", a.key)
}
if err := checkPath(head, a.tail, a.key); err != nil {
@@ -181,17 +172,6 @@ func checkArm(name string, fields map[string]node, wholeToken bool) error {
return nil
}
// hintableRef reports whether {/name} is a reference the grammar accepts, so the
// hint never names a spelling that fails too.
func hintableRef(name string) bool {
for _, seg := range strings.Split(name, ".") {
if checkName(seg) != nil {
return false
}
}
return true
}
// tokenOperands lists the fields one {token} body reads as operands, empty for a
// field token or a builtin that reads none.
func tokenOperands(body string) []string {
@@ -250,35 +230,30 @@ func fieldTokens(format string) []string {
}
// arm is one alternative of a {a|b} token or one operand, split into the key
// naming the node in a template's fields (a sibling field, or the head a
// naming the node in a template's fields (a sibling field, or the "..path" head a
// reference is bound under) and the tail of a dotted path into it. A non-empty
// tail is what makes the arm a bound draw: its head is drawn once per expansion
// (see compileOps).
type arm struct {
name string // as written, for messages
name string // as written, and the key a bound draw's value is held under
key string
tail []string
steps []string // key per level passed through; the head and leaf hold their own
path string // key and tail, the one spelling every way of writing this read shares
}
// splitArm splits one name into key and tail. refs maps a reference to what
// linkRefs bound it to; before linking, a reference is whole.
func splitArm(name string, refs map[string]refBinding) arm {
// splitArm splits one name into key and tail. refs maps a reference to the head
// linkRefs bound it under; before linking, a reference is whole.
func splitArm(name string, refs map[string]string) arm {
if isRef(name) {
b, bound := refs[name]
if !bound || len(b.tail) == 0 {
key := name
if bound {
key = b.key
}
return arm{name: name, key: key, path: key}
key, bound := refs[name]
if !bound || key == name {
return arm{name: name, key: name}
}
return pathArm(name, b.key, b.tail)
return pathArm(name, key, strings.Split(name[len(key)+1:], "."))
}
head, tail, dotted := strings.Cut(name, ".")
if !dotted {
return arm{name: name, key: name, path: name}
return arm{name: name, key: name}
}
return pathArm(name, head, strings.Split(tail, "."))
}
@@ -288,21 +263,62 @@ func pathArm(name, key string, segs []string) arm {
for i := 0; i < len(segs)-1; i++ { // every level except the leaf's own
steps = append(steps, key+"."+strings.Join(segs[:i+1], "."))
}
return arm{name: name, key: key, tail: segs, steps: steps, path: key + "." + strings.Join(segs, ".")}
return arm{name: name, key: key, tail: segs, steps: steps}
}
// splitArms splits a token body's '|' alternatives.
func splitArms(body string, refs map[string]refBinding) []arm {
parts := strings.Split(body, "|")
arms := make([]arm, len(parts))
for i, p := range parts {
arms[i] = splitArm(p, refs)
// checkNoOverlap rejects a format that both renders a level and reads a path into
// it — {p} beside {p.first}, or {p.addr} beside {p.addr.city}. The path reads the
// level's held draw while rendering the level expands it afresh, so their values
// would disagree. Names are compared in sorted order, so which pair is reported
// does not depend on where the tokens sit.
func checkNoOverlap(format string, bound map[string]string, refs map[string]string) error {
names := boundReaders(format, bound, refs)
// Stable over one format-order scan, so two readers of one name (a token and a
// calc operand both naming "p") are reported as the format writes them.
sort.SliceStable(names, func(i, j int) bool { return names[i].name < names[j].name })
for i, level := range names {
for _, path := range names[i+1:] {
if strings.HasPrefix(path.name, level.name+".") {
return fmt.Errorf("%s renders a level that {%s} reads a path into; name the fields you want instead", level.label, path.name)
}
}
}
return arms
return nil
}
// checkSegments rejects an unfinished path: "{a.}" and "{a..b}" each have a
// segment naming nothing. A field really named "" would otherwise make them
// reader is one way a format reaches a bound field, and how to name that spelling.
type reader struct{ name, label string }
// boundReaders lists every way a format reaches a bound field, in the order the
// format writes them. An operand renders its field, so it names a level exactly
// as a token does; one scan finds both, which is what puts them in one order.
func boundReaders(format string, bound map[string]string, refs map[string]string) []reader {
var names []reader
_ = eachToken(format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if fn, _, isFunc := funcCall(t.body); isFunc {
for _, operand := range tokenOperands(t.body) {
a := splitArm(operand, refs)
if _, isBound := bound[a.key]; isBound {
names = append(names, reader{a.name, fmt.Sprintf("%s operand %q", fn, operand)})
}
}
return nil
}
for _, a := range splitArms(t.body, refs) {
if _, isBound := bound[a.key]; isBound {
names = append(names, reader{a.name, "token {" + a.name + "}"})
}
}
return nil
})
return names
}
// checkSegments rejects an unfinished path: "{a.}", "{.b}" and "{a..b}" each have
// a segment naming nothing. A field really named "" would otherwise make them
// resolve, so a typo would read as a path that worked.
func checkSegments(a arm) error {
if len(a.tail) == 0 {
@@ -319,6 +335,16 @@ func checkSegments(a arm) error {
return nil
}
// splitArms splits a token body's '|' alternatives.
func splitArms(body string, refs map[string]string) []arm {
parts := strings.Split(body, "|")
arms := make([]arm, len(parts))
for i, p := range parts {
arms[i] = splitArm(p, refs)
}
return arms
}
// callFn is a builtin bound to one call site: its args already parsed. It reads the
// output emitted so far in the current expansion (a derivation's payload) and the
// values of the operands it named, which expand read for it.
@@ -370,7 +396,7 @@ func (c *formatOps) hold(a arm, label string) {
}
}
func (c *formatOps) function(body string, refs map[string]refBinding) {
func (c *formatOps) function(body string, refs map[string]string) {
name, args, _ := funcCall(body)
var operands []arm
for _, operand := range tokenOperands(body) {
@@ -381,7 +407,7 @@ func (c *formatOps) function(body string, refs map[string]refBinding) {
c.ops = append(c.ops, op{kind: 'b', call: builtins[name].prep(args), operands: operands})
}
func (c *formatOps) field(body string, refs map[string]refBinding) {
func (c *formatOps) field(body string, refs map[string]string) {
arms := splitArms(body, refs)
for _, a := range arms {
if len(a.tail) > 0 {
@@ -393,7 +419,7 @@ func (c *formatOps) field(body string, refs map[string]refBinding) {
// compileOps compiles a format string. Call checkTokens first: it is what proves
// the scan and every token are valid.
func compileOps(format string, refs map[string]refBinding) formatOps {
func compileOps(format string, refs map[string]string) formatOps {
var c formatOps
var lit strings.Builder
flush := func() {
@@ -420,3 +446,27 @@ func compileOps(format string, refs map[string]refBinding) formatOps {
flush()
return c
}
// checkNoRepeatedRead rejects a bare token repeated on a held name: {w} {w} beside
// {uppercase(w)} would read one draw twice, where {w} {w} alone draws twice. The
// error names the single-token spelling.
func checkNoRepeatedRead(format string, c formatOps, refs map[string]string) error {
count := map[string]int{}
return eachToken(format, func(t ftoken) error {
if t.kind != 'b' {
return nil
}
if _, _, isFunc := funcCall(t.body); isFunc {
return nil
}
for _, a := range splitArms(t.body, refs) {
if len(a.tail) > 0 || !c.held[a.key] {
continue
}
if count[a.key]++; count[a.key] > 1 {
return fmt.Errorf("token {%s} is repeated, and %s holds %q to one draw per expansion; write {%s} once", a.name, c.holder[a.key], a.key, a.name)
}
}
return nil
})
}
template_test.go
+14 -82
View File
@@ -8,13 +8,7 @@ import (
)
// engine builds a seeded generator with no loaded categories, for rendering tests.
func engine(seed uint64) *Generator {
s, err := newRand(seed, true)
if err != nil {
panic(err)
}
return &Generator{rand: s}
}
func engine(seed uint64) *Generator { return &Generator{rand: newRand(seed, true)} }
// parse unmarshals a JSON template fragment into its dynamic form.
func parse(t *testing.T, s string) any {
@@ -38,7 +32,7 @@ func compiled(t *testing.T, s string) node {
func mustRender(t *testing.T, f *Generator, s string) string {
t.Helper()
return render(f.rand, compiled(t, s), nil)
return render(f.rand, compiled(t, s))
}
func TestStringIsAFormat(t *testing.T) {
@@ -233,29 +227,17 @@ func TestCompileErrors(t *testing.T) {
`{"format":"x","repeat":1.5}`, // non-integer repeat
`{"format":"x","repeat":"two"}`, // non-numeric repeat
`{"format":"x","repeat":2,"separator":5}`, // non-string separator
`"{nope()}"`, // unknown function
`"{luhn(x)}"`, // function given args it takes none of
`"{luhn(}"`, // malformed function token
`"{int(1)}"`, // wrong arity
`"{int(a,b)}"`, // non-integer args
`"{int(5,1)}"`, // min > max
`"{hex(0)}"`, // count must be positive
`"{nanoid(-1)}"`, // negative count
`"{base64(0)}"`, // count must be positive
`"{float(1,2)}"`, // wrong arity
`"{float(1,2,-1)}"`, // negative decimals
`"{float(NaN,NaN,2)}"`, // bounds must be finite
`"{float(Inf,Inf,2)}"`, // same-sign infinities
`"{float(1,NaN,2)}"`, // one NaN bound
`"{float(-Inf,1,2)}"`, // one infinite bound
`"{digits(+5)}"`, // a count is a plain integer
`"{digits(05)}"`, // no leading zero
`"{int(+1,5)}"`, // a bound is a plain integer
`"{int(5,5)}"`, // a constant is written as text
`"{float(1,1,2)}"`, // a constant is written as text
`{"format":"{x}","x":"v","repeat":2,"separator":""}`, // separator "" is the default
`"{calc(1/0)}"`, // a constant zero divisor
`{"format":"{calc(x/y)}","x":"1","y":"0"}`, // a fixed zero divisor
`"{nope()}"`, // unknown function
`"{luhn(x)}"`, // function given args it takes none of
`"{luhn(}"`, // malformed function token
`"{int(1)}"`, // wrong arity
`"{int(a,b)}"`, // non-integer args
`"{int(5,1)}"`, // min > max
`"{hex(0)}"`, // count must be positive
`"{nanoid(-1)}"`, // negative count
`"{base64(0)}"`, // count must be positive
`"{float(1,2)}"`, // wrong arity
`"{float(1,2,-1)}"`, // negative decimals
`"{iban(US)}"`, // unsupported country
`"{seq(a,b)}"`, // seq takes at most one name
`"{calc()}"`, // calc needs an expression
@@ -315,7 +297,7 @@ func TestGrowIsALowerBound(t *testing.T) {
t.Fatalf("format %q did not compile to a template", format)
}
for i := 0; i < 50; i++ {
if got := len(expand(f.rand, tmpl, nil)); got < tmpl.grow {
if got := len(expand(f.rand, tmpl)); got < tmpl.grow {
t.Errorf("format %q: expand emitted %d bytes, below grow %d", format, got, tmpl.grow)
}
}
@@ -329,53 +311,3 @@ func quote(s string) string {
}
return string(b)
}
func TestHashIsLiteral(t *testing.T) {
f := engine(1)
cases := map[string]string{
`{"format":"","x":"v"}`: "",
`{"format":"#","x":"v"}`: "#",
`{"format":"##","x":"v"}`: "##",
`{"format":"#0#1#A#a","x":"v"}`: "#0#1#A#a",
`{"format":"#{x}","x":"v"}`: "#v",
}
for tmpl, want := range cases {
if got := mustRender(t, f, tmpl); got != want {
t.Errorf("render(%s) = %q, want %q", tmpl, got, want)
}
}
}
func TestMultibyteFormat(t *testing.T) {
// Scanning is rune-aware: multibyte literals coexist with class chars and
// tokens without corrupting indices.
got := mustRender(t, engine(2), `{"format":"Öster{x}-{digits(1)}å","x":"väg"}`)
if !regexp.MustCompile(`^Österväg-[0-9]å$`).MatchString(got) {
t.Fatalf("multibyte format = %q", got)
}
}
func TestAlternationThreeWay(t *testing.T) {
f := engine(4)
seen := map[string]bool{}
for i := 0; i < 200; i++ {
seen[mustRender(t, f, `{"format":"{a|b|c}","a":"A","b":"B","c":"C"}`)] = true
}
if !seen["A"] || !seen["B"] || !seen["C"] || len(seen) != 3 {
t.Fatalf("3-way alternation produced %v, want A, B and C", seen)
}
}
func TestArgErrorsNameTheSpelling(t *testing.T) {
for src, want := range map[string]string{
`"{float(1,2,02)}"`: "write 2",
`{"format":"{calc(a,02)}","a":"1"}`: "write 2",
`"{digits(+5)}"`: "write 5",
`"{hex(99999999999999999999)}"`: "exceeds the maximum",
`"{int(007,9)}"`: "write 7",
} {
if _, err := compile(parse(t, src)); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("compile(%s) = %v, want an error saying %q", src, err, want)
}
}
}
transform_test.go
-6
View File
@@ -51,9 +51,3 @@ func TestTransformArgs(t *testing.T) {
}
}
}
func TestAsciiKeepsDEL(t *testing.T) {
if got := mustRender(t, engine(1), `{"format":"{ascii(x)}","x":"a\u007fb"}`); got != "a\u007fb" {
t.Errorf("ascii over DEL = %q, want it kept: DEL is ASCII", got)
}
}