1179 lines
66 KiB
Markdown
1179 lines
66 KiB
Markdown
# fejkdata
|
||
|
||
Locale-aware fake data for tests and fixtures, generated from JSON templates. Use
|
||
it as a Go library or the CLI — no data on disk, no dependencies, and a seed makes output
|
||
reproducible.
|
||
|
||
```sh
|
||
go install github.com/larvit/fejkdata/cmd/fejkdata@latest
|
||
echo '{/sv_SE.person}' | fejkdata # Sara Eriksson
|
||
```
|
||
|
||
## CLI
|
||
|
||
```sh
|
||
echo '{/sv_SE.person}' | fejkdata # Sara Eriksson
|
||
echo '{/sv_SE.person.last}' | fejkdata # Eriksson
|
||
echo '{/sv_SE.person} hihi' | fejkdata # Sara Eriksson hihi
|
||
echo '{/sv_SE.address}' | fejkdata --seed 42 # the same address every run
|
||
echo '{/sv_SE.word}' | fejkdata -n 3 # nät, barn and sol, a line each
|
||
printf '%s' '{/sv_SE.word}' | fejkdata -n 3 --separator ', ' # nät, barn, sol
|
||
fejkdata --list # every path the data offers
|
||
echo '{/misc.territory[SE].capital}' | fejkdata # Stockholm — a table's row, selected by key or name
|
||
echo '{/geo.SE.locality[Lund].street}' | fejkdata # Fjelievägen — a linked table, drawn inside the row
|
||
echo '{/sv_SE.word}' | fejkdata --data-path ./mydata # layer a directory over the shipped data
|
||
fejkdata --no-shipped-data -d ./mydata --list # only your data
|
||
echo '{"format":"name: {x}","x":["bosse","lina"]}' | fejkdata # name: bosse or name: lina
|
||
fejkdata < template.txt # a template kept in a file
|
||
fejkdata <<'EOF' # born 2003-11-27 — a quoted heredoc passes ', $ and \ as written
|
||
born {date(1990-01-01,2010-12-31,'2006-01-02')}
|
||
EOF
|
||
```
|
||
|
||
fejkdata renders the template on its stdin: a format string, or a JSON object, array, string or `null`. Its `{…}` tokens reach the data by reference from the root,
|
||
`{/sv_SE.person.last}`, so shipped and `--data-path` categories are alike available,
|
||
and everything outside them prints as written. A reference names a category, or a field
|
||
inside one: each dot segment descends one level — folders, then the category (a JSON
|
||
file), then fields — and `[SE]` after a [table](#table) selects its row. `--list` prints
|
||
these paths; a template reads one as `{/path}`.
|
||
|
||
Under `--format text`, the default, what you send is what renders, and fejkdata adds nothing. A format string keeps every
|
||
byte, so `echo`, a file or a heredoc ends each render with its newline, and `printf '%s'`
|
||
sends none; `-n` joins renders with `--separator`, empty by default. A JSON object, array
|
||
or string drops the whitespace around it, and `null` alone renders nothing; `42` or `true`
|
||
alone is a format string, so it prints as written, newline included. An empty or blank stdin is a format string like
|
||
any other, and renders as sent. A JSON template that should end
|
||
each render in a newline writes `\n` in its format, from a quoted heredoc or a file. A
|
||
lone reference, one `{/…}` and nothing else, is the record it names under `--format`, one
|
||
newline after it allowed. Some shells' `echo` reads a backslash as an escape, so pipe a
|
||
template holding one from a quoted heredoc or a file. The rest of stdin's rules are under
|
||
[Decisions](docs/decisions.md#the-cli-renders-exactly-the-template-on-its-stdin-and-adds-nothing).
|
||
|
||
| Flag | |
|
||
|------|--|
|
||
| `-d`, `--data-path D` | a directory to layer over the shipped data; repeatable, the last wins a name clash |
|
||
| `--no-shipped-data` | load only the `--data-path` directories |
|
||
| `-s`, `--seed N` | same seed, version and data: identical output |
|
||
| `-n`, `--repeat N` | render the template N times (0 to 1048576), each an independent draw, streamed; `0` prints nothing, `[]` under `--format json`, or the header under `--format csv` |
|
||
| `--separator S` | between repeated renders (default empty); refused with a record `--format`, which has its own separators |
|
||
| `--format F` | `text` (default), `json`, `ndjson`, `csv` or `sql` — a record's columns, one record per row (json frames them as an array) |
|
||
| `--table T` | the INSERT target for `--format sql`, ignored under any other (default: a lone reference's last segment, such as `users` for `{/users}`, else `records`) |
|
||
| `--list` | print every path, then exit; `--repeat`, `--separator`, `--format` and `--table` do nothing here |
|
||
| `--version`, `-h`, `--help` | print, then exit |
|
||
|
||
`--name value` and `--name=value` both work, a short flag's value attaches or
|
||
follows (`-n3`, `-n 3`) and short flags bundle (`-hn 3`) — see
|
||
[Decisions](docs/decisions.md#flags-follow-getopt_long); flags go anywhere, `--` ends them. Exit codes: `0` success, `1` runtime error (data
|
||
that fails to load, a missing `-d` folder included, a lone reference to nothing under `--format`), `2` misuse — a bad flag, an argument
|
||
other than a flag, an unreadable stdin, nothing piped in, or a
|
||
template that does not compile. From a checkout:
|
||
`go run ./cmd/fejkdata …`.
|
||
|
||
### Your own data
|
||
|
||
A category is a JSON file in a directory. Save this as `mydata/sql.json`:
|
||
|
||
```json
|
||
{
|
||
"format": "INSERT INTO users VALUES({sql-username});",
|
||
"sql-username": {
|
||
"format": "'{username}'",
|
||
"repeat": 3,
|
||
"separator": "),(",
|
||
"username": ["pixelfox", "snork", "turbohund", "blip", "zoom", "wahoo"]
|
||
}
|
||
}
|
||
```
|
||
|
||
```sh
|
||
echo '{/sql}' | fejkdata --seed 1 --data-path ./mydata
|
||
# INSERT INTO users VALUES('zoom'),('wahoo'),('blip');
|
||
echo '{/sql}' | fejkdata --repeat 100 --data-path ./mydata > seed.sql
|
||
```
|
||
|
||
That `format` string is the free-form spelling — you hand-write the whole row.
|
||
For structured output a record writes the row for you.
|
||
|
||
### Records
|
||
|
||
A record is a template seen as columns: its fields are the columns, its `format`
|
||
the whole. `--format json|ndjson|csv|sql` writes the records; the library's
|
||
`FakeRecord` (below) hands back the columns. A column is a string unless it declares a
|
||
[datatype](#datatype), and a [`null`](#null) item draws it as null. Save
|
||
`mydata/users.json`:
|
||
|
||
```json
|
||
{
|
||
"format": "{first} {last}",
|
||
"first": ["Ada", "Bo"],
|
||
"last": ["Lovelace", "Ek"]
|
||
}
|
||
```
|
||
|
||
```sh
|
||
echo '{/users}' | fejkdata --seed 1 -d ./mydata --format json # [{"first":"Bo","last":"Lovelace"}]
|
||
echo '{/users}' | fejkdata --seed 1 -d ./mydata --format ndjson # {"first":"Bo","last":"Lovelace"}
|
||
echo '{/users}' | fejkdata --seed 1 -d ./mydata --format csv # first,last → Bo,Lovelace
|
||
echo '{/users}' | fejkdata --seed 1 -d ./mydata --format sql # INSERT INTO "users" ("first", "last") VALUES ('Bo', 'Lovelace');
|
||
echo '{/users}' | fejkdata --seed 1 -d ./mydata --format sql --table people # INSERT into another table
|
||
```
|
||
|
||
`--repeat` streams that many records — `json` frames them as one array document,
|
||
`ndjson` writes one object per line, `csv` a row after a header, `sql` one INSERT
|
||
per line. Only a category-level template is a record, or a one-item choice holding one; a
|
||
field, a choice of more items or a folder errors, and so does a `repeat` on
|
||
the template itself, which composes the format into one string rather than
|
||
projecting columns — ask for more records with `--repeat`. A `repeat` on a column
|
||
is fine.
|
||
|
||
Columns are written in name order, whatever order the fields appear in. A category
|
||
whose fields are the parts of one value — `sv_SE.price`, `sv_SE.version`, `misc.uuid` —
|
||
projects those parts rather than the value, so for one column holding what `Fake`
|
||
renders, write `{"format":"","price":"{/sv_SE.price}"}` as a fieldless category asks
|
||
for.
|
||
|
||
A column carrying a newline keeps it inside the quoted CSV field or the SQL string
|
||
literal, so a row can span physical lines: read the stream with a CSV or SQL
|
||
parser rather than splitting it on newlines.
|
||
|
||
The SQL is ANSI — identifiers in double quotes, a literal quote doubled (`''`),
|
||
backslashes passed through — so a hyphenated field like `postal-code` stays a
|
||
valid identifier. PostgreSQL and SQLite take it as written; MySQL and MariaDB need
|
||
`ANSI_QUOTES` and `NO_BACKSLASH_ESCAPES` set first, or they read `"users"` as a
|
||
string and a backslash as an escape.
|
||
|
||
A record written only to emit columns still needs a `format` — the grammar's one
|
||
required key — so `"format": ""` carries the fields with an inert format: `Fake` renders
|
||
it as nothing.
|
||
|
||
A record is one render, so its columns read one pick of each [name](#names) its category
|
||
binds: `"code": "{/currency as c}{c.code}"` beside `"symbol": "{c.symbol}"` is one
|
||
currency, while `{/currency.code}` and `{/currency.symbol}` in two columns may name two. A
|
||
column the format binds to a name, as `{place as p}` binds `place`, holds that same pick, so it
|
||
agrees with the columns reading `p`; a column bound only in part, `{place.town as t}`, draws
|
||
afresh, so bind it whole and read `{p.town}`.
|
||
|
||
A reference into a category's own fields, `{/users.first}` inside `users`, is a draw of its
|
||
own, apart from the fields beside it; to share one draw, read the sibling as a field, or bind
|
||
it to a name and read the name
|
||
([Decisions](docs/decisions.md#a-category-may-reference-itself-and-a-records-fences-run-at-load)).
|
||
|
||
## Data
|
||
|
||
The shipped data comes in modules, each a Go package in the folder under [`data/`](data)
|
||
its namespace spells, a dot a slash: one per locale (`data/en_US`, `data/sv_SE`), one per
|
||
country's tree of places (`data/geo/SE`, `data/geo/US`), and the locale-neutral
|
||
`data/misc`. A package's Go name is its namespace in lower case without separators:
|
||
`svse`, `enus`, `geose`, `geous`, `misc`. Each embeds its files, so nothing is read from
|
||
disk. The CLI carries every module, and a library loads those it passes to `WithDataFS`
|
||
([Library](#library)). A directory is a namespace: each JSON file is a category named
|
||
after the file, each subdirectory a dot-path segment, so `mydata/sv_SE/person.json` is
|
||
`sv_SE.person` and replaces the shipped one.
|
||
Sources merge in order; matching folders combine, any other clash is won by the
|
||
last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"`, `/` or ` as `,
|
||
which binds a [name](#names), nor be `-`, which a struct tag reserves. A dot-prefixed
|
||
entry is never a category or a folder, so a data directory can also be a checkout, though
|
||
a category may name a hidden rows file beside it.
|
||
|
||
A source whose root holds `.fejkdata.json` with an `index` loads each category on the
|
||
first call reaching it, so `New` does not check them: load each category once in a test,
|
||
as CI does for the shipped set. Any other source loads in `New`, with the indexed
|
||
categories it reads. An indexed category reading one that another source replaced loads
|
||
on its own first reach, and fails there if the replacement breaks it. The index names
|
||
every category of its source, so a category file it leaves out never loads.
|
||
|
||
The index maps each category's dot path to an entry: `parent`, the table it links to,
|
||
which needs its own entry in the same folder; and `paths`, every path `List` advertises
|
||
below the category, `""` for the category itself. Each shipped module's manifest, such as
|
||
[`data/sv_SE/.fejkdata.json`](data/sv_SE/.fejkdata.json), carries one, so in a copy you
|
||
edit, delete its `index` and keep its `namespace`, or keep the index current.
|
||
A table whose parent column differs from its entry's `parent` fails at first reach, and a
|
||
table linking under another source's table needs a source without an index. `//go:embed`
|
||
of a directory leaves the manifest out, and the source then loads whole in `New` and at
|
||
the root, with no `namespace`; name the manifest in the pattern too.
|
||
|
||
A manifest's `namespace` is the dot path its source's root sits under: a folder holding
|
||
`person.json` with `"namespace": "sv_SE"` provides `sv_SE.person`, its subfolders adding
|
||
segments below that. It needs no index. Each shipped module carries one, so
|
||
`data/sv_SE/person.json` is `sv_SE.person`.
|
||
|
||
Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
|
||
`ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`,
|
||
`username`, `version` and `word`, formatted per locale; `sv_SE` adds `personnummer`
|
||
and `samordningsnummer`, `en_US` adds `ssn`, `itin` and `phone-exchange` (the three
|
||
digits after the area code). `en_US.phone` draws its exchange from `en_US.phone-exchange`,
|
||
which draws evenly from the codes the North American Numbering Plan Administration (NANPA)
|
||
may assign: `200` to `999`, except `211`, `311` … `911`, `555`, `950`, `958`, `959` and
|
||
`976`. `misc` carries `car`, `coordinate`, `creditcard`
|
||
(Luhn-valid), `currency` (ISO 4217), `datetime` (RFC 3339), `emoji`, `httpmethod`,
|
||
`httpstatus`, `language` (ISO 639-1, with its 639-2/T code), `loglevel` (syslog
|
||
severity), `mac`, `mimetype`, `objectid`, `port`, `protocol`, `territory` (ISO
|
||
3166-1), `timezone` (IANA), `tld` (IANA root zone), `useragent` and `uuid` (v4). Many
|
||
carry sub-fields — `misc.currency.symbol`, `misc.territory.alpha2`,
|
||
`misc.httpstatus.code` — which `--list` shows. `car`, `currency`, `httpmethod`,
|
||
`httpstatus`, `language`, `loglevel`, `mimetype`, `port`, `protocol`, `territory`,
|
||
`timezone`, `tld` and `useragent` are [tables](#table), so
|
||
`misc.territory[SE].capital` and `misc.currency[Euro].symbol` select a row; `car` and
|
||
`useragent` carry no key or name, so they are drawn from rather than selected in.
|
||
|
||
Where a script under [`data-import/`](data-import) reads a source, the facts come from
|
||
that source, and the script drops a row only by a rule it states; elsewhere a small hand-written
|
||
set ships, and [`todo.md`](todo.md) carries the step that replaces it (goal 14). Copy
|
||
that states no fact, such as `word` and `sentence`, stays hand-written. Where a category
|
||
has no path that draws only values reaching nothing real (goal 16), write a fixed value such
|
||
as `example.com` into the template; `todo.md` plans the path.
|
||
|
||
`misc.httpmethod`, `misc.protocol` and `misc.port` are IANA's registries.
|
||
`misc.httpmethod` is the eight methods RFC 9110 defines and PATCH; the register's
|
||
other entries, WebDAV and DeltaV among them, do not ship. Its `safe` and
|
||
`idempotent` are `true` or `false`, so a Go `bool` reads them, though `--format json`
|
||
and `sql` write them as text, a table column carrying no datatype. `misc.protocol`
|
||
carries a `number` and a `name`, the keyword itself where the register spells none out,
|
||
and selects by either spelling.
|
||
`misc.port` is the TCP assignments, rendering the number a port field holds with the
|
||
IANA `service` beside it, and selecting by number alone. A draw spans the whole
|
||
register, so pin `misc.port[443]` where a fixture needs a port a reader recognises;
|
||
it stops at 49150, IANA assigning nothing above, so an ephemeral source port is
|
||
`{int(49152,65535)}`.
|
||
`misc.tld` is the root zone's delegated TLDs, keyed with the leading dot
|
||
`misc.territory.tld` already carries, `.se`. Its `type` is the register's —
|
||
`country-code`, `generic`, `generic-restricted`, `infrastructure` or `sponsored` — and
|
||
its `unicode` the form the register displays, `.рф` for `.xn--p1ai` and the key itself
|
||
for an ASCII one, which selects the row too, so `misc.tld[.рф]` renders `.xn--p1ai`. A
|
||
draw spans the whole zone, `.arpa` and `.zuerich` alike, so pin `misc.tld[.com]` where
|
||
a fixture needs one a reader recognises.
|
||
`misc.loglevel` is the eight syslog severities, rendering the keyword a configuration
|
||
writes, `err` and `info`, keyed by the numerical code a PRI encodes and selectable by
|
||
either, so `misc.loglevel[3]` and `misc.loglevel[err]` are one row.
|
||
`misc.loglevel.severity` is the name RFC 5424 spells, `Error` beside `err`; it
|
||
selects nothing, the code and the keyword do.
|
||
[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
|
||
|
||
`misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for most
|
||
of them to dozens for the largest, weighted by the population GeoNames records in each,
|
||
so `misc.territory[US].timezone` draws `America/New_York` far more often than
|
||
`America/Nome`. It links to `misc.territory`, so `misc.territory[SE].timezone` is
|
||
`Europe/Stockholm` and a drawn territory and zone agree. Its `offset` is the zone's
|
||
*standard* offset, so do not pair it with a drawn `misc.datetime`: in a zone that
|
||
observes DST it is the wrong one half the year, which is what storing a zone name
|
||
avoids. A zone tzdb named recently — `Europe/Kyiv`, `America/Ciudad_Juarez` — is
|
||
rejected outright by a consumer resolving it against older tzdata, so where the
|
||
consumer validates the zone, pin one rather than draw it. There is no `UTC` row, tzdb
|
||
giving that name no territory — spell it as the text `"UTC"`. `misc.useragent` carries
|
||
`browser`, `device` and `os` beside the string, and `misc.car` a `make` and a `model`,
|
||
until an international source replaces them.
|
||
|
||
ISO 3166-1 codes territories, not sovereign states, so that is what the table is
|
||
called: Greenland and Åland have codes of their own, and `misc.territory.country`
|
||
names the state each belongs to — `DK` for Greenland, `FI` for Åland, and its own
|
||
code for a sovereign one, or for one the register names no state for.
|
||
|
||
`sex`, `first-name` and `last-name` are tables weighted by bearers, from SCB, the
|
||
SSA and the Census Bureau. `first-name` links to `sex`, so `sv_SE.sex[f].first-name`
|
||
draws a woman's name, and a name both sexes carry is a row under each, so
|
||
`en_US.sex[m].first-name[Taylor]` names the one a `first-name[Taylor]` alone cannot.
|
||
`person` binds one first name and reads its sex through it, `{.first-name as n}` and
|
||
`{n..sex.name}`, so its `first` and `sex` columns agree. `en_US.title` links to `sex`
|
||
too, so a person's prefix never contradicts it. A `personnummer`'s birth number,
|
||
`sv_SE.birth-number` under `sex`, is Skatteverket's test series, 238 for a woman and 239
|
||
for a man, which no real person is ever given.
|
||
|
||
A person of a chosen sex is assembled from the tables — `sex[f].first-name` beside
|
||
`last-name`. `person` and `personnummer` draw apart, so where a name and a number must
|
||
agree, read both through one bound `sex`, `{/sv_SE.sex[f] as s}` for a woman:
|
||
|
||
```json
|
||
{ "format": "{first} {last}, {pnr}",
|
||
"first": "{/sv_SE.sex as s}{s.first-name}",
|
||
"last": "{/sv_SE.last-name}",
|
||
"pnr": "{date(1930-01-01,2025-12-31,'060102')}-{s.birth-number.number}{luhn()}" }
|
||
```
|
||
|
||
Renders e.g. `Anne Grönberg, 011012-2389`.
|
||
|
||
Skatteverket's test series is small: a personnummer is one of about 70,000 values, a day
|
||
in 1930–2025 against the two birth numbers, so a fixture past a few hundred rows repeats
|
||
one and a `UNIQUE` column needs a category of your own. `sv_SE.date` and `en_US.date`
|
||
are uniform over 1970-01-01 to 2029-12-31, `misc.datetime` over 2000-01-01 to 2029-12-31.
|
||
What the two locales do not share: `en_US.address` carries a `region` column the Swedish
|
||
one has no use for, and `sv_SE.title` has no `parent`, so it is selected as
|
||
`sv_SE.title[dr]` rather than inside a sex.
|
||
|
||
The `geo` namespace holds one tree per country under its alpha-2 code: five
|
||
[linked tables](#linked-tables) named alike, and an `address` record binding one locality
|
||
and drawing its street and postal code inside it. The locale's `address` prints that
|
||
record, and each of its columns describes the address printed. Do not assert that a Swedish
|
||
street lies in its postort: the import counts each road segment toward the shipped postort
|
||
owning the postal code whose centre point lies nearest, so a street may sit under a
|
||
neighbouring one.
|
||
|
||
| Table | `geo.SE` | `geo.US` | Weight |
|
||
|-------|----------|----------|--------|
|
||
| `region` | län, by code or name | state, by USPS abbreviation or name; `code` is the FIPS code | population |
|
||
| `municipality` | kommun, by code or name | county, by FIPS code or name | population |
|
||
| `locality` | postort, by name; [`geo-se.py`](data-import/geo-se.py)'s `localities` picks the candidates, and one with no street is dropped | incorporated place, by GEOID or name; Hawaii has none | tätort population, the kommun's where the postort names it, else 200; place population |
|
||
| `postal-code` | postnummer with street delivery, by code | ZCTA, by code | one; address ranges |
|
||
| `street` | gatunamn, every one; [`geo-se.py`](data-import/geo-se.py)'s `street_name` drops a name that is not a street's | street name, per place the quarter with most address ranges, at least ten | segments; address ranges |
|
||
|
||
`geo.SE.region[Skåne län].municipality` draws a kommun in Skåne,
|
||
`geo.SE.locality[Lund].street` a street in Lund, and
|
||
`geo.US.region[IL].locality[Springfield]` settles which Springfield. The US places are
|
||
the incorporated places that hold most of some postal code's land: a tenth of them, drawn
|
||
with odds by population, plus every state capital but Honolulu and every place of 100,000
|
||
or more, so a smaller city may be missing: look one up in
|
||
[`locality.tsv`](data/geo/US/locality.tsv) before selecting it. A region row
|
||
carries its `timezone`, the state's predominant zone, and a locality its `lat` and
|
||
`lon`. What ports across countries is the five table names, the `name` column,
|
||
selection by name, and the `address` record's columns `street`, `street-number`,
|
||
`postal-code` and `locality`; every other column is the country's own, `code` on a
|
||
Swedish region but `abbr` on a US one.
|
||
|
||
## Library
|
||
|
||
```sh
|
||
go get github.com/larvit/fejkdata # Go 1.22+
|
||
```
|
||
|
||
```go
|
||
import (
|
||
"github.com/larvit/fejkdata"
|
||
"github.com/larvit/fejkdata/data"
|
||
)
|
||
|
||
f, err := fejkdata.New(fejkdata.WithDataFS(data.Modules()...), fejkdata.WithSeed(42))
|
||
if err != nil {
|
||
log.Fatal(err)
|
||
}
|
||
v, err := f.Fake("sv_SE.address") // "Trekungagatan 43\n442 42 Kungälv"
|
||
paths := f.List() // the paths Fake renders, sorted; direct descents only
|
||
v, err = f.FakeTemplate("name: {/sv_SE.person.last}") // compile + render in one call
|
||
t, err := f.NewTemplate(`{"format":"name: {x}","x":["bosse","lina"]}`) // compile once
|
||
v = t.Fake() // render many times, no re-parse
|
||
r, err := f.FakeRecord("users") // one record: each field a column
|
||
s := r.JSON() // {"first":"Ada","last":"Lovelace"}
|
||
r, err = f.FakeRecordTemplate(`{"format":"{x}","x":["a","b"]}`) // compile + render inline
|
||
err = f.FakeStruct(&user) // fill a struct's fake:"…" tagged fields
|
||
ok, err := fejkdata.IsTemplate(arg) // an inline template by its shape, else a path
|
||
```
|
||
|
||
| Option | |
|
||
|--------|--|
|
||
| `WithSeed(n)` | same seed, version and data: identical sequence |
|
||
| `WithDataPath(dir)` | layer a directory; repeat to layer several, the last wins a clash |
|
||
| `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `Modules()...`, or your own `embed.FS`; `WithDataFS()` loads none |
|
||
|
||
`data.Modules()` is every shipped module, and links all of them into your program. To link
|
||
only the ones you read, pass a package's `Modules()`: its `FS` and every module it reads, a
|
||
locale's country's places among them, which its package imports:
|
||
|
||
```go
|
||
import (
|
||
"github.com/larvit/fejkdata"
|
||
svse "github.com/larvit/fejkdata/data/sv_SE"
|
||
)
|
||
|
||
f, err := fejkdata.New(fejkdata.WithDataFS(svse.Modules()...))
|
||
```
|
||
|
||
Two locales are `WithDataFS(append(svse.Modules(), enus.Modules()...)...)`. To stand in your
|
||
own places, pass the locale's `FS` alone beside a tree providing `geo.SE.address`, a record
|
||
with a `format` and the columns `sv_SE.address` reads, `street`, `street-number`,
|
||
`postal-code` and `locality`: `WithDataFS(svse.FS, myPlaces)`. `New` fails where no `WithDataFS` or `WithDataPath` names the
|
||
data to load.
|
||
|
||
`New` refuses data that breaks the grammar, could mean two things or cannot render a valid
|
||
value, and `NewTemplate` refuses such a template. A category of an indexed source is
|
||
checked on the first call reaching it instead. Data that fails to load, in `New` or on
|
||
that first call, gives an error matching `ErrLoad`. On a loaded generator:
|
||
|
||
- `Fake` fails only for a path that names nothing, could name two things or reads one
|
||
draw of a level carrying a `repeat`, or for data that fails to load (`ErrLoad`), with
|
||
the same error every call.
|
||
- `FakeStruct` fails only for a non-struct argument, a type its tags do not describe, or
|
||
data that fails to load (`ErrLoad`), with the same error every call.
|
||
- `Template.Fake` cannot fail.
|
||
|
||
A path, wherever a call takes one, may start with one `/`, as a reference does.
|
||
|
||
A `*Record` carries its columns via `Columns()` — each a `Column` of `Name`,
|
||
`DataType`, rendered `Value` and `Null` — and serializes them with `JSON()`
|
||
(one object), `CSVHeader()`/`CSVLine()`, or `SQLInsert(table)` — the shapes the
|
||
CLI's `--format` writes. `FakeRecord` and `FakeRecordTemplate` take a record; a
|
||
path or template that is not one — a bare string, a choice of more than one item, or a
|
||
folder — errors.
|
||
|
||
```go
|
||
type User struct {
|
||
ID int64 `fake:"{seq()}"`
|
||
Last string `fake:"sv_SE.person.last"`
|
||
Age uint8 `fake:"{int(18,99)}"`
|
||
Nick *string `fake:"[null, \"{/sv_SE.username}\"]"`
|
||
Home Address // filled from Address's own tags
|
||
}
|
||
```
|
||
|
||
`FakeStruct` fills a struct through a pointer: each exported field tagged `fake:"…"` is
|
||
a column of one record, its tag a path or an inline template — told apart by
|
||
`IsTemplate` — and its Go type the column's
|
||
[datatype](#datatype): a string, bool, integer or float kind, or a pointer to one,
|
||
which a [`null`](#null) item leaves nil. An integer stays within int64 whatever its
|
||
kind, and a value the kind cannot hold, such as `{int(0,300)}` in a `uint8`, is refused
|
||
naming a kind that holds it. The tags are a record's columns, so they share its
|
||
[names](#names): `First` tagged `{/sv_SE.person as p}{p.first}` and `Sex` tagged `{p.sex}`
|
||
describe one person, while two path tags into `sv_SE.person` are two draws. The fields an
|
||
embedded struct promotes are columns of the same record; a named struct field, or a
|
||
pointer to one, fills from its own tags as a record of its own, so its names and picks are
|
||
its own. `fake:"-"` leaves a field unfilled.
|
||
Untagged fields keep their values, and so does a pointer back to a struct already being filled; a type
|
||
whose fields reach more than 1024 structs is refused, naming `fake:"-"` to cut it. The
|
||
first call for a type compiles its tags and reports what they get wrong, with the same
|
||
error on every later call; a `datatype` in a tag other than the one its Go type sets is
|
||
refused, naming that type.
|
||
|
||
A `*Generator` is safe for concurrent use; a seeded sequence is reproducible only
|
||
when drawn from one goroutine. Changing how a value is composed shifts the seeded
|
||
stream for that value and everything drawn after it.
|
||
|
||
## Data format
|
||
|
||
Every value is a **node**, nestable without limit:
|
||
|
||
| Node | JSON | Renders |
|
||
|------|------|---------|
|
||
| string | `"Malmö"` | its text, with any `{…}` tokens expanded |
|
||
| choice | `["a", "b", …]` | one item, picked at random |
|
||
| template | `{"format": "…", …}` | its format, with `{name}` tokens rendering the named fields |
|
||
| table | `{"format": "…", "rows": "x.tsv", …}` | its format over one row of the TSV beside it ([Table](#table)) |
|
||
| number | `5`, `1.50` | its text as written |
|
||
| boolean | `true` | its text as written |
|
||
|
||
In a record column a bare number or boolean is refused, since it could be the text `"5"`
|
||
or the typed value `{"format": "5", "datatype": "integer"}`. In a struct tag, the
|
||
field's Go type decides.
|
||
|
||
### Format string
|
||
|
||
Every character is literal except a `{…}` token:
|
||
|
||
| Token | Renders |
|
||
|-------|---------|
|
||
| `{name}` | the sibling field `name`, or the pick bound to the name ([Names](#names)) |
|
||
| `{x as name}` | nothing: binds one pick of `x`, a reference, a sibling field or a path into one, to `name` ([Names](#names)) |
|
||
| `{name.field}` | `field` of a fresh draw of the sibling `name`, or of the pick bound to it ([Correlated fields](#correlated-fields)) |
|
||
| `{a\|b}` | one of the named fields, even odds |
|
||
| `{fn(args)}` | a builtin ([Functions](#functions)) |
|
||
| `{/path}`, `{.path}`, `{..path}` | a node reached from the data root, this file's folder, or the folder above ([References](#references)) |
|
||
| `{{`, `}}` | a literal `{` or `}` |
|
||
|
||
Each `{…}` draws afresh, so `{w} {w}`, `{p.a} {p.b}` and `{/person.first} {/person.last}`
|
||
are two draws each. Only a [name](#names) keeps one pick for every read of it.
|
||
|
||
```json
|
||
"100 Main St, Apt {int(1,9)}{upper(1)} — tel {digits(3)}-{digits(4)}"
|
||
```
|
||
|
||
Renders e.g. `100 Main St, Apt 4B — tel 555-0199`. Random characters come from
|
||
the sample functions, so a phone pattern is `070-{digits(3)} {digits(2)} {digits(2)}`.
|
||
A lone `}` is a load error naming `}}`. A repeated arm counts as written, so
|
||
`{a|a|b}` draws `a` two times in three.
|
||
|
||
### Weight
|
||
|
||
An item of a choice may carry a `weight` (default `1`) to skew its odds:
|
||
|
||
```json
|
||
[
|
||
{ "format": "070-{digits(3)} {digits(2)} {digits(2)}", "weight": 10 },
|
||
"08-{digits(3)} {digits(2)} {digits(2)}",
|
||
{ "format": "010-{digits(3)} {digits(2)} {digits(2)}", "weight": 2 }
|
||
]
|
||
```
|
||
|
||
Renders `070-412 38 91` ten times as often as `08-…`. A string item is weighted by
|
||
writing it as `{ "format": "AB", "weight": 3 }`. A repeated item counts as written,
|
||
so `["a", "a", "b"]` draws `a` two times in three, and an item of weight `0` is never
|
||
drawn. A weight outside a choice does nothing, and so does one above `0` on the item of
|
||
a one-item choice. Rejected at load: a weight that is negative, not a number, or too
|
||
close to `0` to tell from it, such as `1e-400`; and a choice whose every weight is `0`, a one-item choice of weight `0` among them.
|
||
|
||
### Repeat
|
||
|
||
A template may carry `repeat` (an integer of 0 or more, default `1`; `0` renders nothing) to render its format
|
||
that many times — each an independent draw — joined by `separator` (default `""`):
|
||
|
||
```json
|
||
{ "format": "{word}", "repeat": 3, "separator": " ", "word": ["foo", "bar", "baz"] }
|
||
```
|
||
|
||
Renders e.g. `bar foo baz`. A `separator` without a `repeat` above `1` does nothing.
|
||
Rejected at load: a `repeat` that multiplies to more than 1 048 576 renders along any
|
||
path of nested repeats.
|
||
|
||
### Datatype
|
||
|
||
A record column may declare `datatype` — `integer`, `number` or `boolean` — so `json`
|
||
writes `42` rather than `"42"` and `sql` a bare literal; a column without one, or with
|
||
`string`, is a string:
|
||
|
||
```json
|
||
{ "format": "",
|
||
"id": { "format": "{seq()}", "datatype": "integer" },
|
||
"paid": { "format": "{p}", "p": ["true", "false"], "datatype": "boolean" },
|
||
"total": { "format": "{calc(net * qty, 2)}", "net": ["19.99", "5.00"], "qty": ["3", "7"], "datatype": "number" } }
|
||
```
|
||
|
||
Writes e.g. `{"id":1,"paid":true,"total":59.97}`. A column is a field of the top-level
|
||
template, or an item of a choice standing in for one; `datatype` anywhere else does
|
||
nothing. A typed column holds one value, alone in its format: a
|
||
literal, one `{int()}`, `{float()}`, `{seq()}` or `{calc()}` call, or a read that lands only on such
|
||
values. `integer` is an int64 written `-?(0|[1-9][0-9]*)` — `{float()}` prints one at
|
||
`0` decimals within int64 — `number` a JSON number, `boolean` `true` or `false`. A value its
|
||
datatype cannot hold is a load error naming it:
|
||
|
||
```text
|
||
order.id: datatype integer: {digits(3)} prints text, not an integer
|
||
order.id: datatype integer: "1{digits(2)}" is not one value; write one literal or one {float()}, {int()}, {seq()} or {calc()}, or read one
|
||
```
|
||
|
||
A column whose format is one read of a record's column and nothing else —
|
||
`"score": "{/src.score}"`, `"score": "{s.score}"` with `s` bound to `/src`, or `"a": "{p}"`
|
||
beside `{place as p}` in its own record — is that column: it takes the column's datatype and
|
||
is null where the column is, and a struct field tagged `src.score` is nil there. A
|
||
`datatype` of its own types the column's values where they prove it, and `"string"`
|
||
reads them as text. Any other read renders the column's text, a null as `""`.
|
||
|
||
A typed column's `{calc()}` must be proven to print a number: each operand a number
|
||
literal, an `{int()}`, `{float()}`, `{seq()}` or `{digits()}` call, a calc, or a read of
|
||
such values, whose bounds keep every divisor from zero and the result within `1e300`.
|
||
What the bounds cannot show is refused — `{calc(a / b)}: divides by b, which is not
|
||
proven nonzero`. The calc fills an `integer` column at `0` decimals, or over whole
|
||
operands with no `/`, while its bounds stay within int64.
|
||
|
||
### Null
|
||
|
||
A `null` item draws a record column as null: `json` writes `null`, `sql` `NULL`, and
|
||
`csv` an empty field, with an empty string written `""` — the convention PostgreSQL's
|
||
`COPY … CSV` reads. A record of one null column is a blank line, which `COPY` reads as
|
||
null but most CSV readers skip, so write such a record as `json` or `sql`. `Fake`
|
||
renders a null as `""`. The other items' weights skew its odds:
|
||
|
||
```json
|
||
{ "format": "", "deleted_at": null, "middle": [null, { "format": "{n}", "n": ["Ann", "Eva"], "weight": 3 }] }
|
||
```
|
||
|
||
`deleted_at` is null every draw, `middle` a name three draws in four. Anywhere but a
|
||
column, `null` renders `""`. Rejected at load: a column whose items hold different
|
||
datatypes.
|
||
|
||
### Table
|
||
|
||
A table is a category whose rows come from a TSV beside its JSON file: the header
|
||
names the columns, each line below it is one row, and the format renders the row
|
||
drawn. Save `mydata/country.tsv` and `mydata/country.json`:
|
||
|
||
```tsv
|
||
alpha2 name population
|
||
DK Denmark 5900000
|
||
NO Norway 5500000
|
||
SE Sweden 10500000
|
||
```
|
||
|
||
```json
|
||
{ "format": "{name} ({alpha2})", "rows": "country.tsv", "key": "alpha2", "name": "name", "weight": "population" }
|
||
```
|
||
|
||
```sh
|
||
echo '{/country}' | fejkdata -d ./mydata # Sweden (SE), about half the time
|
||
echo '{/country.alpha2}' | fejkdata -d ./mydata # NO
|
||
echo '{/country[SE]}' | fejkdata -d ./mydata # Sweden (SE)
|
||
echo '{/country[Norway].alpha2}' | fejkdata -d ./mydata # NO
|
||
echo '{/country}' | fejkdata -d ./mydata --format csv # alpha2,name,population → SE,Sweden,10500000
|
||
```
|
||
|
||
`rows` names the TSV beside the category file; `key` names the column a path
|
||
selects a row by, `name` a column it also selects by, `weight` a column of numbers
|
||
of 0 or more that skews the draw, and `parent` the table a column links to
|
||
([Linked tables](#linked-tables)). The format's `{tokens}` read the columns, and the
|
||
columns are the [record](#records)'s columns, so `--format csv` writes the rows and
|
||
`--list` shows `country.alpha2`. A cell is a string node: `1{digits(2)} {digits(2)}`
|
||
in a cell draws digits and `{/misc.uuid}` reads a reference, while `{name}` in a cell
|
||
is refused, since a cell has no sibling. Each cell may select its own row of another
|
||
table, `{/misc.currency[SEK].symbol}` on one row and `{/misc.currency[EUR].symbol}`
|
||
on the next, since only one row renders. `New` proves the header, the options and every
|
||
cell token, and refuses:
|
||
|
||
- a JSON member other than `format`, `rows`, `key`, `name`, `weight` and `parent`, which
|
||
could be meant as a field or as an option a table does not take;
|
||
- a key cell that is empty or repeats;
|
||
- a weight that is negative, not a number, or too close to `0` to tell from it;
|
||
- every weight `0`, across the table or inside one parent row;
|
||
- a key or name holding `[`, `]`, `{`, `}`, `"` or `|`, which a selector cannot spell.
|
||
|
||
`New` maps the keys, and a name or parent column is mapped on the first draw that selects
|
||
by name or descends through the table. A `name` needs a `key`, since a name naming several rows is reported by
|
||
their keys, or a `parent`, inside whose row a name names one row, so `first-name[Kim]`
|
||
is settled by the `sex` selected before it and a name repeating inside one parent row
|
||
is refused; a name spelling another row's key is refused, since the key would
|
||
select first and the name never. A table's options are its own, so a column may be named
|
||
`name`, as one usually is.
|
||
|
||
### Row selection
|
||
|
||
`[key]` or `[name]` after a table's name selects one row: `misc.territory[SE]` and
|
||
`misc.territory[Sweden]` name one row, and `misc.territory[SE].capital` reads its column.
|
||
A name naming several rows is an error listing their keys, unless a row selected
|
||
before it settles which ([Linked tables](#linked-tables)). A selector is part of the path, so it works
|
||
wherever a path does: `Fake`, `FakeRecord`, a `{/misc.territory[SE].capital}` reference
|
||
and a struct tag. A dot inside the brackets belongs to the key or name, so
|
||
`city[St. Louis]` selects it.
|
||
|
||
### Linked tables
|
||
|
||
A table's `parent` names a column and, by the same name, the table beside it that the
|
||
column links to by key. Save `mydata/city.tsv` and `mydata/city.json` beside the
|
||
`country` table above:
|
||
|
||
```tsv
|
||
name country population
|
||
Copenhagen DK 660000
|
||
Göteborg SE 600000
|
||
Oslo NO 710000
|
||
Stockholm SE 990000
|
||
```
|
||
|
||
```json
|
||
{ "format": "{name}", "rows": "city.tsv", "key": "name", "parent": "country", "weight": "population" }
|
||
```
|
||
|
||
```sh
|
||
echo '{/country[SE].city}' | fejkdata -d ./mydata # Stockholm or Göteborg
|
||
echo '{/country.city.name}' | fejkdata -d ./mydata # a country drawn, then a city inside it
|
||
echo '{/city[Oslo].country}' | fejkdata -d ./mydata # NO — the link column's cell
|
||
echo '{/city as c}{c.name}, {c..country.name}' | fejkdata -d ./mydata # Oslo, Norway — one city and its country
|
||
```
|
||
|
||
A path descends from a row to a linked table by name, at any depth, and `--list`
|
||
advertises each direct step. One path reads one chain of rows: `country.city.name` draws a
|
||
country by its weights, then a city inside it by theirs. Two paths are two draws, so
|
||
`{/city.name}, {/country.name}` may name a city and another country. The reads of one
|
||
[name](#names) agree instead, and the table it binds decides whose weights govern:
|
||
`{/city as c}` draws a city by city weight and `{c..country.name}` is its country, while
|
||
`{/country as k}` draws a country by country weight and `{k.city.name}` a city inside it.
|
||
A name keeps every row its reads draw, so `{k.city.name}` and `{k.city.population}` describe
|
||
one city, and `{k.name}` its country.
|
||
|
||
`New` refuses a link cell that is no key of the parent, a parent row no child links to, a
|
||
chain of parents that closes, and a table named like a column of any table above it.
|
||
|
||
### Step up
|
||
|
||
`..` after a row steps up to the row its link column names. Name the parent table right
|
||
after the `..`:
|
||
|
||
```sh
|
||
echo '{/city[Oslo]..country.name}' | fejkdata -d ./mydata # Norway
|
||
echo '{/city..country}' | fejkdata -d ./mydata # a city drawn, then its country: Sweden (SE)
|
||
echo '{/city[Göteborg]..country.city}' | fejkdata -d ./mydata # Stockholm or Göteborg, drawn afresh
|
||
echo '{/city as c}{c}, {c..country.name}' | fejkdata -d ./mydata # Oslo, Norway
|
||
```
|
||
|
||
It steps up from a selected row, a drawn row or a [name](#names) bound to a row, one
|
||
level per `..`: `geo.SE.locality..municipality..region.name`. `.country` after a row
|
||
still reads the link column's cell, `NO`.
|
||
|
||
After a `..`, `.` steps down again and draws afresh inside the row stepped up to, so
|
||
`city[Göteborg]..country.city` may draw Göteborg again. Under a name, the paths stepping
|
||
down from one `..` read one draw: `{c..country.city.name}` and
|
||
`{c..country.city.population}` describe one city.
|
||
|
||
A `..` that names anything but the parent table is an error naming the parent. A selector
|
||
after a `..` is an error naming the table to select from, since it could name a row outside
|
||
the one stepped up to. `--list` advertises no path with a `..`. At the start of a
|
||
reference, `..` is the folder above ([References](#references)).
|
||
|
||
### Options and fields
|
||
|
||
`format`, `weight`, `repeat`, `separator` and `datatype` are the only options;
|
||
**any other key is a field** (see [Decisions](docs/decisions.md#options-and-fields-share-one-namespace)), and `rows` makes a category
|
||
a [table](#table), so no template carries a field of that name. A [name](#names) shares the
|
||
namespace too.
|
||
|
||
### Functions
|
||
|
||
A `{name(args)}` token calls a builtin. Arguments are checked at `New`: a bad
|
||
count, range, country or expression fails fast; an integer may carry a sign or leading
|
||
zeros (`+5` and `05` are `5`); a count of `0` renders nothing; bounds are finite; and a length, count or decimal place beyond a sane maximum is rejected, so a
|
||
fat-fingered `hex(2000000000)` never tries to allocate gigabytes. Every builtin draws only from the seed — a
|
||
time-based id takes its timestamp from the rng, not the clock — so seeded output
|
||
stays reproducible.
|
||
|
||
| Function | Kind | Renders |
|
||
|----------|------|---------|
|
||
| `{luhn()}` | derivation | Luhn check digit over the digits emitted so far in this expansion |
|
||
| `{mod11()}` | derivation | weighted mod-11 check char (weights 2–7 from the right); `X` for 10 |
|
||
| `{ean()}` | derivation | EAN-13 / UPC-A / ISBN-13 / GTIN check digit |
|
||
| `{digits(n)}` | sample | `n` digits 0–9 |
|
||
| `{upper(n)}`, `{lower(n)}` | sample | `n` letters A–Z, a–z |
|
||
| `{int(min,max)}` | sample | uniform integer in `[min, max]` |
|
||
| `{float(min,max,dp)}` | sample | number in `[min, max]` with `dp` decimals |
|
||
| `{hex(n)}` | sample | `n` lowercase hex digits |
|
||
| `{base64(n)}` | sample | `n` random bytes, base64 |
|
||
| `{uuid()}` | sample | UUID v7 (`misc.uuid` ships v4 as data) |
|
||
| `{ulid()}` | sample | ULID, 26 Crockford base32 chars |
|
||
| `{nanoid(n)}` | sample | URL-safe Nano ID, `n` chars |
|
||
| `{iban(CC)}` | sample | length- and mod-97-valid IBAN for BE, DE, DK, ES, FI, NO or SE |
|
||
| `{date(from,to,'layout')}` | sample | a second between two `YYYY-MM-DD` days, both included, in a quoted Go layout: `'2006-01-02'`, `'January 2, 2006'`, `'060102'`, `'2006-01-02T15:04:05Z'` |
|
||
| `{time('layout')}` | sample | a second within a day: `'15:04'`, `'3:04 PM'` |
|
||
| `{seq()}`, `{seq(name)}` | counter | next integer from 1 in this generator; `name` selects an independent counter |
|
||
| `{calc(expr)}`, `{calc(expr,dp)}` | computation | an arithmetic expression over sibling fields and names ([Computation](#computation)) |
|
||
| `{lowercase(x)}`, `{uppercase(x)}`, `{ascii(x)}` | transform | a field's value rewritten ([Transforms](#transforms)) |
|
||
|
||
A derivation reads what is to its left, so place it after its payload; the
|
||
buffer is per expansion, so a nested template keeps fixed parts out of the sum. A
|
||
Swedish personnummer is a Luhn checksum over the nine digits before it, six of
|
||
them a birthdate:
|
||
|
||
```json
|
||
{ "format": "{date(1930-01-01,2010-12-31,'060102')}-{birth}{luhn()}", "birth": ["238", "239"] }
|
||
```
|
||
|
||
Renders e.g. `811218-2389`. A layout is Go's: the reference time `Mon Jan 2
|
||
15:04:05 MST 2006` spelled as the output should look, quoted, since a layout may
|
||
carry the comma that separates arguments, with English names. Every second
|
||
between the two days is reachable, so a layout with a clock draws the time too, and
|
||
`from` may equal `to`, which is that one day. The instant is UTC, so a zone in the
|
||
layout prints `UTC` or `Z`.
|
||
The quotes delimit a layout outside a selector only, so `[O'Fallon]` in an
|
||
argument stays a name. A layout naming no field prints itself, `date` with a clock-only
|
||
layout prints the drawn instant's clock, and `date` over one day with no clock field prints
|
||
that day.
|
||
Rejected at `New`: a bound that is no calendar date, or after the other; an unquoted
|
||
or double-quoted layout, naming the single-quoted one; and `time` with a layout naming a
|
||
date field, naming `date`, since `time` draws no date. `{seq()}` spans `Fake`
|
||
calls and `repeat`, resets with a new generator, and is the natural primary key for
|
||
the SQL example above.
|
||
|
||
### Computation
|
||
|
||
`{calc(expr)}` evaluates `+ - * /`, parentheses and unary minus over number
|
||
literals, sibling fields and [names](#names), each rendered then read as a number;
|
||
a second argument rounds to that many decimals. A hyphen is always subtraction, so a
|
||
hyphenated field or name can't be an operand.
|
||
|
||
A calc computes from the values its format shows where both read one [name](#names):
|
||
|
||
```json
|
||
{ "format": "{net as n}{qty as q}{n} x {q} = {calc(n * q, 2)}", "net": ["19.99", "5.00"], "qty": ["3", "7"] }
|
||
```
|
||
|
||
Renders e.g. `19.99 x 3 = 59.97`.
|
||
|
||
A result that rounds to zero prints unsigned — `0`, `0.00` — as `{float()}`'s does. An
|
||
operand that can never be a number (`"abc"`, or a choice of such) is rejected at load, as
|
||
is a division by a constant zero (`1/0`, or a fixed `"0"` field); an operand that sometimes is not a number yields
|
||
`NaN`, and a division by one that is not constant `Inf` — both print rather than
|
||
fail, except in a [typed column](#datatype), which must prove neither happens.
|
||
|
||
### Transforms
|
||
|
||
`{lowercase(x)}`, `{uppercase(x)}` and `{ascii(x)}` rewrite the value of `x` — a
|
||
field, a path, a `..path` or a [name](#names) — and nest. `ascii` folds Latin letters (`Åsa Öberg`
|
||
→ `Asa Oberg`) and drops any other non-ASCII rune. An email
|
||
matches the name beside it where both read one name:
|
||
|
||
```json
|
||
{ "format": "{person as p}{p.first} {p.last} <{lowercase(ascii(p.first))}.{lowercase(ascii(p.last))}@example.com>",
|
||
"person": [
|
||
{ "format": "{first} {last}", "first": "Åsa", "last": "Öberg" },
|
||
{ "format": "{first} {last}", "first": "Bo", "last": "Ek" }
|
||
] }
|
||
```
|
||
|
||
Renders `Åsa Öberg <asa.oberg@example.com>` or `Bo Ek <bo.ek@example.com>`, never
|
||
a mix.
|
||
|
||
### References
|
||
|
||
A reference renders a node from elsewhere in the data — the path `Fake` takes,
|
||
across every loaded source — so one category borrows another. The sigil says
|
||
where the path starts, as in a filesystem: `{/en_US.person}` from the data root,
|
||
`{.username}` from the folder this file sits in, `{..username}` from the folder
|
||
above. `sv_SE.email` can therefore read its own locale's `username`
|
||
without naming `sv_SE`:
|
||
|
||
```json
|
||
"Hej, {/en_US.person}!"
|
||
```
|
||
|
||
Renders e.g. `Hej, Pat Smith!`. Each reference draws afresh ([Format string](#format-string)),
|
||
so `{.person.first} {.person.last}` may name two people. Bind the person to a [name](#names) to read one: `{.person as p}{p.first}
|
||
{p.last}`. Rejected at `New`: a path that is
|
||
unknown, names a folder, has no folder above, reads a field not every variant
|
||
of a choice carries, and a reference that leads back to its own value, directly,
|
||
mutually or through a chain.
|
||
|
||
### Names
|
||
|
||
`{x as n}` binds one pick of `x`, a reference, a sibling field or a path into one, to the name
|
||
`n` and prints nothing. `{n}` then
|
||
prints that pick, and `{n.path}` reads a path through it, so every read of `n` describes one
|
||
row:
|
||
|
||
```json
|
||
"{/misc.territory as t}{t.flag} {t}: capital {t.capital}, calling code +{t.calling-code}"
|
||
```
|
||
|
||
Renders e.g. `🇳🇴 Norway: capital Oslo, calling code +47`. A name keeps what each `{n.path}`
|
||
lands on and every level it passes, so `{n}` and `{n.path}` agree, and so does a
|
||
[transform](#transforms) of `n.path`. Everything else under the name draws as it would
|
||
anywhere: if `n`'s category has the format `{w}-{w}`, it draws `w` twice, and `{n.w}` beside
|
||
`{n}` is refused, since it could read either draw. A sibling field is bound the same way, so
|
||
`{place as p}{p.postal-code} {p.locality}` reads one place ([Correlated fields](#correlated-fields)).
|
||
|
||
A name is a pick of its own: `{t.name}` and `{/misc.territory.name}` beside it are two draws,
|
||
and so are two names bound to one category.
|
||
|
||
A name is drawn on its first read, and lives in the category binding it: any field of the
|
||
category may read it, and a record's columns read one pick. Each render of the category picks
|
||
anew, and so does each bare reference to it. A name a `repeat` binds picks again on every
|
||
iteration, and one bound outside the repeat keeps its pick on every line. A name a choice's
|
||
item binds lives in that item, and picks anew each time the item is drawn. A category a
|
||
reference enters does not see the name; bind it in the category that reads it.
|
||
|
||
Refused at `New`, each error naming what to write instead:
|
||
|
||
- a name bound twice outside any `repeat`, twice in one `repeat`, or both inside a `repeat`
|
||
and outside it;
|
||
- a name that is a field or an option;
|
||
- a binding of anything but a reference, or a field of the template binding it or a path into
|
||
one;
|
||
- a read of a name inside the field bound to it: read it outside that field;
|
||
- reading a name outside the choice item that binds it, since the other items leave it
|
||
unbound;
|
||
- a path through a name that selects a row: `{r.municipality[0180]}` could mean municipality
|
||
0180 whatever region `r` picked, or 0180 only where `r` picked the region holding it;
|
||
- `{n}` beside `{n.w}` where `n`'s category reads `w` twice, as in `{w}-{w}` or
|
||
`{w.a} {w.b}`;
|
||
- a binding in a table's format or cell.
|
||
|
||
### Correlated fields
|
||
|
||
`{name.field}` reads a path into a sibling. Several tokens read one draw of it through a
|
||
[name](#names): bind the sibling, and every read of
|
||
the name reads that one row — a locality and the postal code that really covers it:
|
||
|
||
```json
|
||
{ "format": "{place as p}{street} {int(1,99)}\n{p.postal-code} {p.locality}",
|
||
"street": ["Kungsgatan", "Storgatan"],
|
||
"place": [
|
||
{ "format": "{locality}", "locality": "Stockholm", "postal-code": "1{digits(2)} {digits(2)}", "weight": 975 },
|
||
{ "format": "{locality}", "locality": "Tranås", "postal-code": "573 {digits(2)}", "weight": 18 }
|
||
] }
|
||
```
|
||
|
||
Renders e.g. `Kungsgatan 35` / `176 99 Stockholm`, never a Stockholm code beside
|
||
Tranås; each row's `weight` says how often it appears. `{p.geo.town.name}
|
||
{p.geo.town.zip}` share the town too, as a name keeps every level its reads pass. Every variant of a choice
|
||
on the path must carry the rest of it, so a row missing a field is named at load:
|
||
|
||
```text
|
||
token {p.postal-code}: not every variant of this 2-way choice carries "postal-code"; all carry [locality]
|
||
```
|
||
|
||
The sub-fields stay addressable — `Fake("address.place.locality")` renders, and
|
||
`List` advertises it. A path may not read into a level carrying a `repeat`, and `List`
|
||
advertises no such path; read that level whole.
|
||
|
||
### Performance
|
||
|
||
The target is about a microsecond per value (goal 13).
|
||
|
||
A source with an [index](#data), as the shipped set has, loads nothing in `New`: a
|
||
category loads on the first `Fake`, `FakeRecord`, `FakeStruct` or template that reads
|
||
it, with every category it reads and its whole table family. Each file is parsed,
|
||
validated and weight-indexed once, bar a table's name and parent columns, which are
|
||
mapped on the first `Fake` that selects by name or descends through it. A `Fake` call then
|
||
costs about what its output costs: an unweighted pick is O(1) whatever the list's length,
|
||
a weighted one O(log n), and long formats, deep nesting and many tokens add cost in
|
||
proportion to the output.
|
||
|
||
## Versioning
|
||
|
||
Semver tags on `main`, `v0.1.0` first; one version covers the shipped data, the
|
||
library and the CLI, and [`CHANGELOG.md`](CHANGELOG.md) names what each release
|
||
changed. A consumer's data, code and scripts keep working across a minor or a patch. A
|
||
minor only adds, and a major is the only release that changes what exists. Seeded output
|
||
is the exception: any release, a patch too, may change what a seed draws. Where you
|
||
compare seeded output, install one exact version
|
||
(`go install github.com/larvit/fejkdata/cmd/fejkdata@vX.Y.Z`; `go.mod` already pins one),
|
||
and regenerate seeded fixtures and expected values when you raise that version.
|
||
|
||
| Surface | Major | Minor |
|
||
|---------|-------|-------|
|
||
| Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another; change a table's key, name, weight or parent column, or remove a row | a path outside a record's columns, a locale, a value in a list, a row |
|
||
| Records (a table is one) | remove, rename, retype or add a column; let a column be null | a record, as a new category |
|
||
| Data format | a fence: a spelling a load rejects that it accepted, in a category or a manifest; a template option, since it reserves a field name | a builtin |
|
||
| CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format |
|
||
| Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option |
|
||
| Data packages | rename or remove a package; move a category to another package; make a package read another it did not | a package |
|
||
|
||
A patch changes no row of this table: performance, docs, or a fix inside a promised
|
||
behaviour that adds or removes no value and changes no path, format or spelling.
|
||
|
||
An error's wording may improve in a minor; the path, rejected spelling and replacement
|
||
it names may not.
|
||
|
||
Before `v1.0.0` a minor is the breaking unit: `0.(x+1).0` may carry a major's
|
||
changes, each named in the changelog, and a `0.x.y` patch may not. `v1.0.0` is
|
||
cut once the shipped data is in its record shape and one full minor has shipped
|
||
with no breaking change. From `v2` the module path carries `/vN`, so fences ship
|
||
batched into as few majors as possible.
|
||
|
||
[`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) pins every path, each
|
||
template category's format, the categories each category reads, each column's
|
||
datatype and nullability, and each table's key, name, weight and parent columns; a pull request that
|
||
changes it or `data/` adds its `CHANGELOG.md` entry, which CI checks. A removed,
|
||
renamed or retyped line is a major.
|
||
|
||
## Audience
|
||
|
||
App developers writing tests and fixtures, in Go and at a shell:
|
||
|
||
- a **bulk fixture author**, millions of rows into CSV or SQL
|
||
- a **Go test author**, filling a struct with `FakeStruct`
|
||
- a **hand fixture author**, a few values at a shell
|
||
- a **validator-facing author**, who needs a value a real checker accepts
|
||
- a **data author**, who writes categories under `--data-path` and meets every load
|
||
error the fences raise
|
||
- a **custom-CLI builder**, who ships a CLI like `fejkdata` with data and functions of
|
||
their own
|
||
|
||
and a **contributor**, who reads [`todo.md`](todo.md), [`AGENTS.md`](AGENTS.md) and
|
||
the Development section below, and who ships a register the six above then draw from.
|
||
|
||
## Goals
|
||
|
||
Where the project is heading; the sections before Audience document what ships today.
|
||
|
||
1. **The best fake-data tool there is, and the one developers reach for.**
|
||
1. You can install it the way you install your other tools, and people hear about
|
||
it where developers read.
|
||
2. **Enough data ships built in to fill a real test database: the CLI carries all of it, a
|
||
library imports what it needs, and nothing is fetched at run time.**
|
||
1. It covers the ten most-spoken languages and the Nordic countries.
|
||
2. Each locale covers the common categories: names, addresses, phone numbers, ids,
|
||
companies and dates.
|
||
3. The shipped data comes in Go packages, and a program links only the packages it
|
||
imports.
|
||
3. **The code is easy to understand.**
|
||
1. You can go from a bug to the code behind it without asking anyone.
|
||
2. You can change one part without keeping the rest in your head.
|
||
3. Every name says what the thing really is.
|
||
4. Each file makes sense on its own, without another document open beside it.
|
||
4. **Every value is valid.** It passes the same check the real system would apply,
|
||
such as the checksum in a Swedish personal number.
|
||
1. Values that belong together come from the same random pick. You choose which
|
||
ones by giving that pick a name.
|
||
5. **Templates are easy to write and read.** A template is the text you write to
|
||
describe a value, such as `{first} {last}`.
|
||
1. Everything outside `{…}` prints exactly as written.
|
||
2. Each `{…}` makes a new random pick.
|
||
3. A name keeps one pick. `{/person as p}` picks a person and prints nothing.
|
||
`{p}` then prints that person, and `{p.last}` their last name.
|
||
4. A path works like a folder path: `.` goes down a level, and `..` goes up one.
|
||
5. `[SE]` selects one row of a table, and means that row wherever you write it.
|
||
6. If two people could read a template differently, loading it fails, and the
|
||
error shows how to write each meaning.
|
||
7. Whatever is well-formed and means one thing loads, even where part of it does
|
||
nothing. fejkdata does not guess at mistakes.
|
||
8. When a clearer name turns up, it replaces the old one, even if that needs a new
|
||
major version.
|
||
9. Everything fejkdata renders is written as a template: on the command line, in a
|
||
struct tag and in the library.
|
||
6. **Errors surface when the data loads, never mid-render, and say plainly what is
|
||
wrong.**
|
||
1. A template that could render a value its own rules forbid, such as `NaN` or `Inf`,
|
||
fails to load instead.
|
||
2. Data that reads something nobody provided fails to load, and the error names the
|
||
module that provides it by default, or says that none does.
|
||
3. A module from fejkdata is proven whole before it ships; a module from anyone else
|
||
may load on first reach, and its author proves it.
|
||
7. **One command line gets you a value.**
|
||
1. Flags work like in other command-line tools, and can go anywhere on the line.
|
||
2. Your first template of your own needs no escaping and no options.
|
||
3. The CLI fills in what you leave out with a sane default: all the shipped data, the
|
||
system's randomness and today's date.
|
||
8. **Human readable and editable data formats** — source data is saved in JSON or
|
||
TSV formats when possible.
|
||
9. **Flexible and extensible.**
|
||
1. Data and functions come in modules, a Go package or a data folder, from fejkdata
|
||
or anyone else, and fejkdata's load as anyone else's do.
|
||
2. A module names the modules it reads by default, and anything providing the same
|
||
paths can stand in for one.
|
||
3. Loading two things under one name fails, in the library and the CLI alike, unless
|
||
you say the second replaces the first, and then every read of that name reads
|
||
the second.
|
||
4. The builtins ship with fejkdata itself, so a template means the same in every
|
||
program.
|
||
5. A function loaded from a module can ask for the input it needs, and gets the
|
||
randomness and the date the generator uses.
|
||
6. A CLI carrying your own modules is a few lines of Go, and works just like
|
||
`fejkdata`.
|
||
10. **The same seed, date, data and versions always give the same output.**
|
||
1. In the library, the caller supplies everything that can change the output: data,
|
||
seed and date.
|
||
11. **No unnecessary runtime dependencies** — use standard library by default.
|
||
12. **The docs are a map of the template language.**
|
||
1. Every feature has its own heading.
|
||
2. Every example runs as a test and shows its output.
|
||
3. Each rule is stated in one place.
|
||
13. **It is fast enough that you never notice it with the shipped data.**
|
||
1. A render's cost grows only with the data it reaches.
|
||
14. **Strongly prioritize data sourced from good online sources**
|
||
15. **The most common data comes first,** such as names, addresses, phone numbers and
|
||
dates.
|
||
16. **Values look real by default. You can also ask for values that can never reach
|
||
anything real,** such as a domain nobody can register.
|
||
|
||
## Layout
|
||
|
||
```
|
||
doc.go the package doc, and the vocabulary the package is written in
|
||
fejkdata.go Generator, New, options, List
|
||
index.go a source's index: its categories left unloaded, and loading one on the first call reaching it
|
||
node.go the node model and JSON -> node compilation
|
||
table.go tables: their options, format and cells, the link to the parent beside them, and how a path passes and draws a table
|
||
name.go names: {x as n} bindings, the scopes they live in, the reads they answer, and each binding's target
|
||
namefence.go the load fences over names
|
||
pick.go a render's named picks: each level's pick key, the frame each name scope renders in, the memo a pick keeps, and the reads kept under it
|
||
env.go renderEnv: what a render reads names and rows through, the stack of frames it renders in, and the frames a read opens and finds
|
||
path.go the dotted-path walks: the paths List advertises, the sub-paths every variant of a choice carries, proving a path reaches a node, and drawing the path
|
||
render.go Fake, the recursive renderer (choices, format strings, expansions) and the reads its tokens make
|
||
record.go records: Record, the JSON/CSV/SQL serializers, and their entry points
|
||
struct.go structs: FakeStruct, fake tags as one record's columns, and a field's Go type as its column's datatype
|
||
inline.go inline templates: Template, NewTemplate, FakeTemplate, IsTemplate, and their compile and resolve
|
||
format.go a format's tokens checked against their fields, and compiled into ops
|
||
arm.go the compiled read of one {a|b} alternative or operand: its head, the path to its leaf, and the op holding it
|
||
reference.go one reference's resolution: its sigil, its category, its tail
|
||
resolve.go resolving templates against the assembled tree in steps, each over every template before the next
|
||
graph.go the node graph: tree walks, containment and render edges, and the cycle and repeat fences
|
||
sorted.go sortedNames: a map's keys, sorted
|
||
calc.go the {calc()} operand checks: never a number, and a constant zero divisor
|
||
datatype.go column datatypes: DataType, a template's declared datatype, a column's datatype, and a format that only reads a column
|
||
valueproof.go the value proof: what a typed column, calc operand or struct field holds, checked at load
|
||
data.go the load path: the categories internal/datafiles hands over, compiled into one merged tree, and the load pipeline every entry point runs
|
||
cmd/fejkdata/ the fejkdata CLI
|
||
internal/builtinfunc/ the {name()} functions: their checks and draws, checksums, date and time layouts, transforms and calc arithmetic
|
||
internal/datafiles/ the walk of a data tree: its folders, its category files and the rows files beside them, and its manifest
|
||
internal/datatype/ the datatype a record column holds; the root's DataType is an alias of it
|
||
internal/drawstate/ the seeded randomness and the {seq()} counters a generator draws through
|
||
internal/grammar/ how the template language is written: format tokens, paths and selectors, reference sigils, identifiers, calc syntax, and whether a string is a template or a path
|
||
internal/invariant/ the one phrase every package panics with when an invariant breaks
|
||
internal/jsonvalue/ a decoded JSON value, its numbers kept as written, the one input a template compiles from, and the kind an error names it by
|
||
internal/proven/ what a proof knows of a value, and the bounds a calc takes from its operands
|
||
internal/rows/ a table's rows: the TSV, the options proved over it, the links between tables, row selection and draws, and the pin set one path or one named pick fixes
|
||
data/ the shipped modules, each a Go package embedding its categories (JSON, and a TSV per table) and its manifest; and Modules, which lists them
|
||
data-import/ the scripts that rebuild each sourced table (see DATA-LICENSES.md), and the modules they share
|
||
docs/ the decision log, the register research behind the shipped data, the survey of other libraries, panels' scores and rulings on how readable the code is, and what each chunk cost an agent to read
|
||
release-tooling/ the release CI publishes from the changelog heading
|
||
testdata/ the pinned shipped shape (see Versioning)
|
||
```
|
||
|
||
## Development
|
||
|
||
Everything runs in Docker — **no local tooling beyond Docker is needed**.
|
||
Source is bind-mounted; build caches persist in the `gocache` volume.
|
||
|
||
```sh
|
||
docker compose run --rm test # go test -race
|
||
docker compose run --rm cover # tests with coverage
|
||
docker compose run --rm bench # benchmarks
|
||
docker compose run --rm build # compile the library
|
||
docker compose run --rm vet # go vet
|
||
docker compose run --rm cyclo # cyclomatic complexity over 14 (test files excluded)
|
||
docker compose run --rm dev # interactive shell
|
||
```
|
||
|
||
Commands that rewrite source keep your file ownership when run with `--user`:
|
||
|
||
```sh
|
||
docker compose run --rm --user "$(id -u):$(id -g)" fmt # gofmt -w .
|
||
docker compose run --rm --user "$(id -u):$(id -g)" generate # go generate: each data/ module's .fejkdata.json
|
||
docker compose run --rm --user "$(id -u):$(id -g)" tidy # go mod tidy
|
||
```
|
||
|
||
A `--user` command answering `permission denied` on `/cache` has met root-owned
|
||
files in the `gocache` volume: Docker creates the volume root-owned, and every
|
||
command here that omits `--user` writes into it as root. Hand it back, and again
|
||
whenever it recurs:
|
||
|
||
```sh
|
||
docker compose run --rm --user root --entrypoint chown test -R "$(id -u):$(id -g)" /cache
|
||
```
|
||
|
||
Every pull request runs `docker build .` against both the latest and the lowest
|
||
supported Go, and must pass before it can be merged — unless it changes none of
|
||
the files the build and its tests read, nor the workflow itself, in which case
|
||
it's skipped. That build is the whole gate but the
|
||
changelog check, which CI runs against the PR base — vet, complexity, format check
|
||
and tests — so run it locally before pushing. The lowest Go builds the `portable`
|
||
stage, all of that bar the format check:
|
||
|
||
```sh
|
||
docker build . # latest
|
||
docker build --build-arg GO_VERSION=1.22.12 --target portable . # lowest supported
|
||
GO_VERSION=1.22.12 docker compose run --rm test # the same tests, without the image build
|
||
```
|
||
|
||
Goal 3 is scored by a panel of simulated readers, and [`AGENTS.md`](AGENTS.md) says
|
||
what the score asks of a pull request. Goal 5 is checked against simulated template
|
||
writers from the Audience.
|
||
|
||
A change to the shipped data runs `generate` above first and commits each module's
|
||
`.fejkdata.json` with the change, then re-pins
|
||
[`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) in its own commit:
|
||
|
||
```sh
|
||
REPIN=1 docker compose run --rm --user "$(id -u):$(id -g)" test
|
||
```
|
||
|
||
A shipped table built from a source is rebuilt by its script under
|
||
[`data-import/`](data-import), one command per dataset, fetching the source named in
|
||
[`DATA-LICENSES.md`](DATA-LICENSES.md). Downloads are cached under
|
||
`data-import/cache/`, which is ignored by version control and grows past a
|
||
gigabyte, so pass `--cache DIR` to reuse a copy you already have and delete it to
|
||
fetch afresh; `geo-us.py` fetches two
|
||
TIGER/Line files per county it ships, about a gigabyte, `geo-se.py` needs
|
||
a Trafikverket API key, free at [data.trafikverket.se](https://data.trafikverket.se/),
|
||
in `TRAFIKVERKET_API_KEY` or a `--key-file`, and the Census host behind `geo-us.py`
|
||
and `names-us.py` rejects a client for a while after a burst, so `--surnames` takes
|
||
a copy of the surname file:
|
||
|
||
```sh
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" -e TRAFIKVERKET_API_KEY data-import data-import/geo-se.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpmethod.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpstatus.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/language.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/loglevel.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/mimetype.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-se.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-us.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/port.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/protocol.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/territory.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/timezone.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/tld.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/useragent.py
|
||
```
|
||
|
||
To release, head `CHANGELOG.md` with the version's section in place of `Unreleased`
|
||
and merge: once `main` passes the gate, CI tags that commit `vX.Y.Z` and publishes
|
||
the GitHub release with the section as its body. A top heading of `[Unreleased]`
|
||
publishes nothing.
|
||
|
||
## License
|
||
|
||
MIT — see [LICENSE](LICENSE). Forked from [github.com/Timewave-AB/fakes](https://github.com/Timewave-AB/fakes).
|