1092 lines
60 KiB
Markdown
1092 lines
60 KiB
Markdown
# fejkdata
|
||
|
||
Locale-aware fake data for tests and fixtures, generated from JSON templates. Use
|
||
it as a Go library or the CLI — no data on disk, no dependencies, and a seed makes output
|
||
reproducible.
|
||
|
||
```sh
|
||
go install github.com/larvit/fejkdata/cmd/fejkdata@latest
|
||
fejkdata sv_SE.person # Sara Eriksson
|
||
```
|
||
|
||
## CLI
|
||
|
||
```sh
|
||
fejkdata sv_SE.person # Sara Eriksson
|
||
fejkdata sv_SE.person.last # Eriksson
|
||
fejkdata --seed 42 sv_SE.address # the same address every run
|
||
fejkdata -n 3 --separator ', ' sv_SE.word # nät, barn, sol
|
||
fejkdata --list # every path the data offers
|
||
fejkdata 'misc.territory[SE].capital' # Stockholm — a table's row, selected by key or name
|
||
fejkdata 'geo.SE.locality[Lund].street' # Fjelievägen — a linked table, drawn inside the row
|
||
fejkdata --data-path ./mydata sv_SE.word # layer a directory over the shipped data
|
||
fejkdata --no-shipped-data -d ./mydata --list # only your data
|
||
fejkdata 'name: {/sv_SE.person.last}' # name: <a surname> — an inline template
|
||
fejkdata "{date(1990-01-01,2010-12-31,'2006-01-02')}" # 2003-11-27 — the argument in "…", the layout in '…'
|
||
fejkdata '{"format":"name: {x}","x":["bosse","lina"]}' # name: bosse or name: lina
|
||
```
|
||
|
||
A path names a category, or a field inside one: each dot segment descends one
|
||
level — folders, then the category (a JSON file), then fields — and `[SE]` after a
|
||
[table](#table) selects its row. An argument that is
|
||
a JSON object, array or string, or that carries a `{` token, is instead an
|
||
**inline template**: a format string or a JSON value compiled and rendered on the
|
||
spot. Its tokens reach the data by reference from the root —
|
||
`{/sv_SE.person.last}`, so shipped and `--data-path` categories are alike
|
||
available. A path never contains a brace or a quote, and a bracket only as a selector
|
||
after a name, so the two spellings cannot collide; which spellings an inline template
|
||
rejects, and what each names instead, is under
|
||
[Decisions](docs/decisions.md#an-argument-is-a-template-by-its-shape-not-by-a-flag).
|
||
|
||
| Flag | |
|
||
|------|--|
|
||
| `-d`, `--data-path D` | a directory to layer over the shipped data; repeatable, the last wins a name clash |
|
||
| `--no-shipped-data` | load only the `--data-path` directories |
|
||
| `-s`, `--seed N` | same seed, version and data: identical output |
|
||
| `-n`, `--repeat N` | render the value N times (up to 1048576), each an independent draw, streamed |
|
||
| `--separator S` | between repeated values (default a newline) |
|
||
| `--format F` | `text` (default), `json`, `ndjson`, `csv` or `sql` — a record's columns, one record per row (json frames them as an array) |
|
||
| `--table T` | the INSERT target for `--format sql` (default: the path's last segment, or `records` for an inline template) |
|
||
| `--list` | print every path, then exit |
|
||
| `--version`, `-h`, `--help` | print, then exit |
|
||
|
||
`--name value` and `--name=value` both work, a short flag's value attaches or
|
||
follows (`-n3`, `-n 3`) and short flags bundle (`-hn 3`) — see
|
||
[Decisions](docs/decisions.md#flags-follow-getopt_long); flags go anywhere, `--` ends them. Exit codes: `0` success, `1` runtime error (missing
|
||
dir, unknown path), `2` misuse — a bad flag, an argument that names neither a
|
||
template nor a path, or an inline template that does not compile. From a checkout:
|
||
`go run ./cmd/fejkdata …`.
|
||
|
||
### Your own data
|
||
|
||
A category is a JSON file in a directory. Save this as `mydata/sql.json`:
|
||
|
||
```json
|
||
{
|
||
"format": "INSERT INTO users VALUES({sql-username});",
|
||
"sql-username": {
|
||
"format": "'{username}'",
|
||
"repeat": 3,
|
||
"separator": "),(",
|
||
"username": ["pixelfox", "snork", "turbohund", "blip", "zoom", "wahoo"]
|
||
}
|
||
}
|
||
```
|
||
|
||
```sh
|
||
fejkdata --seed 1 --data-path ./mydata sql
|
||
# INSERT INTO users VALUES('zoom'),('wahoo'),('blip');
|
||
fejkdata --repeat 100 --data-path ./mydata sql > seed.sql
|
||
```
|
||
|
||
That `format` string is the free-form spelling — you hand-write the whole row.
|
||
For structured output a record writes the row for you.
|
||
|
||
### Records
|
||
|
||
A record is a template seen as columns: its fields are the columns, its `format`
|
||
the whole. `--format json|ndjson|csv|sql` writes the records; the library's
|
||
`FakeRecord` (below) hands back the columns. A column is a string unless it declares a
|
||
[datatype](#datatype), and a [`null`](#null) item draws it as null. Save
|
||
`mydata/users.json`:
|
||
|
||
```json
|
||
{
|
||
"format": "{first} {last}",
|
||
"first": ["Ada", "Bo"],
|
||
"last": ["Lovelace", "Ek"]
|
||
}
|
||
```
|
||
|
||
```sh
|
||
fejkdata --seed 1 --data-path ./mydata --format json users # [{"first":"Bo","last":"Lovelace"}]
|
||
fejkdata --seed 1 --data-path ./mydata --format ndjson users # {"first":"Bo","last":"Lovelace"}
|
||
fejkdata --seed 1 --data-path ./mydata --format csv users # first,last → Bo,Lovelace
|
||
fejkdata --seed 1 --data-path ./mydata --format sql users # INSERT INTO "users" ("first", "last") VALUES ('Bo', 'Lovelace');
|
||
fejkdata --seed 1 --data-path ./mydata --format sql --table people users # INSERT into another table
|
||
```
|
||
|
||
`--repeat` streams that many records — `json` frames them as one array document,
|
||
`ndjson` writes one object per line, `csv` a row after a header, `sql` one INSERT
|
||
per line. Only a category-level template is a record; a field, choice or folder
|
||
errors, and so does a `repeat` on
|
||
the template itself, which composes the format into one string rather than
|
||
projecting columns — ask for more records with `--repeat`. A `repeat` on a column
|
||
is fine.
|
||
|
||
Columns are written in name order, whatever order the fields appear in. A category
|
||
whose fields are the parts of one value — `sv_SE.price`, `sv_SE.version`, `misc.uuid` —
|
||
projects those parts rather than the value, so for one column holding what `Fake`
|
||
renders, write `{"format":"","price":"{/sv_SE.price}"}` as a fieldless category asks
|
||
for.
|
||
|
||
A column carrying a newline keeps it inside the quoted CSV field or the SQL string
|
||
literal, so a row can span physical lines: read the stream with a CSV or SQL
|
||
parser rather than splitting it on newlines.
|
||
|
||
The SQL is ANSI — identifiers in double quotes, a literal quote doubled (`''`),
|
||
backslashes passed through — so a hyphenated field like `postal-code` stays a
|
||
valid identifier. PostgreSQL and SQLite take it as written; MySQL and MariaDB need
|
||
`ANSI_QUOTES` and `NO_BACKSLASH_ESCAPES` set first, or they read `"users"` as a
|
||
string and a backslash as an escape.
|
||
|
||
A record written only to emit columns still needs a `format` — the grammar's one
|
||
required key — so `"format": ""` carries the fields with an inert format: `Fake` renders
|
||
it as nothing.
|
||
|
||
A record is one render, so its columns read one pick of each [name](#names) its category
|
||
binds: `"code": "{/currency as c}{c.code}"` beside `"symbol": "{c.symbol}"` is one
|
||
currency, while `{/currency.code}` and `{/currency.symbol}` in two columns may name two. A
|
||
column the format binds to a name, as `{place as p}` binds `place`, holds that same pick, so it
|
||
agrees with the columns reading `p`; a column bound only in part, `{place.town as t}`, draws
|
||
afresh, so bind it whole and read `{p.town}`.
|
||
|
||
A category never references itself: `{/users.first}` or a bare `{/users}` inside `users`
|
||
describes a draw other than the fields beside it. Read a sibling as a field, and put a
|
||
value two fields share in its own category, bind it to a name in one field, and read the
|
||
name in both
|
||
([Decisions](docs/decisions.md#a-category-never-references-itself-and-a-records-fences-run-at-load)).
|
||
|
||
## Data
|
||
|
||
The shipped set under [`data/`](data) — one folder per locale (`en_US`, `sv_SE`)
|
||
plus a locale-neutral `misc` folder — is embedded, so the CLI and the library
|
||
work with no data on disk. A directory is a namespace: each JSON file is a
|
||
category named after the file, each subdirectory a dot-path segment, so
|
||
`mydata/sv_SE/person.json` is `sv_SE.person` and replaces the shipped one.
|
||
Sources merge in order; matching folders combine, any other clash is won by the
|
||
last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"`, `/` or ` as `,
|
||
which binds a [name](#names), nor be `-`, which a struct tag reserves; dot-prefixed entries are skipped, so a data directory can also be a checkout.
|
||
|
||
Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
|
||
`ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`,
|
||
`username`, `version` and `word`, formatted per locale; `sv_SE` adds `personnummer`
|
||
and `samordningsnummer`, `en_US` adds `ssn`, `itin` and `phone-exchange` (the three
|
||
digits after the area code). `misc` carries `car`, `coordinate`, `creditcard`
|
||
(Luhn-valid), `currency` (ISO 4217), `datetime` (RFC 3339), `emoji`, `httpmethod`,
|
||
`httpstatus`, `language` (ISO 639-1, with its 639-2/T code), `loglevel` (syslog
|
||
severity), `mac`, `mimetype`, `objectid`, `port`, `protocol`, `territory` (ISO
|
||
3166-1), `timezone` (IANA), `tld` (IANA root zone), `useragent` and `uuid` (v4). Many
|
||
carry sub-fields — `misc.currency.symbol`, `misc.territory.alpha2`,
|
||
`misc.httpstatus.code` — which `--list` shows. `car`, `currency`, `httpmethod`,
|
||
`httpstatus`, `language`, `loglevel`, `mimetype`, `port`, `protocol`, `territory`,
|
||
`timezone`, `tld` and `useragent` are [tables](#table), so
|
||
`misc.territory[SE].capital` and `misc.currency[Euro].symbol` select a row; `car` and
|
||
`useragent` carry no key or name, so they are drawn from rather than selected in.
|
||
|
||
Where a script under [`data-import/`](data-import) reads a source, the facts come from
|
||
that source, and the script drops a row only by a rule it states; elsewhere a small hand-written
|
||
set ships, and [`todo.md`](todo.md) carries the step that replaces it (goal 14). Copy
|
||
that states no fact, such as `word` and `sentence`, stays hand-written. Where a category
|
||
has no path that draws only values reaching nothing real (goal 16), write a fixed value such
|
||
as `example.com` into the template; `todo.md` plans the path.
|
||
|
||
`misc.httpmethod`, `misc.protocol` and `misc.port` are IANA's registries.
|
||
`misc.httpmethod` is the eight methods RFC 9110 defines and PATCH; the register's
|
||
other entries, WebDAV and DeltaV among them, do not ship. Its `safe` and
|
||
`idempotent` are `true` or `false`, so a Go `bool` reads them, though `--format json`
|
||
and `sql` write them as text, a table column carrying no datatype. `misc.protocol`
|
||
carries a `number` and a `name`, the keyword itself where the register spells none out,
|
||
and selects by either spelling.
|
||
`misc.port` is the TCP assignments, rendering the number a port field holds with the
|
||
IANA `service` beside it, and selecting by number alone. A draw spans the whole
|
||
register, so pin `misc.port[443]` where a fixture needs a port a reader recognises;
|
||
it stops at 49150, IANA assigning nothing above, so an ephemeral source port is
|
||
`{int(49152,65535)}`.
|
||
`misc.tld` is the root zone's delegated TLDs, keyed with the leading dot
|
||
`misc.territory.tld` already carries, `.se`. Its `type` is the register's —
|
||
`country-code`, `generic`, `generic-restricted`, `infrastructure` or `sponsored` — and
|
||
its `unicode` the form the register displays, `.рф` for `.xn--p1ai` and the key itself
|
||
for an ASCII one, which selects the row too, so `misc.tld[.рф]` renders `.xn--p1ai`. A
|
||
draw spans the whole zone, `.arpa` and `.zuerich` alike, so pin `misc.tld[.com]` where
|
||
a fixture needs one a reader recognises.
|
||
`misc.loglevel` is the eight syslog severities, rendering the keyword a configuration
|
||
writes, `err` and `info`, keyed by the numerical code a PRI encodes and selectable by
|
||
either, so `misc.loglevel[3]` and `misc.loglevel[err]` are one row.
|
||
`misc.loglevel.severity` is the name RFC 5424 spells, `Error` beside `err`; it
|
||
selects nothing, the code and the keyword do.
|
||
[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
|
||
|
||
`misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for most
|
||
of them to dozens for the largest, weighted by the population GeoNames records in each,
|
||
so `misc.territory[US].timezone` draws `America/New_York` far more often than
|
||
`America/Nome`. It links to `misc.territory`, so `misc.territory[SE].timezone` is
|
||
`Europe/Stockholm` and a drawn territory and zone agree. Its `offset` is the zone's
|
||
*standard* offset, so do not pair it with a drawn `misc.datetime`: in a zone that
|
||
observes DST it is the wrong one half the year, which is what storing a zone name
|
||
avoids. A zone tzdb named recently — `Europe/Kyiv`, `America/Ciudad_Juarez` — is
|
||
rejected outright by a consumer resolving it against older tzdata, so where the
|
||
consumer validates the zone, pin one rather than draw it. There is no `UTC` row, tzdb
|
||
giving that name no territory — spell it as the text `"UTC"`. `misc.useragent` carries
|
||
`browser`, `device` and `os` beside the string, and `misc.car` a `make` and a `model`,
|
||
until an international source replaces them.
|
||
|
||
ISO 3166-1 codes territories, not sovereign states, so that is what the table is
|
||
called: Greenland and Åland have codes of their own, and `misc.territory.country`
|
||
names the state each belongs to — `DK` for Greenland, `FI` for Åland, and its own
|
||
code for a sovereign one, or for one the register names no state for.
|
||
|
||
`sex`, `first-name` and `last-name` are tables weighted by bearers, from SCB, the
|
||
SSA and the Census Bureau. `first-name` links to `sex`, so `sv_SE.sex[f].first-name`
|
||
draws a woman's name, and a name both sexes carry is a row under each, so
|
||
`en_US.sex[m].first-name[Taylor]` names the one a `first-name[Taylor]` alone cannot.
|
||
`person` binds one first name and reads its sex through it, `{.first-name as n}` and
|
||
`{n..sex.name}`, so its `first` and `sex` columns agree. `en_US.title` links to `sex`
|
||
too, so a person's prefix never contradicts it. A `personnummer`'s birth number,
|
||
`sv_SE.birth-number` under `sex`, is Skatteverket's test series, 238 for a woman and 239
|
||
for a man, which no real person is ever given.
|
||
|
||
A person of a chosen sex is assembled from the tables — `sex[f].first-name` beside
|
||
`last-name`. `person` and `personnummer` draw apart, so where a name and a number must
|
||
agree, read both through one bound `sex`, `{/sv_SE.sex[f] as s}` for a woman:
|
||
|
||
```json
|
||
{ "format": "{first} {last}, {pnr}",
|
||
"first": "{/sv_SE.sex as s}{s.first-name}",
|
||
"last": "{/sv_SE.last-name}",
|
||
"pnr": "{date(1930-01-01,2025-12-31,'060102')}-{s.birth-number.number}{luhn()}" }
|
||
```
|
||
|
||
Renders e.g. `Anne Grönberg, 011012-2389`.
|
||
|
||
Skatteverket's test series is small: a personnummer is one of about 70,000 values, a day
|
||
in 1930–2025 against the two birth numbers, so a fixture past a few hundred rows repeats
|
||
one and a `UNIQUE` column needs a category of your own. `sv_SE.date` and `en_US.date`
|
||
are uniform over 1970-01-01 to 2029-12-31, `misc.datetime` over 2000-01-01 to 2029-12-31.
|
||
What the two locales do not share: `en_US.address` carries a `region` column the Swedish
|
||
one has no use for, and `sv_SE.title` has no `parent`, so it is selected as
|
||
`sv_SE.title[dr]` rather than inside a sex.
|
||
|
||
A `geo` folder holds one tree per country under its alpha-2 code: five
|
||
[linked tables](#linked-tables) named alike, and an `address` record binding one locality
|
||
and drawing its street and postal code inside it. The locale's `address` prints that
|
||
record, and each of its columns describes the address printed.
|
||
|
||
| Table | `geo.SE` | `geo.US` | Weight |
|
||
|-------|----------|----------|--------|
|
||
| `region` | län, by code or name | state, by USPS abbreviation or name; `code` is the FIPS code | population |
|
||
| `municipality` | kommun, by code or name | county, by FIPS code or name | population |
|
||
| `locality` | postort, by name; [`geo-se.py`](data-import/geo-se.py)'s `localities` picks which ship | incorporated place of 25,000 people or more with a postal code of its own, by GEOID or name; Hawaii has none | tätort population, the kommun's where the postort names it, else 200; place population |
|
||
| `postal-code` | postnummer with street delivery, by code | ZCTA, by code | one; address ranges |
|
||
| `street` | gatunamn, the ten with most road segments per postort | street name, the ten with most address ranges per place | segments; address ranges |
|
||
|
||
`geo.SE.region[Skåne län].municipality` draws a kommun in Skåne,
|
||
`geo.SE.locality[Lund].street` a street in Lund, and
|
||
`geo.US.region[IL].locality[Springfield]` settles which Springfield. A region row
|
||
carries its `timezone`, the state's predominant zone, and a locality its `lat` and
|
||
`lon`. What ports across countries is the five table names, the `name` column,
|
||
selection by name, and the `address` record's columns `street`, `street-number`,
|
||
`postal-code` and `locality`; every other column is the country's own, `code` on a
|
||
Swedish region but `abbr` on a US one.
|
||
|
||
## Library
|
||
|
||
```sh
|
||
go get github.com/larvit/fejkdata # Go 1.22+
|
||
```
|
||
|
||
```go
|
||
f, err := fejkdata.New(fejkdata.WithSeed(42))
|
||
if err != nil {
|
||
log.Fatal(err)
|
||
}
|
||
v, err := f.Fake("sv_SE.address") // "Räfsalsvägen 43\n442 42 Kungälv"
|
||
paths := f.List() // the paths Fake renders, sorted; direct descents only
|
||
v, err = f.FakeTemplate("name: {/sv_SE.person.last}") // compile + render in one call
|
||
t, err := f.NewTemplate(`{"format":"name: {x}","x":["bosse","lina"]}`) // compile once
|
||
v = t.Fake() // render many times, no re-parse
|
||
r, err := f.FakeRecord("users") // one record: each field a column
|
||
s := r.JSON() // {"first":"Ada","last":"Lovelace"}
|
||
r, err = f.FakeRecordTemplate(`{"format":"{x}","x":["a","b"]}`) // compile + render inline
|
||
err = f.FakeStruct(&user) // fill a struct's fake:"…" tagged fields
|
||
ok, err := fejkdata.IsTemplate(arg) // an inline template by its shape, else a path
|
||
```
|
||
|
||
| Option | |
|
||
|--------|--|
|
||
| `WithSeed(n)` | same seed, version and data: identical sequence |
|
||
| `WithDataPath(dir)` | layer a directory; repeat to layer several, the last wins a clash |
|
||
| `WithDataFS(fsys)` | layer an `fs.FS`, such as your own `embed.FS` |
|
||
| `WithoutShippedData()` | load only what you give |
|
||
|
||
`New` refuses a mistake in the data, and `NewTemplate` one in the template. On a loaded
|
||
generator:
|
||
|
||
- `Fake` fails only for a path that names nothing, could name two things, or reads one
|
||
draw of a level carrying a `repeat`, with the same error every call.
|
||
- `FakeStruct` fails only for a non-struct argument or a type its tags do not describe,
|
||
with the same error every call.
|
||
- `Template.Fake` cannot fail.
|
||
|
||
A `*Record` carries its columns via `Columns()` — each a `Column` of `Name`,
|
||
`DataType`, rendered `Value` and `Null` — and serializes them with `JSON()`
|
||
(one object), `CSVHeader()`/`CSVLine()`, or `SQLInsert(table)` — the shapes the
|
||
CLI's `--format` writes. `FakeRecord` and `FakeRecordTemplate` take a record; a
|
||
path or template that is not one — a bare string, a choice, or a folder — errors.
|
||
|
||
```go
|
||
type User struct {
|
||
ID int64 `fake:"{seq()}"`
|
||
Last string `fake:"sv_SE.person.last"`
|
||
Age uint8 `fake:"{int(18,99)}"`
|
||
Nick *string `fake:"[null, \"{/sv_SE.username}\"]"`
|
||
Home Address // filled from Address's own tags
|
||
}
|
||
```
|
||
|
||
`FakeStruct` fills a struct through a pointer: each exported field tagged `fake:"…"` is
|
||
a column of one record, its tag a path or an inline template — told apart by
|
||
`IsTemplate`, as the CLI tells an argument — and its Go type the column's
|
||
[datatype](#datatype): a string, bool, integer or float kind, or a pointer to one,
|
||
which a [`null`](#null) item leaves nil. An integer stays within int64 whatever its
|
||
kind, and a value the kind cannot hold, such as `{int(0,300)}` in a `uint8`, is refused
|
||
naming a kind that holds it. The tags are a record's columns, so they share its
|
||
[names](#names): `First` tagged `{/sv_SE.person as p}{p.first}` and `Sex` tagged `{p.sex}`
|
||
describe one person, while two path tags into `sv_SE.person` are two draws. The fields an
|
||
embedded struct promotes are columns of the same record; a named struct field, or a
|
||
pointer to one, fills from its own tags as a record of its own, so its names and picks are
|
||
its own. `fake:"-"` leaves a struct field, embedded or named, or a pointer to one, unfilled.
|
||
Untagged fields keep their values, and so does a pointer back to a struct already being filled; a type
|
||
whose fields reach more than 1024 structs is refused, naming `fake:"-"` to cut it. The
|
||
first call for a type compiles its tags and reports what they get wrong, with the same
|
||
error on every later call; a `datatype` in a tag names the Go type that already sets it.
|
||
|
||
A `*Generator` is safe for concurrent use; a seeded sequence is reproducible only
|
||
when drawn from one goroutine. Changing how a value is composed shifts the seeded
|
||
stream for that value and everything drawn after it.
|
||
|
||
## Data format
|
||
|
||
Every value is a **node**, nestable without limit:
|
||
|
||
| Node | JSON | Renders |
|
||
|------|------|---------|
|
||
| string | `"Malmö"` | its text, with any `{…}` tokens expanded |
|
||
| choice | `["a", "b", …]` | one item, picked at random |
|
||
| template | `{"format": "…", …}` | its format, with `{name}` tokens rendering the named fields |
|
||
| table | `{"format": "…", "rows": "x.tsv", …}` | its format over one row of the TSV beside it ([Table](#table)) |
|
||
|
||
### Format string
|
||
|
||
Every character is literal except a `{…}` token:
|
||
|
||
| Token | Renders |
|
||
|-------|---------|
|
||
| `{name}` | the sibling field `name`, or the pick bound to the name ([Names](#names)) |
|
||
| `{x as name}` | nothing: binds one pick of `x`, a reference, a sibling field or a path into one, to `name` ([Names](#names)) |
|
||
| `{name.field}` | `field` of a fresh draw of the sibling `name`, or of the pick bound to it ([Correlated fields](#correlated-fields)) |
|
||
| `{a\|b}` | one of the named fields, even odds |
|
||
| `{fn(args)}` | a builtin ([Functions](#functions)) |
|
||
| `{/path}`, `{.path}`, `{..path}` | a node reached from the data root, this file's folder, or the folder above ([References](#references)) |
|
||
| `{{`, `}}` | a literal `{` or `}` |
|
||
|
||
Each `{…}` draws afresh, so `{w} {w}`, `{p.a} {p.b}` and `{/person.first} {/person.last}`
|
||
are two draws each. Only a [name](#names) keeps one pick for every read of it.
|
||
|
||
```json
|
||
"100 Main St, Apt {int(1,9)}{upper(1)} — tel {digits(3)}-{digits(4)}"
|
||
```
|
||
|
||
Renders e.g. `100 Main St, Apt 4B — tel 555-0199`. Random characters come from
|
||
the sample functions, so a phone pattern is `070-{digits(3)} {digits(2)} {digits(2)}`.
|
||
A lone `}` is a load error naming `}}`; the arms of `{a|b}` must differ, a
|
||
repeated arm being a second spelling of [weight](#weight).
|
||
|
||
### Weight
|
||
|
||
An item of a choice may carry a `weight` (default `1`) to skew its odds:
|
||
|
||
```json
|
||
[
|
||
{ "format": "070-{digits(3)} {digits(2)} {digits(2)}", "weight": 10 },
|
||
"08-{digits(3)} {digits(2)} {digits(2)}",
|
||
{ "format": "010-{digits(3)} {digits(2)} {digits(2)}", "weight": 2 }
|
||
]
|
||
```
|
||
|
||
Renders `070-412 38 91` ten times as often as `08-…`. A string item is weighted by
|
||
writing it as `{ "format": "AB", "weight": 3 }`. Rejected at load: a weight that
|
||
is negative, non-numeric, `0` (never drawn — remove the item), `1` (the default)
|
||
or outside a choice, and a repeated item — `["a", "a", "b"]` is `[{ "format": "a", "weight": 2 }, "b"]`,
|
||
and the error says so.
|
||
|
||
### Repeat
|
||
|
||
A template may carry `repeat` (an integer above `1`) to render its format that
|
||
many times — each an independent draw — joined by `separator` (default `""`):
|
||
|
||
```json
|
||
{ "format": "{word}", "repeat": 3, "separator": " ", "word": ["foo", "bar", "baz"] }
|
||
```
|
||
|
||
Renders e.g. `bar foo baz`. Rejected at load: a `separator` without a `repeat`,
|
||
a `separator` of `""` (the default), and a `repeat` that multiplies to more than
|
||
1 048 576 renders along any path of nested repeats.
|
||
|
||
### Datatype
|
||
|
||
A record column may declare `datatype` — `integer`, `number` or `boolean` — so `json`
|
||
writes `42` rather than `"42"` and `sql` a bare literal; a column without one is a
|
||
string:
|
||
|
||
```json
|
||
{ "format": "",
|
||
"id": { "format": "{seq()}", "datatype": "integer" },
|
||
"paid": { "format": "{p}", "p": ["true", "false"], "datatype": "boolean" },
|
||
"total": { "format": "{calc(net * qty, 2)}", "net": ["19.99", "5.00"], "qty": ["3", "7"], "datatype": "number" } }
|
||
```
|
||
|
||
Writes e.g. `{"id":1,"paid":true,"total":59.97}`. A column is a field of the top-level
|
||
template, or an item of a choice standing in for one; `datatype` anywhere else is a
|
||
load error. A typed column holds one value, alone in its format: a literal, one
|
||
`{int()}`, `{float()}`, `{seq()}` or `{calc()}` call, or a read that lands only on such
|
||
values. `integer` is an int64 written `0|-?[1-9][0-9]*` — `{float()}` prints one at
|
||
`0` decimals within int64 — `number` a JSON number, `boolean` `true` or `false`. A value its
|
||
datatype cannot hold is a load error naming it:
|
||
|
||
```text
|
||
order.id: datatype integer: {digits(3)} prints text, not an integer
|
||
order.id: datatype integer: "1{digits(2)}" is not one value; write one literal or one {float()}, {int()}, {seq()} or {calc()}, or read one
|
||
```
|
||
|
||
A column whose format is one read of a record's column and nothing else —
|
||
`"score": "{/src.score}"`, `"score": "{s.score}"` with `s` bound to `/src`, or `"a": "{p}"`
|
||
beside `{place as p}` in its own record — is that column: it takes the column's datatype and
|
||
is null where the column is, and a struct field tagged `src.score` is nil there. A `datatype` of its own types the
|
||
column's values where they prove it, and one restating the datatype it takes is refused.
|
||
Any other read renders the column's text, a null as `""`.
|
||
|
||
A typed column's `{calc()}` must be proven to print a number: each operand a number
|
||
literal, an `{int()}`, `{float()}`, `{seq()}` or `{digits()}` call, a calc, or a read of
|
||
such values, whose bounds keep every divisor from zero and the result within `1e300`.
|
||
What the bounds cannot show is refused — `{calc(a / b)}: divides by b, which is not
|
||
proven nonzero`. The calc fills an `integer` column at `0` decimals, or over whole
|
||
operands with no `/`, while its bounds stay within int64.
|
||
|
||
### Null
|
||
|
||
A `null` item draws a record column as null: `json` writes `null`, `sql` `NULL`, and
|
||
`csv` an empty field, with an empty string written `""` — the convention PostgreSQL's
|
||
`COPY … CSV` reads. A record of one null column is a blank line, which `COPY` reads as
|
||
null but most CSV readers skip, so write such a record as `json` or `sql`. `Fake`
|
||
renders a null as `""`. The other items' weights skew its odds:
|
||
|
||
```json
|
||
{ "format": "", "deleted_at": null, "middle": [null, { "format": "{n}", "n": ["Ann", "Eva"], "weight": 3 }] }
|
||
```
|
||
|
||
`deleted_at` is null every draw, `middle` a name three draws in four. Rejected at
|
||
load: `null` anywhere but a column, naming `""`, and a column whose items hold
|
||
different datatypes.
|
||
|
||
### Table
|
||
|
||
A table is a category whose rows come from a TSV beside its JSON file: the header
|
||
names the columns, each line below it is one row, and the format renders the row
|
||
drawn. Save `mydata/country.tsv` and `mydata/country.json`:
|
||
|
||
```tsv
|
||
alpha2 name population
|
||
DK Denmark 5900000
|
||
NO Norway 5500000
|
||
SE Sweden 10500000
|
||
```
|
||
|
||
```json
|
||
{ "format": "{name} ({alpha2})", "rows": "country.tsv", "key": "alpha2", "name": "name", "weight": "population" }
|
||
```
|
||
|
||
```sh
|
||
fejkdata -d ./mydata country # Sweden (SE), about half the time
|
||
fejkdata -d ./mydata country.alpha2 # NO
|
||
fejkdata -d ./mydata 'country[SE]' # Sweden (SE)
|
||
fejkdata -d ./mydata 'country[Norway].alpha2' # NO
|
||
fejkdata -d ./mydata --format csv country # alpha2,name,population → SE,Sweden,10500000
|
||
```
|
||
|
||
`rows` names the TSV beside the category file; `key` names the column a path
|
||
selects a row by, `name` a column it also selects by, `weight` a column of positive
|
||
numbers that skews the draw, and `parent` the table a column links to
|
||
([Linked tables](#linked-tables)). The format's `{tokens}` read the columns, and the
|
||
columns are the [record](#records)'s columns, so `--format csv` writes the rows and
|
||
`--list` shows `country.alpha2`. A cell is a string node: `1{digits(2)} {digits(2)}`
|
||
in a cell draws digits and `{/misc.uuid}` reads a reference, while `{name}` in a cell
|
||
is refused, since a cell has no sibling. Each cell may select its own row of another
|
||
table, `{/misc.currency[SEK].symbol}` on one row and `{/misc.currency[EUR].symbol}`
|
||
on the next, since only one row renders. `New` proves the header, the options and every
|
||
cell token, and refuses a TSV no category names, a key that is empty or repeats, a
|
||
weight that is not a positive number, and a key or name holding `[`, `]`, `{`, `}`,
|
||
`"` or `|`, which a selector cannot spell; `New` maps the keys, and a name or parent column
|
||
is mapped on the first draw that selects by name or descends through the table. A `name` needs a `key`, since a name naming several rows is reported by
|
||
their keys, or a `parent`, inside whose row a name names one row, so `first-name[Kim]`
|
||
is settled by the `sex` selected before it and a name repeating inside one parent row
|
||
is refused; a name spelling another row's key is refused, since the key would
|
||
select first and the name never. The table's options are its own — `rows`, `key`,
|
||
`name`, `weight` and `parent` — so a column may be named `name`, as one usually is.
|
||
|
||
A choice of templates sharing one format and one set of string fields is a table
|
||
written by hand, and `New` refuses it in a data file naming the TSV to write; an
|
||
inline template has no file beside it, so there it stays a choice.
|
||
|
||
### Row selection
|
||
|
||
`[key]` or `[name]` after a table's name selects one row: `misc.territory[SE]` and
|
||
`misc.territory[Sweden]` name one row, and `misc.territory[SE].capital` reads its column.
|
||
A name naming several rows is an error listing their keys, unless a row selected
|
||
before it settles which ([Linked tables](#linked-tables)). A selector is part of the path, so it works
|
||
wherever a path does: `Fake`, `FakeRecord`, a `{/misc.territory[SE].capital}` reference
|
||
and a struct tag. A dot inside the brackets belongs to the key or name, so
|
||
`city[St. Louis]` selects it. A path starts with a name, and `[` still opens a JSON
|
||
array at the start of a CLI argument, so `'[SE]'` alone names nothing.
|
||
|
||
### Linked tables
|
||
|
||
A table's `parent` names a column and, by the same name, the table beside it that the
|
||
column links to by key. Save `mydata/city.tsv` and `mydata/city.json` beside the
|
||
`country` table above:
|
||
|
||
```tsv
|
||
name country population
|
||
Copenhagen DK 660000
|
||
Göteborg SE 600000
|
||
Oslo NO 710000
|
||
Stockholm SE 990000
|
||
```
|
||
|
||
```json
|
||
{ "format": "{name}", "rows": "city.tsv", "key": "name", "parent": "country", "weight": "population" }
|
||
```
|
||
|
||
```sh
|
||
fejkdata -d ./mydata 'country[SE].city' # Stockholm or Göteborg
|
||
fejkdata -d ./mydata country.city.name # a country drawn, then a city inside it
|
||
fejkdata -d ./mydata 'city[Oslo].country' # NO — the link column's cell
|
||
fejkdata -d ./mydata '{/city as c}{c.name}, {c..country.name}' # Oslo, Norway — one city and its country
|
||
```
|
||
|
||
A path descends from a row to a linked table by name, at any depth, and `--list`
|
||
advertises each direct step. One path reads one chain of rows: `country.city.name` draws a
|
||
country by its weights, then a city inside it by theirs. Two paths are two draws, so
|
||
`{/city.name}, {/country.name}` may name a city and another country. The reads of one
|
||
[name](#names) agree instead, and the table it binds decides whose weights govern:
|
||
`{/city as c}` draws a city by city weight and `{c..country.name}` is its country, while
|
||
`{/country as k}` draws a country by country weight and `{k.city.name}` a city inside it.
|
||
A name keeps every row its reads draw, so `{k.city.name}` and `{k.city.population}` describe
|
||
one city, and `{k.name}` its country.
|
||
|
||
`New` refuses a link cell that is no key of the parent, a parent row no child links to, a
|
||
chain of parents that closes, and a table named like a column of any table above it.
|
||
|
||
### Step up
|
||
|
||
`..` after a row steps up to the row its link column names. Name the parent table right
|
||
after the `..`:
|
||
|
||
```sh
|
||
fejkdata -d ./mydata 'city[Oslo]..country.name' # Norway
|
||
fejkdata -d ./mydata 'city..country' # a city drawn, then its country: Sweden (SE)
|
||
fejkdata -d ./mydata 'city[Göteborg]..country.city' # Stockholm or Göteborg, drawn afresh
|
||
fejkdata -d ./mydata '{/city as c}{c}, {c..country.name}' # Oslo, Norway
|
||
```
|
||
|
||
It steps up from a selected row, a drawn row or a [name](#names) bound to a row, one
|
||
level per `..`: `geo.SE.locality..municipality..region.name`. `.country` after a row
|
||
still reads the link column's cell, `NO`.
|
||
|
||
After a `..`, `.` steps down again and draws afresh inside the row stepped up to, so
|
||
`city[Göteborg]..country.city` may draw Göteborg again. Under a name, the paths stepping
|
||
down from one `..` read one draw: `{c..country.city.name}` and
|
||
`{c..country.city.population}` describe one city.
|
||
|
||
A `..` that names anything but the parent table is an error naming the parent. A selector
|
||
after a `..` is an error naming the table to select from, since it could name a row outside
|
||
the one stepped up to. `--list` advertises no path with a `..`. At the start of a
|
||
reference, `..` is the folder above ([References](#references)).
|
||
|
||
### Options and fields
|
||
|
||
`format`, `weight`, `repeat`, `separator` and `datatype` are the only options;
|
||
**any other key is a field** (see [Decisions](docs/decisions.md#options-and-fields-share-one-namespace)), and `rows` makes a category
|
||
a [table](#table), so no template carries a field of that name. A [name](#names) shares the
|
||
namespace too. An object that does nothing a
|
||
string can't — only a `format` — is rejected naming the string, as is a one-item
|
||
choice naming its item.
|
||
|
||
### Functions
|
||
|
||
A `{name(args)}` token calls a builtin. Arguments are checked at `New`: a bad
|
||
count, range, country or expression fails fast; an integer is written plain
|
||
(`5`, not `+5` or `05`); bounds are finite; a sample that could only ever emit one
|
||
value (`int(5,5)`, `float(1,1,2)`) is rejected naming the text to write instead;
|
||
and a length, count or decimal place beyond a sane maximum is rejected, so a
|
||
fat-fingered `hex(2000000000)` never tries to allocate gigabytes. Every builtin draws only from the seed — a
|
||
time-based id takes its timestamp from the rng, not the clock — so seeded output
|
||
stays reproducible.
|
||
|
||
| Function | Kind | Renders |
|
||
|----------|------|---------|
|
||
| `{luhn()}` | derivation | Luhn check digit over the digits emitted so far in this expansion |
|
||
| `{mod11()}` | derivation | weighted mod-11 check char (weights 2–7 from the right); `X` for 10 |
|
||
| `{ean()}` | derivation | EAN-13 / UPC-A / ISBN-13 / GTIN check digit |
|
||
| `{digits(n)}` | sample | `n` digits 0–9 |
|
||
| `{upper(n)}`, `{lower(n)}` | sample | `n` letters A–Z, a–z |
|
||
| `{int(min,max)}` | sample | uniform integer in `[min, max]` |
|
||
| `{float(min,max,dp)}` | sample | number in `[min, max]` with `dp` decimals |
|
||
| `{hex(n)}` | sample | `n` lowercase hex digits |
|
||
| `{base64(n)}` | sample | `n` random bytes, base64 |
|
||
| `{uuid()}` | sample | UUID v7 (`misc.uuid` ships v4 as data) |
|
||
| `{ulid()}` | sample | ULID, 26 Crockford base32 chars |
|
||
| `{nanoid(n)}` | sample | URL-safe Nano ID, `n` chars |
|
||
| `{iban(CC)}` | sample | length- and mod-97-valid IBAN for BE, DE, DK, ES, FI, NO or SE |
|
||
| `{date(from,to,'layout')}` | sample | a second between two `YYYY-MM-DD` days, both included, in a quoted Go layout: `'2006-01-02'`, `'January 2, 2006'`, `'060102'`, `'2006-01-02T15:04:05Z'` |
|
||
| `{time('layout')}` | sample | a second within a day: `'15:04'`, `'3:04 PM'` |
|
||
| `{seq()}`, `{seq(name)}` | counter | next integer from 1 in this generator; `name` selects an independent counter |
|
||
| `{calc(expr)}`, `{calc(expr,dp)}` | computation | an arithmetic expression over sibling fields and names ([Computation](#computation)) |
|
||
| `{lowercase(x)}`, `{uppercase(x)}`, `{ascii(x)}` | transform | a field's value rewritten ([Transforms](#transforms)) |
|
||
|
||
A derivation reads what is to its left, so place it after its payload; the
|
||
buffer is per expansion, so a nested template keeps fixed parts out of the sum. A
|
||
Swedish personnummer is a Luhn checksum over the nine digits before it, six of
|
||
them a birthdate:
|
||
|
||
```json
|
||
{ "format": "{date(1930-01-01,2010-12-31,'060102')}-{birth}{luhn()}", "birth": ["238", "239"] }
|
||
```
|
||
|
||
Renders e.g. `811218-2389`. A layout is Go's: the reference time `Mon Jan 2
|
||
15:04:05 MST 2006` spelled as the output should look, quoted, since a layout may
|
||
carry the comma that separates arguments, with English names. Every second
|
||
between the two days is reachable, so a layout with a clock draws the time too, and
|
||
`from` may equal `to`, which is that one day. The instant is UTC, so a zone in the
|
||
layout prints `UTC` or `Z`.
|
||
The quotes delimit a layout outside a selector only, so `[O'Fallon]` in an
|
||
argument stays a name. Rejected at `New`: a bound that is no calendar date, or not
|
||
before the other; an unquoted layout, naming the single-quoted one; a layout naming no field, which is
|
||
text, as is one day in a layout with no clock; for `date` a layout naming no date
|
||
field, naming `time`; and for `time` a layout naming a date field, naming `date`. `{seq()}` spans `Fake`
|
||
calls and `repeat`, resets with a new generator, and is the natural primary key for
|
||
the SQL example above.
|
||
|
||
### Computation
|
||
|
||
`{calc(expr)}` evaluates `+ - * /`, parentheses and unary minus over number
|
||
literals, sibling fields and [names](#names), each rendered then read as a number;
|
||
a second argument rounds to that many decimals. A hyphen is always subtraction, so a
|
||
hyphenated field or name can't be an operand.
|
||
|
||
A calc computes from the values its format shows where both read one [name](#names):
|
||
|
||
```json
|
||
{ "format": "{net as n}{qty as q}{n} x {q} = {calc(n * q, 2)}", "net": ["19.99", "5.00"], "qty": ["3", "7"] }
|
||
```
|
||
|
||
Renders e.g. `19.99 x 3 = 59.97`.
|
||
|
||
A result that rounds to zero prints unsigned — `0`, `0.00` — as `{float()}`'s does. An
|
||
operand that can never be a number (`"abc"`, or a choice of such) is rejected at load, as
|
||
is a division by a constant zero (`1/0`, or a fixed `"0"` field); an operand that sometimes is not a number yields
|
||
`NaN`, and a division by one that is not constant `Inf` — both print rather than
|
||
fail, except in a [typed column](#datatype), which must prove neither happens.
|
||
|
||
### Transforms
|
||
|
||
`{lowercase(x)}`, `{uppercase(x)}` and `{ascii(x)}` rewrite the value of `x` — a
|
||
field, a path, a `..path` or a [name](#names) — and nest. `ascii` folds Latin letters (`Åsa Öberg`
|
||
→ `Asa Oberg`) and drops any other non-ASCII rune. An email
|
||
matches the name beside it where both read one name:
|
||
|
||
```json
|
||
{ "format": "{person as p}{p.first} {p.last} <{lowercase(ascii(p.first))}.{lowercase(ascii(p.last))}@example.com>",
|
||
"person": [
|
||
{ "format": "{first} {last}", "first": "Åsa", "last": "Öberg" },
|
||
{ "format": "{first} {last}", "first": "Bo", "last": "Ek" }
|
||
] }
|
||
```
|
||
|
||
Renders `Åsa Öberg <asa.oberg@example.com>` or `Bo Ek <bo.ek@example.com>`, never
|
||
a mix.
|
||
|
||
### References
|
||
|
||
A reference renders a node from elsewhere in the data — the path `Fake` takes,
|
||
across every loaded source — so one category borrows another. The sigil says
|
||
where the path starts, as in a filesystem: `{/en_US.person}` from the data root,
|
||
`{.username}` from the folder this file sits in, `{..username}` from the folder
|
||
above. `data/sv_SE/email.json` can therefore read its own locale's `username`
|
||
without naming `sv_SE`:
|
||
|
||
```json
|
||
"Hej, {/en_US.person}!"
|
||
```
|
||
|
||
Renders e.g. `Hej, Pat Smith!`. Each reference draws afresh ([Format string](#format-string)),
|
||
so `{.person.first} {.person.last}` may name two people. Bind the person to a [name](#names) to read one: `{.person as p}{p.first}
|
||
{p.last}`. Rejected at `New`: a path that is
|
||
unknown, names a folder, has no folder above, reads a field not every variant
|
||
of a choice carries, or names the category the reference sits in, and a reference
|
||
that leads back to its own value, directly, mutually or through a chain.
|
||
|
||
### Names
|
||
|
||
`{x as n}` binds one pick of `x`, a reference, a sibling field or a path into one, to the name
|
||
`n` and prints nothing. `{n}` then
|
||
prints that pick, and `{n.path}` reads a path through it, so every read of `n` describes one
|
||
row:
|
||
|
||
```json
|
||
"{/misc.territory as t}{t.flag} {t}: capital {t.capital}, calling code +{t.calling-code}"
|
||
```
|
||
|
||
Renders e.g. `🇳🇴 Norway: capital Oslo, calling code +47`. A name keeps what each `{n.path}`
|
||
lands on and every level it passes, so `{n}` and `{n.path}` agree, and so does a
|
||
[transform](#transforms) of `n.path`. Everything else under the name draws as it would
|
||
anywhere: if `n`'s category has the format `{w}-{w}`, it draws `w` twice, and `{n.w}` beside
|
||
`{n}` is refused, since it could read either draw. A sibling field is bound the same way, so
|
||
`{place as p}{p.postal-code} {p.locality}` reads one place ([Correlated fields](#correlated-fields)).
|
||
|
||
A name is a pick of its own: `{t.name}` and `{/misc.territory.name}` beside it are two draws,
|
||
and so are two names bound to one category.
|
||
|
||
A name is drawn on its first read, and lives in the category binding it: any field of the
|
||
category may read it, and a record's columns read one pick. Each render of the category picks
|
||
anew, and so does each bare reference to it. A name a `repeat` binds picks again on every
|
||
iteration, and one bound outside the repeat keeps its pick on every line.
|
||
|
||
Refused at `New`, each error naming what to write instead:
|
||
|
||
- a name bound twice outside any `repeat`, twice in one `repeat`, or both inside a `repeat`
|
||
and outside it;
|
||
- a name that is a field or an option;
|
||
- a binding of anything but a reference, or a field of the template binding it or a path into
|
||
one;
|
||
- a read of a name inside the field bound to it: read it outside that field;
|
||
- a binding in a choice's item, which every other item would leave unbound;
|
||
- a binding nothing reads, or one read only once, unless that read sits in a `repeat`
|
||
nested where it is bound: write what it binds there instead, `{/word}` for
|
||
`{/word as w}{w}` and `{/region.name}` for `{/region as r}{r.name}`; a calc cannot read a
|
||
reference or a path, so a calc's one read of a name bound to either is not refused;
|
||
- a path through a name that selects a row;
|
||
- `{n}` beside `{n.w}` where `n`'s category reads `w` twice, as in `{w}-{w}` or
|
||
`{w.a} {w.b}`;
|
||
- a binding in a table's format or cell.
|
||
|
||
### Correlated fields
|
||
|
||
`{name.field}` reads a path into a sibling. Several tokens read one draw of it through a
|
||
[name](#names): bind the sibling, and every read of
|
||
the name reads that one row — a locality and the postal code that really covers it:
|
||
|
||
```json
|
||
{ "format": "{place as p}{street} {int(1,99)}\n{p.postal-code} {p.locality}",
|
||
"street": ["Kungsgatan", "Storgatan"],
|
||
"place": [
|
||
{ "format": "{locality}", "locality": "Stockholm", "postal-code": "1{digits(2)} {digits(2)}", "weight": 975 },
|
||
{ "format": "{locality}", "locality": "Tranås", "postal-code": "573 {digits(2)}", "weight": 18 }
|
||
] }
|
||
```
|
||
|
||
Renders e.g. `Kungsgatan 35` / `176 99 Stockholm`, never a Stockholm code beside
|
||
Tranås; each row's `weight` says how often it appears. `{p.geo.town.name}
|
||
{p.geo.town.zip}` share the town too, as a name keeps every level its reads pass. Every variant of a choice
|
||
on the path must carry the rest of it, so a row missing a field is named at load:
|
||
|
||
```text
|
||
token {p.postal-code}: not every variant of this 2-way choice carries "postal-code"; all carry [locality]
|
||
```
|
||
|
||
The sub-fields stay addressable — `Fake("address.place.locality")` renders, and
|
||
`List` advertises it. A path may not read into a level carrying a `repeat`, and `List`
|
||
advertises no such path; read that level whole.
|
||
|
||
### Performance
|
||
|
||
The target is about a microsecond per value (goal 13).
|
||
|
||
With only the shipped set, `New` loads nothing: a category loads on the first `Fake`,
|
||
`FakeRecord`, `FakeStruct` or template that reads it, with every category it reads and
|
||
its whole table family. Beside `--data-path`, `WithDataPath` or `WithDataFS`, every
|
||
category loads in `New`. Each file is parsed, validated and weight-indexed once, bar a
|
||
table's name and parent columns, which are mapped on the first `Fake` that selects by
|
||
name or descends through it. A `Fake` call then costs about what its output
|
||
costs: an unweighted pick is O(1) whatever the list's length, a weighted one
|
||
O(log n), and long formats, deep nesting and many tokens add cost in proportion to
|
||
the output.
|
||
|
||
## Versioning
|
||
|
||
Semver tags on `main`, `v0.1.0` first; one version covers the shipped data, the
|
||
library and the CLI, and [`CHANGELOG.md`](CHANGELOG.md) names what each release
|
||
changed. A consumer's data, code and scripts keep working across a minor or a patch. A
|
||
minor only adds, and a major is the only release that changes what exists. Seeded output
|
||
is the exception: any release, a patch too, may change what a seed draws. Where you
|
||
compare seeded output, install one exact version
|
||
(`go install github.com/larvit/fejkdata/cmd/fejkdata@vX.Y.Z`; `go.mod` already pins one),
|
||
and regenerate seeded fixtures and expected values when you raise that version.
|
||
|
||
| Surface | Major | Minor |
|
||
|---------|-------|-------|
|
||
| Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another; change a table's key, name, weight or parent column, or remove a row | a path outside a record's columns, a locale, a value in a list, a row |
|
||
| Records (a table is one) | remove, rename, retype or add a column; let a column be null | a record, as a new category |
|
||
| Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin |
|
||
| CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format |
|
||
| Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option |
|
||
|
||
A patch changes no row of this table: performance, docs, or a fix inside a promised
|
||
behaviour that adds or removes no value and changes no path, format or spelling.
|
||
|
||
An error's wording may improve in a minor; the path, rejected spelling and replacement
|
||
it names may not.
|
||
|
||
Before `v1.0.0` a minor is the breaking unit: `0.(x+1).0` may carry a major's
|
||
changes, each named in the changelog, and a `0.x.y` patch may not. `v1.0.0` is
|
||
cut once the shipped data is in its record shape and one full minor has shipped
|
||
with no breaking change. From `v2` the module path carries `/vN`, so fences ship
|
||
batched into as few majors as possible.
|
||
|
||
[`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) pins every path, each
|
||
template category's format, the categories each category reads, each column's
|
||
datatype and nullability, and each table's key, name, weight and parent columns; a pull request that
|
||
changes it or `data/` adds its `CHANGELOG.md` entry, which CI checks. A removed,
|
||
renamed or retyped line is a major.
|
||
|
||
## Audience
|
||
|
||
App developers writing tests and fixtures, in Go and at a shell:
|
||
|
||
- a **bulk fixture author**, millions of rows into CSV or SQL
|
||
- a **Go test author**, filling a struct with `FakeStruct`
|
||
- a **hand fixture author**, a few values at a shell
|
||
- a **validator-facing author**, who needs a value a real checker accepts
|
||
- a **data author**, who writes categories under `--data-path` and meets every load
|
||
error the fences raise
|
||
- a **custom-CLI builder**, who ships a CLI like `fejkdata` with data and functions of
|
||
their own
|
||
|
||
and a **contributor**, who reads [`todo.md`](todo.md), [`AGENTS.md`](AGENTS.md) and
|
||
the Development section below, and who ships a register the six above then draw from.
|
||
|
||
## Goals
|
||
|
||
Where the project is heading; the sections before Audience document what ships today.
|
||
|
||
1. **The best fake-data tool there is, and the one developers reach for.**
|
||
1. You can install it the way you install your other tools, and people hear about
|
||
it where developers read.
|
||
2. **Enough data ships built in to fill a real test database: the CLI carries all of it, a
|
||
library imports what it needs, and nothing is fetched at run time.**
|
||
1. It covers the ten most-spoken languages and the Nordic countries.
|
||
2. Each locale covers the common categories: names, addresses, phone numbers, ids,
|
||
companies and dates.
|
||
3. The shipped data comes in Go modules, and a library gets only the modules it is
|
||
handed.
|
||
3. **The code is easy to understand.**
|
||
1. You can go from a bug to the code behind it without asking anyone.
|
||
2. You can change one part without keeping the rest in your head.
|
||
3. Every name says what the thing really is.
|
||
4. Each file makes sense on its own, without another document open beside it.
|
||
4. **Every value is valid.** It passes the same check the real system would apply,
|
||
such as the checksum in a Swedish personal number.
|
||
1. Values that belong together come from the same random pick. You choose which
|
||
ones by giving that pick a name.
|
||
5. **Templates are easy to write and read.** A template is the text you write to
|
||
describe a value, such as `{first} {last}`.
|
||
1. Everything outside `{…}` prints exactly as written.
|
||
2. Each `{…}` makes a new random pick.
|
||
3. A name keeps one pick. `{/person as p}` picks a person and prints nothing.
|
||
`{p}` then prints that person, and `{p.last}` their last name.
|
||
4. A path works like a folder path: `.` goes down a level, and `..` goes up one.
|
||
5. `[SE]` selects one row of a table, and means that row wherever you write it.
|
||
6. If two people could read a template differently, loading it fails, and the
|
||
error shows how to write each meaning.
|
||
7. Each result has one obvious way to write it. A spelling with a part that doesn't
|
||
change anything — a default written out, an item listed twice in a choice, a
|
||
wrapper around what could stand alone — fails to load, and the error shows the
|
||
spelling to write instead.
|
||
8. When a clearer name turns up, it replaces the old one, even if that needs a new
|
||
major version.
|
||
9. Everything fejkdata renders is written as a template: on the command line, in a
|
||
struct tag and in the library.
|
||
6. **Mistakes are caught when the data loads, and the error says plainly what is
|
||
wrong.**
|
||
1. A template that could render a value its own rules forbid, such as `NaN` or `Inf`,
|
||
fails to load instead.
|
||
2. Data that reads something nobody provided fails to load, and the error names the
|
||
module that provides it by default, or says that none does.
|
||
7. **One command gets you a value.**
|
||
1. Flags work like in other command-line tools, and can go anywhere on the line.
|
||
2. Your first template of your own needs no escaping and no options.
|
||
3. The CLI fills in what you leave out with a sane default: all the shipped data, the
|
||
system's randomness and today's date.
|
||
8. **Human readable and editable data formats** — source data is saved in JSON or
|
||
TSV formats when possible.
|
||
9. **Flexible and extensible.**
|
||
1. Data and functions come in modules, a Go module or a data folder, from fejkdata
|
||
or anyone else, and fejkdata's load as anyone else's do.
|
||
2. A module names the modules it reads by default, and anything providing the same
|
||
paths can stand in for one.
|
||
3. Loading two things under one name fails, in the library and the CLI alike, unless
|
||
you say the second replaces the first, and then every read of that name reads
|
||
the second.
|
||
4. The builtins ship with fejkdata itself, so a template means the same in every
|
||
program.
|
||
5. A function loaded from a module can ask for the input it needs, and gets the
|
||
randomness and the date the generator uses.
|
||
6. A CLI carrying your own modules is a few lines of Go, and works just like
|
||
`fejkdata`.
|
||
10. **The same seed, date, data and versions always give the same output.**
|
||
1. In the library, the caller supplies everything that can change the output: data,
|
||
seed and date.
|
||
11. **No unnecessary runtime dependencies** — use standard library by default.
|
||
12. **The docs are a map of the template language.**
|
||
1. Every feature has its own heading.
|
||
2. Every example runs as a test and shows its output.
|
||
3. Each rule is stated in one place.
|
||
13. **It is fast enough that you never notice it with the shipped data.**
|
||
1. A render's cost grows only with the data it reaches.
|
||
14. **Strongly prioritize data sourced from good online sources**
|
||
15. **The most common data comes first,** such as names, addresses, phone numbers and
|
||
dates.
|
||
16. **Values look real by default. You can also ask for values that can never reach
|
||
anything real,** such as a domain nobody can register.
|
||
|
||
## Layout
|
||
|
||
```
|
||
doc.go the package doc, and the vocabulary the package is written in
|
||
fejkdata.go Generator, New, options, List
|
||
shipped.go the embedded data set, and loading a shipped category on the first call reaching it
|
||
shippedindex.go the index of shipped categories, generated from data/
|
||
node.go the node model and JSON -> node compilation
|
||
table.go tables: their options, format and cells, the link to the parent beside them, and how a path passes and draws a table
|
||
name.go names: {x as n} bindings, the scopes they live in, and the reads they answer
|
||
namefence.go the load fences over names
|
||
pick.go a render's named picks: the frames of the name scopes rendering, the memo a pick keeps, and the reads kept under it
|
||
env.go renderEnv: what a render reads names and rows through, and the frames a read opens and finds
|
||
path.go the dotted-path walks, and proving a path reaches a node
|
||
render.go Fake, the recursive renderer (choices, format strings, expansions) and the reads its tokens make
|
||
record.go records: Record, the JSON/CSV/SQL serializers, and their entry points
|
||
struct.go structs: FakeStruct, fake tags as one record's columns, and a field's Go type as its column's datatype
|
||
inline.go inline templates: Template, NewTemplate, FakeTemplate, IsTemplate, and their compile and resolve
|
||
format.go a format's tokens checked against their fields, and compiled into ops
|
||
arm.go the compiled read of one {a|b} alternative or operand: its head, the path to its leaf, and the op holding it
|
||
reference.go one reference's resolution: its sigil, its category, its tail
|
||
resolve.go resolving templates against the assembled tree in steps, each over every template before the next
|
||
graph.go the node graph: tree walks, containment and render edges, and the cycle and repeat fences
|
||
calc.go the {calc()} operand checks: never a number, and a constant zero divisor
|
||
datatype.go column datatypes: DataType, where datatype and null may sit, a column's datatype
|
||
valueproof.go the value proof: what a typed column, calc operand or struct field holds, checked at load
|
||
data.go the load path: the categories internal/datafiles hands over, compiled into one merged tree, and the load pipeline every entry point runs
|
||
cmd/fejkdata/ the fejkdata CLI
|
||
internal/builtinfunc/ the {name()} functions: their checks and draws, checksums, date and time layouts, transforms and calc arithmetic
|
||
internal/datafiles/ the walk of a data tree: its folders, its category files and the rows files beside them
|
||
internal/datatype/ the datatype a record column holds; the root's DataType is an alias of it
|
||
internal/drawstate/ the seeded randomness and the {seq()} counters a generator draws through
|
||
internal/grammar/ how the template language is written: format tokens, paths and selectors, reference sigils, names, calc syntax, and whether an argument is a template or a path
|
||
internal/invariant/ the one phrase every package panics with when an invariant breaks
|
||
internal/proven/ what a proof knows of a value, and the bounds a calc takes from its operands
|
||
internal/rows/ a table's rows: the TSV, the options proved over it, the links between tables, row selection and draws, and the pin set one path or one named pick fixes
|
||
data/ shipped data (JSON, and a TSV per table), embedded at build: locale folders, geo, misc
|
||
data-import/ the scripts that rebuild each sourced table (see DATA-LICENSES.md)
|
||
docs/ the decision log, the register research behind the shipped data, and the survey of other libraries
|
||
release-tooling/ the release CI publishes from the changelog heading
|
||
testdata/ the pinned shipped shape (see Versioning)
|
||
```
|
||
|
||
## Development
|
||
|
||
Everything runs in Docker — **no local tooling beyond Docker is needed**.
|
||
Source is bind-mounted; build caches persist in the `gocache` volume.
|
||
|
||
```sh
|
||
docker compose run --rm test # go test -race
|
||
docker compose run --rm cover # tests with coverage
|
||
docker compose run --rm bench # benchmarks
|
||
docker compose run --rm build # compile the library
|
||
docker compose run --rm vet # go vet
|
||
docker compose run --rm cyclo # cyclomatic complexity over 14 (test files excluded)
|
||
docker compose run --rm dev # interactive shell
|
||
```
|
||
|
||
Commands that rewrite source keep your file ownership when run with `--user`:
|
||
|
||
```sh
|
||
docker compose run --rm --user "$(id -u):$(id -g)" fmt # gofmt -w .
|
||
docker compose run --rm --user "$(id -u):$(id -g)" generate # go generate: shippedindex.go from data/
|
||
docker compose run --rm --user "$(id -u):$(id -g)" tidy # go mod tidy
|
||
```
|
||
|
||
A `--user` command answering `permission denied` on `/cache` has met root-owned
|
||
files in the `gocache` volume: Docker creates the volume root-owned, and every
|
||
command here that omits `--user` writes into it as root. Hand it back, and again
|
||
whenever it recurs:
|
||
|
||
```sh
|
||
docker compose run --rm --user root --entrypoint chown test -R "$(id -u):$(id -g)" /cache
|
||
```
|
||
|
||
Every pull request runs `docker build .` against both the latest and the lowest
|
||
supported Go, and must pass before it can be merged — unless it changes none of
|
||
the files the build and its tests read, nor the workflow itself, in which case
|
||
it's skipped. That build is the whole gate but the
|
||
changelog check, which CI runs against the PR base — vet, complexity, format check
|
||
and tests — so run it locally before pushing. The lowest Go builds the `portable`
|
||
stage, all of that bar the format check:
|
||
|
||
```sh
|
||
docker build . # latest
|
||
docker build --build-arg GO_VERSION=1.22.12 --target portable . # lowest supported
|
||
GO_VERSION=1.22.12 docker compose run --rm test # the same tests, without the image build
|
||
```
|
||
|
||
Goal 3 is scored by a panel of simulated readers, and [`AGENTS.md`](AGENTS.md) says
|
||
what the score asks of a pull request. Goal 5 is checked against simulated template
|
||
writers from the Audience.
|
||
|
||
A change to the shipped data runs `generate` above first and commits
|
||
[`shippedindex.go`](shippedindex.go) with the change, then re-pins
|
||
[`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) in its own commit:
|
||
|
||
```sh
|
||
REPIN=1 docker compose run --rm --user "$(id -u):$(id -g)" test
|
||
```
|
||
|
||
A shipped table built from a source is rebuilt by its script under
|
||
[`data-import/`](data-import), one command per dataset, fetching the source named in
|
||
[`DATA-LICENSES.md`](DATA-LICENSES.md). Downloads are cached under
|
||
`data-import/cache/`, which is ignored by version control and grows past a
|
||
gigabyte, so pass `--cache DIR` to reuse a copy you already have and delete it to
|
||
fetch afresh; `geo-us.py` fetches two
|
||
TIGER/Line files per county it ships, a few hundred megabytes, `geo-se.py` needs
|
||
a Trafikverket API key, free at [data.trafikverket.se](https://data.trafikverket.se/),
|
||
in `TRAFIKVERKET_API_KEY` or a `--key-file`, and the Census host behind `geo-us.py`
|
||
and `names-us.py` rejects a client for a while after a burst, so `--surnames` takes
|
||
a copy of the surname file:
|
||
|
||
```sh
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" -e TRAFIKVERKET_API_KEY data-import data-import/geo-se.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpmethod.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpstatus.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/language.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/loglevel.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/mimetype.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-se.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-us.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/port.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/protocol.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/territory.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/timezone.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/tld.py
|
||
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/useragent.py
|
||
```
|
||
|
||
To release, head `CHANGELOG.md` with the version's section in place of `Unreleased`
|
||
and merge: once `main` passes the gate, CI tags that commit `vX.Y.Z` and publishes
|
||
the GitHub release with the section as its body. A top heading of `[Unreleased]`
|
||
publishes nothing.
|
||
|
||
## License
|
||
|
||
MIT — see [LICENSE](LICENSE). Forked from [github.com/Timewave-AB/fakes](https://github.com/Timewave-AB/fakes).
|