Format text is literal; class runs are {digits(n)} {upper(n)} {lower(n)}; a string is a format

This commit is contained in:
2026-09-02 12:37:21 +02:00
parent 99d8aa0fb7
commit ff4b243148
29 changed files with 308 additions and 267 deletions
+22 -27
View File
@@ -63,7 +63,7 @@ own and render it — no code change. Save this as `mydata/sql.json`:
```json ```json
{ {
"format": "INSERT INTO users V#ALUES({sql-username});", "format": "INSERT INTO users VALUES({sql-username});",
"sql-username": { "sql-username": {
"format": "'{username}'", "format": "'{username}'",
"repeat": 3, "repeat": 3,
@@ -74,9 +74,7 @@ own and render it — no code change. Save this as `mydata/sql.json`:
``` ```
`sql-username` renders `'{username}'` `repeat` times and joins the results with `sql-username` renders `'{username}'` `repeat` times and joins the results with
the `),(` separator; the outer `V#ALUES(…)` wraps that into one valid row list. the `),(` separator; the outer `VALUES(…)` wraps that into one valid row list.
(`#A` escapes the literal `A`, which a format string would otherwise read as a
letter token — see [Data format](#data-format).)
```sh ```sh
fejkdata --seed 1 --data-path ./mydata sql fejkdata --seed 1 --data-path ./mydata sql
@@ -160,7 +158,7 @@ when drawn from one goroutine.
The shipped set under [`data/`](data) — one folder per locale (`en_US`, `sv_SE`) The shipped set under [`data/`](data) — one folder per locale (`en_US`, `sv_SE`)
plus a locale-neutral `misc` folder — is embedded in the library and the CLI, so plus a locale-neutral `misc` folder — is embedded in the library and the CLI, so
both work with no data on disk. Layer your own directories over it; a category or both work with no data on disk. Layer your own directories over it; a category or
folder name must not use `.`, `|`, `(` or `}` (see [Data format](#data-format)), folder name must not use `.`, `|`, `(`, `{` or `}` (see [Data format](#data-format)),
and dot-prefixed entries are skipped, so a data directory can also be a checkout. and dot-prefixed entries are skipped, so a data directory can also be a checkout.
A directory is just a namespace. Each JSON file is a category named after the A directory is just a namespace. Each JSON file is a category named after the
@@ -206,7 +204,7 @@ Every value is a **node**, one of three shapes, nestable without limit:
| Node | JSON | Meaning | | Node | JSON | Meaning |
|------|------|---------| |------|------|---------|
| literal | `"Malmö"` | emitted verbatim — never formatted | | string | `"Malmö"` | a format with no fields: text, or tokens that need none (`"{digits(3)}"`, `"{..path}"`) |
| choice | `["a", "b", …]` | one element, picked at random | | choice | `["a", "b", …]` | one element, picked at random |
| template | `{"format": "…", …}` | a format string plus the named sub-nodes it references | | template | `{"format": "…", …}` | a format string plus the named sub-nodes it references |
@@ -215,9 +213,9 @@ within a choice:
```json ```json
[ [
{ "format": "#070-000 00 00", "weight": 10 }, { "format": "070-{digits(3)} {digits(2)} {digits(2)}", "weight": 10 },
{ "format": "#01-000 00 00" }, { "format": "01-{digits(3)} {digits(2)} {digits(2)}" },
{ "format": "#010-000 00 00" } { "format": "010-{digits(3)} {digits(2)} {digits(2)}" }
] ]
``` ```
@@ -242,8 +240,8 @@ is a field**. So write `seperator` and you get a field by that name while the
option stays unset. `New` rejects an option that cannot take effect — a option stays unset. `New` rejects an option that cannot take effect — a
`separator` without a `repeat` above 1, a `weight` outside a choice — and a name `separator` without a `repeat` above 1, a `weight` outside a choice — and a name
using a character the grammars reserve: `.` separates the segments of a path, `|` using a character the grammars reserve: `.` separates the segments of a path, `|`
the arms of a token, `(` opens a function call and `}` ends the token, so a name the arms of a token, `(` opens a function call and `{` `}` delimit the token, so a
carrying one is a name no format could ever spell. An empty name goes the same name carrying one is a name no format could ever spell. An empty name goes the same
way — it is no path segment at all, so `List` never offers it. That holds for a way — it is no path segment at all, so `List` never offers it. That holds for a
category, a folder and a field alike. category, a folder and a field alike.
@@ -254,7 +252,7 @@ argument counts are rejected at `New`. This is what makes a generated Swedish
personnummer valid — its last digit is a Luhn checksum over the nine before it: personnummer valid — its last digit is a Luhn checksum over the nine before it:
```json ```json
{ "format": "00{mmdd}-000{luhn()}", "mmdd": [ ] } { "format": "{digits(2)}{mmdd}-{digits(3)}{luhn()}", "mmdd": [ ] }
``` ```
renders e.g. `811218-987`, then `{luhn()}` appends `6``811218-9876`. Place it renders e.g. `811218-987`, then `{luhn()}` appends `6``811218-9876`. Place it
@@ -264,7 +262,7 @@ form prefixes the century outside the checksummed core:
```json ```json
{ "format": "{century}{core}", "century": ["19", "20"], { "format": "{century}{core}", "century": ["19", "20"],
"core": { "format": "00{mmdd}-000{luhn()}", "mmdd": [ ] } } "core": { "format": "{digits(2)}{mmdd}-{digits(3)}{luhn()}", "mmdd": [ ] } }
``` ```
A function must be deterministic (no wall-clock), so a seeded generator stays A function must be deterministic (no wall-clock), so a seeded generator stays
@@ -289,6 +287,9 @@ fat-fingered `hex(2000000000)` can't try to allocate gigabytes at render.
| `{ulid()}` | sample | ULID, 26-char Crockford base32 | | `{ulid()}` | sample | ULID, 26-char Crockford base32 |
| `{nanoid(n)}` | sample | URL-safe Nano ID, `n` chars | | `{nanoid(n)}` | sample | URL-safe Nano ID, `n` chars |
| `{hex(n)}` | sample | `n` lowercase hex digits | | `{hex(n)}` | sample | `n` lowercase hex digits |
| `{digits(n)}` | sample | `n` digits 09 |
| `{upper(n)}` | sample | `n` letters AZ |
| `{lower(n)}` | sample | `n` letters az |
| `{base64(n)}` | sample | `n` random bytes, base64 | | `{base64(n)}` | sample | `n` random bytes, base64 |
| `{int(min,max)}` | sample | uniform integer in `[min, max]` | | `{int(min,max)}` | sample | uniform integer in `[min, max]` |
| `{float(min,max,dp)}` | sample | number in `[min, max]` with `dp` decimals | | `{float(min,max,dp)}` | sample | number in `[min, max]` with `dp` decimals |
@@ -349,8 +350,8 @@ locality and the postal code that really covers it, stay together:
```json ```json
{ "format": "{street} {number}\n{place.postal-code} {place.locality}", { "format": "{street} {number}\n{place.postal-code} {place.locality}",
"place": [ "place": [
{ "format": "{locality}", "locality": "Stockholm", "postal-code": { "format": "#100 00" }, "weight": 975 }, { "format": "{locality}", "locality": "Stockholm", "postal-code": "1{digits(2)} {digits(2)}", "weight": 975 },
{ "format": "{locality}", "locality": "Tranås", "postal-code": { "format": "#5#7#3 00" }, "weight": 18 } { "format": "{locality}", "locality": "Tranås", "postal-code": "573 {digits(2)}", "weight": 18 }
] } ] }
``` ```
@@ -411,31 +412,25 @@ carries "postal-code"; all carry [locality]
The sub-fields stay addressable on their own — `Fake("address.place.locality")` The sub-fields stay addressable on their own — `Fake("address.place.locality")`
renders, and `List` advertises it. renders, and `List` advertises it.
**Format string.** Every character is literal except: **Format string.** Every character is literal except a `{…}` token:
| Token | Expands to | | Token | Expands to |
|-------|-----------| |-------|-----------|
| `0` | digit 09 |
| `1` | digit 19 |
| `A` | letter AZ |
| `a` | letter az |
| `#` | escape — the next char is literal (`#0``0`, `##``#`) |
| `{name}` | render the sibling field `name` | | `{name}` | render the sibling field `name` |
| `{name.field}` | render `field` of one draw of the sibling `name` (see **Correlated fields**) | | `{name.field}` | render `field` of one draw of the sibling `name` (see **Correlated fields**) |
| `{name()}` | call a built-in function (see **Functions**) | | `{name()}` | call a built-in function (see **Functions**) |
| `{..path}` | render the node at a dot path from the data root (see **References**) | | `{..path}` | render the node at a dot path from the data root (see **References**) |
| `{{`, `}}` | a literal `{` or `}` |
`{a|b}` renders one of the sibling fields `a` or `b`, chosen at random; an arm `{a|b}` renders one of the sibling fields `a` or `b`, chosen at random; an arm
may be a `{..path}` reference too (`{name|..en_US.person}`). The arms are picked may be a `{..path}` reference too (`{name|..en_US.person}`). The arms are picked
evenly and must differ — `{a|a|b}` would skew the odds, which is what `weight` is evenly and must differ — `{a|a|b}` would skew the odds, which is what `weight` is
for, so a repeated arm is a load error. for, so a repeated arm is a load error.
Inside a `format`, `0 1 A a` are **always** character classes — so a fixed `0`, Text means what it says: `100 Main St` renders `100 Main St`. Random characters
`1`, `A` or `a` must be escaped (`#1`, `#A`) or it becomes random. A format of come from the sample functions — `{digits(3)}`, `{upper(1)}`, `{lower(2)}`,
`100 Main St` renders e.g. `506 Mdin St` — the `1`, `0`, `0` and `a` were random, `{int(10,99)}` — so a phone pattern is `070-{digits(3)} {digits(2)} {digits(2)}`.
the `M`, `in` and `St` were not. A half-fixed string is the trap: `555-0000` A lone `}` is a load error naming `}}`.
keeps `555-` and randomises the last four digits. For a value with no
tokens at all, use a bare string node (`"100 Main St"`), emitted verbatim.
**Putting it together** (`person.json`): **Putting it together** (`person.json`):
+27 -35
View File
@@ -16,27 +16,21 @@ const (
maxDecimals = 1024 maxDecimals = 1024
) )
// builtins is the registry of {name(args)} functions. Derivations read the // builtins is the registry of {name(args)} functions. Derivations read the digits
// digits emitted so far in the current expansion (luhn, mod11, ean — place them // emitted so far in the current expansion (place them after their payload);
// after their payload); samples read only the rng (uuid, ulid, ...). All // samples read only the rng. A time-based id (uuid v7, ulid) draws its timestamp
// must stay pure over (rng, emitted, args) so seeded output is reproducible — a // from the rng, not the wall clock, so seeded output stays reproducible.
// time-based id (uuid v7, ulid) draws its timestamp from the rng, not the wall
// clock. Add a builtin only for what data can't express: a random v4 UUID and a
// 24-hex ObjectID both ship as data, so the uuid builtin is v7.
var builtins = map[string]builtin{ var builtins = map[string]builtin{
"luhn": {arity: 0, prep: derive(func(e string) string { return string(rune('0' + luhnCheck(e))) })}, "luhn": {arity: 0, prep: derive(func(e string) string { return string(rune('0' + luhnCheck(e))) })},
"mod11": {arity: 0, prep: derive(mod11Check)}, "mod11": {arity: 0, prep: derive(mod11Check)},
"ean": {arity: 0, prep: derive(eanCheck)}, "ean": {arity: 0, prep: derive(eanCheck)},
"uuid": {arity: 0, prep: sample(uuidV7)}, "uuid": {arity: 0, prep: sample(uuidV7)},
"ulid": {arity: 0, prep: sample(ulid)}, "ulid": {arity: 0, prep: sample(ulid)},
"nanoid": {arity: 1, check: posIntArg, prep: func(a []string) callFn { "nanoid": {arity: 1, check: posIntArg, prep: chars(nanoidAlphabet)},
n := atoi(a[0]) "hex": {arity: 1, check: posIntArg, prep: chars(hexDigits)},
return func(s *session, _ string, _ []string) string { return nanoid(s, n) } "digits": {arity: 1, check: posIntArg, prep: chars("0123456789")},
}}, "upper": {arity: 1, check: posIntArg, prep: chars("ABCDEFGHIJKLMNOPQRSTUVWXYZ")},
"hex": {arity: 1, check: posIntArg, prep: func(a []string) callFn { "lower": {arity: 1, check: posIntArg, prep: chars("abcdefghijklmnopqrstuvwxyz")},
n := atoi(a[0])
return func(s *session, _ string, _ []string) string { return randHex(s, n) }
}},
"base64": {arity: 1, check: posIntArg, prep: func(a []string) callFn { "base64": {arity: 1, check: posIntArg, prep: func(a []string) callFn {
n := atoi(a[0]) n := atoi(a[0])
return func(s *session, _ string, _ []string) string { return func(s *session, _ string, _ []string) string {
@@ -74,9 +68,9 @@ var builtins = map[string]builtin{
}}, }},
} }
// derive and sample are the two argument-free builtin shapes the README names: a // derive and sample are the two argument-free builtin shapes: a derivation reads
// derivation reads the output emitted so far, a sample reads only the rng. Each // the output emitted so far, a sample reads only the rng. chars is the shape of a
// lifts that one function into the prep every registry entry supplies. // sample of n characters drawn from an alphabet.
func derive(f func(emitted string) string) func([]string) callFn { func derive(f func(emitted string) string) func([]string) callFn {
return func([]string) callFn { return func([]string) callFn {
return func(_ *session, emitted string, _ []string) string { return f(emitted) } return func(_ *session, emitted string, _ []string) string { return f(emitted) }
@@ -89,6 +83,13 @@ func sample(f func(rng) string) func([]string) callFn {
} }
} }
func chars(alphabet string) func([]string) callFn {
return func(a []string) callFn {
n := atoi(a[0])
return func(s *session, _ string, _ []string) string { return randChars(s, n, alphabet) }
}
}
const hexDigits = "0123456789abcdef" const hexDigits = "0123456789abcdef"
// atoi parses an arg a builtin's check already validated. It panics rather than // atoi parses an arg a builtin's check already validated. It panics rather than
@@ -119,10 +120,10 @@ func randBytes(r rng, n int) []byte {
return b return b
} }
func randHex(r rng, n int) string { func randChars(r rng, n int, alphabet string) string {
b := make([]byte, n) b := make([]byte, n)
for i := range b { for i := range b {
b[i] = hexDigits[r.IntN(16)] b[i] = alphabet[r.IntN(len(alphabet))]
} }
return string(b) return string(b)
} }
@@ -182,18 +183,9 @@ func seqArg(_ map[string]node, a []string) error {
return nil return nil
} }
// nanoidAlphabet is the 64-char URL-safe set Nano IDs use (order is irrelevant // nanoidAlphabet is the 64-char URL-safe set Nano IDs use.
// to the uniform pick).
const nanoidAlphabet = "_-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ" const nanoidAlphabet = "_-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
func nanoid(r rng, n int) string {
b := make([]byte, n)
for i := range b {
b[i] = nanoidAlphabet[r.IntN(len(nanoidAlphabet))]
}
return string(b)
}
// uuidV7 builds an RFC 9562 v7 UUID. The 48-bit timestamp field is drawn from // uuidV7 builds an RFC 9562 v7 UUID. The 48-bit timestamp field is drawn from
// the rng (not the clock) to stay reproducible, then the version (7) and variant // the rng (not the clock) to stay reproducible, then the version (7) and variant
// (10) bits are forced; the rest is random. // (10) bits are forced; the rest is random.
-10
View File
@@ -8,16 +8,6 @@ import (
"unicode" "unicode"
) )
// calc is the {calc(expr[, dp])} token: an arithmetic expression over number
// literals and sibling-field names, with + - * /, unary minus and parentheses. It is
// a registry builtin like any other {name(args)} function — the one that names
// operands, which expand reads for it (once per expansion, so the value computed is
// the value the format showed) and hands over as strings. The value prints in
// minimal decimal form, or rounded to dp when given. An operand that isn't a number
// becomes NaN, which propagates and prints as "NaN" — visible, never a render error.
// The expression is parsed once at New into an AST every render shares and none
// mutates, so render cannot fail and must not carry per-render state.
// calcNode is a parsed expression node. It evaluates over the operand values expand // calcNode is a parsed expression node. It evaluates over the operand values expand
// read, so the evaluator touches neither the rng nor the node tree. // read, so the evaluator touches neither the rng nor the node tree.
type calcNode interface { type calcNode interface {
+6 -6
View File
@@ -9,16 +9,16 @@
} }
], ],
"street-number": [ "street-number": [
{ "format": "10" }, { "format": "{int(10,99)}" },
{ "format": "100", "weight": 2 }, { "format": "{int(100,999)}", "weight": 2 },
{ "format": "1000", "weight": 0.5 }, { "format": "{int(1000,9999)}", "weight": 0.5 },
{ "format": "10100", "weight": 0.3 } { "format": "{int(10,99)}{int(100,999)}", "weight": 0.3 }
], ],
"locality": ["Albany", "Atlanta", "Austin", "Baltimore", "Boston", "Charlotte", "Chicago", "Cincinnati", "Cleveland", "Columbus", "Dallas", "Denver", "Detroit", "El Paso", "Fort Worth", "Fresno", "Houston", "Indianapolis", "Jacksonville", "Kansas City", "Las Vegas", "Long Beach", "Los Angeles", "Memphis", "Mesa", "Miami", "Milwaukee", "Minneapolis", "Nashville", "New Orleans", "Oakland", "Oklahoma City", "Omaha", "Orlando", "Philadelphia", "Phoenix", "Pittsburgh", "Portland", "Raleigh", "Sacramento", "San Antonio", "San Diego", "San Jose", "Seattle", "St. Louis", "Tampa", "Tucson", "Tulsa"], "locality": ["Albany", "Atlanta", "Austin", "Baltimore", "Boston", "Charlotte", "Chicago", "Cincinnati", "Cleveland", "Columbus", "Dallas", "Denver", "Detroit", "El Paso", "Fort Worth", "Fresno", "Houston", "Indianapolis", "Jacksonville", "Kansas City", "Las Vegas", "Long Beach", "Los Angeles", "Memphis", "Mesa", "Miami", "Milwaukee", "Minneapolis", "Nashville", "New Orleans", "Oakland", "Oklahoma City", "Omaha", "Orlando", "Philadelphia", "Phoenix", "Pittsburgh", "Portland", "Raleigh", "Sacramento", "San Antonio", "San Diego", "San Jose", "Seattle", "St. Louis", "Tampa", "Tucson", "Tulsa"],
"region": ["AL", "AZ", "CA", "CO", "CT", "FL", "GA", "IL", "IN", "KY", "LA", "MA", "MD", "MI", "MN", "MO", "NC", "NJ", "NV", "NY", "OH", "OK", "OR", "PA", "TN", "TX", "VA", "WA", "WI"], "region": ["AL", "AZ", "CA", "CO", "CT", "FL", "GA", "IL", "IN", "KY", "LA", "MA", "MD", "MI", "MN", "MO", "NC", "NJ", "NV", "NY", "OH", "OK", "OR", "PA", "TN", "TX", "VA", "WA", "WI"],
"postal-code": [ "postal-code": [
{ "format": "10000" }, { "format": "{int(10000,99999)}" },
{ "format": "10000-0000", "weight": 0.3 } { "format": "{int(10000,99999)}-{digits(4)}", "weight": 0.3 }
] ]
} }
] ]
+1 -1
View File
@@ -1,6 +1,6 @@
[ [
{ {
"format": "##{x}{x}{x}{x}{x}{x}", "format": "#{x}{x}{x}{x}{x}{x}",
"x": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "d", "e", "f"], "x": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "d", "e", "f"],
"weight": 2 "weight": 2
}, },
+6 -6
View File
@@ -4,12 +4,12 @@
"month": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"], "month": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"],
"day": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28"], "day": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28"],
"year": [ "year": [
{ "format": "#1970" }, { "format": "197{digits(1)}" },
{ "format": "#1980" }, { "format": "198{digits(1)}" },
{ "format": "#1990", "weight": 2 }, { "format": "199{digits(1)}", "weight": 2 },
{ "format": "2#0#00", "weight": 3 }, { "format": "200{digits(1)}", "weight": 3 },
{ "format": "2#0#10", "weight": 3 }, { "format": "201{digits(1)}", "weight": 3 },
{ "format": "2#020", "weight": 2 } { "format": "202{digits(1)}", "weight": 2 }
] ]
} }
] ]
+1 -1
View File
@@ -2,7 +2,7 @@
{ {
"format": "{o}.{o}.{o}.{o}", "format": "{o}.{o}.{o}.{o}",
"weight": 8, "weight": 8,
"o": [{ "format": "0", "weight": 3 }, { "format": "10", "weight": 4 }, { "format": "#100", "weight": 2 }] "o": [{ "format": "{digits(1)}", "weight": 3 }, { "format": "{int(10,99)}", "weight": 4 }, { "format": "1{digits(2)}", "weight": 2 }]
}, },
{ {
"format": "{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}" "format": "{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}"
+4 -4
View File
@@ -3,13 +3,13 @@
"format": "({area}) {exch}-{line}", "format": "({area}) {exch}-{line}",
"weight": 2, "weight": 2,
"area": ["202", "212", "213", "305", "312", "404", "415", "469", "503", "512", "602", "617", "646", "702", "718", "720", "737", "773", "786", "808", "917"], "area": ["202", "212", "213", "305", "312", "404", "415", "469", "503", "512", "602", "617", "646", "702", "718", "720", "737", "773", "786", "808", "917"],
"exch": [{ "format": "100" }], "exch": [{ "format": "{int(100,999)}" }],
"line": [{ "format": "0000" }] "line": [{ "format": "{digits(4)}" }]
}, },
{ {
"format": "{area}-{exch}-{line}", "format": "{area}-{exch}-{line}",
"area": ["202", "212", "213", "305", "312", "404", "415", "469", "503", "512", "602", "617", "646", "702", "718", "720", "737", "773", "786", "808", "917"], "area": ["202", "212", "213", "305", "312", "404", "415", "469", "503", "512", "602", "617", "646", "702", "718", "720", "737", "773", "786", "808", "917"],
"exch": [{ "format": "100" }], "exch": [{ "format": "{int(100,999)}" }],
"line": [{ "format": "0000" }] "line": [{ "format": "{digits(4)}" }]
} }
] ]
+7 -7
View File
@@ -2,15 +2,15 @@
{ {
"format": "${amt}.{cents}", "format": "${amt}.{cents}",
"amt": [ "amt": [
{ "format": "1", "weight": 2 }, { "format": "{int(1,9)}", "weight": 2 },
{ "format": "10", "weight": 4 }, { "format": "{int(10,99)}", "weight": 4 },
{ "format": "100", "weight": 3 }, { "format": "{int(100,999)}", "weight": 3 },
{ "format": "1#,000", "weight": 1 }, { "format": "{int(1,9)},{digits(3)}", "weight": 1 },
{ "format": "10#,000", "weight": 0.4 }, { "format": "{int(10,99)},{digits(3)}", "weight": 0.4 },
{ "format": "100#,000", "weight": 0.1 } { "format": "{int(100,999)},{digits(3)}", "weight": 0.1 }
], ],
"cents": [ "cents": [
{ "format": "00", "weight": 3 }, { "format": "{digits(2)}", "weight": 3 },
"49", "49",
"95", "95",
"99" "99"
+1 -1
View File
@@ -1,3 +1,3 @@
[ [
{ "format": "100-00-0000" } { "format": "{int(100,999)}-{digits(2)}-{digits(4)}" }
] ]
+1 -1
View File
@@ -2,7 +2,7 @@
{ {
"format": "{hour}:{minute} {ampm}", "format": "{hour}:{minute} {ampm}",
"hour": ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12"], "hour": ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12"],
"minute": [{ "format": "{t}0", "t": ["0", "1", "2", "3", "4", "5"] }], "minute": [{ "format": "{t}{digits(1)}", "t": ["0", "1", "2", "3", "4", "5"] }],
"ampm": ["AM", "PM"] "ampm": ["AM", "PM"]
} }
] ]
+1 -1
View File
@@ -1,7 +1,7 @@
[ [
{ {
"format": "{pre}{n}.{n}.{n}{suffix}", "format": "{pre}{n}.{n}.{n}{suffix}",
"n": [{ "format": "0", "weight": 3 }, { "format": "10" }], "n": [{ "format": "{digits(1)}", "weight": 3 }, { "format": "{int(10,99)}" }],
"pre": ["", { "format": "v", "weight": 0.4 }], "pre": ["", { "format": "v", "weight": 0.4 }],
"suffix": ["", { "format": "-{tag}.{m}", "tag": ["alpha", "beta", "rc"], "m": ["1", "2", "3"], "weight": 0.3 }] "suffix": ["", { "format": "-{tag}.{m}", "tag": ["alpha", "beta", "rc"], "m": ["1", "2", "3"], "weight": 0.3 }]
} }
+3 -3
View File
@@ -1,5 +1,5 @@
[ [
{ "format": "4{d}{luhn()}", "d": { "format": "0", "repeat": 14 }, "weight": 4 }, { "format": "4{d}{luhn()}", "d": { "format": "{digits(1)}", "repeat": 14 }, "weight": 4 },
{ "format": "5{m}{d}{luhn()}", "m": ["1", "2", "3", "4", "5"], "d": { "format": "0", "repeat": 13 }, "weight": 3 }, { "format": "5{m}{d}{luhn()}", "m": ["1", "2", "3", "4", "5"], "d": { "format": "{digits(1)}", "repeat": 13 }, "weight": 3 },
{ "format": "3{m}{d}{luhn()}", "m": ["4", "7"], "d": { "format": "0", "repeat": 12 }, "weight": 1 } { "format": "3{m}{d}{luhn()}", "m": ["4", "7"], "d": { "format": "{digits(1)}", "repeat": 12 }, "weight": 1 }
] ]
+7 -7
View File
@@ -10,15 +10,15 @@
["Avenyn", "Birger Jarlsgatan", "Promenaden", "Staby", "Sveavägen", "Vintjärn"] ["Avenyn", "Birger Jarlsgatan", "Promenaden", "Staby", "Sveavägen", "Vintjärn"]
], ],
"street-number": [ "street-number": [
{ "format": "1" }, { "format": "{int(1,9)}" },
{ "format": "10" }, { "format": "{int(10,99)}" },
{ "format": "100", "weight": 0.2 }, { "format": "{int(100,999)}", "weight": 0.2 },
{ "format": "1A", "weight": 0.2 }, { "format": "{int(1,9)}{upper(1)}", "weight": 0.2 },
{ "format": "10A", "weight": 0.1 }, { "format": "{int(10,99)}{upper(1)}", "weight": 0.1 },
{ "format": "100A", "weight": 0.05 } { "format": "{int(100,999)}{upper(1)}", "weight": 0.05 }
], ],
"postal-code": [ "postal-code": [
{ "format": "100 10" } { "format": "{int(100,999)} {int(10,99)}" }
], ],
"locality": ["Alingsås", "Alvesta", "Ängelholm", "Arboga", "Arvika", "Avesta", "Boden", "Bollnäs", "Borås", "Borlänge", "Enköping", "Eskilstuna", "Eslöv", "Fagersta", "Falkenberg", "Falköping", "Falun", "Finspång", "Gällivare", "Gävle", "Göteborg", "Halmstad", "Haparanda", "Härnösand", "Hässleholm", "Helsingborg", "Huddinge", "Hudiksvall", "Jönköping", "Kalmar", "Karlshamn", "Karlskoga", "Karlskrona", "Karlstad", "Katrineholm", "Kiruna", "Köping", "Kramfors", "Kristianstad", "Kristinehamn", "Landskrona", "Lidingö", "Lidköping", "Lindesberg", "Linköping", "Ljungby", "Ludvika", "Luleå", "Lund", "Lycksele", "Malmö", "Mariestad", "Mjölby", "Mölndal", "Mora", "Motala", "Nacka", "Nässjö", "Norrköping", "Norrtälje", "Nyköping", "Nynäshamn", "Örebro", "Örnsköldsvik", "Oskarshamn", "Östersund", "Piteå", "Rabbalshede", "Ronneby", "Säffle", "Sandviken", "Sävsjö", "Sigtuna", "Skara", "Skellefteå", "Skövde", "Söderhamn", "Södertälje", "Sollentuna", "Solna", "Sölvesborg", "Stockholm", "Strängnäs", "Sundbyberg", "Sundsvall", "Täby", "Tierp", "Tranås", "Trelleborg", "Trollhättan", "Uddevalla", "Ulricehamn", "Umeå", "Upplands Väsby", "Uppsala", "Vänersborg", "Varberg", "Värnamo", "Västerås", "Västervik", "Växjö", "Vetlanda", "Vimmerby", "Visby", "Ystad"] "locality": ["Alingsås", "Alvesta", "Ängelholm", "Arboga", "Arvika", "Avesta", "Boden", "Bollnäs", "Borås", "Borlänge", "Enköping", "Eskilstuna", "Eslöv", "Fagersta", "Falkenberg", "Falköping", "Falun", "Finspång", "Gällivare", "Gävle", "Göteborg", "Halmstad", "Haparanda", "Härnösand", "Hässleholm", "Helsingborg", "Huddinge", "Hudiksvall", "Jönköping", "Kalmar", "Karlshamn", "Karlskoga", "Karlskrona", "Karlstad", "Katrineholm", "Kiruna", "Köping", "Kramfors", "Kristianstad", "Kristinehamn", "Landskrona", "Lidingö", "Lidköping", "Lindesberg", "Linköping", "Ljungby", "Ludvika", "Luleå", "Lund", "Lycksele", "Malmö", "Mariestad", "Mjölby", "Mölndal", "Mora", "Motala", "Nacka", "Nässjö", "Norrköping", "Norrtälje", "Nyköping", "Nynäshamn", "Örebro", "Örnsköldsvik", "Oskarshamn", "Östersund", "Piteå", "Rabbalshede", "Ronneby", "Säffle", "Sandviken", "Sävsjö", "Sigtuna", "Skara", "Skellefteå", "Skövde", "Söderhamn", "Södertälje", "Sollentuna", "Solna", "Sölvesborg", "Stockholm", "Strängnäs", "Sundbyberg", "Sundsvall", "Täby", "Tierp", "Tranås", "Trelleborg", "Trollhättan", "Uddevalla", "Ulricehamn", "Umeå", "Upplands Väsby", "Uppsala", "Vänersborg", "Varberg", "Värnamo", "Västerås", "Västervik", "Växjö", "Vetlanda", "Vimmerby", "Visby", "Ystad"]
} }
+1 -1
View File
@@ -1,6 +1,6 @@
[ [
{ {
"format": "##{x}{x}{x}{x}{x}{x}", "format": "#{x}{x}{x}{x}{x}{x}",
"x": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "d", "e", "f"], "x": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "d", "e", "f"],
"weight": 2 "weight": 2
}, },
+6 -6
View File
@@ -4,12 +4,12 @@
"month": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"], "month": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"],
"day": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28"], "day": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28"],
"year": [ "year": [
{ "format": "#1970" }, { "format": "197{digits(1)}" },
{ "format": "#1980" }, { "format": "198{digits(1)}" },
{ "format": "#1990", "weight": 2 }, { "format": "199{digits(1)}", "weight": 2 },
{ "format": "2#0#00", "weight": 3 }, { "format": "200{digits(1)}", "weight": 3 },
{ "format": "2#0#10", "weight": 3 }, { "format": "201{digits(1)}", "weight": 3 },
{ "format": "2#020", "weight": 2 } { "format": "202{digits(1)}", "weight": 2 }
] ]
} }
] ]
+1 -1
View File
@@ -2,7 +2,7 @@
{ {
"format": "{o}.{o}.{o}.{o}", "format": "{o}.{o}.{o}.{o}",
"weight": 8, "weight": 8,
"o": [{ "format": "0", "weight": 3 }, { "format": "10", "weight": 4 }, { "format": "#100", "weight": 2 }] "o": [{ "format": "{digits(1)}", "weight": 3 }, { "format": "{int(10,99)}", "weight": 4 }, { "format": "1{digits(2)}", "weight": 2 }]
}, },
{ {
"format": "{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}" "format": "{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}:{hex(4)}"
+6 -6
View File
@@ -3,15 +3,15 @@
"format": "{prefix}-{a} {b} {c}", "format": "{prefix}-{a} {b} {c}",
"weight": 10, "weight": 10,
"prefix": ["070", "072", "073", "076", "079"], "prefix": ["070", "072", "073", "076", "079"],
"a": [{ "format": "000" }], "a": [{ "format": "{digits(3)}" }],
"b": [{ "format": "00" }], "b": [{ "format": "{digits(2)}" }],
"c": [{ "format": "00" }] "c": [{ "format": "{digits(2)}" }]
}, },
{ {
"format": "{prefix}-{a} {b} {c}", "format": "{prefix}-{a} {b} {c}",
"prefix": ["08", "011", "013", "018", "019", "021", "023", "026", "031", "033", "035", "036", "040", "042", "044", "046", "054", "060", "063", "090"], "prefix": ["08", "011", "013", "018", "019", "021", "023", "026", "031", "033", "035", "036", "040", "042", "044", "046", "054", "060", "063", "090"],
"a": [{ "format": "000" }], "a": [{ "format": "{digits(3)}" }],
"b": [{ "format": "00" }], "b": [{ "format": "{digits(2)}" }],
"c": [{ "format": "00" }] "c": [{ "format": "{digits(2)}" }]
} }
] ]
+6 -6
View File
@@ -2,12 +2,12 @@
{ {
"format": "{amt}{ore} kr", "format": "{amt}{ore} kr",
"amt": [ "amt": [
{ "format": "1", "weight": 2 }, { "format": "{int(1,9)}", "weight": 2 },
{ "format": "10", "weight": 4 }, { "format": "{int(10,99)}", "weight": 4 },
{ "format": "100", "weight": 3 }, { "format": "{int(100,999)}", "weight": 3 },
{ "format": "1 000", "weight": 1 }, { "format": "{int(1,9)} {digits(3)}", "weight": 1 },
{ "format": "10 000", "weight": 0.3 } { "format": "{int(10,99)} {digits(3)}", "weight": 0.3 }
], ],
"ore": ["", { "format": ",{c}", "c": [{ "format": "00" }], "weight": 0.4 }] "ore": ["", { "format": ",{c}", "c": [{ "format": "{digits(2)}" }], "weight": 0.4 }]
} }
] ]
+1 -1
View File
@@ -1,6 +1,6 @@
[ [
{ {
"format": "00{mmdd}-000{luhn()}", "format": "{digits(2)}{mmdd}-{digits(3)}{luhn()}",
"mmdd": [ "mmdd": [
{ {
"format": "{m}{d}", "format": "{m}{d}",
+3 -3
View File
@@ -1,8 +1,8 @@
[ [
{ {
"format": "{hour}:{minute}{sec}", "format": "{hour}:{minute}{sec}",
"hour": [{ "format": "#00", "weight": 4 }, { "format": "#10", "weight": 4 }, { "format": "2{t}", "t": ["0", "1", "2", "3"], "weight": 2 }], "hour": [{ "format": "0{digits(1)}", "weight": 4 }, { "format": "1{digits(1)}", "weight": 4 }, { "format": "2{t}", "t": ["0", "1", "2", "3"], "weight": 2 }],
"minute": [{ "format": "{t}0", "t": ["0", "1", "2", "3", "4", "5"] }], "minute": [{ "format": "{t}{digits(1)}", "t": ["0", "1", "2", "3", "4", "5"] }],
"sec": ["", { "format": ":{s}", "s": [{ "format": "{t}0", "t": ["0", "1", "2", "3", "4", "5"] }], "weight": 0.4 }] "sec": ["", { "format": ":{s}", "s": [{ "format": "{t}{digits(1)}", "t": ["0", "1", "2", "3", "4", "5"] }], "weight": 0.4 }]
} }
] ]
+1 -1
View File
@@ -1,7 +1,7 @@
[ [
{ {
"format": "{pre}{n}.{n}.{n}{suffix}", "format": "{pre}{n}.{n}.{n}{suffix}",
"n": [{ "format": "0", "weight": 3 }, { "format": "10" }], "n": [{ "format": "{digits(1)}", "weight": 3 }, { "format": "{int(10,99)}" }],
"pre": ["", { "format": "v", "weight": 0.4 }], "pre": ["", { "format": "v", "weight": 0.4 }],
"suffix": ["", { "format": "-{tag}.{m}", "tag": ["alpha", "beta", "rc"], "m": ["1", "2", "3"], "weight": 0.3 }] "suffix": ["", { "format": "-{tag}.{m}", "tag": ["alpha", "beta", "rc"], "m": ["1", "2", "3"], "weight": 0.3 }]
} }
-2
View File
@@ -156,8 +156,6 @@ func paths(n node) []string {
out = append(out, p) out = append(out, p)
} }
return out return out
case literal:
return []string{""}
} }
return nil return nil
} }
+72
View File
@@ -0,0 +1,72 @@
#!/usr/bin/env python3
"""Rewrite data files from the class-char format grammar to the literal-text one.
convert.py data/sv_SE/person.json ...
Old: 0 1 A a are character classes, # escapes. New: text is literal; a run of
0/A/a becomes {digits(n)}/{upper(n)}/{lower(n)}, 1 followed by n zeros becomes
{int(10^n, 10^(n+1)-1)}, a literal brace becomes {{ or }}.
"""
import re
import sys
RUN = {"0": "{{digits({})}}", "A": "{{upper({})}}", "a": "{{lower({})}}"}
def literal(c):
return {"{": "{{", "}": "}}"}.get(c, c)
def convert_format(f):
out, i, n = [], 0, len(f)
while i < n:
c = f[i]
if c == "#":
i += 1
if i < n:
out.append(literal(f[i]))
i += 1
else:
out.append("#")
elif c == "{":
j = f.find("}", i)
if j < 0:
out.append(f[i:])
break
out.append(f[i:j + 1])
i = j + 1
elif c in RUN:
k = 0
while i < n and f[i] == c:
k += 1
i += 1
out.append(RUN[c].format(k))
elif c == "1":
i += 1
k = 0
while i < n and f[i] == "0":
k += 1
i += 1
out.append("{{int({},{})}}".format(10 ** k, 10 ** (k + 1) - 1))
else:
out.append(literal(c))
i += 1
return "".join(out)
FORMAT_VALUE = re.compile(r'("format":\s*")((?:[^"\\]|\\.)*)(")')
def convert_text(text):
return FORMAT_VALUE.sub(lambda m: m.group(1) + convert_format(m.group(2)) + m.group(3), text)
if __name__ == "__main__":
for path in sys.argv[1:]:
with open(path) as fh:
before = fh.read()
after = convert_text(before)
if after != before:
with open(path, "w") as fh:
fh.write(after)
print("converted", path)
+40 -20
View File
@@ -7,16 +7,11 @@ import (
"strings" "strings"
) )
// node is a compiled template element: literal, choice, or template. Compiling // node is a compiled template element: a choice or a template. Compiling JSON
// JSON into these once (see compile) means rendering never re-inspects the raw // into these once (see compile) means rendering never re-inspects the raw JSON or
// JSON or re-sums weights. // re-sums weights.
type node interface{ isNode() } type node interface{ isNode() }
// literal is emitted verbatim, never formatted.
type literal string
func (literal) isNode() {}
// group is a namespace of named children, built from a directory of JSON files // group is a namespace of named children, built from a directory of JSON files
// and subdirectories. It has no value of its own: descend into a named child by // and subdirectories. It has no value of its own: descend into a named child by
// dot path; rendering one is an error (see Fake). // dot path; rendering one is an error (see Fake).
@@ -36,16 +31,19 @@ type choice struct {
func (*choice) isNode() {} func (*choice) isNode() {}
// template renders a format string, substituting {tokens} from fields. repeat // template renders a format string, substituting {tokens} from fields. A bare
// (default 1) renders that format that many times and joins the results with // JSON string is a template with no fields. repeat (default 1) renders that format
// separator (default ""), each render an independent pick. // that many times and joins the results with separator (default ""), each render
// an independent pick.
type template struct { type template struct {
format string format string
fields map[string]node fields map[string]node
repeat int repeat int
separator string separator string
ops []op // format compiled once (see compileOps); what expand walks ops []op // format compiled once (see compileOps); what expand walks
grow int // minimum output size, to size the render buffer grow int // minimum output size, to size the render buffer
fixed bool // no op varies, so every render is lit
lit string // the whole output when fixed
// bound maps each field the format addresses by dotted path to one path token // bound maps each field the format addresses by dotted path to one path token
// reading it, which is the half of an overlap the fences name. nil when the // reading it, which is the half of an overlap the fences name. nil when the
// format takes no path. // format takes no path.
@@ -72,7 +70,7 @@ func compile(v any) (node, error) {
func compileItem(v any) (node, error) { func compileItem(v any) (node, error) {
switch v := v.(type) { switch v := v.(type) {
case string: case string:
return literal(v), nil return compileString(v)
case []any: case []any:
return compileChoice(v) return compileChoice(v)
case map[string]any: case map[string]any:
@@ -82,6 +80,29 @@ func compileItem(v any) (node, error) {
} }
} }
func compileString(s string) (node, error) {
if err := checkTokens(s, nil); err != nil {
return nil, err
}
t := &template{format: s, repeat: 1}
t.compileFormat()
return t, nil
}
// compileFormat compiles the format into ops once every field is in place.
func (t *template) compileFormat() {
t.ops, t.grow, t.bound, t.held = compileOps(t.format)
t.fixed = true
for _, o := range t.ops {
if o.kind != 'l' {
t.fixed = false
}
}
if t.fixed && len(t.ops) == 1 {
t.lit = t.ops[0].lit
}
}
func compileChoice(items []any) (node, error) { func compileChoice(items []any) (node, error) {
if len(items) == 0 { if len(items) == 0 {
return nil, fmt.Errorf("empty choice") return nil, fmt.Errorf("empty choice")
@@ -163,7 +184,7 @@ func compileTemplate(m map[string]any) (node, error) {
if err := checkTokens(format, t.fields); err != nil { if err := checkTokens(format, t.fields); err != nil {
return nil, err return nil, err
} }
t.ops, t.grow, t.bound, t.held = compileOps(format) t.compileFormat()
if err := checkNoOverlap(format, t.bound); err != nil { if err := checkNoOverlap(format, t.bound); err != nil {
return nil, err return nil, err
} }
@@ -205,8 +226,6 @@ func checkPath(n node, tail []string, level string) error {
return nil return nil
} }
return checkPath(n.items[0], tail, level) return checkPath(n.items[0], tail, level)
case literal:
return fmt.Errorf("a plain string has no field %q", tail[0])
default: default:
return fmt.Errorf("cannot descend into %T at %q", n, tail[0]) return fmt.Errorf("cannot descend into %T at %q", n, tail[0])
} }
@@ -255,9 +274,10 @@ func weightOf(raw any) (float64, error) {
// reservedInName is what a category, folder or field name may not contain: a dot // reservedInName is what a category, folder or field name may not contain: a dot
// separates the segments of a path, '|' the arms of a token, '(' opens a function // separates the segments of a path, '|' the arms of a token, '(' opens a function
// call and '}' ends the token. A name carrying one is reachable by no format, so it // call and braces delimit the token. A name carrying one is reachable by no format,
// is rejected where it is authored rather than at the token that cannot reach it. // so it is rejected where it is authored rather than at the token that cannot
const reservedInName = ".|(}" // reach it.
const reservedInName = ".|({}"
// reservedList spells reservedInName for an error message, so the two cannot drift. // reservedList spells reservedInName for an error message, so the two cannot drift.
var reservedList = strings.Join(strings.Split(reservedInName, ""), " ") var reservedList = strings.Join(strings.Split(reservedInName, ""), " ")
+12 -8
View File
@@ -90,11 +90,10 @@ func checkBoundLevelsHeld(root map[string]node) error {
} }
// cover collects what one held draw of a level answers for: the level and // cover collects what one held draw of a level answers for: the level and
// everything contained in it, since a path may read any of it. Literals are left // everything contained in it, since a path may read any of it. A fixed string is
// out — one fixed string cannot disagree with itself, and a literal is a value, so // left out — it cannot disagree with itself.
// two that spell the same text are indistinguishable.
func cover(n node, into map[node]bool) { func cover(n node, into map[node]bool) {
if _, fixed := n.(literal); fixed { if isFixed(n) {
return return
} }
into[n] = true into[n] = true
@@ -111,10 +110,9 @@ func cover(n node, into map[node]bool) {
// The walk stops at a {..path} edge, which is where the operand's own value ends // The walk stops at a {..path} edge, which is where the operand's own value ends
// and a shared source begins: two names referencing one category are two draws, the // and a shared source begins: two names referencing one category are two draws, the
// same rule {word} {word} follows. cover stops there too, by way of named, so both // same rule {word} {word} follows. cover stops there too, by way of named, so both
// halves of the fence end at the same boundary. A literal is left out for the // halves of the fence end at the same boundary.
// reason cover leaves one out.
func operandDraw(n node, into map[node]bool) { func operandDraw(n node, into map[node]bool) {
if _, fixed := n.(literal); fixed { if isFixed(n) {
return return
} }
if into[n] { if into[n] {
@@ -129,6 +127,12 @@ func operandDraw(n node, into map[node]bool) {
} }
} }
// isFixed is a string that varies nothing: fixed text with no fields to read into.
func isFixed(n node) bool {
t, ok := n.(*template)
return ok && t.fixed && len(t.fields) == 0
}
// renders reports whether rendering n can reach anything in want, following the // renders reports whether rendering n can reach anything in want, following the
// same edges expand does. seen keeps a node shared by several routes from being // same edges expand does. seen keeps a node shared by several routes from being
// walked twice; checkNoCycles has already proved the graph is a DAG, so the walk // walked twice; checkNoCycles has already proved the graph is a DAG, so the walk
@@ -294,7 +298,7 @@ func (e renderEdge) reached() string {
// renderEdges lists the children rendering n recurses into, mirroring expand: a // renderEdges lists the children rendering n recurses into, mirroring expand: a
// choice's items, and a template's field/reference tokens plus its calc operands. // choice's items, and a template's field/reference tokens plus its calc operands.
// A literal or group renders nothing, so it has no edges. // A group renders nothing, so it has no edges.
func renderEdges(n node) []renderEdge { func renderEdges(n node) []renderEdge {
switch n := n.(type) { switch n := n.(type) {
case *choice: case *choice:
+3 -21
View File
@@ -59,8 +59,6 @@ func descend(s *session, n node, segments []string) (node, error) {
} }
} }
return descend(s, pick(s, n), segments) return descend(s, pick(s, n), segments)
case literal:
return nil, fmt.Errorf("a plain string has no field %q", segments[0])
default: default:
return nil, fmt.Errorf("cannot descend into %T at %q", n, segments[0]) return nil, fmt.Errorf("cannot descend into %T at %q", n, segments[0])
} }
@@ -85,12 +83,13 @@ func unreachableInChoice(c *choice, want string) error {
// front, so rendering a compiled tree cannot fail. // front, so rendering a compiled tree cannot fail.
func render(s *session, n node) string { func render(s *session, n node) string {
switch n := n.(type) { switch n := n.(type) {
case literal:
return string(n)
case *choice: case *choice:
return render(s, pick(s, n)) return render(s, pick(s, n))
case *template: case *template:
if n.repeat == 1 { if n.repeat == 1 {
if n.fixed {
return n.lit
}
return expand(s, n) return expand(s, n)
} }
var b strings.Builder var b strings.Builder
@@ -119,21 +118,6 @@ func pick(r rng, c *choice) node {
return c.items[i] return c.items[i]
} }
// classChar randomises one character-class rune: '0' digit 0-9, '1' digit 1-9,
// 'A' letter A-Z, 'a' letter a-z.
func classChar(s *session, c rune) byte {
switch c {
case '0':
return byte('0' + s.IntN(10))
case '1':
return byte('1' + s.IntN(9))
case 'A':
return byte('A' + s.IntN(26))
default: // 'a'
return byte('a' + s.IntN(26))
}
}
// expand renders a template's compiled ops. compile validated every token, so this // expand renders a template's compiled ops. compile validated every token, so this
// cannot fail. // cannot fail.
func expand(s *session, t *template) string { func expand(s *session, t *template) string {
@@ -153,8 +137,6 @@ func expand(s *session, t *template) string {
switch o.kind { switch o.kind {
case 'l': case 'l':
b.WriteString(o.lit) b.WriteString(o.lit)
case 'c':
b.WriteByte(classChar(s, o.r))
case 'f': case 'f':
b.WriteString(readField(s, t, held, o.arms[s.IntN(len(o.arms))])) b.WriteString(readField(s, t, held, o.arms[s.IntN(len(o.arms))]))
case 'b': case 'b':
+43 -55
View File
@@ -6,41 +6,33 @@ import (
"strings" "strings"
) )
// This file is the {token} grammar of a format string: how it is scanned // ftoken is one unit of a scanned format string: a literal rune or the body of a
// (eachToken), how a function token is parsed (funcCall) and validated // {…} token.
// (checkFunc, checkTokens), and the builtin contract its functions implement.
// render.go evaluates these tokens; node.go compiles the surrounding JSON;
// reference.go binds {..path} tokens across the tree.
// ftoken is one unit of a scanned format string: a literal rune to emit, a class
// char to randomise ('0' '1' 'A' 'a'), or the body of a {…} token.
type ftoken struct { type ftoken struct {
kind byte // 'l' literal rune, 'c' class char, 'b' brace body kind byte // 'l' literal rune, 'b' brace body
r rune // for kinds 'l' and 'c' r rune
body string body string
} }
// eachToken scans a format string once and calls fn for each unit, the single // eachToken scans a format string once and calls fn for each unit, the single
// source of truth for how '#' escapes and {…} braces are read — compileOps, // source of truth for how braces are read: "{{" and "}}" are literal braces, a "{"
// checkTokens, fieldTokens and refTokens all drive off it so the grammar can't // opens a token that must reach its "}", and a lone "}" is an error.
// drift between the validator and the renderer. '#' escapes the next char to a
// literal ("#0" -> '0', "##" -> '#'); a '{' must reach a '}' (else an error); an
// unmatched '}' is an ordinary literal. The error stops the scan early.
func eachToken(format string, fn func(ftoken) error) error { func eachToken(format string, fn func(ftoken) error) error {
rs := []rune(format) rs := []rune(format)
for i := 0; i < len(rs); i++ { for i := 0; i < len(rs); i++ {
var t ftoken var t ftoken
switch c := rs[i]; c { switch c := rs[i]; c {
case '#':
t.kind, t.r = 'l', '#'
if i++; i < len(rs) {
t.r = rs[i]
}
case '0', '1', 'A', 'a':
t.kind, t.r = 'c', c
case '{': case '{':
if i+1 < len(rs) && rs[i+1] == '{' {
t.kind, t.r = 'l', '{'
i++
break
}
end := i + 1 end := i + 1
for end < len(rs) && rs[end] != '}' { for end < len(rs) && rs[end] != '}' {
if rs[end] == '{' {
return fmt.Errorf("'{' inside a token in %q; a literal brace is written {{", format)
}
end++ end++
} }
if end >= len(rs) { if end >= len(rs) {
@@ -48,6 +40,13 @@ func eachToken(format string, fn func(ftoken) error) error {
} }
t.kind, t.body = 'b', string(rs[i+1:end]) t.kind, t.body = 'b', string(rs[i+1:end])
i = end i = end
case '}':
if i+1 < len(rs) && rs[i+1] == '}' {
t.kind, t.r = 'l', '}'
i++
break
}
return fmt.Errorf("lone '}' in %q; a literal brace is written }}", format)
default: default:
t.kind, t.r = 'l', c t.kind, t.r = 'l', c
} }
@@ -61,13 +60,10 @@ func eachToken(format string, fn func(ftoken) error) error {
// builtin is a format-string function invoked as {name(args)}. It receives the // builtin is a format-string function invoked as {name(args)}. It receives the
// session (its rng, and the {seq()} counters), the output emitted so far in the // session (its rng, and the {seq()} counters), the output emitted so far in the
// current expansion (for derivations such as a checksum over preceding digits), and // current expansion (for derivations such as a checksum over preceding digits), and
// the values of the operands it named (only calc names any). // the values of the operands it named (only calc names any). All must stay pure
// Almost all are pure over (rng, emitted, args) — no wall-clock, no crypto/rand — // over (rng, emitted, args) so seeded output is reproducible; seq advances
// so seeding stays reproducible; a time-based id derives its time from the rng. // per-session counter state, which is itself deterministic. arity is the exact arg
// seq is the one exception: it advances per-session counter state, which is itself // count, or -1 for variadic (then check does all the validation).
// deterministic (1, 2, 3 …). arity is the exact arg count, or -1 for variadic
// (then check does all the validation). The optional check validates args at
// compile time (their values, beyond the count). The registry lives in builtins.go.
type builtin struct { type builtin struct {
arity int arity int
// prep parses validated args once, at compile time, into the closure expand calls. // prep parses validated args once, at compile time, into the closure expand calls.
@@ -165,11 +161,9 @@ func checkTokens(format string, fields map[string]node) error {
}) })
} }
// checkNoRepeatedArm rejects {a|a|b}. An alternation picks its arms evenly, so a // checkNoRepeatedArm rejects {a|a|b}: an alternation picks its arms evenly, so a
// repeated one skews the odds — a third spelling of what weight is for, and one an // repeated one is a second spelling of weight. The error names the spelling that
// author is far likelier to have typed by accident than meant. The error names the // does skew a pick.
// spelling that does skew a pick. It runs over the arms as written, so a repeated
// reference arm ({..a|..a}) is caught alongside a repeated sibling.
func checkNoRepeatedArm(body string, names []string) error { func checkNoRepeatedArm(body string, names []string) error {
if len(names) < 2 { if len(names) < 2 {
return nil return nil
@@ -228,10 +222,9 @@ func splitArm(name string) arm {
} }
// checkNoOverlap rejects a format that both renders a level and reads a path into // checkNoOverlap rejects a format that both renders a level and reads a path into
// it — {p} beside {p.first}, or {p.addr} beside {p.addr.city}. The two spell one // it — {p} beside {p.first}, or {p.addr} beside {p.addr.city}. The path reads the
// draw two ways: the path reads the level's held draw, while rendering the level // level's held draw while rendering the level expands it afresh, so their values
// expands it afresh, so their values disagree. One spelling, so there is nothing // would disagree. Names are compared in sorted order, so which pair is reported
// to get wrong. Names are compared in sorted order, so which pair is reported
// does not depend on where the tokens sit. // does not depend on where the tokens sit.
func checkNoOverlap(format string, bound map[string]string) error { func checkNoOverlap(format string, bound map[string]string) error {
names := boundReaders(format, bound) names := boundReaders(format, bound)
@@ -311,13 +304,12 @@ func splitArms(body string) []arm {
// values of the operands it named, which expand read for it. // values of the operands it named, which expand read for it.
type callFn func(s *session, emitted string, operands []string) string type callFn func(s *session, emitted string, operands []string) string
// op is one compiled unit of a format string: a literal run, a class char, a field // op is one compiled unit of a format string: a literal run, a field alternation,
// alternation, or a builtin already bound to its args. compile builds these so // or a builtin already bound to its args. compile builds these so render never
// render never re-scans the format. // re-scans the format.
type op struct { type op struct {
kind byte // 'l' literal run, 'c' class char, 'f' field alternation, 'b' builtin kind byte // 'l' literal run, 'f' field alternation, 'b' builtin
lit string // kind 'l' lit string // kind 'l'
r rune // kind 'c'
arms []arm // kind 'f': the '|' alternatives, split into key and path once arms []arm // kind 'f': the '|' alternatives, split into key and path once
call callFn call callFn
// operands names the sibling fields a {calc()} reads, in the order calcVars // operands names the sibling fields a {calc()} reads, in the order calcVars
@@ -325,14 +317,14 @@ type op struct {
operands []string operands []string
} }
// compileOps turns a format string into ops, and returns the smallest output it can // compileOps turns a format string into ops, and returns the size of its literal
// produce (literals plus one byte per class char) to size the render buffer, plus // text to size the render buffer, plus two sets. bound is the levels the format
// two sets. bound is the levels the format addresses by dotted path, each mapped to // addresses by dotted path, each mapped to the first path reading it, which is what
// the first path reading it, which is what the overlap fences name. held is every // the overlap fences name. held is every name drawn once per expansion — those
// name drawn once per expansion — those levels, plus the siblings a {calc()} reads, // levels, plus the siblings a {calc()} reads, so an operand shown is the operand
// so an operand shown is the operand computed. Both are nil when the format needs // computed. Both are nil when the format needs neither, so data that uses neither
// neither, so data that uses neither carries no render-time cost. Call checkTokens // carries no render-time cost. Call checkTokens first: it is what proves the scan
// first: it is what proves the scan and every token are valid. // and every token are valid.
func compileOps(format string) ([]op, int, map[string]string, map[string]bool) { func compileOps(format string) ([]op, int, map[string]string, map[string]bool) {
var ops []op var ops []op
var lit strings.Builder var lit strings.Builder
@@ -356,10 +348,6 @@ func compileOps(format string) ([]op, int, map[string]string, map[string]bool) {
switch t.kind { switch t.kind {
case 'l': case 'l':
lit.WriteRune(t.r) lit.WriteRune(t.r)
case 'c':
flush()
grow++
ops = append(ops, op{kind: 'c', r: t.r})
case 'b': case 'b':
flush() flush()
if name, args, ok := funcCall(t.body); ok { if name, args, ok := funcCall(t.body); ok {
+26 -26
View File
@@ -219,32 +219,32 @@ func TestCompileErrors(t *testing.T) {
`[{"format":"A","weight":0},{"format":"B","weight":0}]`, // weights sum to zero `[{"format":"A","weight":0},{"format":"B","weight":0}]`, // weights sum to zero
`[{"format":"A","weight":1e308},{"format":"B","weight":1e308}]`, // weights overflow to +Inf `[{"format":"A","weight":1e308},{"format":"B","weight":1e308}]`, // weights overflow to +Inf
`[{"format":"A","weight":"heavy"}]`, // non-numeric weight `[{"format":"A","weight":"heavy"}]`, // non-numeric weight
`{"format":"x","repeat":0}`, // repeat below 1 `{"format":"x","repeat":0}`, // repeat below 1
`{"format":"x","repeat":-2}`, // negative repeat `{"format":"x","repeat":-2}`, // negative repeat
`{"format":"x","repeat":1.5}`, // non-integer repeat `{"format":"x","repeat":1.5}`, // non-integer repeat
`{"format":"x","repeat":"two"}`, // non-numeric repeat `{"format":"x","repeat":"two"}`, // non-numeric repeat
`{"format":"x","repeat":2,"separator":5}`, // non-string separator `{"format":"x","repeat":2,"separator":5}`, // non-string separator
`{"format":"{nope()}"}`, // unknown function `{"format":"{nope()}"}`, // unknown function
`{"format":"{luhn(x)}"}`, // function given args it takes none of `{"format":"{luhn(x)}"}`, // function given args it takes none of
`{"format":"{luhn(}"}`, // malformed function token `{"format":"{luhn(}"}`, // malformed function token
`{"format":"{int(1)}"}`, // wrong arity `{"format":"{int(1)}"}`, // wrong arity
`{"format":"{int(a,b)}"}`, // non-integer args `{"format":"{int(a,b)}"}`, // non-integer args
`{"format":"{int(5,1)}"}`, // min > max `{"format":"{int(5,1)}"}`, // min > max
`{"format":"{hex(0)}"}`, // count must be positive `{"format":"{hex(0)}"}`, // count must be positive
`{"format":"{nanoid(-1)}"}`, // negative count `{"format":"{nanoid(-1)}"}`, // negative count
`{"format":"{base64(0)}"}`, // count must be positive `{"format":"{base64(0)}"}`, // count must be positive
`{"format":"{float(1,2)}"}`, // wrong arity `{"format":"{float(1,2)}"}`, // wrong arity
`{"format":"{float(1,2,-1)}"}`, // negative decimals `{"format":"{float(1,2,-1)}"}`, // negative decimals
`{"format":"{iban(US)}"}`, // unsupported country `{"format":"{iban(US)}"}`, // unsupported country
`{"format":"{seq(a,b)}"}`, // seq takes at most one name `{"format":"{seq(a,b)}"}`, // seq takes at most one name
`{"format":"{calc()}"}`, // calc needs an expression `{"format":"{calc()}"}`, // calc needs an expression
`{"format":"{calc(1 +)}"}`, // dangling operator `{"format":"{calc(1 +)}"}`, // dangling operator
`{"format":"{calc((1 + 2)}"}`, // unbalanced parenthesis `{"format":"{calc((1 + 2)}"}`, // unbalanced parenthesis
`{"format":"{calc(1 2)}"}`, // two operands, no operator `{"format":"{calc(1 2)}"}`, // two operands, no operator
`{"format":"{calc(price)}"}`, // operand names no field `{"format":"{calc(price)}"}`, // operand names no field
`{"format":"{calc(1, 2, 3)}"}`, // too many args `{"format":"{calc(1, 2, 3)}"}`, // too many args
`{"format":"{calc(1, x)}"}`, // decimals arg not an integer `{"format":"{calc(1, x)}"}`, // decimals arg not an integer
`{"format":"{calc(1, -1)}"}`, // decimals negative `{"format":"{calc(1, -1)}"}`, // decimals negative
} { } {
if _, err := compile(parse(t, bad)); err == nil { if _, err := compile(parse(t, bad)); err == nil {
t.Errorf("compile(%s) = nil error, want error", bad) t.Errorf("compile(%s) = nil error, want error", bad)