Compare commits

...

7 Commits

41 changed files with 18755 additions and 214 deletions
+15 -2
View File
@@ -13,8 +13,9 @@ replacement, and each removed path, column or flag.
`key`, `name`, `weight` and `parent`; a path selects a row by key or name,
`misc.country[SE]`, and descends to a linked table by name; linked tables draw
consistently within one render and draw group. `rows` is an option, so no
template may carry a field of that name. Refused at `New`: a `name` without a
`key`, a name spelling another row's key, a table named like a column of any
template may carry a field of that name. A `name` without a `key` resolves
inside the table's `parent`. Refused at `New`: a `name` without a `key` or a
`parent`, a name repeating inside one parent row, a name spelling another row's key, a table named like a column of any
table above it, a table whose format or cell references its own family, and,
within one render and draw group, a path drawing a table another path selects a
row of, or two paths pinning different rows of one table.
@@ -32,3 +33,15 @@ replacement, and each removed path, column or flag.
`address` record over one consistent draw of them. `sv_SE.address` and
`en_US.address` read those records, so `en_US.address.street` no longer carries
`name` and `suffix`, and a locale folder loads only beside `geo`.
- `{date(from,to,'layout')}` and `{time('layout')}`: a second between two days, or
within one, in a quoted Go layout. `sv_SE.date`, `en_US.date`, `sv_SE.time` and
`en_US.time` render through them, so `date.year`, `date.month`, `date.day` and
`time.hour` are no longer paths, and `misc.datetime` is an RFC 3339 instant.
- `sex`, `first-name` and `last-name` tables in `sv_SE` and `en_US`, weighted by
bearers from SCB, the SSA and the Census Bureau; `first-name` links to `sex`, and a
name both sexes carry is a row under each. `person` reads them, so its columns are
`first`, `last`, `prefix` and `sex`, and `person.femalefirst` and
`person.malefirst` are no longer paths.
- `sv_SE.personnummer` and `sv_SE.samordningsnummer`, Skatteverket's test series
over the sex the render drew, in place of `sv_SE.ssn`; `en_US.ssn` in the ranges
the SSA assigns, and `en_US.itin`.
+4
View File
@@ -11,6 +11,10 @@ Every shipped dataset, its source, its licence and the attribution it asks for.
| `geo/SE/street.tsv` | [Trafikverket NVDB](https://www.trafikverket.se/) Gatunamn, through the open API | CC0 1.0 | none required | `data-import/geo-se.py` |
| `geo/US/region.tsv`, `municipality.tsv`, `locality.tsv` | [Census Bureau](https://www.census.gov/) Gazetteer 2026 and population estimates 2025 | [public domain](https://www.usa.gov/government-works) | none required | `data-import/geo-us.py` |
| `geo/US/postal-code.tsv`, `street.tsv` | Census Bureau ZCTA to place relationships 2020 and TIGER/Line 2025 address ranges and feature names | public domain | none required | `data-import/geo-us.py` |
| `sv_SE/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` |
| `en_US/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` |
| `en_US/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` |
| `sv_SE/sex.tsv`, `en_US/sex.tsv` | curated (Skatteverket's test birth numbers are facts) | — | — | — |
| `misc/country.tsv` | [datasets/country-codes](https://github.com/datasets/country-codes) | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/country.py` |
| `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` |
| `misc/httpstatus.tsv` | curated (IANA HTTP status codes are facts) | — | — | — |
+68 -20
View File
@@ -217,17 +217,27 @@ Sources merge in order; matching folders combine, any other clash is won by the
last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"` or `/`, nor be
`-`, which a struct tag reserves; dot-prefixed entries are skipped, so a data directory can also be a checkout.
Each locale carries `address`, `color`, `company`, `date`, `email`, `ip`,
`person`, `phone`, `price`, `sentence`, `ssn`, `time`, `url`, `username`,
`version` and `word`, formatted per locale. `misc` carries `car`, `coordinate`,
`country` (ISO 3166), `creditcard` (Luhn-valid), `currency` (ISO 4217), `emoji`,
`httpstatus`, `language` (ISO 639), `mac`, `mimetype`, `objectid`, `timezone`
(IANA), `useragent` and `uuid` (v4). Many carry sub-fields — `misc.currency.symbol`,
Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
`ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`,
`username`, `version` and `word`, formatted per locale; `sv_SE` adds
`personnummer` and `samordningsnummer`, `en_US` adds `ssn` and `itin`. `misc`
carries `car`, `coordinate`, `country` (ISO 3166), `creditcard` (Luhn-valid),
`currency` (ISO 4217), `datetime` (RFC 3339), `emoji`, `httpstatus`, `language`
(ISO 639), `mac`, `mimetype`, `objectid`, `timezone` (IANA), `useragent` and
`uuid` (v4). Many carry sub-fields — `misc.currency.symbol`,
`misc.country.alpha2`, `misc.httpstatus.code` — which `--list` shows. `country`,
`currency`, `httpstatus`, `language` and `mimetype` are [tables](#table), so
`misc.country[SE].capital` and `misc.currency[Euro].symbol` select a row;
[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
`sex`, `first-name` and `last-name` are tables weighted by bearers, from SCB, the
SSA and the Census Bureau. `first-name` links to `sex`, so `sv_SE.sex[f].first-name`
draws a woman's name, and a name both sexes carry is a row under each, so
`en_US.sex[m].first-name[Taylor]` names the one a `first-name[Taylor]` alone cannot.
`person` reads one draw of the three, so its `first` and `sex` columns agree, and so
does a `personnummer` in the same render: its birth number is Skatteverket's test
series, 238 for a woman and 239 for a man, which no real person is ever given.
A `geo` folder holds one tree per country under its alpha-2 code: five
[linked tables](#linked-tables) named alike, and an `address` record over one
consistent draw of them, which the locale's `address` reads.
@@ -407,7 +417,9 @@ cell token, and refuses a TSV no category names, a key that is empty or repeats,
weight that is not a positive number, and a key or name holding `[`, `]`, `{`, `}`,
`"` or `|`, which a selector cannot spell; the rows are indexed on the first draw that
selects one. A `name` needs a `key`, since a name naming several rows is reported by
their keys, and a name spelling another row's key is refused, since the key would
their keys, or a `parent`, inside whose row a name names one row, so `first-name[Kim]`
is settled by the `sex` selected before it and a name repeating inside one parent row
is refused; a name spelling another row's key is refused, since the key would
select first and the name never. The table's options are its own — `rows`, `key`,
`name`, `weight` and `parent` — so a column may be named `name`, as one usually is.
@@ -505,21 +517,30 @@ stays reproducible.
| `{ulid()}` | sample | ULID, 26 Crockford base32 chars |
| `{nanoid(n)}` | sample | URL-safe Nano ID, `n` chars |
| `{iban(CC)}` | sample | length- and mod-97-valid IBAN for BE, DE, DK, ES, FI, NO or SE |
| `{date(from,to,'layout')}` | sample | a second between two `YYYY-MM-DD` days, both included, in a quoted Go layout: `'2006-01-02'`, `'January 2, 2006'`, `'060102'`, `'2006-01-02T15:04:05Z'` |
| `{time('layout')}` | sample | a second within a day: `'15:04'`, `'3:04 PM'` |
| `{seq()}`, `{seq(name)}` | counter | next integer from 1 in this generator; `name` selects an independent counter |
| `{calc(expr)}`, `{calc(expr,dp)}` | computation | an arithmetic expression over sibling fields ([Computation](#computation)) |
| `{lowercase(x)}`, `{uppercase(x)}`, `{ascii(x)}` | transform | a field's value rewritten ([Transforms](#transforms)) |
A derivation reads what is to its left, so place it after its payload; the
buffer is per expansion, so a nested template keeps fixed parts out of the sum. A
Swedish personnummer is a Luhn checksum over the nine digits before it:
Swedish personnummer is a Luhn checksum over the nine digits before it, six of
them a birthdate:
```json
{ "format": "{century}{core}", "century": ["19", "20"],
"core": { "format": "{digits(2)}{mmdd}-{digits(3)}{luhn()}", "mmdd": ["0115", "0704", "1218"] } }
{ "format": "{date(1930-01-01,2010-12-31,'060102')}-{birth}{luhn()}", "birth": ["238", "239"] }
```
Renders e.g. `19811218-9876`. `{seq()}` spans `Fake` calls and `repeat`, resets
with a new generator, and is the natural primary key for the SQL example above.
Renders e.g. `811218-2389`. A layout is Go's: the reference time `Mon Jan 2
15:04:05 MST 2006` spelled as the output should look, quoted, since a layout may
carry the comma that separates arguments, with English names. Every second
between the two days is reachable, so a layout with a clock draws the time too.
Rejected at `New`: a bound that is no calendar date, or not before the other; an
unquoted layout, naming the quoted one; a layout naming no field, which is text;
and for `time` a layout naming a date field, naming `date`. `{seq()}` spans `Fake`
calls and `repeat`, resets with a new generator, and is the natural primary key for
the SQL example above.
### Computation
@@ -573,9 +594,9 @@ without naming `sv_SE`:
Renders e.g. `Hej, Pat Smith!`. A reference path into a category is held like a
[correlated](#correlated-fields) path, but for the whole render — one `Fake`, or one
record — rather than one format: `{.person.femalefirst} {.person.last}` name one
record — rather than one format: `{.person.first} {.person.last}` name one
person, as do the same two references in sibling fields or a nested template, and
`{lowercase(.person.femalefirst)}` reads that same draw. Each `repeat` iteration is
`{lowercase(.person.first)}` reads that same draw. Each `repeat` iteration is
a render of its own, in no group, so it draws anew, and a [draw group](#draw-group) holds a
draw apart. A bare reference names no field and makes its own picks each time —
`{/misc.uuid} {/misc.uuid}` is two draws — while the reference paths inside what it
@@ -594,8 +615,8 @@ its groups by name; the unnamed group spans them all.
```json
{ "format": "{payer} pays {payee}; signed {signature}",
"payer": { "format": "{/sv_SE.person.femalefirst} {/sv_SE.person.last}", "drawGroup": "payer" },
"payee": "{/sv_SE.person.femalefirst} {/sv_SE.person.last}",
"payer": { "format": "{/sv_SE.person.first} {/sv_SE.person.last}", "drawGroup": "payer" },
"payee": "{/sv_SE.person.first} {/sv_SE.person.last}",
"signature": { "format": "{/sv_SE.person.last}", "drawGroup": "payer" } }
```
@@ -728,8 +749,7 @@ renamed or retyped line is a major.
key, or prefixing options, would tax every template to guard against a
misspelt option.
- **`{a|b}` stays beside nested choices.** `[[…], […]]` picks the same way, but
its arms are anonymous; `{femalefirst|malefirst}` keeps `person.femalefirst`
addressable.
its arms are anonymous; `{female|male}` keeps `person.female` addressable.
- **Flags follow getopt_long.** `--name value` and `--name=value` both work; a
short flag's value attaches or follows (`-s42`, `-s 42`) and short flags bundle
(`-hn 3`), as every shell user expects. A single-dash long flag is rejected
@@ -1040,6 +1060,30 @@ renamed or retyped line is a major.
bigger neighbour ship no address; counting the land outside every place too would
drop a quarter of the places, whose codes straddle unincorporated land, for a
postal city the USPS mostly names the same way.
- **A layout is always quoted.** A layout may carry the comma that separates
arguments, `'January 2, 2006'`, and one spelling for every layout beats a rule
about which ones need the quotes, so the bare spelling is refused naming the
quoted one. The layout is Go's reference time because the library renders with
it and a Go caller already knows it; its names are English, and a locale's own
month and weekday names are data.
- **No builtin reads the clock, so a date is bounded by days, never by an age.**
An `age(min,max)` would make a seeded fixture change with the day it runs on,
which is what a seed exists to prevent; a birthdate for someone 20 to 60 is
`date(1966-01-01,2006-12-31,…)`, re-pinned as any fixture is.
- **A name column without a key resolves inside its parent.** A given name both
sexes carry is a row under each, so `name` cannot be the key; the parent's row
tells the two apart, `sex[f].first-name[Kim]`, the ambiguity error spells each
row inside its parent, and a name repeating inside one parent row is refused at
load, since nothing could then select it.
- **The Swedish ids draw Skatteverket's test series.** A Luhn-valid personnummer
over a random birth number may be a living person's; 238 and 239 after any date
are blocked from assignment, so the shipped `personnummer` and
`samordningsnummer` use those, read from the `sex` table's `birth-number`
column so the number and the name agree on sex.
- **The US given names come from a mirror of the SSA file.** ssa.gov refuses a
client outside the US, so `names-us.py` reads a GitHub copy that ends at 2020,
which a count over the births since 1930 barely feels; `--names` takes the
official zip.
- **`List` advertises direct descents only.** `region.municipality.locality` is
listed, and `region.locality` resolves too but is not: the set of every descent
through a chain of five tables is every subsequence of it, and the direct chain is
@@ -1091,15 +1135,19 @@ A shipped table built from a source is rebuilt by its script under
[`data-import/`](data-import), one command per dataset, fetching the source named in
[`DATA-LICENSES.md`](DATA-LICENSES.md). Downloads are cached under
`data-import/cache/`, so delete it to fetch afresh; `geo-us.py` fetches two
TIGER/Line files per county it ships, a few hundred megabytes, and `geo-se.py` needs
TIGER/Line files per county it ships, a few hundred megabytes, `geo-se.py` needs
a Trafikverket API key, free at [data.trafikverket.se](https://data.trafikverket.se/),
in `TRAFIKVERKET_API_KEY` or a `--key-file`:
in `TRAFIKVERKET_API_KEY` or a `--key-file`, and the Census host behind `geo-us.py`
and `names-us.py` rejects a client for a while after a burst, so `--surnames` takes
a copy of the surname file:
```sh
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/country.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py
docker compose run --rm --user "$(id -u):$(id -g)" -e TRAFIKVERKET_API_KEY data-import data-import/geo-se.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-se.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-us.py
```
To release, head `CHANGELOG.md` with the version's section in place of `Unreleased`
+1 -1
View File
@@ -30,7 +30,7 @@ func BenchmarkPerson(b *testing.B) { benchPath(b, "data", "sv_SE.person") }
func BenchmarkAddress(b *testing.B) { benchPath(b, "data", "sv_SE.address") }
func BenchmarkWord(b *testing.B) { benchPath(b, "data", "sv_SE.word") }
func BenchmarkCreditcard(b *testing.B) { benchPath(b, "data", "misc.creditcard") }
func BenchmarkSSN(b *testing.B) { benchPath(b, "data", "sv_SE.ssn") }
func BenchmarkSSN(b *testing.B) { benchPath(b, "data", "sv_SE.personnummer") }
func BenchmarkUUIDv7(b *testing.B) { benchPath(b, "data", "misc.uuid") }
func tmpData(b *testing.B, name, body string) string {
+106
View File
@@ -7,6 +7,7 @@ import (
"math"
"strconv"
"strings"
"time"
"unicode"
)
@@ -60,6 +61,8 @@ var builtins = map[string]builtin{
cc := a[0]
return func(s *session, _ string, _ []string) string { return iban(s, cc) }
}},
"date": {arity: -1, check: dateArgs, prep: datePrep},
"time": {arity: -1, check: timeArg, prep: timePrep},
"calc": {arity: -1, check: checkCalc, prep: calcPrep, operands: calcOperands},
"lowercase": {arity: 1, check: transformArg, prep: transformPrep(strings.ToLower), operands: transformOperand},
"uppercase": {arity: 1, check: transformArg, prep: transformPrep(strings.ToUpper), operands: transformOperand},
@@ -466,3 +469,106 @@ func iban(r rng, cc string) string {
feed(0)
return fmt.Sprintf("%s%02d%s", cc, 98-rem, bban)
}
const dayLayout = "2006-01-02"
// The two instants a layout is proved against: alike in nothing, so a layout that
// formats them alike names no field, and one that tells their days apart at the same
// clock names a date field.
var (
layoutProbe = time.Date(2001, 2, 3, 4, 5, 6, 0, time.UTC)
layoutProbe2 = time.Date(2010, 11, 12, 13, 14, 15, 0, time.UTC)
layoutDay = time.Date(2010, 11, 12, 4, 5, 6, 0, time.UTC)
)
// layoutArg is the Go layout a quoted arg holds; unquoted, it is refused naming the
// quoted spelling, since a layout may carry the comma that splits args.
func layoutArg(a string) (string, error) {
if len(a) < 2 || a[0] != '\'' || a[len(a)-1] != '\'' {
return "", fmt.Errorf("layout %s is not quoted; write '%s'", a, strings.Trim(a, "'"))
}
layout := a[1 : len(a)-1]
if layoutProbe.Format(layout) == layoutProbe2.Format(layout) {
return "", fmt.Errorf("layout '%s' names no field, so it is the constant %q; write it as text", layout, layout)
}
return layout, nil
}
// layoutOf is layoutArg for an arg a check already validated.
func layoutOf(a string) string {
layout, err := layoutArg(a)
if err != nil {
panic(fmt.Sprintf("fejkdata: builtin arg %q reached prep unvalidated: %v", a, err))
}
return layout
}
// layoutArity checks a call that ends in a layout takes n args; an unquoted layout
// carrying a comma splits into more, so the error names its quoted spelling.
func layoutArity(name string, n int, a []string) error {
if len(a) == n {
return nil
}
hint := ""
if len(a) > n {
hint = fmt.Sprintf("; a layout holding a comma is quoted: '%s'", strings.Join(a[n-1:], ", "))
}
return fmt.Errorf("%s takes %d args, got %d%s", name, n, len(a), hint)
}
func dateArgs(_ map[string]node, a []string) error {
if err := layoutArity("date", 3, a); err != nil {
return err
}
from, err := time.Parse(dayLayout, a[0])
if err != nil {
return fmt.Errorf("date(from,to,layout): from %q is not a YYYY-MM-DD date", a[0])
}
to, err := time.Parse(dayLayout, a[1])
if err != nil {
return fmt.Errorf("date(from,to,layout): to %q is not a YYYY-MM-DD date", a[1])
}
if !from.Before(to) {
return fmt.Errorf("date(from,to,layout): from %s is not before to %s", a[0], a[1])
}
_, err = layoutArg(a[2])
return err
}
func timeArg(_ map[string]node, a []string) error {
if err := layoutArity("time", 1, a); err != nil {
return err
}
layout, err := layoutArg(a[0])
if err != nil {
return err
}
if layoutProbe.Format(layout) != layoutDay.Format(layout) {
return fmt.Errorf("time(layout): '%s' names a date field; write date(from,to,layout)", layout)
}
return nil
}
// datePrep draws a second in [from 00:00:00, to 23:59:59] UTC; the span is counted
// in seconds, since a Duration overflows past 292 years.
func datePrep(a []string) callFn {
from, err := time.Parse(dayLayout, a[0])
if err != nil {
panic(fmt.Sprintf("fejkdata: builtin arg %q reached prep unvalidated: %v", a[0], err))
}
to, err := time.Parse(dayLayout, a[1])
if err != nil {
panic(fmt.Sprintf("fejkdata: builtin arg %q reached prep unvalidated: %v", a[1], err))
}
layout, start, span := layoutOf(a[2]), from.Unix(), int(to.Unix()-from.Unix())+86400
return func(s *session, _ string, _ []string) string {
return time.Unix(start+int64(s.IntN(span)), 0).UTC().Format(layout)
}
}
func timePrep(a []string) callFn {
layout := layoutOf(a[0])
return func(s *session, _ string, _ []string) string {
return time.Unix(int64(s.IntN(86400)), 0).UTC().Format(layout)
}
}
+84
View File
@@ -4,7 +4,9 @@ import (
"encoding/base64"
"regexp"
"strconv"
"strings"
"testing"
"time"
)
func TestBuiltinIDGenerators(t *testing.T) {
@@ -41,6 +43,7 @@ func TestBuiltinSamplesReproducible(t *testing.T) {
`"{uuid()}"`, `"{ulid()}"`,
`"{nanoid(12)}"`, `"{int(1,1000000)}"`,
`"{float(0,1,6)}"`, `"{base64(12)}"`, `"{iban(SE)}"`,
`"{date(2000-01-01,2020-12-31,'2006-01-02 15:04:05')}"`, `"{time('15:04:05')}"`,
} {
if a, b := mustRender(t, engine(7), tmpl), mustRender(t, engine(7), tmpl); a != b {
t.Fatalf("%s not reproducible: %q != %q", tmpl, a, b)
@@ -233,3 +236,84 @@ func TestClassBuiltinArgs(t *testing.T) {
}
}
}
// TestBuiltinDateAndTime pins the two clock-free samples: date draws a second in
// [from 00:00:00, to 23:59:59], both days reachable, and renders it in the quoted Go
// layout, commas and English names included; time draws a second within one day.
func TestBuiltinDateAndTime(t *testing.T) {
f := engine(1)
seen := map[string]bool{}
for i := 0; i < 500; i++ {
got := mustRender(t, f, `"{date(1990-01-01,1990-12-31,'2006-01-02')}"`)
if d, err := time.Parse("2006-01-02", got); err != nil || d.Year() != 1990 {
t.Fatalf("date = %q, want a 1990 calendar date (err %v)", got, err)
}
seen[got] = true
}
if len(seen) < 200 {
t.Fatalf("date drew %d distinct days of 365 in 500, want a uniform spread", len(seen))
}
lo, hi := false, false
for i := 0; i < 200; i++ {
got := mustRender(t, f, `"{date(2020-02-28,2020-02-29,'2006-01-02')}"`)
if got != "2020-02-28" && got != "2020-02-29" {
t.Fatalf("date(2020-02-28,2020-02-29) = %q, out of range", got)
}
lo, hi = lo || got == "2020-02-28", hi || got == "2020-02-29"
}
if !lo || !hi {
t.Fatalf("date never hit a bound: lo=%v hi=%v (bounds must be inclusive)", lo, hi)
}
if got := mustRender(t, f, `"{date(2020-07-04,2020-07-05,'January 2, 2006')}"`); got != "July 4, 2020" && got != "July 5, 2020" {
t.Fatalf("date with a comma in its layout = %q", got)
}
rfc := regexp.MustCompile(`^2021-\d\d-\d\dT\d\d:\d\d:\d\dZ$`)
clock := regexp.MustCompile(`^([01]\d|2[0-3]):[0-5]\d$`)
ampm := regexp.MustCompile(`^(1[0-2]|[1-9]):[0-5]\d (AM|PM)$`)
seconds := map[string]bool{}
for i := 0; i < 300; i++ {
got := mustRender(t, f, `"{date(2021-01-01,2021-12-31,'2006-01-02T15:04:05Z07:00')}"`)
if !rfc.MatchString(got) {
t.Fatalf("date in an RFC 3339 layout = %q, want %s", got, rfc)
}
seconds[got[17:19]] = true
if got := mustRender(t, f, `"{time('15:04')}"`); !clock.MatchString(got) {
t.Fatalf("time('15:04') = %q, want %s", got, clock)
}
if got := mustRender(t, f, `"{time('3:04 PM')}"`); !ampm.MatchString(got) {
t.Fatalf("time('3:04 PM') = %q, want %s", got, ampm)
}
got = mustRender(t, f, `"{date(1950-01-01,2000-12-31,'060102')}-238{luhn()}"`)
if d := digitsOnly(got); len(d) != 10 || !luhnValid(d) {
t.Fatalf("a personnummer over date() = %q, want ten Luhn-valid digits", got)
}
}
if len(seconds) < 30 {
t.Fatalf("date drew %d distinct seconds in 300, want the whole day, not midnight", len(seconds))
}
}
// TestBuiltinDateArgs pins the New-time checks: bounds are calendar dates in order,
// the layout is quoted, names a field, and for time names no date field.
func TestBuiltinDateArgs(t *testing.T) {
for tmpl, want := range map[string]string{
`"{date(1990-13-01,1990-12-31,'2006-01-02')}"`: "1990-13-01",
`"{date(1990-12-31,1990-01-01,'2006-01-02')}"`: "before",
`"{date(1990-01-01,1990-01-01,'2006-01-02')}"`: "before",
`"{date(1990-01-01,1990-12-31,2006-01-02)}"`: "'2006-01-02'",
`"{date(1990-01-01,1990-12-31,'January 2, 2006)}"`: "'",
`"{date(1990-01-01,1990-12-31,'x')}"`: "text",
`"{date(1990-01-01,1990-12-31,'')}"`: "text",
`"{date(1990-01-01,1990-12-31)}"`: "3 args",
`"{date(1990-01-01,1990-12-31,January 2, 2006)}"`: "'January 2, 2006'",
`"{time(3:04 PM, Mon)}"`: "'3:04 PM, Mon'",
`"{time(15:04)}"`: "'15:04'",
`"{time('2006-01-02 15:04')}"`: "date(",
`"{time('x')}"`: "text",
} {
_, err := compile(parse(t, tmpl))
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("compile(%s) = %v, want an error mentioning %q", tmpl, err, want)
}
}
}
+1 -17
View File
@@ -13,7 +13,6 @@ import os
import re
import sys
import urllib.request
import xml.etree.ElementTree as ET
import zipfile
from pathlib import Path
@@ -38,24 +37,9 @@ CACHE = Path(__file__).resolve().parent / "cache"
TIMEZONE = "Europe/Stockholm"
ONE_POSITION = {"Stockholm", "Göteborg", "Malmö"}
UNMATCHED_POPULATION = 200
XLSX_NS = {"m": "http://schemas.openxmlformats.org/spreadsheetml/2006/main"}
def xlsx_rows(data):
z = zipfile.ZipFile(io.BytesIO(data))
strings = ["".join(t.text or "" for t in si.iter("{%s}t" % XLSX_NS["m"])) for si in ET.fromstring(z.read("xl/sharedStrings.xml")).findall("m:si", XLSX_NS)]
sheet = ET.fromstring(z.read("xl/worksheets/sheet1.xml"))
for row in sheet.findall(".//m:row", XLSX_NS):
cells = []
for c in row.findall("m:c", XLSX_NS):
v = c.find("m:v", XLSX_NS)
cells.append("" if v is None else strings[int(v.text)] if c.get("t") == "s" else v.text)
yield cells
def scb_codes(cache):
regions, municipalities = {}, {}
for cells in xlsx_rows(tsv.fetch(CODES, cache, "kommunlankod.xlsx", magic=b"PK")):
for cells in tsv.xlsx_rows(tsv.fetch(CODES, cache, "kommunlankod.xlsx", magic=b"PK")):
if len(cells) < 2 or not re.fullmatch(r"\d{2}|\d{4}", cells[0]):
continue
(regions if len(cells[0]) == 2 else municipalities)[cells[0]] = cells[1].strip()
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env python3
"""Rebuild data/sv_SE/first-name.tsv and last-name.tsv from SCB's whole-population name counts (CC0, "Källa: SCB").
data-import/names-se.py [--source URL_OR_FILE] [--first N] [--last N] [--cache DIR] [--out DIR]
A tilltalsnamn goes under each sex that carries it, so a name both sexes carry has two rows.
"""
import argparse
import re
from pathlib import Path
import tsv
SOURCE = "https://www.scb.se/contentassets/9fe7dbb460994c72b835163dbc491ef9/namn-med-minst-tva-barare-31-december-2022.xlsx"
OUT = Path(__file__).resolve().parent.parent / "data" / "sv_SE"
CACHE = Path(__file__).resolve().parent / "cache"
SHEETS = {"f": "Tilltalsnamn kvinnor", "m": "Tilltalsnamn män"}
SURNAMES = "Efternamn"
NAME = re.compile(r"^[^\W\d_]{2,}([ '-][^\W\d_]{2,})*$")
PARTICLES = {"af", "av", "de", "den", "der", "di", "du", "la", "le", "van", "von"}
def cased(name):
"""SCB's uppercase name as it is written: each part capitalised, a particle before another part lowercased."""
parts = name.lower().split(" ")
return " ".join(w if w in PARTICLES and len(parts) > 1 else w.title() for w in parts)
def counted(rows, top):
"""The top names of a sheet by bearers, initials and single letters dropped."""
names = [(cells[0], int(cells[1])) for cells in rows if len(cells) >= 2 and cells[1].isdigit() and NAME.match(cells[0])]
return sorted(names, key=lambda n: (-n[1], n[0]))[:top]
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--cache", default=str(CACHE))
p.add_argument("--first", type=int, default=2000, help="names per sex")
p.add_argument("--last", type=int, default=5000)
p.add_argument("--out", default=str(OUT))
p.add_argument("--source", default=SOURCE)
a = p.parse_args()
data = tsv.fetch(a.source, a.cache, "scb-namn-2022.xlsx", magic=b"PK")
first = [{"name": cased(n), "sex": sex, "count": c} for sex, sheet in SHEETS.items() for n, c in counted(tsv.xlsx_rows(data, sheet), a.first)]
last = [{"name": cased(n), "count": c} for n, c in counted(tsv.xlsx_rows(data, SURNAMES), a.last)]
out = Path(a.out)
tsv.write(out / "first-name.tsv", ["name", "sex", "count"], sorted(first, key=lambda r: (r["name"], r["sex"])))
tsv.write(out / "last-name.tsv", ["name", "count"], sorted(last, key=lambda r: r["name"]))
if __name__ == "__main__":
main()
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env python3
"""Rebuild data/en_US/first-name.tsv and last-name.tsv from SSA baby names and the Census 2010 surnames (public domain).
data-import/names-us.py [--names URL_OR_FILE] [--surnames URL_OR_FILE] [--from-year YEAR] [--first N] [--last N] [--cache DIR] [--out DIR]
A given name is counted over the births of --from-year and later, under each sex it was given to, so a
name both sexes carry has two rows. --names takes SSA's names.zip or a merged name,sex,count,year file;
--surnames takes the Census names.zip or its Names_2010Census.csv.
"""
import argparse
import collections
import csv
import io
import re
import zipfile
from pathlib import Path
import tsv
NAMES = "https://raw.githubusercontent.com/hackerb9/ssa-baby-names/master/alldata.txt"
SURNAMES = "https://www2.census.gov/topics/genealogy/2010surnames/names.zip"
OUT = Path(__file__).resolve().parent.parent / "data" / "en_US"
CACHE = Path(__file__).resolve().parent / "cache"
MC = re.compile(r"^Mc([a-z])")
def csv_rows(data, member):
"""The rows of a CSV, or of every member of a zip named like member, name,sex,count[,year]."""
if data.startswith(b"PK"):
z = zipfile.ZipFile(io.BytesIO(data))
for n in sorted(z.namelist()):
if re.fullmatch(member, n):
year = re.sub(r"\D", "", n)
for r in csv.reader(io.StringIO(z.read(n).decode("utf-8-sig"))):
yield r + [year] if year else r
return
yield from csv.reader(io.StringIO(data.decode("utf-8-sig")))
def given(data, from_year):
counts = collections.Counter()
for r in csv_rows(data, r"yob\d{4}\.txt"):
if len(r) >= 4 and r[3].isdigit() and int(r[3]) >= from_year:
counts[(r[0], r[1].lower())] += int(r[2])
return counts
def surname(name):
"""A Census uppercase surname as it is written: capitalised, and Mc before a capital."""
return MC.sub(lambda m: "Mc" + m.group(1).upper(), name.title())
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--cache", default=str(CACHE))
p.add_argument("--first", type=int, default=2000, help="names per sex")
p.add_argument("--from-year", type=int, default=1930)
p.add_argument("--last", type=int, default=5000)
p.add_argument("--names", default=NAMES)
p.add_argument("--out", default=str(OUT))
p.add_argument("--surnames", default=SURNAMES)
a = p.parse_args()
counts = given(tsv.fetch(a.names, a.cache, "ssa-names.txt"), a.from_year)
first = []
for sex in ("f", "m"):
top = sorted(((n, c) for (n, s), c in counts.items() if s == sex), key=lambda n: (-n[1], n[0]))[:a.first]
first += [{"name": n, "sex": sex, "count": c} for n, c in top]
rows = csv_rows(tsv.fetch(a.surnames, a.cache, "census-surnames-2010.zip"), r"Names_2010Census\.csv")
last = [{"name": surname(r[0]), "count": int(r[2])} for r in rows if len(r) >= 3 and r[2].isdigit() and r[0].isalpha()]
last = sorted(last, key=lambda r: (-r["count"], r["name"]))[:a.last]
out = Path(a.out)
tsv.write(out / "first-name.tsv", ["name", "sex", "count"], sorted(first, key=lambda r: (r["name"], r["sex"])))
tsv.write(out / "last-name.tsv", ["name", "count"], sorted(last, key=lambda r: r["name"]))
if __name__ == "__main__":
main()
+20
View File
@@ -1,10 +1,15 @@
"""A source fetched once into the cache, and a table written as the loader admits it."""
import io
import re
import sys
import time
import urllib.request
import xml.etree.ElementTree as ET
import zipfile
from pathlib import Path
XLSX_NS = {"m": "http://schemas.openxmlformats.org/spreadsheetml/2006/main", "r": "http://schemas.openxmlformats.org/officeDocument/2006/relationships"}
def fetch(source, cache, name, magic=b"", data=None, headers=None):
"""The bytes of a URL, downloaded into cache/name once, or of a local file."""
@@ -28,6 +33,21 @@ def fetch(source, cache, name, magic=b"", data=None, headers=None):
sys.exit(f"{source}: no valid download in 5 attempts")
def xlsx_rows(data, sheet=None):
"""The rows of an xlsx sheet named sheet, the first sheet by default, each a list of cell texts."""
z = zipfile.ZipFile(io.BytesIO(data))
strings = ["".join(t.text or "" for t in si.iter("{%s}t" % XLSX_NS["m"])) for si in ET.fromstring(z.read("xl/sharedStrings.xml")).findall("m:si", XLSX_NS)]
rels = {r.get("Id"): r.get("Target") for r in ET.fromstring(z.read("xl/_rels/workbook.xml.rels"))}
sheets = {s.get("name"): rels[s.get("{%s}id" % XLSX_NS["r"])] for s in ET.fromstring(z.read("xl/workbook.xml")).iter("{%s}sheet" % XLSX_NS["m"])}
target = sheets[sheet] if sheet else next(iter(sheets.values()))
for row in ET.fromstring(z.read("xl/" + target)).findall(".//m:row", XLSX_NS):
cells = []
for c in row.findall("m:c", XLSX_NS):
v = c.find("m:v", XLSX_NS)
cells.append("" if v is None else strings[int(v.text)] if c.get("t") == "s" else v.text)
yield cells
def write(path, columns, rows):
"""Write the rows as a TSV; every cell must be non-empty and free of tabs, newlines and braces."""
lines = ["\t".join(columns)]
+1 -13
View File
@@ -1,13 +1 @@
{
"format": "{month}/{day}/{year}",
"month": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"],
"day": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28"],
"year": [
"197{digits(1)}",
"198{digits(1)}",
{ "format": "199{digits(1)}", "weight": 2 },
{ "format": "200{digits(1)}", "weight": 3 },
{ "format": "201{digits(1)}", "weight": 3 },
{ "format": "202{digits(1)}", "weight": 2 }
]
}
"{date(1970-01-01,2029-12-31,'01/02/2006')}"
+1
View File
@@ -0,0 +1 @@
{ "format": "{name}", "rows": "first-name.tsv", "name": "name", "parent": "sex", "weight": "count" }
File diff suppressed because it is too large Load Diff
+15
View File
@@ -0,0 +1,15 @@
{
"format": "9{digits(2)}-{group}-{serial}",
"group": [
{ "format": "{int(50,65)}", "weight": 16 },
{ "format": "{int(70,88)}", "weight": 19 },
{ "format": "{int(90,92)}", "weight": 3 },
{ "format": "{int(94,99)}", "weight": 6 }
],
"serial": [
{ "format": "000{int(1,9)}", "weight": 9 },
{ "format": "00{int(10,99)}", "weight": 90 },
{ "format": "0{int(100,999)}", "weight": 900 },
{ "format": "{int(1000,9999)}", "weight": 9000 }
]
}
+1
View File
@@ -0,0 +1 @@
{ "format": "{name}", "rows": "last-name.tsv", "key": "name", "weight": "count" }
File diff suppressed because it is too large Load Diff
+5 -5
View File
@@ -1,7 +1,7 @@
{
"format": "{prefix}{femalefirst|malefirst} {last}",
"femalefirst": ["Abigail", "Addison", "Amelia", "Aria", "Aubrey", "Audrey", "Aurora", "Ava", "Bella", "Brooklyn", "Camila", "Caroline", "Charlotte", "Chloe", "Claire", "Eleanor", "Elizabeth", "Ella", "Ellie", "Emily", "Emma", "Evelyn", "Gianna", "Grace", "Hannah", "Harper", "Hazel", "Isabella", "Layla", "Leah", "Lillian", "Lily", "Lucy", "Luna", "Madison", "Maya", "Mia", "Mila", "Naomi", "Natalie", "Nora", "Olivia", "Paisley", "Penelope", "Riley", "Savannah", "Scarlett", "Sofia", "Sophia", "Stella", "Victoria", "Violet", "Zoe"],
"malefirst": ["Aiden", "Alexander", "Andrew", "Anthony", "Asher", "Benjamin", "Caleb", "Carter", "Charles", "Christopher", "Daniel", "David", "Dylan", "Elijah", "Ethan", "Ezra", "Gabriel", "Grayson", "Henry", "Isaac", "Jack", "Jackson", "Jacob", "James", "Jayden", "John", "Joseph", "Joshua", "Julian", "Levi", "Liam", "Lincoln", "Logan", "Lucas", "Luke", "Mason", "Mateo", "Matthew", "Michael", "Nathan", "Noah", "Oliver", "Owen", "Samuel", "Sebastian", "Theodore", "Thomas", "William", "Wyatt"],
"last": ["Adams", "Allen", "Anderson", "Bailey", "Baker", "Bell", "Bennett", "Brooks", "Brown", "Campbell", "Carter", "Clark", "Collins", "Cook", "Cooper", "Cox", "Davis", "Edwards", "Evans", "Flores", "Foster", "Garcia", "Gonzalez", "Gray", "Green", "Hall", "Harris", "Hayes", "Hernandez", "Hill", "Howard", "Hughes", "Jackson", "James", "Jenkins", "Johnson", "Jones", "Kelly", "King", "Lee", "Lewis", "Long", "Lopez", "Martin", "Martinez", "Miller", "Mitchell", "Moore", "Morgan", "Morris", "Murphy", "Nelson", "Parker", "Perez", "Perry", "Peterson", "Phillips", "Powell", "Price", "Ramirez", "Reed", "Richardson", "Rivera", "Roberts", "Robinson", "Rodriguez", "Rogers", "Ross", "Russell", "Sanchez", "Sanders", "Scott", "Smith", "Stewart", "Sullivan", "Taylor", "Thomas", "Thompson", "Torres", "Turner", "Walker", "Ward", "Watson", "White", "Williams", "Wilson", "Wood", "Wright", "Young"],
"prefix": ["", { "format": "{title} ", "title": ["Dr", "Miss", "Mr", "Mrs", "Ms", "Mx", "Prof"], "weight": 0.1 }]
"format": "{prefix}{first} {last}",
"first": "{.first-name.name}",
"last": "{.last-name.name}",
"prefix": ["", { "format": "{title} ", "title": ["Dr", "Miss", "Mr", "Mrs", "Ms", "Mx", "Prof"], "weight": 0.1 }],
"sex": "{.sex.name}"
}
+1
View File
@@ -0,0 +1 @@
{ "format": "{name}", "rows": "sex.tsv", "key": "code", "name": "name" }
+3
View File
@@ -0,0 +1,3 @@
code name
f female
m male
1 code name
2 f female
3 m male
+19 -1
View File
@@ -1 +1,19 @@
"{int(100,999)}-{digits(2)}-{digits(4)}"
{
"format": "{area}-{group}-{serial}",
"area": [
{ "format": "00{int(1,9)}", "weight": 9 },
{ "format": "0{int(10,99)}", "weight": 90 },
{ "format": "{int(100,665)}", "weight": 566 },
{ "format": "{int(667,899)}", "weight": 233 }
],
"group": [
{ "format": "0{int(1,9)}", "weight": 9 },
{ "format": "{int(10,99)}", "weight": 90 }
],
"serial": [
{ "format": "000{int(1,9)}", "weight": 9 },
{ "format": "00{int(10,99)}", "weight": 90 },
{ "format": "0{int(100,999)}", "weight": 900 },
{ "format": "{int(1000,9999)}", "weight": 9000 }
]
}
+1 -6
View File
@@ -1,6 +1 @@
{
"format": "{hour}:{minute} {ampm}",
"hour": ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12"],
"minute": { "format": "{t}{digits(1)}", "t": ["0", "1", "2", "3", "4", "5"] },
"ampm": ["AM", "PM"]
}
"{time('3:04 PM')}"
+1
View File
@@ -0,0 +1 @@
"{date(2000-01-01,2029-12-31,'2006-01-02T15:04:05Z')}"
+1 -13
View File
@@ -1,13 +1 @@
{
"format": "{year}-{month}-{day}",
"month": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"],
"day": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28"],
"year": [
"197{digits(1)}",
"198{digits(1)}",
{ "format": "199{digits(1)}", "weight": 2 },
{ "format": "200{digits(1)}", "weight": 3 },
{ "format": "201{digits(1)}", "weight": 3 },
{ "format": "202{digits(1)}", "weight": 2 }
]
}
"{date(1970-01-01,2029-12-31,'2006-01-02')}"
+1
View File
@@ -0,0 +1 @@
{ "format": "{name}", "rows": "first-name.tsv", "name": "name", "parent": "sex", "weight": "count" }
File diff suppressed because it is too large Load Diff
+1
View File
@@ -0,0 +1 @@
{ "format": "{name}", "rows": "last-name.tsv", "key": "name", "weight": "count" }
File diff suppressed because it is too large Load Diff
+5 -30
View File
@@ -1,32 +1,7 @@
{
"format": "{prefix}{femalefirst|malefirst} {last}",
"femalefirst": ["Agnes", "Alice", "Alicia", "Alma", "Amanda", "Anna", "Astrid", "Cornelia", "Ebba", "Elin", "Ella", "Ellen", "Elsa", "Emilia", "Emma", "Ester", "Eva", "Frida", "Hanna", "Hedda", "Ida", "Ingrid", "Isabelle", "Julia", "Karin", "Klara", "Kristina", "Lena", "Linnéa", "Lova", "Maja", "Maria", "Moa", "Molly", "Märta", "Nellie", "Nora", "Olivia", "Saga", "Sara", "Selma", "Signe", "Siri", "Sofia", "Stina", "Tilde", "Tuva", "Wilma", "Ylva", "Åsa"],
"malefirst": ["Adam", "Albin", "Alexander", "Alfred", "Anders", "Anton", "Arvid", "Axel", "Bengt", "Bo", "Carl", "David", "Edvin", "Elias", "Emil", "Erik", "Filip", "Folke", "Fredrik", "Gustav", "Göran", "Hampus", "Hans", "Henrik", "Hugo", "Isak", "Johan", "Jonas", "Karl", "Kjell", "Lars", "Leo", "Linus", "Love", "Magnus", "Mats", "Mattias", "Mikael", "Nils", "Olle", "Oskar", "Otto", "Patrik", "Per", "Pontus", "Rasmus", "Samuel", "Sixten", "Stefan", "Sten", "Sven", "Theodor", "Tobias", "Viktor", "William", "Åke"],
"last": [
{
"format": "{first}sson",
"weight": 5,
"first": ["Ander", "Arvid", "Bengt", "Daniel", "David", "Erik", "Fredrik", "Gustaf", "Henrik", "Håkan", "Isak", "Jakob", "Johan", "Jon", "Jön", "Karl", "Lar", "Magnu", "Matt", "Mikael", "Mårten", "Nil", "Ol", "Per", "Petter", "Samuel", "Sven"]
},
{
"format": "{first}{last}",
"weight": 3.6,
"first": ["Ahl", "Berg", "Blom", "Ceder", "Dahl", "Ek", "Eng", "Falk", "Fors", "Gran", "Hag", "Hed", "Holm", "Lind", "Lund", "Mal", "Ny", "Rosen", "Sand", "Sjö", "Skog", "Söder", "Sten", "Strand", "Sund", "Wik", "Öst"],
"last": ["berg", "blad", "crona", "dahl", "ed", "fors", "gren", "holm", "in", "kvist", "löf", "lund", "man", "mark", "qvist", "roth", "stedt", "sten", "strand", "ström", "vall"]
},
{
"format": "{first}{last}",
"weight": 0.267,
"first": ["Hell", "Wall"],
"last": ["berg", "blad", "crona", "dahl", "ed", "fors", "gren", "holm", "in", "kvist", "man", "mark", "qvist", "roth", "stedt", "sten", "strand", "ström", "vall"]
},
{
"format": "{first}{last}",
"weight": 0.133,
"first": "Norr",
"last": ["berg", "blad", "crona", "dahl", "ed", "fors", "gren", "holm", "in", "kvist", "löf", "lund", "man", "mark", "qvist", "stedt", "sten", "strand", "ström", "vall"]
},
["Berg", "Blom", "Eismar", "Falk", "Holm", "Lind", "Norberg", "Strand", "Ström", "von Flemming", "Åberg", "Öberg"]
],
"prefix": ["", { "format": "{string} ", "string": ["dr", "prof"], "weight": 0.05 }]
"format": "{prefix}{first} {last}",
"first": "{.first-name.name}",
"last": "{.last-name.name}",
"prefix": ["", { "format": "{string} ", "string": ["dr", "prof"], "weight": 0.05 }],
"sex": "{.sex.name}"
}
+1
View File
@@ -0,0 +1 @@
"{date(1930-01-01,2025-12-31,'060102')}-{.sex.birth-number}{luhn()}"
+1
View File
@@ -0,0 +1 @@
"{date(1930-01-01,2025-12-31,'0601')}{int(61,88)}-{.sex.birth-number}{luhn()}"
+1
View File
@@ -0,0 +1 @@
{ "format": "{name}", "rows": "sex.tsv", "key": "code", "name": "name" }
+3
View File
@@ -0,0 +1,3 @@
code name birth-number
f kvinna 238
m man 239
1 code name birth-number
2 f kvinna 238
3 m man 239
-22
View File
@@ -1,22 +0,0 @@
{
"format": "{digits(2)}{mmdd}-{digits(3)}{luhn()}",
"mmdd": [
{
"format": "{m}{d}",
"weight": 7,
"m": ["01", "03", "05", "07", "08", "10", "12"],
"d": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31"]
},
{
"format": "{m}{d}",
"weight": 4,
"m": ["04", "06", "09", "11"],
"d": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30"]
},
{
"format": "{m}{d}",
"m": "02",
"d": ["01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28"]
}
]
}
+1 -17
View File
@@ -1,17 +1 @@
{
"format": "{hour}:{minute}{sec}",
"hour": [
{ "format": "0{digits(1)}", "weight": 4 },
{ "format": "1{digits(1)}", "weight": 4 },
{ "format": "2{t}", "t": ["0", "1", "2", "3"], "weight": 2 }
],
"minute": { "format": "{t}{digits(1)}", "t": ["0", "1", "2", "3", "4", "5"] },
"sec": [
"",
{
"format": ":{s}",
"s": { "format": "{t}{digits(1)}", "t": ["0", "1", "2", "3", "4", "5"] },
"weight": 0.4
}
]
}
"{time('15:04')}"
+110 -18
View File
@@ -10,7 +10,7 @@ import (
// TestShippedDataCategories asserts every shipped locale emits only well-formed
// values for each data category. en and sv patterns differ where the format is
// locale-specific (date, time, ssn, company, price, ...); the rest are shared.
// locale-specific (date, time, company, price, ...); the rest are shared.
func TestShippedDataCategories(t *testing.T) {
letters := regexp.MustCompile(`^[\pL'-]+$`)
semver := regexp.MustCompile(`^v?\d+\.\d+\.\d+(-(alpha|beta|rc)\.\d+)?$`)
@@ -33,10 +33,7 @@ func TestShippedDataCategories(t *testing.T) {
regexp.MustCompile(`^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])$`)},
{"time",
regexp.MustCompile(`^([1-9]|1[0-2]):[0-5]\d (AM|PM)$`),
regexp.MustCompile(`^([01]\d|2[0-3]):[0-5]\d(:[0-5]\d)?$`)},
{"ssn",
regexp.MustCompile(`^[1-9]\d{2}-\d{2}-\d{4}$`),
regexp.MustCompile(`^\d{2}(0[1-9]|1[0-2])(0[1-9]|[12]\d|3[01])-\d{4}$`)},
regexp.MustCompile(`^([01]\d|2[0-3]):[0-5]\d$`)},
{"version", semver, semver},
{"email", email, email},
{"ip", ip, ip},
@@ -72,7 +69,11 @@ func TestShippedMiscCategories(t *testing.T) {
v4 := regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`)
mac := regexp.MustCompile(`^([0-9a-f]{2}:){5}[0-9a-f]{2}$`)
objectid := regexp.MustCompile(`^[0-9a-f]{24}$`)
datetime := regexp.MustCompile(`^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$`)
for i := 0; i < 200; i++ {
if v := fake(t, f, "datetime"); !datetime.MatchString(v) {
t.Fatalf("misc datetime %q is not an RFC 3339 UTC instant", v)
}
if v := fake(t, f, "objectid"); !objectid.MatchString(v) {
t.Fatalf("misc objectid %q is not 24 hex chars", v)
}
@@ -147,30 +148,121 @@ func TestSwedishPersonNamesHaveNoTripleLetter(t *testing.T) {
}
}
// TestSwedishPersonnummer checks the two rules the shape regex can't: the date
// is a real calendar date (so month-length variants never emit e.g. Apr 31 or
// Feb 30) and the trailing digit is a valid Luhn checksum over the other nine.
// TestSwedishPersonnummer checks what the shape regex can't: the date is a real
// calendar date, the birth number is Skatteverket's test series (238 female, 239
// male), the trailing digit is a Luhn checksum over the other nine, and a
// samordningsnummer is the same with 60 added to the day.
func TestSwedishPersonnummer(t *testing.T) {
sv := newGenerator(t, "data", WithSeed(1))
sawLongMonthEnd := false
re := regexp.MustCompile(`^\d{6}-23[89]\d$`)
sawLongMonthEnd, birth := false, map[string]bool{}
for i := 0; i < 2000; i++ {
v := fake(t, sv, "sv_SE.ssn")
v := fake(t, sv, "sv_SE.personnummer")
d := digitsOnly(v)
if len(d) != 10 {
t.Fatalf("ssn %q has %d digits, want 10", v, len(d))
if !re.MatchString(v) {
t.Fatalf("personnummer %q, want YYMMDD-238C or YYMMDD-239C", v)
}
if _, err := time.Parse("060102", d[:6]); err != nil { // 2-digit year, real-date check
t.Fatalf("ssn %q is not a valid calendar date: %v", v, err)
if _, err := time.Parse("060102", d[:6]); err != nil {
t.Fatalf("personnummer %q is not a valid calendar date: %v", v, err)
}
if !luhnValid(d) {
t.Fatalf("ssn %q fails the Luhn check", v)
t.Fatalf("personnummer %q fails the Luhn check", v)
}
if d[4:6] == "31" {
sawLongMonthEnd = true
sawLongMonthEnd = sawLongMonthEnd || d[4:6] == "31"
birth[d[6:9]] = true
s := fake(t, sv, "sv_SE.samordningsnummer")
sd := digitsOnly(s)
day, err := strconv.Atoi(sd[4:6])
if !re.MatchString(s) || err != nil || day < 61 || day > 88 || !luhnValid(sd) {
t.Fatalf("samordningsnummer %q, want YYMM(61-88)-23[89]C, Luhn-valid", s)
}
if _, err := time.Parse("0601", sd[:4]); err != nil {
t.Fatalf("samordningsnummer %q is not a valid year and month: %v", s, err)
}
}
if !sawLongMonthEnd {
t.Fatal("never generated a 31st — 31-day months are not reaching their last day")
t.Fatal("never generated a 31st")
}
if !birth["238"] || !birth["239"] {
t.Fatalf("birth numbers drawn %v, want both 238 and 239", birth)
}
for i := 0; i < 200; i++ {
got := fakeTemplate(t, sv, `{/sv_SE.person.sex} {/sv_SE.personnummer}`)
if f, m := strings.HasPrefix(got, "kvinna ") && got[14:17] == "238", strings.HasPrefix(got, "man ") && got[11:14] == "239"; !f && !m {
t.Fatalf("%q: a person and a personnummer in one render disagree on sex", got)
}
}
}
// TestShippedUSTaxIds pins the SSA and IRS ranges: an SSN's area is 001-899 but
// 666, its group 01-99 and its serial 0001-9999; an ITIN is 9XX-GG-XXXX with GG
// in 50-65, 70-88, 90-92 or 94-99.
func TestShippedUSTaxIds(t *testing.T) {
f := newGenerator(t, "data", WithSeed(5))
ssn := regexp.MustCompile(`^(\d{3})-(\d{2})-(\d{4})$`)
itin := regexp.MustCompile(`^9\d{2}-(5\d|6[0-5]|7\d|8[0-8]|9[0-24-9])-\d{4}$`)
for i := 0; i < 2000; i++ {
v := fake(t, f, "en_US.ssn")
m := ssn.FindStringSubmatch(v)
if m == nil {
t.Fatalf("ssn %q, want AAA-GG-SSSS", v)
}
area, _ := strconv.Atoi(m[1])
group, _ := strconv.Atoi(m[2])
serial, _ := strconv.Atoi(m[3])
if area < 1 || area > 899 || area == 666 || group < 1 || serial < 1 {
t.Fatalf("ssn %q is in a range the SSA never assigns", v)
}
if v := fake(t, f, "en_US.itin"); !itin.MatchString(v) {
t.Fatalf("itin %q, want %s", v, itin)
}
}
}
// TestShippedPersonNames pins the name tables: a sex pins its first names, a name
// both sexes carry appears under both, and the record's sex agrees with its name.
func TestShippedPersonNames(t *testing.T) {
f := newGenerator(t, "data", WithSeed(9))
for path, want := range map[string]string{
"sv_SE.sex[f]": "kvinna",
"sv_SE.sex[man].code": "m",
"en_US.sex[f]": "female",
"sv_SE.sex[f].first-name[Anna]": "Anna",
"sv_SE.sex[m].first-name[Erik].sex": "m",
"en_US.sex[m].first-name[James]": "James",
"en_US.sex[f].first-name[Taylor].sex": "f",
"en_US.sex[m].first-name[Taylor].sex": "m",
"sv_SE.last-name[Andersson]": "Andersson",
"sv_SE.last-name[Andersson].count": "",
"en_US.last-name[Smith]": "Smith",
} {
got := fake(t, f, path)
if want == "" && !regexp.MustCompile(`^[1-9]\d*$`).MatchString(got) {
t.Errorf("Fake(%q) = %q, want a count", path, got)
} else if want != "" && got != want {
t.Errorf("Fake(%q) = %q, want %q", path, got, want)
}
}
if _, err := f.Fake("en_US.first-name[Taylor]"); err == nil || !strings.Contains(err.Error(), "sex[f].first-name[Taylor]") {
t.Fatalf("Fake(en_US.first-name[Taylor]) = %v, want both sexes' rows listed", err)
}
for _, locale := range []string{"sv_SE", "en_US"} {
seen := map[string]bool{}
for i := 0; i < 2000; i++ {
seen[fake(t, f, locale+".sex[f].first-name")] = true
first, sex := fake(t, f, locale+".person.first"), fake(t, f, locale+".person.sex")
if first == "" || sex == "" {
t.Fatalf("%s.person lacks a first name or a sex", locale)
}
}
if len(seen) < 300 || !seen["Anna"] && locale == "sv_SE" || !seen["Mary"] && locale == "en_US" {
t.Fatalf("%s female first names: %d distinct in 2000, want a weighted register", locale, len(seen))
}
got := fakeTemplate(t, f, `{/`+locale+`.sex.code}|{/`+locale+`.person.first}`)
code, first, _ := strings.Cut(got, "|")
if v := fake(t, f, locale+".sex["+code+"].first-name["+first+"]"); v != first {
t.Fatalf("%s: %q drawn as a %s name is not one", locale, first, code)
}
}
}
+11
View File
@@ -81,3 +81,14 @@ func TestDifferentSeedsDiffer(t *testing.T) {
}
t.Fatal("seeds 1 and 2 produced identical sequences")
}
// fakeTemplate renders an inline template against a loaded generator, so its
// references resolve.
func fakeTemplate(t *testing.T, f *Generator, s string) string {
t.Helper()
got, err := f.FakeTemplate(s)
if err != nil {
t.Fatalf("FakeTemplate(%s) = %v", s, err)
}
return got
}
+29 -6
View File
@@ -222,8 +222,8 @@ func (t *table) bindOptions(o tableOptionValues) error {
}
*opt.into = i
}
if t.name >= 0 && t.key < 0 {
return fmt.Errorf("name selects a row as a key does, and lists the rows it matches by their keys, so it needs a key column; add key")
if t.name >= 0 && t.key < 0 && t.parent < 0 {
return fmt.Errorf("name selects a row as a key does, and lists the rows it matches by their keys, so it needs a key column, or a parent inside which each name is one row; add key")
}
if err := t.indexKeys(); err != nil {
return err
@@ -231,9 +231,18 @@ func (t *table) bindOptions(o tableOptionValues) error {
return t.sumWeights()
}
// indexKeys proves every key names one row, and keeps the index a link is proved by.
// indexKeys proves every key names one row, and keeps the index a link is proved by;
// without a key, a name names one row inside its parent.
func (t *table) indexKeys() error {
if t.key < 0 {
inside := make(map[string]int, t.rows())
for r := 0; r < t.rows() && t.name >= 0; r++ {
k := t.cell(r, t.parent) + "\t" + t.cell(r, t.name)
if first, dup := inside[k]; dup {
return fmt.Errorf("%s line %d: name %q repeats line %d inside %s %q; a name selects one row inside its parent", t.file, r+2, t.cell(r, t.name), first+2, t.columns[t.parent], t.cell(r, t.parent))
}
inside[k] = r
}
return nil
}
t.byKey = make(map[string]int, t.rows())
@@ -488,8 +497,8 @@ func (t *table) under(r int, a *table, pr int) bool {
// find is the row a selector names: by key first, then by name, where a name
// naming several rows resolves inside the ancestors pinned in d.
func (t *table) find(sel string, d *draws) (int, error) {
if t.key < 0 {
return 0, fmt.Errorf("%s has no key column to select a row by", t.category)
if t.key < 0 && t.name < 0 {
return 0, fmt.Errorf("%s has no key or name column to select a row by", t.category)
}
if r, ok := t.byKey[sel]; ok {
return r, nil
@@ -506,8 +515,15 @@ func (t *table) find(sel string, d *draws) (int, error) {
}
keys := make([]string, len(rows))
for i, r := range rows {
if t.key < 0 {
keys[i] = t.selectorSpelling(r)
} else {
keys[i] = t.cell(r, t.key)
}
}
if t.key < 0 {
return 0, fmt.Errorf("%q names %d rows of %s; select it inside its %s, one of %v", sel, len(rows), t.category, t.parentT.category, keys)
}
inside := ""
if t.parentT != nil {
inside = fmt.Sprintf(", or select it inside its %s", t.parentT.category)
@@ -515,7 +531,14 @@ func (t *table) find(sel string, d *draws) (int, error) {
return 0, fmt.Errorf("%q names %d rows of %s; select one by key, one of %v%s", sel, len(rows), t.category, keys, inside)
}
// selectorSpelling is how a path writes a selected row, for messages.
// selectorSpelling is how a path writes a selected row, for messages: by key, or
// by name inside its parent's row.
func (t *table) selectorSpelling(r int) string {
switch {
case t.key >= 0:
return t.category + "[" + t.cell(r, t.key) + "]"
case t.name >= 0:
return t.parentT.selectorSpelling(t.parentRow(r)) + "." + t.category + "[" + t.cell(r, t.name) + "]"
}
return fmt.Sprintf("%s line %d", t.file, r+2)
}
+35
View File
@@ -440,6 +440,7 @@ func TestTableFences(t *testing.T) {
"a bracket in a key": {map[string]string{"t.json": `{"format":"{a}","rows":"t.tsv","key":"a"}`, "t.tsv": "a\nx[1]\ny\n"}, `"["`},
"a brace in a name": {map[string]string{"t.json": `{"format":"{a}","rows":"t.tsv","key":"a","name":"n"}`, "t.tsv": "a\tn\nx\tx{1}\ny\ty\n"}, `"{"`},
"name without a key": {map[string]string{"t.json": `{"format":"{a}","rows":"t.tsv","name":"a"}`, "t.tsv": "a\nx\nx\n"}, "key"},
"a name repeating inside one parent row": {with(siblings(), map[string]string{"street.json": `{"format":"{name}","rows":"street.tsv","name":"name","parent":"locality"}`, "street.tsv": "name\tlocality\nAvenyn\tL1\nAvenyn\tL1\nStorgatan\tL2\nStorgatan\tL3\nStorgatan\tL4\nStorgatan\tL5\nStorgatan\tL6\nStorgatan\tL7\nStorgatan\tL8\n"}), `"Avenyn"`},
"a name that is another row's key": {map[string]string{"t.json": `{"format":"{a}","rows":"t.tsv","key":"a","name":"n"}`, "t.tsv": "a\tn\nx\ty\ny\tz\n"}, `"y"`},
"a cell reading its family": {with(geo(), map[string]string{"locality.tsv": "code\tname\tmunicipality\tnote\nL1\tStockholm\t0180\t{/municipality.code}\nL2\tSolna\t0184\t-\nL3\tMalmö\t1280\t-\nL4\tLund\t1281\t-\nL5\tGöteborg\t1480\t-\n"}), "family"},
"a format reading its family": {with(geo(), map[string]string{"locality.json": `{"format":"{name} {/region.name}","rows":"locality.tsv","key":"code","name":"name","parent":"municipality"}`}), "family"},
@@ -709,3 +710,37 @@ func TestShippedTables(t *testing.T) {
t.Fatalf("country draws %d distinct rows in 5000, want the full register", len(count))
}
}
// TestNamedTableWithoutAKeyResolvesInsideItsParent pins a table whose rows are
// told apart only by their parent: a name selects a row inside the parent pinned
// before it, an ambiguous one is listed by its parent's spelling, and a name
// repeating inside one parent row is a load error (see TestTableFences).
func TestNamedTableWithoutAKeyResolvesInsideItsParent(t *testing.T) {
files := siblings()
files["street.json"] = `{"format":"{name}","rows":"street.tsv","name":"name","parent":"locality","weight":"segments"}`
f := newGenerator(t, writeFiles(t, files), WithSeed(1))
for path, want := range map[string]string{
"street[Avenyn]": "Avenyn",
"street[Avenyn].locality": "L5",
"locality[L7].street[Sandbyvägen]": "Sandbyvägen",
"locality[L7].street[Sandbyvägen].segments": "2",
"municipality[0184].locality[Sandby].street[Sandbyvägen]": "Sandbyvägen",
"municipality[0184].locality[Sandby].street[Sandbyvägen].locality": "L8",
} {
if got := fake(t, f, path); got != want {
t.Errorf("Fake(%q) = %q, want %q", path, got, want)
}
}
for path, want := range map[string]string{
"street[Sandbyvägen]": "locality[L7].street[Sandbyvägen]",
"locality[L1].street[Avenyn]": "not inside",
"street[Kungsgatan]": `"Kungsgatan"`,
} {
if _, err := f.Fake(path); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("Fake(%q) = %v, want an error mentioning %s", path, err, want)
}
}
if got := fakeTemplate(t, f, `{/locality[L8].name}: {/locality[L8].street[Sandbyvägen].name}`); got != "Sandby: Sandbyvägen" {
t.Fatalf("a name selected inside a pinned parent = %q", got)
}
}
+17 -6
View File
@@ -88,17 +88,28 @@ func funcCall(body string) (name string, args []string, ok bool) {
return body[:lp], splitArgs(body[lp+1 : len(body)-1]), true
}
// splitArgs parses a function arg list: comma-separated outside a selector,
// trimmed; empty -> none.
// splitArgs parses a function arg list: comma-separated outside a selector or a
// quoted layout, trimmed; empty -> none.
func splitArgs(s string) []string {
if strings.TrimSpace(s) == "" {
return nil
}
args := splitOutside(s, ',')
for i := range args {
args[i] = strings.TrimSpace(args[i])
var args []string
depth, quoted, start := 0, false, 0
for i := 0; i < len(s); i++ {
switch c := s[i]; {
case c == '\'':
quoted = !quoted
case quoted:
case c == '[':
depth++
case c == ']' && depth > 0:
depth--
case c == ',' && depth == 0:
args, start = append(args, strings.TrimSpace(s[start:i])), i+1
}
return args
}
return append(args, strings.TrimSpace(s[start:]))
}
// checkFunc validates a function token at compile time: well-formed, naming a
+49 -29
View File
@@ -8,20 +8,27 @@ en_US.color
en_US.company format "{base} {suffix}"
en_US.company.base string
en_US.company.suffix string
en_US.date format "{month}/{day}/{year}"
en_US.date.day string
en_US.date.month string
en_US.date.year string
en_US.date format "{date(1970-01-01,2029-12-31,'01/02/2006')}"
en_US.email format "{local}@{domain}"
en_US.email.domain string
en_US.email.local string
en_US.email.local.n
en_US.first-name format "{name}" name name weight count parent sex
en_US.first-name.count string
en_US.first-name.name string
en_US.first-name.sex string
en_US.ip
en_US.person format "{prefix}{femalefirst|malefirst} {last}"
en_US.person.femalefirst string
en_US.itin format "9{digits(2)}-{group}-{serial}"
en_US.itin.group string
en_US.itin.serial string
en_US.last-name format "{name}" key name weight count
en_US.last-name.count string
en_US.last-name.name string
en_US.person format "{prefix}{first} {last}" reads en_US.first-name en_US.last-name en_US.sex
en_US.person.first string
en_US.person.last string
en_US.person.malefirst string
en_US.person.prefix string
en_US.person.sex string
en_US.phone
en_US.phone.area
en_US.phone.exch
@@ -34,12 +41,18 @@ en_US.sentence.adj
en_US.sentence.noun
en_US.sentence.prep
en_US.sentence.verb
en_US.ssn format "{int(100,999)}-{digits(2)}-{digits(4)}"
en_US.time format "{hour}:{minute} {ampm}"
en_US.time.ampm string
en_US.time.hour string
en_US.time.minute string
en_US.time.minute.t
en_US.sex format "{name}" key code name name
en_US.sex.code string
en_US.sex.first-name
en_US.sex.first-name.count
en_US.sex.first-name.name
en_US.sex.first-name.sex
en_US.sex.name string
en_US.ssn format "{area}-{group}-{serial}"
en_US.ssn.area string
en_US.ssn.group string
en_US.ssn.serial string
en_US.time format "{time('3:04 PM')}"
en_US.url format "https://{host}{path}"
en_US.url.host string
en_US.url.path string
@@ -216,6 +229,7 @@ misc.currency.decimals string
misc.currency.name string
misc.currency.numeric string
misc.currency.symbol string
misc.datetime format "{date(2000-01-01,2029-12-31,'2006-01-02T15:04:05Z')}"
misc.emoji
misc.httpstatus format "{code} {reason}" key code name reason
misc.httpstatus.code string
@@ -241,20 +255,25 @@ sv_SE.color
sv_SE.company format "{base} {suffix}"
sv_SE.company.base string
sv_SE.company.suffix string
sv_SE.date format "{year}-{month}-{day}"
sv_SE.date.day string
sv_SE.date.month string
sv_SE.date.year string
sv_SE.date format "{date(1970-01-01,2029-12-31,'2006-01-02')}"
sv_SE.email format "{local}@{domain}"
sv_SE.email.domain string
sv_SE.email.local string
sv_SE.email.local.n
sv_SE.first-name format "{name}" name name weight count parent sex
sv_SE.first-name.count string
sv_SE.first-name.name string
sv_SE.first-name.sex string
sv_SE.ip
sv_SE.person format "{prefix}{femalefirst|malefirst} {last}"
sv_SE.person.femalefirst string
sv_SE.last-name format "{name}" key name weight count
sv_SE.last-name.count string
sv_SE.last-name.name string
sv_SE.person format "{prefix}{first} {last}" reads sv_SE.first-name sv_SE.last-name sv_SE.sex
sv_SE.person.first string
sv_SE.person.last string
sv_SE.person.malefirst string
sv_SE.person.prefix string
sv_SE.person.sex string
sv_SE.personnummer format "{date(1930-01-01,2025-12-31,'060102')}-{.sex.birth-number}{luhn()}" reads sv_SE.sex
sv_SE.phone
sv_SE.phone.a
sv_SE.phone.b
@@ -263,20 +282,21 @@ sv_SE.phone.prefix
sv_SE.price format "{amt}{ore} kr"
sv_SE.price.amt string
sv_SE.price.ore string
sv_SE.samordningsnummer format "{date(1930-01-01,2025-12-31,'0601')}{int(61,88)}-{.sex.birth-number}{luhn()}" reads sv_SE.sex
sv_SE.sentence
sv_SE.sentence.adj
sv_SE.sentence.noun
sv_SE.sentence.prep
sv_SE.sentence.verb
sv_SE.ssn format "{digits(2)}{mmdd}-{digits(3)}{luhn()}"
sv_SE.ssn.mmdd string
sv_SE.ssn.mmdd.d
sv_SE.ssn.mmdd.m
sv_SE.time format "{hour}:{minute}{sec}"
sv_SE.time.hour string
sv_SE.time.minute string
sv_SE.time.minute.t
sv_SE.time.sec string
sv_SE.sex format "{name}" key code name name
sv_SE.sex.birth-number string
sv_SE.sex.code string
sv_SE.sex.first-name
sv_SE.sex.first-name.count
sv_SE.sex.first-name.name
sv_SE.sex.first-name.sex
sv_SE.sex.name string
sv_SE.time format "{time('15:04')}"
sv_SE.url format "https://{host}{path}"
sv_SE.url.host string
sv_SE.url.path string
+6 -5
View File
@@ -42,7 +42,7 @@ value is composed; TSV says which values exist.
and a `data-import/` directory of Python scripts that rebuild each TSV from its
source, so a refresh is one command per dataset.
- Make every shipped category a record with its building blocks as columns
(`sv_SE.person` → `femalefirst`, `malefirst`; `misc.uuid` → `variant`).
(`sv_SE.person` → `first`, `last`, `sex`, done in step 3; `misc.uuid` → `variant`).
- Share the handle lists between `email.local` and `username` only if that is a
clean win — a reference between shipped categories is a major once tagged.
@@ -75,9 +75,8 @@ countries; the README maps each to the native term.
#### Builtins the data cannot express
- `{date(from,to,layout)}`: a date in `[from, to]` in a Go layout, and `{age(min,max)}`
as the birthdate spelling; `{time(layout)}`. Personnummer, SSN, birthdate, card
expiry, unix time and ISO datetime all build on it.
- `{date(from,to,'layout')}` and `{time('layout')}` shipped in step 3; `age()` is
rejected, README Decisions. Add a `unix` layout once something needs it.
- Derivations: `{isin()}` (Luhn over letters expanded to digits), `{cusip()}`,
`{aba()}` (3-7-1 weights), `{vin()}` (position 9 over the whole; a sample taking
the WMI, since the check sits mid-string).
@@ -165,7 +164,9 @@ address, phone, national id, company and date names each.
1. Table node, key and name selection, parent links, consistent draws, the
choice-of-rows fence, `DATA-LICENSES.md`, `data-import/` — done.
2. `geo/SE` and `geo/US`, and `address` in both locales on top of them — done.
3. Weighted person names and valid ids in both locales; `date()`.
3. Weighted person names and valid ids in both locales; `date()` — done. Middle
names wait for a draw group that shares its family's pins, so a second name
is drawn under the same sex.
4. `misc` conversions and the new `misc` tables.
5. The remaining locale categories: company, phone, finance, vehicle, words.
6. Records with building-block columns across the shipped set; shape re-pin.