larvit/fejkdata
3
0
Fork 0
Code Pull Requests 1 Actions Activity

Make great documentation goal 1.2, and plan the README's reorganisation and the nested-template gaps #33

Open
lilleman wants to merge 966 commits from worktree-todo-relative-dates-exact-counts into main
pull from: worktree-todo-relative-dates-exact-counts
merge into: larvit:main
Conversation 0 Commits 966 Files Changed 167
+712 -681
6 changed files with 712 additions and 681 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 35f1eeffb2 - Show all commits
CHANGELOG.md
+4
View File
@@ -137,6 +137,10 @@ replacement, and each removed path, column or flag.
it reads: `svse.Modules()` holds the Swedish places too. `New` loads the modules `WithDataFS(modules...)` names and it reads: `svse.Modules()` holds the Swedish places too. `New` loads the modules `WithDataFS(modules...)` names and
the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option
is given. A manifest's `namespace` mounts its source under a dot path. is given. A manifest's `namespace` mounts its source under a dot path.
- Package `github.com/larvit/fejkdata/cli` is the `fejkdata` command: `cli.Main(modules...)`
runs it over the modules given, and returns its exit code, so a `main` package passing
`append(data.Modules(), mydata)` builds a CLI carrying your own data. `--version` prints
the fejkdata module's version, `devel` in a build from a checkout.
- A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each - A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
category on the first call reaching it, as each shipped module does. Data failing to load, category on the first call reaching it, as each shipped module does. Data failing to load,
in `New` or on that first call, gives an error matching `ErrLoad`, and the CLI exits 1. in `New` or on that first call, gives an error matching `ErrLoad`, and the CLI exits 1.
README.md
+33 -17
View File
@@ -337,10 +337,10 @@ go get github.com/larvit/fejkdata # Go 1.22+
```go ```go
import ( import (
"github.com/larvit/fejkdata" "github.com/larvit/fejkdata"
"github.com/larvit/fejkdata/data" svse "github.com/larvit/fejkdata/data/sv_SE"
) )
f, err := fejkdata.New(fejkdata.WithDataFS(data.Modules()...), fejkdata.WithSeed(42)) f, err := fejkdata.New(fejkdata.WithDataFS(svse.Modules()...), fejkdata.WithSeed(42))
if err != nil { if err != nil {
log.Fatal(err) log.Fatal(err)
} }
@@ -362,20 +362,9 @@ ok, err := fejkdata.IsTemplate(arg) // an inline template by its shape, el
| `WithDataPath(dir)` | layer a directory; repeat to layer several, the last wins a clash | | `WithDataPath(dir)` | layer a directory; repeat to layer several, the last wins a clash |
| `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `Modules()...`, or your own `embed.FS`; `WithDataFS()` loads none | | `WithDataFS(modules...)` | layer modules in order, each an `fs.FS`: a shipped package's `Modules()...`, or your own `embed.FS`; `WithDataFS()` loads none |
`data.Modules()` is every shipped module, and links all of them into your program. To link A shipped package's `Modules()` is its `FS` and every module it reads, a locale's country's
only the ones you read, pass a package's `Modules()`: its `FS` and every module it reads, a places among them, which its package imports, so your program links only what you read.
locale's country's places among them, which its package imports: `data.Modules()` is every shipped module, and links all of them. Two locales are `WithDataFS(append(svse.Modules(), enus.Modules()...)...)`. To stand in your
```go
import (
"github.com/larvit/fejkdata"
svse "github.com/larvit/fejkdata/data/sv_SE"
)
f, err := fejkdata.New(fejkdata.WithDataFS(svse.Modules()...))
```
Two locales are `WithDataFS(append(svse.Modules(), enus.Modules()...)...)`. To stand in your
own places, pass the locale's `FS` alone beside a tree providing `geo.SE.address`, a record own places, pass the locale's `FS` alone beside a tree providing `geo.SE.address`, a record
with a `format` and the columns `sv_SE.address` reads, `street`, `street-number`, with a `format` and the columns `sv_SE.address` reads, `street`, `street-number`,
`postal-code` and `locality`: `WithDataFS(svse.FS, myPlaces)`. `New` fails where no `WithDataFS` or `WithDataPath` names the `postal-code` and `locality`: `WithDataFS(svse.FS, myPlaces)`. `New` fails where no `WithDataFS` or `WithDataPath` names the
@@ -1034,6 +1023,32 @@ Where the project is heading; the sections before Audience document what ships t
16. **Values look real by default. You can also ask for values that can never reach 16. **Values look real by default. You can also ask for values that can never reach
anything real,** such as a domain nobody can register. anything real,** such as a domain nobody can register.
### Your own CLI
Package `cli` is the `fejkdata` command, and its `Main` takes the modules the command
loads, so a CLI carrying your own data is a `main` package of your own:
```go
package main
import (
"embed"
"os"
"github.com/larvit/fejkdata/cli"
"github.com/larvit/fejkdata/data"
)
//go:embed *.json
var mydata embed.FS
func main() { os.Exit(cli.Main(append(data.Modules(), mydata)...)) }
```
It takes `fejkdata`'s flags and exit codes, `--no-shipped-data` leaves out the modules
passed to `Main`, and `--version` prints the fejkdata version it is built with.
`fejkdata`'s own `main` is `os.Exit(cli.Main(data.Modules()...))`.
## Layout ## Layout
``` ```
@@ -1061,7 +1076,8 @@ calc.go the {calc()} operand checks: never a number, and a constant zero
datatype.go column datatypes: DataType, a template's declared datatype, a column's datatype, and a format that only reads a column datatype.go column datatypes: DataType, a template's declared datatype, a column's datatype, and a format that only reads a column
valueproof.go the value proof: what a typed column, calc operand or struct field holds, checked at load valueproof.go the value proof: what a typed column, calc operand or struct field holds, checked at load
data.go the load path: the categories internal/datafiles hands over, compiled into one merged tree, and the load pipeline every entry point runs data.go the load path: the categories internal/datafiles hands over, compiled into one merged tree, and the load pipeline every entry point runs
cmd/fejkdata/ the fejkdata CLI cli/ the fejkdata command, as the library call Main
cmd/fejkdata/ the fejkdata binary: Main over every shipped module
internal/builtinfunc/ the {name()} functions: their checks and draws, checksums, date and time layouts, transforms and calc arithmetic internal/builtinfunc/ the {name()} functions: their checks and draws, checksums, date and time layouts, transforms and calc arithmetic
internal/datafiles/ the walk of a data tree: its folders, its category files and the rows files beside them, and its manifest internal/datafiles/ the walk of a data tree: its folders, its category files and the rows files beside them, and its manifest
internal/datatype/ the datatype a record column holds; the root's DataType is an alias of it internal/datatype/ the datatype a record column holds; the root's DataType is an alias of it
cli/cli.go
+665
View File
@@ -0,0 +1,665 @@
// Package cli is the fejkdata command: it renders the template on its stdin with fake
// values from the modules it is given and any directories layered over them. A program
// passing its own modules to Main is a fejkdata CLI carrying them.
package cli
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"os"
"runtime/debug"
"sort"
"strconv"
"strings"
"github.com/larvit/fejkdata"
)
const usage = `Usage: <template> | fejkdata [flags]
fejkdata --list | --help | --version
fejkdata reads one template from stdin and renders it:
echo 'name: {/sv_SE.person.last}' | fejkdata
echo '{"format":"{x}","x":["bosse","lina"]}' | fejkdata
fejkdata < template.txt
fejkdata <<'EOF'
born {date(1990-01-01,2010-12-31,'2006-01-02')}
EOF
A template is a format string, or a JSON object, array, string or null. Its {…} tokens
reach the data by reference from the root, {/sv_SE.person.last}, whether the data
is shipped or layered with --data-path. A reference names a category or a dotted
path into one ({/sv_SE.person.last}), a table's row by key or name
({/misc.territory[SE]}), and .. steps up from a row to the row it links to
({/geo.SE.locality..municipality.name}).
Under --format text, the default, what you send is what renders, and fejkdata adds
nothing: a format string keeps every byte, so echo's newline ends each render, and
printf '%s' sends none. -n joins renders with --separator, empty by default. A JSON
object, array or string drops the whitespace around it, and null alone renders nothing;
42 or true alone is a format string, printed as written. An empty or blank stdin is a
format string like any other, and renders as sent. A quoted heredoc, <<'EOF', passes $,
a backslash or a quote as written.
With --format json, ndjson, csv or sql the template must be a record — a JSON
template whose fields are its columns, or a lone reference such as {/sv_SE.person}:
one reference and nothing else, with one newline after it allowed — and the rows are
written as one JSON array, one JSON object per line, one CSV row (after a header), or
one INSERT.
-d, --data-path D a data directory to layer over the shipped data (repeatable; last wins on a clash)
--format F output form: text (default), json, ndjson, csv or sql
-h, --help print this help, then exit
--list list the paths the data offers, each written after {/ in a template, then exit;
--repeat, --separator, --format and --table do nothing here
--no-shipped-data load only the --data-path directories
-n, --repeat N render the template N times, 0..1048576 (default 1); 0 prints nothing,
[] under --format json, or the header under --format csv
-s, --seed N same seed, version and data: identical output
--separator S string between repeated renders (default empty); refused with a record
--format, which has its own separators
--table T the INSERT target for --format sql, ignored under any other
(default: a lone reference's last segment, else records)
--version print the version, then exit
A short flag's value attaches or follows (-n3, -n 3); short flags bundle (-hn 3);
-- ends the flags.
`
type invocation struct {
args []string
dirs []string
format string
help bool
list bool
noShipped bool
repeat int
seed uint64
seeded bool
separator string
separatorSet bool
table string
tableSet bool
version bool
}
type flagDef struct {
long string
short string
value bool
set func(*invocation, string) error
}
var flagDefs = []flagDef{
{"data-path", "d", true, func(in *invocation, v string) error { in.dirs = append(in.dirs, v); return nil }},
{"format", "", true, func(in *invocation, v string) error { in.format = v; return nil }},
{"help", "h", false, func(in *invocation, _ string) error { in.help = true; return nil }},
{"list", "", false, func(in *invocation, _ string) error { in.list = true; return nil }},
{"no-shipped-data", "", false, func(in *invocation, _ string) error { in.noShipped = true; return nil }},
{"repeat", "n", true, func(in *invocation, v string) error {
n, err := strconv.Atoi(v)
if err != nil || n < 0 || n > fejkdata.MaxRepeat {
return fmt.Errorf("--repeat needs an integer in 0..%d, got %q", fejkdata.MaxRepeat, v)
}
in.repeat = n
return nil
}},
{"seed", "s", true, func(in *invocation, v string) error {
n, err := strconv.ParseUint(v, 10, 64)
if err != nil {
return fmt.Errorf("--seed needs an unsigned integer, got %q", v)
}
in.seed, in.seeded = n, true
return nil
}},
{"separator", "", true, func(in *invocation, v string) error { in.separator, in.separatorSet = v, true; return nil }},
{"table", "", true, func(in *invocation, v string) error { in.table, in.tableSet = v, true; return nil }},
{"version", "", false, func(in *invocation, _ string) error { in.version = true; return nil }},
}
func flagByLong(name string) *flagDef {
for i := range flagDefs {
if flagDefs[i].long == name {
return &flagDefs[i]
}
}
return nil
}
func flagByShort(name string) *flagDef {
for i := range flagDefs {
if flagDefs[i].short == name {
return &flagDefs[i]
}
}
return nil
}
// flagArg is one flag as written: its definition and the value attached to it,
// if any.
type flagArg struct {
def *flagDef
value string
inline bool
}
// splitFlags reads one argv element as flags: --name or --name=value, or -abc
// where each letter is a short flag and the first that takes a value takes the
// rest of the element.
func splitFlags(arg string) ([]flagArg, error) {
if strings.HasPrefix(arg, "--") {
name, value, inline := strings.Cut(arg[2:], "=")
def := flagByLong(name)
if def == nil {
return nil, fmt.Errorf("unknown flag --%s", name)
}
return []flagArg{{def, value, inline}}, nil
}
if name, _, _ := strings.Cut(arg[1:], "="); len(name) > 1 && flagByLong(name) != nil {
return nil, fmt.Errorf("unknown flag %s; use --%s", arg, name)
}
var flags []flagArg
letters := arg[1:]
for i, r := range letters {
letter := string(r)
def := flagByShort(letter)
if def == nil {
return nil, fmt.Errorf("unknown flag -%s", letter)
}
if !def.value {
flags = append(flags, flagArg{def: def})
continue
}
rest := letters[i+len(letter):]
if strings.HasPrefix(rest, "=") {
return nil, fmt.Errorf("-%s takes its value attached (-%s%s) or next (-%s %s); = belongs to --%s=%s",
letter, letter, rest[1:], letter, rest[1:], def.long, rest[1:])
}
return append(flags, flagArg{def, rest, rest != ""}), nil
}
return flags, nil
}
func parseArgs(argv []string) (invocation, error) {
in := invocation{repeat: 1, format: "text"}
for i := 0; i < len(argv); i++ {
arg := argv[i]
if arg == "--" {
in.args = append(in.args, argv[i+1:]...)
return in, nil
}
if len(arg) < 2 || arg[0] != '-' {
in.args = append(in.args, arg)
continue
}
flags, err := splitFlags(arg)
if err != nil {
return in, err
}
for _, fl := range flags {
if !fl.def.value {
if fl.inline {
return in, fmt.Errorf("--%s takes no value", fl.def.long)
}
_ = fl.def.set(&in, "")
continue
}
value := fl.value
if !fl.inline {
if i++; i >= len(argv) {
return in, fmt.Errorf("--%s needs a value", fl.def.long)
}
value = argv[i]
}
if err := fl.def.set(&in, value); err != nil {
return in, err
}
}
}
return in, nil
}
// recordFormat is one way to write a record out: the line each record renders,
// the header that precedes the first one, and the open/close frame plus the
// between-record separator a document form needs.
type recordFormat struct {
header func(*fejkdata.Record) string
line func(r *fejkdata.Record, table string) string
open string
close string
sep string
writes string // what it writes between values, which --separator would seem to set
}
// recordFormats is every --format that writes records. json frames the records
// as one array document; ndjson writes one object per line.
var recordFormats = map[string]recordFormat{
"csv": {header: (*fejkdata.Record).CSVHeader, line: func(r *fejkdata.Record, _ string) string { return r.CSVLine() }, sep: "\n", writes: `always separates fields with ","`},
"json": {line: jsonLine, open: "[", close: "]", sep: ",\n", writes: "writes the records as one JSON array"},
"ndjson": {line: jsonLine, sep: "\n", writes: "writes one record per line"},
"sql": {line: func(r *fejkdata.Record, table string) string { return r.SQLInsert(table) }, sep: "\n", writes: "writes one INSERT per line"},
}
func jsonLine(r *fejkdata.Record, _ string) string { return r.JSON() }
func (in invocation) writesRecords() bool {
return in.format != "text"
}
// formatNames lists the --format values, text included, for the misuse error.
func formatNames() string {
names := []string{"text"}
for name := range recordFormats {
names = append(names, name)
}
sort.Strings(names)
return strings.Join(names[:len(names)-1], ", ") + " or " + names[len(names)-1]
}
// checkFlags rejects a flag value or combination that cannot run.
func (in invocation) checkFlags() error {
if _, ok := recordFormats[in.format]; !ok && in.format != "text" {
return fmt.Errorf("--format takes %s, got %q", formatNames(), in.format)
}
if in.tableSet && in.table == "" && in.format == "sql" {
return errors.New("--table names the INSERT target, so it cannot be empty")
}
if f, records := recordFormats[in.format]; records && in.separatorSet && !in.list {
return fmt.Errorf("--separator joins text renders; --format %s %s", in.format, f.writes)
}
return nil
}
const pipeHint = "echo '{/sv_SE.person}' | fejkdata"
// operandError refuses an argument that is no flag, naming the spelling that reads what
// it most likely means: dropping it for -, a file for a file's name, else a template
// piped in.
func operandError(arg string) error {
unexpected := "unexpected argument '" + arg + "'"
if arg == "-" {
return fmt.Errorf("%s; stdin is read already, so drop it", unexpected)
}
if fi, err := os.Stat(arg); err == nil && fi.Mode().IsRegular() {
return fmt.Errorf("%s; read a file from stdin: fejkdata < %s", unexpected, shellQuoted(arg))
}
ref := "{/" + strings.TrimPrefix(arg, "/") + "}"
_, argLone := loneReference(arg)
inline, err := fejkdata.IsTemplate(arg)
_, refLone := loneReference(ref)
example := pipeHint
switch {
case argLone || inline:
example = piped(arg)
case err == nil && refLone:
example = piped(ref)
}
return fmt.Errorf("%s; the template is read from stdin: %s", unexpected, example)
}
// check rejects a flag combination or an operand that cannot run, before any data is
// read.
func (in invocation) check() error {
if err := in.checkFlags(); err != nil {
return err
}
if len(in.args) > 0 {
return operandError(in.args[0])
}
return nil
}
func (in invocation) options(modules []fs.FS) []fejkdata.Option {
if in.noShipped {
modules = nil
}
opts := []fejkdata.Option{fejkdata.WithDataFS(modules...)}
for _, dir := range in.dirs {
opts = append(opts, fejkdata.WithDataPath(dir))
}
if in.seeded {
opts = append(opts, fejkdata.WithSeed(in.seed))
}
return opts
}
// write streams the input's renders to w, repeat of them joined by the separator; a write
// failure surfaces from Flush, bufio keeping the first one.
func (in invocation) write(f *fejkdata.Generator, src input, w io.Writer) error {
if in.writesRecords() {
return in.writeRecords(f, src, w)
}
t, err := f.NewTemplate(src.template)
if err != nil {
return templateError(err)
}
out := bufio.NewWriter(w)
for i := 0; i < in.repeat; i++ {
if i > 0 {
out.WriteString(in.separator)
}
out.WriteString(t.Fake())
}
return out.Flush()
}
// recordStream builds the record drawer for the input, plus the INSERT table
// a sql format names.
func (in invocation) recordStream(f *fejkdata.Generator, src input) (func() (*fejkdata.Record, error), string, error) {
record := func() (*fejkdata.Record, error) { return f.FakeRecord(src.record) }
if src.record == "" {
t, err := f.NewRecordTemplate(src.template)
if err != nil {
return nil, "", templateError(spacedReference(src.template, err))
}
record = func() (*fejkdata.Record, error) { return t.Fake(), nil }
}
table := in.table
if table == "" {
table = defaultTable(src.record)
}
return record, table, nil
}
// writeRecords streams a record per line in the chosen format, framing a document
// form with its open/close brackets and a header preceding the first record.
func (in invocation) writeRecords(f *fejkdata.Generator, src input, w io.Writer) error {
record, table, err := in.recordStream(f, src)
if err != nil {
return err
}
format := recordFormats[in.format]
if in.repeat == 0 {
return writeNoRecord(record, format, w)
}
out := bufio.NewWriter(w)
if format.open != "" {
out.WriteString(format.open)
out.WriteByte('\n')
}
for i := 0; i < in.repeat; i++ {
r, err := record()
if err != nil {
return err
}
if i == 0 && format.header != nil {
out.WriteString(format.header(r))
out.WriteByte('\n')
}
if i > 0 {
out.WriteString(format.sep)
}
out.WriteString(format.line(r, table))
}
if format.close != "" {
out.WriteByte('\n')
out.WriteString(format.close)
}
out.WriteByte('\n')
return out.Flush()
}
// writeNoRecord writes zero records: the header where the format writes one, an empty document
// where it frames one, else nothing. It draws one record, which names the header's columns, so a
// lone reference to nothing fails as it would with records.
func writeNoRecord(record func() (*fejkdata.Record, error), format recordFormat, w io.Writer) error {
r, err := record()
switch {
case err != nil:
return err
case format.header != nil:
_, err = io.WriteString(w, format.header(r)+"\n")
case format.open != "":
_, err = io.WriteString(w, format.open+format.close+"\n")
}
return err
}
// defaultTable names the INSERT target when --table is absent: the path's last
// segment, or "records" when stdin is not a lone reference.
func defaultTable(path string) string {
if path == "" {
return "records"
}
segments := strings.Split(withoutSelectors(path), ".")
return segments[len(segments)-1]
}
// withoutSelectors is path with its [selectors] cut out.
func withoutSelectors(path string) string {
var names strings.Builder
for depth, i := 0, 0; i < len(path); i++ {
switch {
case path[i] == '[':
depth++
case path[i] == ']':
depth--
case depth == 0:
names.WriteByte(path[i])
}
}
return names.String()
}
// inputError marks a failure that is the input's own fault — nothing piped in, a template
// that does not compile, or a stdin that cannot be read. run reports it as misuse (exit 2, with a
// pointer to --help), unlike a lone reference to nothing under --format, which is a runtime
// error (exit 1).
type inputError struct{ error }
func (e inputError) Unwrap() error { return e.error }
// jsonSpace is the whitespace JSON allows around a value.
const jsonSpace = " \t\r\n"
// input is what stdin holds: the template, and the path a lone reference names as a
// record.
type input struct {
template string
record string
}
// parseInput keeps stdin whole as the template, and names a record where stdin is a lone
// reference once JSON's surrounding whitespace, or one final newline, is cut.
func parseInput(raw string) input {
trimmed := strings.Trim(raw, jsonSpace)
text := raw
if json.Valid([]byte(trimmed)) {
text = trimmed
}
text, found := strings.CutSuffix(text, "\n")
if found {
text = strings.TrimSuffix(text, "\r")
}
record, _ := loneReference(text)
return input{template: raw, record: record}
}
// loneReference returns the path that text reads when it is one reference from the root
// and nothing else, bare or as a JSON string.
func loneReference(text string) (string, bool) {
var unquoted string
if strings.HasPrefix(text, `"`) && strings.HasSuffix(text, `"`) && json.Unmarshal([]byte(text), &unquoted) == nil {
text = unquoted
}
path, opens := strings.CutPrefix(text, "{/")
path, closes := strings.CutSuffix(path, "}")
if !opens || !closes || !isPath(path) {
return "", false
}
return path, true
}
// isPath reports whether a reference's body after its / is a path alone: no second /, no
// folder sigil, no empty segment, and none of the |, ( or " as " a token reads outside a selector.
// It copies the grammar, since the CLI reads only the library's public API.
func isPath(body string) bool {
if body == "" || strings.ContainsAny(body[:1], "/.") {
return false
}
names := withoutSelectors(body)
if strings.ContainsAny(names, "|(") || strings.Contains(names, " as ") || hasEmptySegment(names) {
return false
}
inline, err := fejkdata.IsTemplate(body)
return !inline && err == nil
}
// hasEmptySegment reports whether a dotted path holds a segment with no name: two dots are a
// step up, and only a step up may end the path.
func hasEmptySegment(path string) bool {
segments := strings.Split(path, ".")
stepUp := false
for i, seg := range segments[1:] {
last := i == len(segments)-2
switch {
case seg != "":
stepUp = false
case stepUp != last:
return true
default:
stepUp = !stepUp
}
}
return false
}
// spacedReference adds to a record refusal that whitespace around a lone reference made
// the template text.
func spacedReference(template string, err error) error {
if trimmed := strings.TrimSpace(template); trimmed != template && errors.Is(err, fejkdata.ErrNoColumns) {
if path, lone := loneReference(trimmed); lone {
return fmt.Errorf("%w; the whitespace around {/%s} makes it text, so remove it to write the record %s", err, path, path)
}
}
return err
}
// piped is the command piping template to fejkdata; some shells' echo reads a backslash
// as an escape, so a template holding one goes through printf.
func piped(template string) string {
if strings.ContainsRune(template, '\\') {
return "printf '%s' " + shellQuoted(template) + " | fejkdata"
}
return "echo " + shellQuoted(template) + " | fejkdata"
}
// shellQuoted is s as a shell reads it back: bare where it holds nothing a shell expands.
func shellQuoted(s string) string {
if s != "" && !strings.ContainsAny(s, " \t\r\n'\"$`\\*?[]{}()<>|&;#~!") {
return s
}
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
}
// Main runs the command on the process's arguments and standard streams over modules, each
// passed to [fejkdata.WithDataFS], and returns the exit code: 0 ok, 1 runtime error,
// 2 misuse. --no-shipped-data leaves modules out.
func Main(modules ...fs.FS) int {
var stdin io.Reader = os.Stdin
if fi, err := os.Stdin.Stat(); err == nil && fi.Mode()&os.ModeCharDevice != 0 {
stdin = nil
}
return run(modules, os.Args[1:], stdin, os.Stdout, os.Stderr)
}
// run is Main over the given streams. A nil stdin is a terminal, which holds no template.
func run(modules []fs.FS, args []string, stdin io.Reader, stdout, stderr io.Writer) int {
in, err := parseArgs(args)
if err != nil {
return misuse(stderr, err)
}
if in.help {
io.WriteString(stdout, usage)
return 0
}
if in.version {
fmt.Fprintln(stdout, "fejkdata "+buildVersion())
return 0
}
if err := in.check(); err != nil {
return misuse(stderr, err)
}
var src input
if !in.list {
if src, err = readStdin(stdin); err != nil {
return fail(stderr, err)
}
}
f, err := fejkdata.New(in.options(modules)...)
if err != nil {
fmt.Fprintln(stderr, err)
return 1
}
if in.list {
for _, p := range f.List() {
fmt.Fprintln(stdout, p)
}
return 0
}
if err := in.write(f, src, stdout); err != nil {
return fail(stderr, err)
}
return 0
}
func readStdin(stdin io.Reader) (input, error) {
if stdin == nil {
return input{}, inputError{fmt.Errorf("the template comes from stdin, and nothing is piped in: %s", pipeHint)}
}
raw, err := io.ReadAll(stdin)
if err != nil {
return input{}, inputError{fmt.Errorf("stdin cannot be read: %w", err)}
}
return parseInput(string(raw)), nil
}
// templateError is misuse unless the data the template reads failed to load.
func templateError(err error) error {
if errors.Is(err, fejkdata.ErrLoad) {
return err
}
return inputError{err}
}
func fail(stderr io.Writer, err error) int {
var te inputError
if errors.As(err, &te) {
return misuse(stderr, te.error)
}
fmt.Fprintln(stderr, err)
if errors.Is(err, fejkdata.ErrNoColumns) {
fmt.Fprintln(stderr, "wrap that JSON in single quotes, which keep a shell from expanding its braces")
}
return 1
}
func misuse(stderr io.Writer, err error) int {
// A library error already names the program, so the prefix is not doubled.
fmt.Fprintf(stderr, "fejkdata: %s\ntry 'fejkdata --help'\n", strings.TrimPrefix(err.Error(), "fejkdata: "))
return 2
}
// buildVersion is the version of the fejkdata module go install stamps into the binary,
// whichever module's main package carries the CLI, or "devel".
func buildVersion() string {
info, _ := debug.ReadBuildInfo()
return moduleVersion(info)
}
func moduleVersion(info *debug.BuildInfo) string {
if info != nil {
for _, m := range append([]*debug.Module{&info.Main}, info.Deps...) {
if m.Path == "github.com/larvit/fejkdata" && m.Version != "" && m.Version != "(devel)" {
return m.Version
}
}
}
return "devel"
}
cmd/fejkdata/main.go
+4 -651
View File
@@ -1,659 +1,12 @@
// Command fejkdata renders the template on its stdin with fake values from the shipped // Command fejkdata renders the template on its stdin with fake values from every shipped
// data and any directories layered over it. // module; package cli documents it.
//
// echo '{/sv_SE.person}' | fejkdata # a full person
// echo 'name: {/sv_SE.person.last}' | fejkdata # text around a surname
// echo '{/sv_SE.person}' | fejkdata --data-path ./mydata # layer custom data; the last dir wins
// echo '{/sv_SE.address}' | fejkdata --seed 42
package main package main
import ( import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"os" "os"
"runtime/debug"
"sort"
"strconv"
"strings"
"github.com/larvit/fejkdata" "github.com/larvit/fejkdata/cli"
"github.com/larvit/fejkdata/data" "github.com/larvit/fejkdata/data"
) )
const usage = `Usage: <template> | fejkdata [flags] func main() { os.Exit(cli.Main(data.Modules()...)) }
fejkdata --list | --help | --version
fejkdata reads one template from stdin and renders it:
echo 'name: {/sv_SE.person.last}' | fejkdata
echo '{"format":"{x}","x":["bosse","lina"]}' | fejkdata
fejkdata < template.txt
fejkdata <<'EOF'
born {date(1990-01-01,2010-12-31,'2006-01-02')}
EOF
A template is a format string, or a JSON object, array, string or null. Its {…} tokens
reach the data by reference from the root, {/sv_SE.person.last}, whether the data
is shipped or layered with --data-path. A reference names a category or a dotted
path into one ({/sv_SE.person.last}), a table's row by key or name
({/misc.territory[SE]}), and .. steps up from a row to the row it links to
({/geo.SE.locality..municipality.name}).
Under --format text, the default, what you send is what renders, and fejkdata adds
nothing: a format string keeps every byte, so echo's newline ends each render, and
printf '%s' sends none. -n joins renders with --separator, empty by default. A JSON
object, array or string drops the whitespace around it, and null alone renders nothing;
42 or true alone is a format string, printed as written. An empty or blank stdin is a
format string like any other, and renders as sent. A quoted heredoc, <<'EOF', passes $,
a backslash or a quote as written.
With --format json, ndjson, csv or sql the template must be a record — a JSON
template whose fields are its columns, or a lone reference such as {/sv_SE.person}:
one reference and nothing else, with one newline after it allowed — and the rows are
written as one JSON array, one JSON object per line, one CSV row (after a header), or
one INSERT.
-d, --data-path D a data directory to layer over the shipped data (repeatable; last wins on a clash)
--format F output form: text (default), json, ndjson, csv or sql
-h, --help print this help, then exit
--list list the paths the data offers, each written after {/ in a template, then exit;
--repeat, --separator, --format and --table do nothing here
--no-shipped-data load only the --data-path directories
-n, --repeat N render the template N times, 0..1048576 (default 1); 0 prints nothing,
[] under --format json, or the header under --format csv
-s, --seed N same seed, version and data: identical output
--separator S string between repeated renders (default empty); refused with a record
--format, which has its own separators
--table T the INSERT target for --format sql, ignored under any other
(default: a lone reference's last segment, else records)
--version print the version, then exit
A short flag's value attaches or follows (-n3, -n 3); short flags bundle (-hn 3);
-- ends the flags.
`
type invocation struct {
args []string
dirs []string
format string
help bool
list bool
noShipped bool
repeat int
seed uint64
seeded bool
separator string
separatorSet bool
table string
tableSet bool
version bool
}
type flagDef struct {
long string
short string
value bool
set func(*invocation, string) error
}
var flagDefs = []flagDef{
{"data-path", "d", true, func(in *invocation, v string) error { in.dirs = append(in.dirs, v); return nil }},
{"format", "", true, func(in *invocation, v string) error { in.format = v; return nil }},
{"help", "h", false, func(in *invocation, _ string) error { in.help = true; return nil }},
{"list", "", false, func(in *invocation, _ string) error { in.list = true; return nil }},
{"no-shipped-data", "", false, func(in *invocation, _ string) error { in.noShipped = true; return nil }},
{"repeat", "n", true, func(in *invocation, v string) error {
n, err := strconv.Atoi(v)
if err != nil || n < 0 || n > fejkdata.MaxRepeat {
return fmt.Errorf("--repeat needs an integer in 0..%d, got %q", fejkdata.MaxRepeat, v)
}
in.repeat = n
return nil
}},
{"seed", "s", true, func(in *invocation, v string) error {
n, err := strconv.ParseUint(v, 10, 64)
if err != nil {
return fmt.Errorf("--seed needs an unsigned integer, got %q", v)
}
in.seed, in.seeded = n, true
return nil
}},
{"separator", "", true, func(in *invocation, v string) error { in.separator, in.separatorSet = v, true; return nil }},
{"table", "", true, func(in *invocation, v string) error { in.table, in.tableSet = v, true; return nil }},
{"version", "", false, func(in *invocation, _ string) error { in.version = true; return nil }},
}
func flagByLong(name string) *flagDef {
for i := range flagDefs {
if flagDefs[i].long == name {
return &flagDefs[i]
}
}
return nil
}
func flagByShort(name string) *flagDef {
for i := range flagDefs {
if flagDefs[i].short == name {
return &flagDefs[i]
}
}
return nil
}
// flagArg is one flag as written: its definition and the value attached to it,
// if any.
type flagArg struct {
def *flagDef
value string
inline bool
}
// splitFlags reads one argv element as flags: --name or --name=value, or -abc
// where each letter is a short flag and the first that takes a value takes the
// rest of the element.
func splitFlags(arg string) ([]flagArg, error) {
if strings.HasPrefix(arg, "--") {
name, value, inline := strings.Cut(arg[2:], "=")
def := flagByLong(name)
if def == nil {
return nil, fmt.Errorf("unknown flag --%s", name)
}
return []flagArg{{def, value, inline}}, nil
}
if name, _, _ := strings.Cut(arg[1:], "="); len(name) > 1 && flagByLong(name) != nil {
return nil, fmt.Errorf("unknown flag %s; use --%s", arg, name)
}
var flags []flagArg
letters := arg[1:]
for i, r := range letters {
letter := string(r)
def := flagByShort(letter)
if def == nil {
return nil, fmt.Errorf("unknown flag -%s", letter)
}
if !def.value {
flags = append(flags, flagArg{def: def})
continue
}
rest := letters[i+len(letter):]
if strings.HasPrefix(rest, "=") {
return nil, fmt.Errorf("-%s takes its value attached (-%s%s) or next (-%s %s); = belongs to --%s=%s",
letter, letter, rest[1:], letter, rest[1:], def.long, rest[1:])
}
return append(flags, flagArg{def, rest, rest != ""}), nil
}
return flags, nil
}
func parseArgs(argv []string) (invocation, error) {
in := invocation{repeat: 1, format: "text"}
for i := 0; i < len(argv); i++ {
arg := argv[i]
if arg == "--" {
in.args = append(in.args, argv[i+1:]...)
return in, nil
}
if len(arg) < 2 || arg[0] != '-' {
in.args = append(in.args, arg)
continue
}
flags, err := splitFlags(arg)
if err != nil {
return in, err
}
for _, fl := range flags {
if !fl.def.value {
if fl.inline {
return in, fmt.Errorf("--%s takes no value", fl.def.long)
}
_ = fl.def.set(&in, "")
continue
}
value := fl.value
if !fl.inline {
if i++; i >= len(argv) {
return in, fmt.Errorf("--%s needs a value", fl.def.long)
}
value = argv[i]
}
if err := fl.def.set(&in, value); err != nil {
return in, err
}
}
}
return in, nil
}
// recordFormat is one way to write a record out: the line each record renders,
// the header that precedes the first one, and the open/close frame plus the
// between-record separator a document form needs.
type recordFormat struct {
header func(*fejkdata.Record) string
line func(r *fejkdata.Record, table string) string
open string
close string
sep string
writes string // what it writes between values, which --separator would seem to set
}
// recordFormats is every --format that writes records. json frames the records
// as one array document; ndjson writes one object per line.
var recordFormats = map[string]recordFormat{
"csv": {header: (*fejkdata.Record).CSVHeader, line: func(r *fejkdata.Record, _ string) string { return r.CSVLine() }, sep: "\n", writes: `always separates fields with ","`},
"json": {line: jsonLine, open: "[", close: "]", sep: ",\n", writes: "writes the records as one JSON array"},
"ndjson": {line: jsonLine, sep: "\n", writes: "writes one record per line"},
"sql": {line: func(r *fejkdata.Record, table string) string { return r.SQLInsert(table) }, sep: "\n", writes: "writes one INSERT per line"},
}
func jsonLine(r *fejkdata.Record, _ string) string { return r.JSON() }
func (in invocation) writesRecords() bool {
return in.format != "text"
}
// formatNames lists the --format values, text included, for the misuse error.
func formatNames() string {
names := []string{"text"}
for name := range recordFormats {
names = append(names, name)
}
sort.Strings(names)
return strings.Join(names[:len(names)-1], ", ") + " or " + names[len(names)-1]
}
// checkFlags rejects a flag value or combination that cannot run.
func (in invocation) checkFlags() error {
if _, ok := recordFormats[in.format]; !ok && in.format != "text" {
return fmt.Errorf("--format takes %s, got %q", formatNames(), in.format)
}
if in.tableSet && in.table == "" && in.format == "sql" {
return errors.New("--table names the INSERT target, so it cannot be empty")
}
if f, records := recordFormats[in.format]; records && in.separatorSet && !in.list {
return fmt.Errorf("--separator joins text renders; --format %s %s", in.format, f.writes)
}
return nil
}
const pipeHint = "echo '{/sv_SE.person}' | fejkdata"
// operandError refuses an argument that is no flag, naming the spelling that reads what
// it most likely means: dropping it for -, a file for a file's name, else a template
// piped in.
func operandError(arg string) error {
unexpected := "unexpected argument '" + arg + "'"
if arg == "-" {
return fmt.Errorf("%s; stdin is read already, so drop it", unexpected)
}
if fi, err := os.Stat(arg); err == nil && fi.Mode().IsRegular() {
return fmt.Errorf("%s; read a file from stdin: fejkdata < %s", unexpected, shellQuoted(arg))
}
ref := "{/" + strings.TrimPrefix(arg, "/") + "}"
_, argLone := loneReference(arg)
inline, err := fejkdata.IsTemplate(arg)
_, refLone := loneReference(ref)
example := pipeHint
switch {
case argLone || inline:
example = piped(arg)
case err == nil && refLone:
example = piped(ref)
}
return fmt.Errorf("%s; the template is read from stdin: %s", unexpected, example)
}
// check rejects a flag combination or an operand that cannot run, before any data is
// read.
func (in invocation) check() error {
if err := in.checkFlags(); err != nil {
return err
}
if len(in.args) > 0 {
return operandError(in.args[0])
}
return nil
}
func (in invocation) options() []fejkdata.Option {
var modules []fs.FS
if !in.noShipped {
modules = data.Modules()
}
opts := []fejkdata.Option{fejkdata.WithDataFS(modules...)}
for _, dir := range in.dirs {
opts = append(opts, fejkdata.WithDataPath(dir))
}
if in.seeded {
opts = append(opts, fejkdata.WithSeed(in.seed))
}
return opts
}
// write streams the input's renders to w, repeat of them joined by the separator; a write
// failure surfaces from Flush, bufio keeping the first one.
func (in invocation) write(f *fejkdata.Generator, src input, w io.Writer) error {
if in.writesRecords() {
return in.writeRecords(f, src, w)
}
t, err := f.NewTemplate(src.template)
if err != nil {
return templateError(err)
}
out := bufio.NewWriter(w)
for i := 0; i < in.repeat; i++ {
if i > 0 {
out.WriteString(in.separator)
}
out.WriteString(t.Fake())
}
return out.Flush()
}
// recordStream builds the record drawer for the input, plus the INSERT table
// a sql format names.
func (in invocation) recordStream(f *fejkdata.Generator, src input) (func() (*fejkdata.Record, error), string, error) {
record := func() (*fejkdata.Record, error) { return f.FakeRecord(src.record) }
if src.record == "" {
t, err := f.NewRecordTemplate(src.template)
if err != nil {
return nil, "", templateError(spacedReference(src.template, err))
}
record = func() (*fejkdata.Record, error) { return t.Fake(), nil }
}
table := in.table
if table == "" {
table = defaultTable(src.record)
}
return record, table, nil
}
// writeRecords streams a record per line in the chosen format, framing a document
// form with its open/close brackets and a header preceding the first record.
func (in invocation) writeRecords(f *fejkdata.Generator, src input, w io.Writer) error {
record, table, err := in.recordStream(f, src)
if err != nil {
return err
}
format := recordFormats[in.format]
if in.repeat == 0 {
return writeNoRecord(record, format, w)
}
out := bufio.NewWriter(w)
if format.open != "" {
out.WriteString(format.open)
out.WriteByte('\n')
}
for i := 0; i < in.repeat; i++ {
r, err := record()
if err != nil {
return err
}
if i == 0 && format.header != nil {
out.WriteString(format.header(r))
out.WriteByte('\n')
}
if i > 0 {
out.WriteString(format.sep)
}
out.WriteString(format.line(r, table))
}
if format.close != "" {
out.WriteByte('\n')
out.WriteString(format.close)
}
out.WriteByte('\n')
return out.Flush()
}
// writeNoRecord writes zero records: the header where the format writes one, an empty document
// where it frames one, else nothing. It draws one record, which names the header's columns, so a
// lone reference to nothing fails as it would with records.
func writeNoRecord(record func() (*fejkdata.Record, error), format recordFormat, w io.Writer) error {
r, err := record()
switch {
case err != nil:
return err
case format.header != nil:
_, err = io.WriteString(w, format.header(r)+"\n")
case format.open != "":
_, err = io.WriteString(w, format.open+format.close+"\n")
}
return err
}
// defaultTable names the INSERT target when --table is absent: the path's last
// segment, or "records" when stdin is not a lone reference.
func defaultTable(path string) string {
if path == "" {
return "records"
}
segments := strings.Split(withoutSelectors(path), ".")
return segments[len(segments)-1]
}
// withoutSelectors is path with its [selectors] cut out.
func withoutSelectors(path string) string {
var names strings.Builder
for depth, i := 0, 0; i < len(path); i++ {
switch {
case path[i] == '[':
depth++
case path[i] == ']':
depth--
case depth == 0:
names.WriteByte(path[i])
}
}
return names.String()
}
// inputError marks a failure that is the input's own fault — nothing piped in, a template
// that does not compile, or a stdin that cannot be read. run reports it as misuse (exit 2, with a
// pointer to --help), unlike a lone reference to nothing under --format, which is a runtime
// error (exit 1).
type inputError struct{ error }
func (e inputError) Unwrap() error { return e.error }
// jsonSpace is the whitespace JSON allows around a value.
const jsonSpace = " \t\r\n"
// input is what stdin holds: the template, and the path a lone reference names as a
// record.
type input struct {
template string
record string
}
// parseInput keeps stdin whole as the template, and names a record where stdin is a lone
// reference once JSON's surrounding whitespace, or one final newline, is cut.
func parseInput(raw string) input {
trimmed := strings.Trim(raw, jsonSpace)
text := raw
if json.Valid([]byte(trimmed)) {
text = trimmed
}
text, found := strings.CutSuffix(text, "\n")
if found {
text = strings.TrimSuffix(text, "\r")
}
record, _ := loneReference(text)
return input{template: raw, record: record}
}
// loneReference returns the path that text reads when it is one reference from the root
// and nothing else, bare or as a JSON string.
func loneReference(text string) (string, bool) {
var unquoted string
if strings.HasPrefix(text, `"`) && strings.HasSuffix(text, `"`) && json.Unmarshal([]byte(text), &unquoted) == nil {
text = unquoted
}
path, opens := strings.CutPrefix(text, "{/")
path, closes := strings.CutSuffix(path, "}")
if !opens || !closes || !isPath(path) {
return "", false
}
return path, true
}
// isPath reports whether a reference's body after its / is a path alone: no second /, no
// folder sigil, no empty segment, and none of the |, ( or " as " a token reads outside a selector.
// It copies the grammar, since the CLI reads only the library's public API.
func isPath(body string) bool {
if body == "" || strings.ContainsAny(body[:1], "/.") {
return false
}
names := withoutSelectors(body)
if strings.ContainsAny(names, "|(") || strings.Contains(names, " as ") || hasEmptySegment(names) {
return false
}
inline, err := fejkdata.IsTemplate(body)
return !inline && err == nil
}
// hasEmptySegment reports whether a dotted path holds a segment with no name: two dots are a
// step up, and only a step up may end the path.
func hasEmptySegment(path string) bool {
segments := strings.Split(path, ".")
stepUp := false
for i, seg := range segments[1:] {
last := i == len(segments)-2
switch {
case seg != "":
stepUp = false
case stepUp != last:
return true
default:
stepUp = !stepUp
}
}
return false
}
// spacedReference adds to a record refusal that whitespace around a lone reference made
// the template text.
func spacedReference(template string, err error) error {
if trimmed := strings.TrimSpace(template); trimmed != template && errors.Is(err, fejkdata.ErrNoColumns) {
if path, lone := loneReference(trimmed); lone {
return fmt.Errorf("%w; the whitespace around {/%s} makes it text, so remove it to write the record %s", err, path, path)
}
}
return err
}
// piped is the command piping template to fejkdata; some shells' echo reads a backslash
// as an escape, so a template holding one goes through printf.
func piped(template string) string {
if strings.ContainsRune(template, '\\') {
return "printf '%s' " + shellQuoted(template) + " | fejkdata"
}
return "echo " + shellQuoted(template) + " | fejkdata"
}
// shellQuoted is s as a shell reads it back: bare where it holds nothing a shell expands.
func shellQuoted(s string) string {
if s != "" && !strings.ContainsAny(s, " \t\r\n'\"$`\\*?[]{}()<>|&;#~!") {
return s
}
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
}
func main() {
var stdin io.Reader = os.Stdin
if fi, err := os.Stdin.Stat(); err == nil && fi.Mode()&os.ModeCharDevice != 0 {
stdin = nil
}
os.Exit(run(os.Args[1:], stdin, os.Stdout, os.Stderr))
}
// run returns the exit code: 0 ok, 1 runtime error, 2 misuse. A nil stdin is a
// terminal, which holds no template.
func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
in, err := parseArgs(args)
if err != nil {
return misuse(stderr, err)
}
if in.help {
io.WriteString(stdout, usage)
return 0
}
if in.version {
fmt.Fprintln(stdout, "fejkdata "+buildVersion())
return 0
}
if err := in.check(); err != nil {
return misuse(stderr, err)
}
var src input
if !in.list {
if src, err = readStdin(stdin); err != nil {
return fail(stderr, err)
}
}
f, err := fejkdata.New(in.options()...)
if err != nil {
fmt.Fprintln(stderr, err)
return 1
}
if in.list {
for _, p := range f.List() {
fmt.Fprintln(stdout, p)
}
return 0
}
if err := in.write(f, src, stdout); err != nil {
return fail(stderr, err)
}
return 0
}
func readStdin(stdin io.Reader) (input, error) {
if stdin == nil {
return input{}, inputError{fmt.Errorf("the template comes from stdin, and nothing is piped in: %s", pipeHint)}
}
raw, err := io.ReadAll(stdin)
if err != nil {
return input{}, inputError{fmt.Errorf("stdin cannot be read: %w", err)}
}
return parseInput(string(raw)), nil
}
// templateError is misuse unless the data the template reads failed to load.
func templateError(err error) error {
if errors.Is(err, fejkdata.ErrLoad) {
return err
}
return inputError{err}
}
func fail(stderr io.Writer, err error) int {
var te inputError
if errors.As(err, &te) {
return misuse(stderr, te.error)
}
fmt.Fprintln(stderr, err)
if errors.Is(err, fejkdata.ErrNoColumns) {
fmt.Fprintln(stderr, "wrap that JSON in single quotes, which keep a shell from expanding its braces")
}
return 1
}
func misuse(stderr io.Writer, err error) int {
// A library error already names the program, so the prefix is not doubled.
fmt.Fprintf(stderr, "fejkdata: %s\ntry 'fejkdata --help'\n", strings.TrimPrefix(err.Error(), "fejkdata: "))
return 2
}
// buildVersion is the module version go install stamps into the binary, or "devel".
func buildVersion() string {
if info, ok := debug.ReadBuildInfo(); ok && info.Main.Version != "" {
return info.Main.Version
}
return "devel"
}
docs/decisions.md
+5 -4
View File
@@ -96,9 +96,10 @@ template per run.
## The CLI reads only the library's public API ## The CLI reads only the library's public API
2026-10-07, Lilleman auf Larv. Serves goal 9.6: a CLI carrying your own modules is a few 2026-10-07, Lilleman auf Larv; revised 2026-10-11 to name package `cli`. Serves goal 9.6: a
lines of Go and works just like `fejkdata`, so `fejkdata`'s own CLI is such a CLI and CLI carrying your own modules is a few lines of Go over `cli.Main` and works just like
its non-test code imports nothing under `internal/`. Until `todo.md` item 129 moves it into `fejkdata`, so `fejkdata`'s own CLI is such a CLI, and `cli`'s non-test code imports nothing
under `internal/`. Until `todo.md` item 129 moves it into
the library, the CLI copies how the grammar tells a lone reference, in `isPath`, the library, the CLI copies how the grammar tells a lone reference, in `isPath`,
`hasEmptySegment` and `withoutSelectors`, and a test pins the copy to the grammar. `hasEmptySegment` and `withoutSelectors`, and a test pins the copy to the grammar.
@@ -138,7 +139,7 @@ A directory a machine happens to have would make `--seed 42` machine-dependent;
## The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0 ## The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0
2026-10-04, Lilleman auf Larv; revised 2026-10-09 to put the packages in the core's Go module. Serves goals 2.2, 6.2, 7.3, 9 and 10.1. Carried out by `todo.md` items 181, 184, 185, 186 and 196. The CLI binary may grow to hundreds of MB. 2026-10-04, Lilleman auf Larv; revised 2026-10-09 to put the packages in the core's Go module. Serves goals 2.2, 6.2, 7.3, 9 and 10.1. Carried out by `todo.md` items 181, 185, 186 and 196. The CLI binary may grow to hundreds of MB.
- One Go package per locale, one per country's `geo` tree, and one for `misc`, all in the core's Go module. A package registers nothing when imported. - One Go package per locale, one per country's `geo` tree, and one for `misc`, all in the core's Go module. A package registers nothing when imported.
- A bare `New()` loads no data and fails, naming the option to add. `WithoutShippedData` goes. - A bare `New()` loads no data and fails, naming the option to add. `WithoutShippedData` goes.
todo.md
+1 -9
View File
@@ -70,7 +70,6 @@
| 192 | 0.1.0 | decision | **Fail the gate when the shipped data passes 100 MiB, naming item 193.** | 1 | 2 | 6 | 8 | 2, 1.1 | 26.5 | | 192 | 0.1.0 | decision | **Fail the gate when the shipped data passes 100 MiB, naming item 193.** | 1 | 2 | 6 | 8 | 2, 1.1 | 26.5 |
| 210 | 0.1.0 | | **Add `fr_FR` with its `geo/FR` tree.** | 3 | 8 | 8 | 8 | 2 | 26.2 | | 210 | 0.1.0 | | **Add `fr_FR` with its `geo/FR` tree.** | 3 | 8 | 8 | 8 | 2 | 26.2 |
| 15 | 0.1.0 | | **Add `misc` tables and templates, one row of its detail's table per chunk.** | 3 | 9 | 8 | 8 | 2, 15, 14 | 25.9 | | 15 | 0.1.0 | | **Add `misc` tables and templates, one row of its detail's table per chunk.** | 3 | 9 | 8 | 8 | 2, 15, 14 | 25.9 |
| 184 | 0.1.0 | decision | **Make the CLI the library call `cli.Main(modules ...fs.FS) int`, with `fejkdata`'s own `main` one line over `data.Modules()`.** | 4 | 5 | 7 | 9 | 9.6, 2 | 25.8 |
| 201 | 0.1.0 | | **Place each Swedish street in the postort whose area holds it, so a big postort such as Stockholm carries its own streets.** | 4 | 6 | 8 | 8 | 2, 16 | 25.7 | | 201 | 0.1.0 | | **Place each Swedish street in the postort whose area holds it, so a big postort such as Stockholm carries its own streets.** | 4 | 6 | 8 | 8 | 2, 16 | 25.7 |
| 129 | 0.1.0 | decision | **Take a template wherever the library renders, so `fake:"{/sv_SE.person.first}"` and `Fake("{/sv_SE.person}")` render, and text with no `{…}` prints as written, as the CLI already does.** | 5 | 8 | 8 | 10 | 5.9, 7 | 25.0 | | 129 | 0.1.0 | decision | **Take a template wherever the library renders, so `fake:"{/sv_SE.person.first}"` and `Fake("{/sv_SE.person}")` render, and text with no `{…}` prints as written, as the CLI already does.** | 5 | 8 | 8 | 10 | 5.9, 7 | 25.0 |
| 163 | 0.1.0 | | **Lift the CLI's `--repeat` cap, since the CLI streams each render, and give the builtins' count and decimal caps a technical reason in the README.** | 2 | 2 | 7 | 7 | 2, 5.7 | 24.8 | | 163 | 0.1.0 | | **Lift the CLI's `--repeat` cap, since the CLI streams each render, and give the builtins' count and decimal caps a technical reason in the README.** | 2 | 2 | 7 | 7 | 2, 5.7 | 24.8 |
@@ -191,7 +190,7 @@
### 185. Seat a panel of the README's audience personas to pull data in through the module API and build a custom CLI, and fix what it names. ### 185. Seat a panel of the README's audience personas to pull data in through the module API and build a custom CLI, and fix what it names.
Needs item 184; 0.1.0 is not cut before it. Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Seats: the Go test author, the bulk fixture author, a custom CLI's builder, and a data author with a `-d` folder and the replace flag, each in a scratch Go module against the checkout or a pre-release tag. Bench `data.Modules()` and the `.fejkdata.json` manifest a data author writes by hand in use; the package names, the manifest's `reads` and `namespace` and the errors naming the import set were benched on 2026-10-11. Decoding the manifest is most of `New`'s cost (about 0.55 ms and 2449 allocations on 2026-10-10, 2864 allocations once each module carried its own manifest). An entry lists every path below its category, its linked tables' paths too, so `geo.*.region` repeats its whole family. An entry holding only its own paths, with `List` composing a family from the entries naming it as `parent`, would shrink it. Record the ruling in `docs/`. 0.1.0 is not cut before it. Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0". Seats: the Go test author, the bulk fixture author, a custom CLI's builder, and a data author with a `-d` folder and the replace flag, each in a scratch Go module against the checkout or a pre-release tag. Bench `data.Modules()` and the `.fejkdata.json` manifest a data author writes by hand in use; the package names, the manifest's `reads` and `namespace` and the errors naming the import set were benched on 2026-10-11. Decoding the manifest is most of `New`'s cost (about 0.55 ms and 2449 allocations on 2026-10-10, 2864 allocations once each module carried its own manifest). An entry lists every path below its category, its linked tables' paths too, so `geo.*.region` repeats its whole family. An entry holding only its own paths, with `List` composing a family from the entries naming it as `parent`, would shrink it. Record the ruling in `docs/`.
### 16. Add locale categories: company, phone, finance, vehicle, words. ### 16. Add locale categories: company, phone, finance, vehicle, words.
@@ -258,13 +257,6 @@ Shape: T = table, t = template, c = choice.
| `lorem`, `hacker`, `hipster`, `catchphrase`, `buzzword`, `quote` | T/c | lorem ipsum, LLM-written | — | | `lorem`, `hacker`, `hipster`, `catchphrase`, `buzzword`, `quote` | T/c | lorem ipsum, LLM-written | — |
| `direction`, `continent`, `ulid` | c/t | — | — | | `direction`, `continent`, `ulid` | c/t | — | — |
### 184. Make the CLI the library call `cli.Main(modules ...fs.FS) int`, with `fejkdata`'s own `main` one line over `data.Modules()`.
Carries out the decision "The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0".
- `cmd/fejkdata/main.go` moves into package `github.com/larvit/fejkdata/cli` in the core module, and `imports_test.go` keeps `cli` off `internal/`. `cli`'s tests use fixture modules; `cmd/fejkdata` keeps one end-to-end test over the real packages.
- The README's Library quick start shows the whole import block for one locale, and a section shows a custom CLI as example code. Layout and Development follow.
- Revises the decision "The CLI reads only the library's public API" to name `cli`.
### 201. Place each Swedish street in the postort whose area holds it, so a big postort such as Stockholm carries its own streets. ### 201. Place each Swedish street in the postort whose area holds it, so a big postort such as Stockholm carries its own streets.
GeoNames gives Stockholm's 1,139 postal codes 2 points and Göteborg's 758 one, and `streets()` in `geo-se.py` counts each road segment toward the nearest postort's point. Every street then ships at most 363 names under Stockholm, the rest going to Bromma, Hägersten and other neighbours. Find an open source of areas first, such as SCB's tätort polygons (CC0), and how a tätort maps to a postort; record the source in `DATA-LICENSES.md` and the method in the decision on the default embed. The README's warning that a street may sit under a neighbouring postort is revised to match. GeoNames gives Stockholm's 1,139 postal codes 2 points and Göteborg's 758 one, and `streets()` in `geo-se.py` counts each road segment toward the nearest postort's point. Every street then ships at most 363 names under Stockholm, the rest going to Bromma, Hägersten and other neighbours. Find an open source of areas first, such as SCB's tätort polygons (CC0), and how a tätort maps to a postort; record the source in `DATA-LICENSES.md` and the method in the decision on the default embed. The README's warning that a street may sit under a neighbouring postort is revised to match.