.github/workflows/test.yml

@@ -26,11 +26,19 @@ jobs:
           else
             base='${{ github.event.before }}'
           fi
-          if git diff --quiet "$base"...HEAD -- '*.go' go.mod go.sum Dockerfile .dockerignore .github data README.md; then
+          if git diff --quiet "$base"...HEAD -- '*.go' go.mod go.sum Dockerfile .dockerignore .github data testdata README.md; then
             echo "code=false" >> "$GITHUB_OUTPUT"
           else
             echo "code=true" >> "$GITHUB_OUTPUT"
           fi
+          echo "base=$base" >> "$GITHUB_OUTPUT"
+      - name: CHANGELOG entry for data changes
+        run: |
+          base='${{ steps.changes.outputs.base }}'
+          if ! git diff --quiet "$base"...HEAD -- data testdata/shipped_shape.txt && git diff --quiet "$base"...HEAD -- CHANGELOG.md; then
+            echo "::error::the shipped data changed without a CHANGELOG.md entry (see the README's Versioning)"
+            exit 1
+          fi
       # `docker build` streams the context to the daemon. A compose bind-mount of
       # `.` mounts an empty host dir instead, because the job is itself a container.
       - name: Latest supported Go
@@ -41,3 +49,21 @@ jobs:
       - name: Lowest supported Go
         if: ${{ !cancelled() && steps.changes.outputs.code == 'true' }}
         run: docker build --build-arg GO_VERSION=1.22.12 .
+
+  release:
+    name: Gitea release from CHANGELOG.md
+    needs: test
+    if: github.event_name == 'push'
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+      - run: python3 release-tooling/publish_release.py
+        env:
+          GITEA_API_URL: ${{ github.api_url }}
+          GITEA_REPOSITORY: ${{ github.repository }}
+          GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          SHA: ${{ github.sha }}

AGENTS.md

@@ -1,7 +1,8 @@
 # Rules
 
-- Go runs only through `docker compose run --rm <test|vet|fmt|build>`; the merge gate is `docker build .`.
-- Tests first, in their own commit; the implementation follows in the next. A re-pin of seeded output is its own commit.
+- Go runs only through `docker compose run --rm <test|vet|fmt|build>`; the merge gate is `docker build .` plus CI's changelog check.
+- Tests first, in their own commit; the implementation follows in the next. A re-pin of seeded output or of `testdata/shipped_shape.txt` is its own commit.
+- A change under `data/` or to the shape pin adds its `CHANGELOG.md` entry under `Unreleased` in the same PR, and so does a change to a flag, an exit code, an exported name, a fence, a builtin or the lowest Go; what is major is the README's Versioning table.
 - One-line commit messages: no ticket prefix, no repo name, no authorship trailers.
 - Hard tabs. No comment by default; delete a restatement, a rationale, history, or a file preamble.
 - One spelling per result: reject the other at `New`, and let the error name the spelling to use.

CHANGELOG.md

@@ -0,0 +1,11 @@
+# Changelog
+
+Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). A release's
+`Breaking` section comes first and names each rejected spelling with its
+replacement, and each removed path, column or flag.
+
+## [Unreleased]
+
+### Added
+
+- First release: the CLI, the library and the shipped data set.

README.md

@@ -529,6 +529,41 @@ then costs about what its output costs: an unweighted pick is O(1) whatever the
 list's length, a weighted one O(log n), and long formats, deep nesting and many
 tokens add cost in proportion to the output.
 
+## Versioning
+
+Semver tags on `main`, `v0.1.0` first; one version covers the shipped data, the
+library and the CLI, and [`CHANGELOG.md`](CHANGELOG.md) names what each release
+changed. A consumer's data, code and scripts keep working across a minor or a patch:
+a minor only adds, and a major is the only release that changes what exists.
+
+| Surface | Major | Minor |
+|---------|-------|-------|
+| Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another | a path outside a record's columns, a locale, a value in a list |
+| Records | remove, rename, retype or add a column; let a column be null | a record, as a new category |
+| Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin |
+| CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format |
+| Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option |
+
+A patch changes no row of this table: performance, docs, or a fix inside a promised
+behaviour that changes no value, path, format or spelling.
+
+Seeded output is a promise within one version: same seed, same version, same
+data, same output. Any release may shift a stream, since a value added to a list
+moves every draw after it, so pin fixtures per version. An error's wording may
+improve in a minor; the path, rejected spelling and replacement it names may not.
+
+Before `v1.0.0` a minor is the breaking unit: `0.(x+1).0` may carry a major's
+changes, each named in the changelog, and a `0.x.y` patch may not. `v1.0.0` is
+cut once the shipped data is in its record shape and one full minor has shipped
+with no breaking change. From `v2` the module path carries `/vN`, so fences ship
+batched into as few majors as possible.
+
+[`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) pins every path, each
+template category's format, the categories each category reads, and each column's
+datatype and nullability; a pull request that
+changes it or `data/` adds its `CHANGELOG.md` entry, which CI checks. A removed,
+renamed or retyped line is a major.
+
 ## Goals
 
 1. **Valid by construction** — every value passes the check its real consumer
@@ -613,14 +648,31 @@ tokens add cost in proportion to the output.
   folder, `..` the folder above — what those spellings already mean to anyone who
   has typed a path. A locale's files reach each other without naming the locale,
   so a folder renames and copies without editing its references.
-- **After the first tag, a new fence is a major version.** Data files are the
-  public API, and one spelling per result grows by tightening, so every fence
-  invalidates some file. Each such release names the rejected spelling and its
-  replacement in the changelog and in the load error, and that is the whole
-  migration: a fence rejects one spelling with one replacement, so the fix is
-  local to each site. A fence that would need a non-local rewrite ships a
-  converter with its release instead. Before the first tag there is no
-  compatibility promise.
+- **A change to what exists is a major; a minor only adds.** Data files, the CLI
+  and the Go API are the public API, and a consumer must be able to take a minor
+  without an edit — so an added column is a major, since it changes the CSV header
+  and the `INSERT` column list, as is a removed value, which changes what a fixture
+  holds, and a new option, which reserves a field name. One spelling per result
+  grows by tightening, so every fence invalidates some file. Each such release
+  names the rejected spelling and its replacement in the changelog and in the load
+  error, and that is the whole migration: a fence rejects one spelling with one
+  replacement, so the fix is local to each site. A fence that would need a
+  non-local rewrite ships a converter with its release instead. Before `v1.0.0` a
+  minor carries what a major would.
+- **Seeded output is promised within one version.** Any edit to a category shifts
+  its stream and everything drawn after it, so a promise across versions would
+  freeze every shipped list; a fixture is re-pinned on a bump, as this repo's own are.
+- **An error is a contract by what it names, not its bytes.** A script branches on
+  the exit code and reads the named path or spelling, so those hold; wording improves
+  in a minor.
+- **Raising the lowest supported Go is a major.** A consumer building on it breaks,
+  which is the one test every rule above applies; Go's convention of a minor is not
+  followed.
+- **The changelog heading is the one spelling of a release; CI cuts the tag.** A
+  tag pushed by hand is served by `go get` at once, so a tag whose commit lacks its
+  heading is burnt, not fixed. The heading on a gate-passed `main` commit is the
+  trigger instead: the tag can land only there, and the Gitea release the same job
+  publishes keeps one text as its body and is where prebuilt binaries will attach.
 - **A `--data-path` override rebinds every reference to the category it
   replaces.** References bind against the merged tree, so once shipped data uses
   `{.person}`, a consumer's `sv_SE/person.json` is what every shipped reference
@@ -770,8 +822,9 @@ docker compose run --rm --user "$(id -u):$(id -g)" tidy  # go mod tidy
 Every pull request runs `docker build .` against both the latest and the lowest
 supported Go, and must pass before it can be merged — unless it changes none of
 the files the build and its tests read, nor the workflow itself, in which case
-it's skipped (see [Decisions](#decisions)). That build is the whole gate — vet,
-complexity, format check and tests — so run it locally before pushing:
+it's skipped (see [Decisions](#decisions)). That build is the whole gate but the
+changelog check, which CI runs against the PR base — vet, complexity, format check
+and tests — so run it locally before pushing:
 
 ```sh
 docker build .                                  # latest
@@ -779,6 +832,18 @@ docker build --build-arg GO_VERSION=1.22.12 .   # lowest supported
 GO_VERSION=1.22.12 docker compose run --rm test # the same tests, without the image build
 ```
 
+A change to the shipped data re-pins [`testdata/shipped_shape.txt`](testdata/shipped_shape.txt)
+in its own commit:
+
+```sh
+REPIN=1 docker compose run --rm --user "$(id -u):$(id -g)" test
+```
+
+To release, head `CHANGELOG.md` with the version's section in place of `Unreleased`
+and merge: once `main` passes the gate, CI tags that commit `vX.Y.Z` and publishes
+the Gitea release with the section as its body. A top heading of `[Unreleased]`
+publishes nothing.
+
 ## Layout
 
 ```
@@ -801,6 +866,8 @@ value.go        the value proof: what a typed column or calc operand holds, chec
 data.go         data loading: fs.FS folders/files -> namespace tree, multi-source merge
 cmd/fejkdata/   the fejkdata CLI
 data/           shipped data (JSON), embedded at build: locale folders + a misc folder
+release-tooling/ the release CI publishes from the changelog heading
+testdata/       the pinned shipped shape (see Versioning)
 ```
 
 ## License

compose.yaml

@@ -24,6 +24,7 @@ x-go: &go
     HOME: /cache
     GOCACHE: /cache/build
     GOMODCACHE: /cache/mod
+    REPIN: ${REPIN:-}
 
 services:
   test:

release-tooling/publish_release.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Publish the Gitea release that CHANGELOG.md's top heading names, tagging SHA.
+
+A top heading of `[Unreleased]`, or a version already released, publishes nothing.
+Env: GITEA_API_URL, GITEA_REPOSITORY (owner/repo), GITEA_TOKEN, SHA.
+"""
+
+import json
+import os
+import re
+import sys
+import urllib.error
+import urllib.request
+
+HEADING = re.compile(r"^## \[([^\]]+)\].*?$\n?(.*?)(?=^## \[|\Z)", re.M | re.S)
+
+
+def request(path: str, data: dict | None = None):
+	req = urllib.request.Request(
+		f"{os.environ['GITEA_API_URL']}/repos/{os.environ['GITEA_REPOSITORY']}/{path}",
+		data=json.dumps(data).encode() if data else None,
+		headers={"Authorization": f"token {os.environ['GITEA_TOKEN']}", "Content-Type": "application/json"},
+	)
+	with urllib.request.urlopen(req) as resp:
+		return json.load(resp)
+
+
+def main() -> int:
+	with open("CHANGELOG.md", encoding="utf-8") as f:
+		top = HEADING.search(f.read())
+	if top is None:
+		print("CHANGELOG.md has no `## [...]` heading", file=sys.stderr)
+		return 1
+	version, body = top.group(1), top.group(2).strip()
+	if version == "Unreleased":
+		print("top heading is Unreleased; nothing to publish")
+		return 0
+	if not re.fullmatch(r"\d+\.\d+\.\d+", version):
+		print(f"top heading `[{version}]` is neither Unreleased nor X.Y.Z", file=sys.stderr)
+		return 1
+	tag = f"v{version}"
+	try:
+		print(f"{tag} already published: {request(f'releases/tags/{tag}')['html_url']}")
+		return 0
+	except urllib.error.HTTPError as e:
+		if e.code != 404:
+			raise
+	sha = os.environ["SHA"]
+	try:
+		at = request(f"tags/{tag}")["commit"]["sha"]
+		if at != sha:
+			print(f"{tag} exists at {at}, not {sha}; the version is burnt, bump the heading", file=sys.stderr)
+			return 1
+	except urllib.error.HTTPError as e:
+		if e.code != 404:
+			raise
+	release = request("releases", {"body": body, "name": tag, "tag_name": tag, "target_commitish": sha})
+	print(release["html_url"])
+	return 0
+
+
+if __name__ == "__main__":
+	sys.exit(main())

shape_test.go

@@ -0,0 +1,114 @@
+package fejkdata
+
+import (
+	"os"
+	"sort"
+	"strconv"
+	"strings"
+	"testing"
+)
+
+const shapePin = "testdata/shipped_shape.txt"
+
+// REPIN=1 rewrites the pin.
+func TestShippedShapeIsPinned(t *testing.T) {
+	f, err := New(WithSeed(1))
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := shippedShape(f)
+	if os.Getenv("REPIN") == "1" {
+		if err := os.WriteFile(shapePin, []byte(got), 0o644); err != nil {
+			t.Fatal(err)
+		}
+		return
+	}
+	want, err := os.ReadFile(shapePin)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got != string(want) {
+		t.Fatalf("the shipped shape differs from %s: a removed, renamed or retyped line is a breaking change; add the CHANGELOG.md entry, then repin with REPIN=1", shapePin)
+	}
+}
+
+func shippedShape(f *Generator) string {
+	facts := map[string]string{}
+	var walk func(prefix string, n node)
+	walk = func(prefix string, n node) {
+		switch n := n.(type) {
+		case *folder:
+			for _, name := range sortedNames(n.children) {
+				walk(join(prefix, name), n.children[name])
+			}
+		case *choice:
+			facts[prefix] = reads(n)
+		case *template:
+			facts[prefix] = "\tformat " + strconv.Quote(n.format) + reads(n)
+			if _, columns, err := recordOf(n); err == nil {
+				for _, c := range columns {
+					fact := "\t" + c.DataType.String()
+					if _, nullable := columnItems(n.fields[c.Name]); nullable {
+						fact += " null"
+					}
+					facts[join(prefix, c.Name)] = fact
+				}
+			}
+		}
+	}
+	for _, name := range sortedNames(f.categories) {
+		walk(name, f.categories[name])
+	}
+	var b strings.Builder
+	for _, p := range f.List() {
+		b.WriteString(p + facts[p] + "\n")
+	}
+	return b.String()
+}
+
+// reads names the categories any template under n references, sorted.
+func reads(n node) string {
+	set := map[string]bool{}
+	var collect func(node)
+	collect = func(n node) {
+		switch n := n.(type) {
+		case *choice:
+			for _, it := range n.items {
+				collect(it)
+			}
+		case *template:
+			for _, b := range n.refs {
+				set[strings.TrimPrefix(b.key, "/")] = true
+			}
+			for name, field := range n.fields {
+				if !isRef(name) {
+					collect(field)
+				}
+			}
+		}
+	}
+	collect(n)
+	if len(set) == 0 {
+		return ""
+	}
+	keys := make([]string, 0, len(set))
+	for k := range set {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+	return "\treads " + strings.Join(keys, " ")
+}
+
+func TestShippedShapeNamesReads(t *testing.T) {
+	f := newGenerator(t, writeData(t, map[string]string{
+		"a":     `{"format":"{x}","x":["{/b}",{"format":"{/c.v}","weight":2}]}`,
+		"b":     `"y"`,
+		"c":     `{"format":"{v} {n}","n":[null,{"format":"{int(1,9)}","datatype":"integer"}],"v":["z","w"]}`,
+		"d/pos": `["{.q}","{/b}"]`,
+		"d/q":   `"r"`,
+	}))
+	want := "a\tformat \"{x}\"\treads b c\na.x\tstring\nb\tformat \"y\"\nc\tformat \"{v} {n}\"\nc.n\tinteger null\nc.v\tstring\nd.pos\treads b d.q\nd.q\tformat \"r\"\n"
+	if got := shippedShape(f); got != want {
+		t.Fatalf("shippedShape =\n%s\nwant\n%s", got, want)
+	}
+}

testdata/shipped_shape.txt

@@ -0,0 +1,141 @@
+en_US.address	format "{street-number} {street}\n{locality}, {region} {postal-code}"
+en_US.address.locality	string
+en_US.address.postal-code	string
+en_US.address.region	string
+en_US.address.street	string
+en_US.address.street-number	string
+en_US.address.street.name
+en_US.address.street.suffix
+en_US.color
+en_US.company	format "{base} {suffix}"
+en_US.company.base	string
+en_US.company.suffix	string
+en_US.date	format "{month}/{day}/{year}"
+en_US.date.day	string
+en_US.date.month	string
+en_US.date.year	string
+en_US.email	format "{local}@{domain}"
+en_US.email.domain	string
+en_US.email.local	string
+en_US.email.local.n
+en_US.ip
+en_US.person	format "{prefix}{femalefirst|malefirst} {last}"
+en_US.person.femalefirst	string
+en_US.person.last	string
+en_US.person.malefirst	string
+en_US.person.prefix	string
+en_US.phone
+en_US.phone.area
+en_US.phone.exch
+en_US.phone.line
+en_US.price	format "${amt}.{cents}"
+en_US.price.amt	string
+en_US.price.cents	string
+en_US.sentence
+en_US.sentence.adj
+en_US.sentence.noun
+en_US.sentence.prep
+en_US.sentence.verb
+en_US.ssn	format "{int(100,999)}-{digits(2)}-{digits(4)}"
+en_US.time	format "{hour}:{minute} {ampm}"
+en_US.time.ampm	string
+en_US.time.hour	string
+en_US.time.minute	string
+en_US.time.minute.t
+en_US.url	format "https://{host}{path}"
+en_US.url.host	string
+en_US.url.path	string
+en_US.username
+en_US.username.n
+en_US.version	format "{pre}{n}.{n}.{n}{suffix}"
+en_US.version.n	string
+en_US.version.pre	string
+en_US.version.suffix	string
+en_US.word
+misc.car
+misc.car.maker
+misc.car.model
+misc.coordinate	format "{lat}, {lon}"
+misc.coordinate.lat	string
+misc.coordinate.lon	string
+misc.country
+misc.country.alpha2
+misc.country.alpha3
+misc.country.name
+misc.creditcard
+misc.creditcard.d
+misc.currency
+misc.currency.code
+misc.currency.name
+misc.currency.symbol
+misc.emoji
+misc.httpstatus
+misc.httpstatus.code
+misc.httpstatus.reason
+misc.language
+misc.language.code
+misc.language.name
+misc.mac	format "{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}"
+misc.mimetype
+misc.mimetype.ext
+misc.mimetype.type
+misc.objectid	format "{hex(24)}"
+misc.timezone
+misc.useragent
+misc.uuid	format "{hex(8)}-{hex(4)}-4{hex(3)}-{variant}{hex(3)}-{hex(12)}"
+misc.uuid.variant	string
+sv_SE.address	format "{street} {street-number}\n{postal-code} {locality}"
+sv_SE.address.locality	string
+sv_SE.address.postal-code	string
+sv_SE.address.street	string
+sv_SE.address.street-number	string
+sv_SE.color
+sv_SE.company	format "{base} {suffix}"
+sv_SE.company.base	string
+sv_SE.company.suffix	string
+sv_SE.date	format "{year}-{month}-{day}"
+sv_SE.date.day	string
+sv_SE.date.month	string
+sv_SE.date.year	string
+sv_SE.email	format "{local}@{domain}"
+sv_SE.email.domain	string
+sv_SE.email.local	string
+sv_SE.email.local.n
+sv_SE.ip
+sv_SE.person	format "{prefix}{femalefirst|malefirst} {last}"
+sv_SE.person.femalefirst	string
+sv_SE.person.last	string
+sv_SE.person.malefirst	string
+sv_SE.person.prefix	string
+sv_SE.phone
+sv_SE.phone.a
+sv_SE.phone.b
+sv_SE.phone.c
+sv_SE.phone.prefix
+sv_SE.price	format "{amt}{ore} kr"
+sv_SE.price.amt	string
+sv_SE.price.ore	string
+sv_SE.sentence
+sv_SE.sentence.adj
+sv_SE.sentence.noun
+sv_SE.sentence.prep
+sv_SE.sentence.verb
+sv_SE.ssn	format "{digits(2)}{mmdd}-{digits(3)}{luhn()}"
+sv_SE.ssn.mmdd	string
+sv_SE.ssn.mmdd.d
+sv_SE.ssn.mmdd.m
+sv_SE.time	format "{hour}:{minute}{sec}"
+sv_SE.time.hour	string
+sv_SE.time.minute	string
+sv_SE.time.minute.t
+sv_SE.time.sec	string
+sv_SE.url	format "https://{host}{path}"
+sv_SE.url.host	string
+sv_SE.url.path	string
+sv_SE.username
+sv_SE.username.n
+sv_SE.version	format "{pre}{n}.{n}.{n}{suffix}"
+sv_SE.version.n	string
+sv_SE.version.pre	string
+sv_SE.version.suffix	string
+sv_SE.word

todo.md

@@ -7,24 +7,15 @@
 - Major data update. Shipped categories render as records with their building
   blocks as columns (`sv_SE.person` → `femalefirst`, `malefirst`; `misc.uuid` →
   `variant`), and `sv_SE.address` draws its postal code apart from its locality.
-  It also settles what a version promises about shipped data: its paths, its
-  record columns with their datatypes and nulls — a column of one reference alone
-  takes both, so typing a column or adding a null breaks its readers — and whether
-  a seed renders the same output across versions.
 - `email.local` and `username` share their handle lists, while their name
-  variants differ on purpose. Share the lists only if that is a clean win.
+  variants differ on purpose. Share the lists only if that is a clean win — a
+  reference between shipped categories is a major once tagged.
 
 ### Release
 
-- Versioning — semver tags, starting at `v0.1.0`; `v1.0.0` once the grammar
-  settles. From v2 the module path carries `/vN` (`go.mod`, imports, the README's
-  install lines), so fences ship batched into as few majors as possible. Reword
-  the Decision "After the first tag, a new fence is a major version" to match:
-  before `v1.0.0` a fence ships in a minor.
-- Changelog — `CHANGELOG.md`, started with `v0.1.0`; the fence Decision's
-  "changelog" links there.
-- CLI without Go — investigate prebuilt binaries: GoReleaser publishing to Gitea
-  releases, a container image, Homebrew and Scoop.
+- CLI without Go — investigate prebuilt binaries: GoReleaser attaching them to
+  the Gitea release the tag workflow publishes, a container image, Homebrew and
+  Scoop. A checkout build prints `devel` for `--version`; the binaries carry the stamped tag.
 - Homepage — a simple page for fejkdata with an in-browser generator: the library
   compiled to WebAssembly, so visitors generate as much data as they like in their
   own browser.