diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index d6336ba..7569186 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -26,11 +26,19 @@ jobs: else base='${{ github.event.before }}' fi - if git diff --quiet "$base"...HEAD -- '*.go' go.mod go.sum Dockerfile .dockerignore .github data README.md; then + if git diff --quiet "$base"...HEAD -- '*.go' go.mod go.sum Dockerfile .dockerignore .github data testdata README.md; then echo "code=false" >> "$GITHUB_OUTPUT" else echo "code=true" >> "$GITHUB_OUTPUT" fi + echo "base=$base" >> "$GITHUB_OUTPUT" + - name: CHANGELOG entry for data changes + run: | + base='${{ steps.changes.outputs.base }}' + if ! git diff --quiet "$base"...HEAD -- data testdata/shipped_shape.txt && git diff --quiet "$base"...HEAD -- CHANGELOG.md; then + echo "::error::the shipped data changed without a CHANGELOG.md entry (see the README's Versioning)" + exit 1 + fi # `docker build` streams the context to the daemon. A compose bind-mount of # `.` mounts an empty host dir instead, because the job is itself a container. - name: Latest supported Go @@ -41,3 +49,21 @@ jobs: - name: Lowest supported Go if: ${{ !cancelled() && steps.changes.outputs.code == 'true' }} run: docker build --build-arg GO_VERSION=1.22.12 . + + release: + name: Gitea release from CHANGELOG.md + needs: test + if: github.event_name == 'push' + runs-on: ubuntu-24.04 + permissions: + contents: write + steps: + - uses: actions/checkout@v6 + with: + persist-credentials: false + - run: python3 release-tooling/publish_release.py + env: + GITEA_API_URL: ${{ github.api_url }} + GITEA_REPOSITORY: ${{ github.repository }} + GITEA_TOKEN: ${{ secrets.GITHUB_TOKEN }} + SHA: ${{ github.sha }} diff --git a/AGENTS.md b/AGENTS.md index 28cf18d..5bab122 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,8 @@ # Rules -- Go runs only through `docker compose run --rm `; the merge gate is `docker build .`. -- Tests first, in their own commit; the implementation follows in the next. A re-pin of seeded output is its own commit. +- Go runs only through `docker compose run --rm `; the merge gate is `docker build .` plus CI's changelog check. +- Tests first, in their own commit; the implementation follows in the next. A re-pin of seeded output or of `testdata/shipped_shape.txt` is its own commit. +- A change under `data/` or to the shape pin adds its `CHANGELOG.md` entry under `Unreleased` in the same PR, and so does a change to a flag, an exit code, an exported name, a fence, a builtin or the lowest Go; what is major is the README's Versioning table. - One-line commit messages: no ticket prefix, no repo name, no authorship trailers. - Hard tabs. No comment by default; delete a restatement, a rationale, history, or a file preamble. - One spelling per result: reject the other at `New`, and let the error name the spelling to use. diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..b1d7908 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,11 @@ +# Changelog + +Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). A release's +`Breaking` section comes first and names each rejected spelling with its +replacement, and each removed path, column or flag. + +## [Unreleased] + +### Added + +- First release: the CLI, the library and the shipped data set. diff --git a/README.md b/README.md index 3a11981..61c3db2 100644 --- a/README.md +++ b/README.md @@ -529,6 +529,41 @@ then costs about what its output costs: an unweighted pick is O(1) whatever the list's length, a weighted one O(log n), and long formats, deep nesting and many tokens add cost in proportion to the output. +## Versioning + +Semver tags on `main`, `v0.1.0` first; one version covers the shipped data, the +library and the CLI, and [`CHANGELOG.md`](CHANGELOG.md) names what each release +changed. A consumer's data, code and scripts keep working across a minor or a patch: +a minor only adds, and a major is the only release that changes what exists. + +| Surface | Major | Minor | +|---------|-------|-------| +| Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another | a path outside a record's columns, a locale, a value in a list | +| Records | remove, rename, retype or add a column; let a column be null | a record, as a new category | +| Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin | +| CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format | +| Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option | + +A patch changes no row of this table: performance, docs, or a fix inside a promised +behaviour that changes no value, path, format or spelling. + +Seeded output is a promise within one version: same seed, same version, same +data, same output. Any release may shift a stream, since a value added to a list +moves every draw after it, so pin fixtures per version. An error's wording may +improve in a minor; the path, rejected spelling and replacement it names may not. + +Before `v1.0.0` a minor is the breaking unit: `0.(x+1).0` may carry a major's +changes, each named in the changelog, and a `0.x.y` patch may not. `v1.0.0` is +cut once the shipped data is in its record shape and one full minor has shipped +with no breaking change. From `v2` the module path carries `/vN`, so fences ship +batched into as few majors as possible. + +[`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) pins every path, each +template category's format, the categories each category reads, and each column's +datatype and nullability; a pull request that +changes it or `data/` adds its `CHANGELOG.md` entry, which CI checks. A removed, +renamed or retyped line is a major. + ## Goals 1. **Valid by construction** — every value passes the check its real consumer @@ -613,14 +648,31 @@ tokens add cost in proportion to the output. folder, `..` the folder above — what those spellings already mean to anyone who has typed a path. A locale's files reach each other without naming the locale, so a folder renames and copies without editing its references. -- **After the first tag, a new fence is a major version.** Data files are the - public API, and one spelling per result grows by tightening, so every fence - invalidates some file. Each such release names the rejected spelling and its - replacement in the changelog and in the load error, and that is the whole - migration: a fence rejects one spelling with one replacement, so the fix is - local to each site. A fence that would need a non-local rewrite ships a - converter with its release instead. Before the first tag there is no - compatibility promise. +- **A change to what exists is a major; a minor only adds.** Data files, the CLI + and the Go API are the public API, and a consumer must be able to take a minor + without an edit — so an added column is a major, since it changes the CSV header + and the `INSERT` column list, as is a removed value, which changes what a fixture + holds, and a new option, which reserves a field name. One spelling per result + grows by tightening, so every fence invalidates some file. Each such release + names the rejected spelling and its replacement in the changelog and in the load + error, and that is the whole migration: a fence rejects one spelling with one + replacement, so the fix is local to each site. A fence that would need a + non-local rewrite ships a converter with its release instead. Before `v1.0.0` a + minor carries what a major would. +- **Seeded output is promised within one version.** Any edit to a category shifts + its stream and everything drawn after it, so a promise across versions would + freeze every shipped list; a fixture is re-pinned on a bump, as this repo's own are. +- **An error is a contract by what it names, not its bytes.** A script branches on + the exit code and reads the named path or spelling, so those hold; wording improves + in a minor. +- **Raising the lowest supported Go is a major.** A consumer building on it breaks, + which is the one test every rule above applies; Go's convention of a minor is not + followed. +- **The changelog heading is the one spelling of a release; CI cuts the tag.** A + tag pushed by hand is served by `go get` at once, so a tag whose commit lacks its + heading is burnt, not fixed. The heading on a gate-passed `main` commit is the + trigger instead: the tag can land only there, and the Gitea release the same job + publishes keeps one text as its body and is where prebuilt binaries will attach. - **A `--data-path` override rebinds every reference to the category it replaces.** References bind against the merged tree, so once shipped data uses `{.person}`, a consumer's `sv_SE/person.json` is what every shipped reference @@ -770,8 +822,9 @@ docker compose run --rm --user "$(id -u):$(id -g)" tidy # go mod tidy Every pull request runs `docker build .` against both the latest and the lowest supported Go, and must pass before it can be merged — unless it changes none of the files the build and its tests read, nor the workflow itself, in which case -it's skipped (see [Decisions](#decisions)). That build is the whole gate — vet, -complexity, format check and tests — so run it locally before pushing: +it's skipped (see [Decisions](#decisions)). That build is the whole gate but the +changelog check, which CI runs against the PR base — vet, complexity, format check +and tests — so run it locally before pushing: ```sh docker build . # latest @@ -779,6 +832,18 @@ docker build --build-arg GO_VERSION=1.22.12 . # lowest supported GO_VERSION=1.22.12 docker compose run --rm test # the same tests, without the image build ``` +A change to the shipped data re-pins [`testdata/shipped_shape.txt`](testdata/shipped_shape.txt) +in its own commit: + +```sh +REPIN=1 docker compose run --rm --user "$(id -u):$(id -g)" test +``` + +To release, head `CHANGELOG.md` with the version's section in place of `Unreleased` +and merge: once `main` passes the gate, CI tags that commit `vX.Y.Z` and publishes +the Gitea release with the section as its body. A top heading of `[Unreleased]` +publishes nothing. + ## Layout ``` @@ -801,6 +866,8 @@ value.go the value proof: what a typed column or calc operand holds, chec data.go data loading: fs.FS folders/files -> namespace tree, multi-source merge cmd/fejkdata/ the fejkdata CLI data/ shipped data (JSON), embedded at build: locale folders + a misc folder +release-tooling/ the release CI publishes from the changelog heading +testdata/ the pinned shipped shape (see Versioning) ``` ## License diff --git a/compose.yaml b/compose.yaml index bb47873..ca3af11 100644 --- a/compose.yaml +++ b/compose.yaml @@ -24,6 +24,7 @@ x-go: &go HOME: /cache GOCACHE: /cache/build GOMODCACHE: /cache/mod + REPIN: ${REPIN:-} services: test: diff --git a/release-tooling/publish_release.py b/release-tooling/publish_release.py new file mode 100644 index 0000000..313b84e --- /dev/null +++ b/release-tooling/publish_release.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python3 +"""Publish the Gitea release that CHANGELOG.md's top heading names, tagging SHA. + +A top heading of `[Unreleased]`, or a version already released, publishes nothing. +Env: GITEA_API_URL, GITEA_REPOSITORY (owner/repo), GITEA_TOKEN, SHA. +""" + +import json +import os +import re +import sys +import urllib.error +import urllib.request + +HEADING = re.compile(r"^## \[([^\]]+)\].*?$\n?(.*?)(?=^## \[|\Z)", re.M | re.S) + + +def request(path: str, data: dict | None = None): + req = urllib.request.Request( + f"{os.environ['GITEA_API_URL']}/repos/{os.environ['GITEA_REPOSITORY']}/{path}", + data=json.dumps(data).encode() if data else None, + headers={"Authorization": f"token {os.environ['GITEA_TOKEN']}", "Content-Type": "application/json"}, + ) + with urllib.request.urlopen(req) as resp: + return json.load(resp) + + +def main() -> int: + with open("CHANGELOG.md", encoding="utf-8") as f: + top = HEADING.search(f.read()) + if top is None: + print("CHANGELOG.md has no `## [...]` heading", file=sys.stderr) + return 1 + version, body = top.group(1), top.group(2).strip() + if version == "Unreleased": + print("top heading is Unreleased; nothing to publish") + return 0 + if not re.fullmatch(r"\d+\.\d+\.\d+", version): + print(f"top heading `[{version}]` is neither Unreleased nor X.Y.Z", file=sys.stderr) + return 1 + tag = f"v{version}" + try: + print(f"{tag} already published: {request(f'releases/tags/{tag}')['html_url']}") + return 0 + except urllib.error.HTTPError as e: + if e.code != 404: + raise + sha = os.environ["SHA"] + try: + at = request(f"tags/{tag}")["commit"]["sha"] + if at != sha: + print(f"{tag} exists at {at}, not {sha}; the version is burnt, bump the heading", file=sys.stderr) + return 1 + except urllib.error.HTTPError as e: + if e.code != 404: + raise + release = request("releases", {"body": body, "name": tag, "tag_name": tag, "target_commitish": sha}) + print(release["html_url"]) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/shape_test.go b/shape_test.go new file mode 100644 index 0000000..ca12de4 --- /dev/null +++ b/shape_test.go @@ -0,0 +1,114 @@ +package fejkdata + +import ( + "os" + "sort" + "strconv" + "strings" + "testing" +) + +const shapePin = "testdata/shipped_shape.txt" + +// REPIN=1 rewrites the pin. +func TestShippedShapeIsPinned(t *testing.T) { + f, err := New(WithSeed(1)) + if err != nil { + t.Fatal(err) + } + got := shippedShape(f) + if os.Getenv("REPIN") == "1" { + if err := os.WriteFile(shapePin, []byte(got), 0o644); err != nil { + t.Fatal(err) + } + return + } + want, err := os.ReadFile(shapePin) + if err != nil { + t.Fatal(err) + } + if got != string(want) { + t.Fatalf("the shipped shape differs from %s: a removed, renamed or retyped line is a breaking change; add the CHANGELOG.md entry, then repin with REPIN=1", shapePin) + } +} + +func shippedShape(f *Generator) string { + facts := map[string]string{} + var walk func(prefix string, n node) + walk = func(prefix string, n node) { + switch n := n.(type) { + case *folder: + for _, name := range sortedNames(n.children) { + walk(join(prefix, name), n.children[name]) + } + case *choice: + facts[prefix] = reads(n) + case *template: + facts[prefix] = "\tformat " + strconv.Quote(n.format) + reads(n) + if _, columns, err := recordOf(n); err == nil { + for _, c := range columns { + fact := "\t" + c.DataType.String() + if _, nullable := columnItems(n.fields[c.Name]); nullable { + fact += " null" + } + facts[join(prefix, c.Name)] = fact + } + } + } + } + for _, name := range sortedNames(f.categories) { + walk(name, f.categories[name]) + } + var b strings.Builder + for _, p := range f.List() { + b.WriteString(p + facts[p] + "\n") + } + return b.String() +} + +// reads names the categories any template under n references, sorted. +func reads(n node) string { + set := map[string]bool{} + var collect func(node) + collect = func(n node) { + switch n := n.(type) { + case *choice: + for _, it := range n.items { + collect(it) + } + case *template: + for _, b := range n.refs { + set[strings.TrimPrefix(b.key, "/")] = true + } + for name, field := range n.fields { + if !isRef(name) { + collect(field) + } + } + } + } + collect(n) + if len(set) == 0 { + return "" + } + keys := make([]string, 0, len(set)) + for k := range set { + keys = append(keys, k) + } + sort.Strings(keys) + return "\treads " + strings.Join(keys, " ") +} + +func TestShippedShapeNamesReads(t *testing.T) { + f := newGenerator(t, writeData(t, map[string]string{ + "a": `{"format":"{x}","x":["{/b}",{"format":"{/c.v}","weight":2}]}`, + "b": `"y"`, + "c": `{"format":"{v} {n}","n":[null,{"format":"{int(1,9)}","datatype":"integer"}],"v":["z","w"]}`, + "d/pos": `["{.q}","{/b}"]`, + "d/q": `"r"`, + })) + want := "a\tformat \"{x}\"\treads b c\na.x\tstring\nb\tformat \"y\"\nc\tformat \"{v} {n}\"\nc.n\tinteger null\nc.v\tstring\nd.pos\treads b d.q\nd.q\tformat \"r\"\n" + if got := shippedShape(f); got != want { + t.Fatalf("shippedShape =\n%s\nwant\n%s", got, want) + } +} diff --git a/testdata/shipped_shape.txt b/testdata/shipped_shape.txt new file mode 100644 index 0000000..75f4e9c --- /dev/null +++ b/testdata/shipped_shape.txt @@ -0,0 +1,141 @@ +en_US.address format "{street-number} {street}\n{locality}, {region} {postal-code}" +en_US.address.locality string +en_US.address.postal-code string +en_US.address.region string +en_US.address.street string +en_US.address.street-number string +en_US.address.street.name +en_US.address.street.suffix +en_US.color +en_US.company format "{base} {suffix}" +en_US.company.base string +en_US.company.suffix string +en_US.date format "{month}/{day}/{year}" +en_US.date.day string +en_US.date.month string +en_US.date.year string +en_US.email format "{local}@{domain}" +en_US.email.domain string +en_US.email.local string +en_US.email.local.n +en_US.ip +en_US.person format "{prefix}{femalefirst|malefirst} {last}" +en_US.person.femalefirst string +en_US.person.last string +en_US.person.malefirst string +en_US.person.prefix string +en_US.phone +en_US.phone.area +en_US.phone.exch +en_US.phone.line +en_US.price format "${amt}.{cents}" +en_US.price.amt string +en_US.price.cents string +en_US.sentence +en_US.sentence.adj +en_US.sentence.noun +en_US.sentence.prep +en_US.sentence.verb +en_US.ssn format "{int(100,999)}-{digits(2)}-{digits(4)}" +en_US.time format "{hour}:{minute} {ampm}" +en_US.time.ampm string +en_US.time.hour string +en_US.time.minute string +en_US.time.minute.t +en_US.url format "https://{host}{path}" +en_US.url.host string +en_US.url.path string +en_US.username +en_US.username.n +en_US.version format "{pre}{n}.{n}.{n}{suffix}" +en_US.version.n string +en_US.version.pre string +en_US.version.suffix string +en_US.word +misc.car +misc.car.maker +misc.car.model +misc.coordinate format "{lat}, {lon}" +misc.coordinate.lat string +misc.coordinate.lon string +misc.country +misc.country.alpha2 +misc.country.alpha3 +misc.country.name +misc.creditcard +misc.creditcard.d +misc.currency +misc.currency.code +misc.currency.name +misc.currency.symbol +misc.emoji +misc.httpstatus +misc.httpstatus.code +misc.httpstatus.reason +misc.language +misc.language.code +misc.language.name +misc.mac format "{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}" +misc.mimetype +misc.mimetype.ext +misc.mimetype.type +misc.objectid format "{hex(24)}" +misc.timezone +misc.useragent +misc.uuid format "{hex(8)}-{hex(4)}-4{hex(3)}-{variant}{hex(3)}-{hex(12)}" +misc.uuid.variant string +sv_SE.address format "{street} {street-number}\n{postal-code} {locality}" +sv_SE.address.locality string +sv_SE.address.postal-code string +sv_SE.address.street string +sv_SE.address.street-number string +sv_SE.color +sv_SE.company format "{base} {suffix}" +sv_SE.company.base string +sv_SE.company.suffix string +sv_SE.date format "{year}-{month}-{day}" +sv_SE.date.day string +sv_SE.date.month string +sv_SE.date.year string +sv_SE.email format "{local}@{domain}" +sv_SE.email.domain string +sv_SE.email.local string +sv_SE.email.local.n +sv_SE.ip +sv_SE.person format "{prefix}{femalefirst|malefirst} {last}" +sv_SE.person.femalefirst string +sv_SE.person.last string +sv_SE.person.malefirst string +sv_SE.person.prefix string +sv_SE.phone +sv_SE.phone.a +sv_SE.phone.b +sv_SE.phone.c +sv_SE.phone.prefix +sv_SE.price format "{amt}{ore} kr" +sv_SE.price.amt string +sv_SE.price.ore string +sv_SE.sentence +sv_SE.sentence.adj +sv_SE.sentence.noun +sv_SE.sentence.prep +sv_SE.sentence.verb +sv_SE.ssn format "{digits(2)}{mmdd}-{digits(3)}{luhn()}" +sv_SE.ssn.mmdd string +sv_SE.ssn.mmdd.d +sv_SE.ssn.mmdd.m +sv_SE.time format "{hour}:{minute}{sec}" +sv_SE.time.hour string +sv_SE.time.minute string +sv_SE.time.minute.t +sv_SE.time.sec string +sv_SE.url format "https://{host}{path}" +sv_SE.url.host string +sv_SE.url.path string +sv_SE.username +sv_SE.username.n +sv_SE.version format "{pre}{n}.{n}.{n}{suffix}" +sv_SE.version.n string +sv_SE.version.pre string +sv_SE.version.suffix string +sv_SE.word diff --git a/todo.md b/todo.md index 2d65711..fbc3d8c 100644 --- a/todo.md +++ b/todo.md @@ -7,24 +7,15 @@ - Major data update. Shipped categories render as records with their building blocks as columns (`sv_SE.person` → `femalefirst`, `malefirst`; `misc.uuid` → `variant`), and `sv_SE.address` draws its postal code apart from its locality. - It also settles what a version promises about shipped data: its paths, its - record columns with their datatypes and nulls — a column of one reference alone - takes both, so typing a column or adding a null breaks its readers — and whether - a seed renders the same output across versions. - `email.local` and `username` share their handle lists, while their name - variants differ on purpose. Share the lists only if that is a clean win. + variants differ on purpose. Share the lists only if that is a clean win — a + reference between shipped categories is a major once tagged. ### Release -- Versioning — semver tags, starting at `v0.1.0`; `v1.0.0` once the grammar - settles. From v2 the module path carries `/vN` (`go.mod`, imports, the README's - install lines), so fences ship batched into as few majors as possible. Reword - the Decision "After the first tag, a new fence is a major version" to match: - before `v1.0.0` a fence ships in a minor. -- Changelog — `CHANGELOG.md`, started with `v0.1.0`; the fence Decision's - "changelog" links there. -- CLI without Go — investigate prebuilt binaries: GoReleaser publishing to Gitea - releases, a container image, Homebrew and Scoop. +- CLI without Go — investigate prebuilt binaries: GoReleaser attaching them to + the Gitea release the tag workflow publishes, a container image, Homebrew and + Scoop. A checkout build prints `devel` for `--version`; the binaries carry the stamped tag. - Homepage — a simple page for fejkdata with an in-browser generator: the library compiled to WebAssembly, so visitors generate as much data as they like in their own browser.