Emitter 2a: the corpus runner, the canonical serializer and the CommonMark subset #9

2026-08-24T15:37:48+02:00

lilleman commented

2026-08-24 15:37:48 +02:00

First real code: adfToMarkdown emits the CommonMark subset, and the corpus stops being hand-traced assertions and becomes a test suite. 112 tests, all 17 commonmark-subset/ pairs emitting byte for byte, all 44 corpus JSON files gated against the library's own canonical serializer. Floors in the test script so npm test and the gate are one path: 100% lines, 100% functions, 93% branches.

The runner cannot go vacuously green: a new corpus directory, an orphaned fixture, an emptied directory and a non-byte-equal emission each fail a distinct test.

What the review flow changed

An architecture pass and four stability rounds ran against this branch. The architecture pass blocked on four findings, all fixed: one commonmark-grammar.ts now owns every recognizer both directions need — the line-claim predicate had been implemented twice and had already diverged; opensConstruct was split; the type guard no longer admits sparse arrays it claims are AdfNode[].

Twelve §2 losslessness breaks were found and closed. The ones worth naming, because each emitted successfully and read back as different ADF:

Two distinct documents emitting identical bytes — em over a_ and em over a plus a literal _ both spelled _a__. The escaper asked whether a literal character opened a construct, never whether it merged with the syntax emitted beside it.
The emitter never checked that its own **/_/~~ delimiters could open or close where they landed. un + strong("-real") + istic spelled un**-real**istic, which CommonMark reads as plain text — the ** is followed by punctuation and preceded by a letter, so it is not left-flanking. Every mark was lost. Delimiters are now judged by the run they sit in, against cmark's scan_delims rules.
A rule inside a bullet spelled - ---, a thematic break, destroying the list and all its siblings.
A nested list under a paragraph in a list item: CommonMark only lets a list interrupt a paragraph when its first item is non-blank and, if ordered, starts at 1 — so every nested ordered list was swallowed as paragraph text.
A ten-digit ordered-list marker, an empty list emitting nothing, \r and U+0000 passing through raw, and four character classes that had drifted from the CommonMark definitions they implement.

isAdfDocument and isJsonValue now walk iteratively, and blocks, marks and attribute values are held to 500 levels, so the published "nothing throws" promise holds against a deep document.

The error surface

ConvertError is { code, message, path } — the path names the node, from the document root. ConvertErrorCode is a closed list a consumer may switch on exhaustively; adding a member is breaking, and the list is complete at 0.1.0 (AGENTS.md §8).

Open, and recorded rather than guessed

A nested list that cannot interrupt the block above it is refused, not spelled. Whether the canonical form separates it with a blank line, always or only where tight would not survive, is a format decision (§8) waiting on the maintainer; the answer turns that refusal into an emission. Recorded against milestone 2b, along with the separationBetween seam it lands in.

The emitter validates delimiter flanking exactly but does not model CommonMark's matching — the multiple-of-3 rule and the way a run splits across openers. No reachable violation was found by hand; the corpus property test at 2e is what settles it. Recorded there.

First real code: `adfToMarkdown` emits the CommonMark subset, and the corpus stops being hand-traced assertions and becomes a test suite. 112 tests, all 17 `commonmark-subset/` pairs emitting byte for byte, all 44 corpus JSON files gated against the library's own canonical serializer. Floors in the `test` script so `npm test` and the gate are one path: 100% lines, 100% functions, 93% branches. The runner cannot go vacuously green: a new corpus directory, an orphaned fixture, an emptied directory and a non-byte-equal emission each fail a distinct test. ## What the review flow changed An architecture pass and four stability rounds ran against this branch. The architecture pass blocked on four findings, all fixed: one `commonmark-grammar.ts` now owns every recognizer both directions need — the line-claim predicate had been implemented twice and had already diverged; `opensConstruct` was split; the type guard no longer admits sparse arrays it claims are `AdfNode[]`. Twelve §2 losslessness breaks were found and closed. The ones worth naming, because each emitted successfully and read back as different ADF: - Two distinct documents emitting identical bytes — `em` over `a_` and `em` over `a` plus a literal `_` both spelled `_a__`. The escaper asked whether a literal character *opened* a construct, never whether it *merged with* the syntax emitted beside it. - The emitter never checked that its own `**`/`_`/`~~` delimiters could open or close where they landed. `un` + `strong("-real")` + `istic` spelled `un**-real**istic`, which CommonMark reads as plain text — the `**` is followed by punctuation and preceded by a letter, so it is not left-flanking. Every mark was lost. Delimiters are now judged by the run they sit in, against cmark's `scan_delims` rules. - A `rule` inside a bullet spelled `- ---`, a thematic break, destroying the list and all its siblings. - A nested list under a paragraph in a list item: CommonMark only lets a list interrupt a paragraph when its first item is non-blank and, if ordered, starts at 1 — so every nested ordered list was swallowed as paragraph text. - A ten-digit ordered-list marker, an empty list emitting nothing, `\r` and U+0000 passing through raw, and four character classes that had drifted from the CommonMark definitions they implement. `isAdfDocument` and `isJsonValue` now walk iteratively, and blocks, marks and attribute values are held to 500 levels, so the published "nothing throws" promise holds against a deep document. ## The error surface `ConvertError` is `{ code, message, path }` — the path names the node, from the document root. `ConvertErrorCode` is a closed list a consumer may switch on exhaustively; adding a member is breaking, and the list is complete at `0.1.0` (AGENTS.md §8). ## Open, and recorded rather than guessed A nested list that cannot interrupt the block above it is **refused**, not spelled. Whether the canonical form separates it with a blank line, always or only where tight would not survive, is a format decision (§8) waiting on the maintainer; the answer turns that refusal into an emission. Recorded against milestone 2b, along with the `separationBetween` seam it lands in. The emitter validates delimiter *flanking* exactly but does not model CommonMark's *matching* — the multiple-of-3 rule and the way a run splits across openers. No reachable violation was found by hand; the corpus property test at 2e is what settles it. Recorded there.

lilleman added 1 commit 2026-08-24 15:37:49 +02:00

Emitter 2a: the corpus runner, the canonical serializer and the CommonMark subset

CI / gate (push) Successful in 5s

Details

0728ea1cfb

lilleman added 1 commit 2026-08-24 16:18:53 +02:00

Share the CommonMark grammar, close the guard's hole, refuse the lists no marker spells

CI / gate (push) Successful in 4s

Details

e92484eb85

lilleman added 1 commit 2026-08-25 09:00:49 +02:00

Name the node every refusal came from, and close the error list