Compare commits

1 Commits

Author SHA1 Message Date
lilleman 636fe19e59 3d — the online sandbox, lossy conversion, and @atlaskit/adf-schema evaluation
CI / gate (push) Successful in 18s
CI / publish (push) Has been skipped
2026-09-05 16:39:48 +02:00
9 changed files with 19 additions and 6427 deletions
+5 -12
View File
@@ -17,10 +17,8 @@ When losslessness and readability conflict, losslessness wins.
The other direction is a canonical fixpoint, not byte-identity: human markdown normalizes, the way
back yields the library's canonical spelling, and that spelling round-trips byte-identically —
where there is a way back. CommonMark spells some things the flavour has no escape for — a link
destination or title holding a backslash or newline, a paragraph opening with a code span whose
backticks read back as a fence — so a parse succeeding does not imply a spellable document;
`corpus/commonmark-spec/exceptions.json` names those.
where there is a way back. CommonMark spells link destinations the flavour has no escape for, so a
parse succeeding does not imply a spellable document; `todo.md` 3k's exception list names those.
"Equals" is structural equality over editor-normal ADF — adjacent text nodes with identical marks
merged, JSON number semantics, an empty attrs object, marks array or content array the absent
@@ -59,12 +57,8 @@ Round-trip equality is a property tested over a corpus, not a claim made in pros
why ~20 lines of own code cannot do the job, who maintains it, and what auditing it costs. So the
CommonMark and HTML parsers are written in this repo. A table a standard fixes is data rather than
a dependency: HTML5's 2125 semicolon-terminated character references ship packed in their own
module, so entity decoding is complete without one. The CommonMark spec suite is the same shape of
data and ships vendored at `corpus/commonmark-spec/` rather than as the `commonmark-spec` dev
dependency — that package is CommonJS-only, and Renovate auto-bumping a spec version would silently
point the vendored exception list's example numbers at a renumbered suite. A spec bump is a
deliberate re-pin, exceptions re-derived by hand beside it. `devDependencies`: few, each earning its
keep; they never reach a consumer.
module, so entity decoding is complete without one. `devDependencies`: few, each earning its keep;
they never reach a consumer.
## 6. The package contract
@@ -204,8 +198,7 @@ resolver maps them, under `NodeNext` alone; a `.d.ts` reader that is not `tsc` s
`node-floor.js` round-trips the installed package under a Node pinned to `engines.node`'s floor.
A fourth engine reads the build rather than the source: a headless Firefox loads `dist/index.js`
over HTTP and converts the round-trip, normalization and error fixtures — the `commonmark-spec`
sort is the Node suite's to check — which is §6's browser half and the only SpiderMonkey
over HTTP and converts the whole corpus, which is §6's browser half and the only SpiderMonkey
there is — the gate's other three engines are two V8s and a JavaScriptCore that is not Safari's.
A WebDriver session is what carries a verdict back out, the driver and the page's server sharing
one network namespace so each is the other's `127.0.0.1`; `--headless --screenshot` has no such
-1
View File
@@ -72,7 +72,6 @@ assert.deepEqual(
readdirSync(corpusRoot, { withFileTypes: true })
.filter((entry) => entry.isDirectory())
.map((entry) => entry.name)
.filter((name) => name !== 'commonmark-spec')
.sort(),
['errors', 'normalization', 'round-trip'],
'a corpus kind the browser leg does not convert',
+1 -9
View File
@@ -12,13 +12,5 @@ One directory per contract kind, each landing with its milestone:
pins which error.
- `real-payloads/``<name>.json`: sanitized live ADF, round-tripped ADF→markdown→ADF. No
expected markdown.
- `commonmark-spec/` — the CommonMark suite run against `markdownToAdf` by three checks. `spec.json`
is the suite; `refusals.json` pins each refusing example to its error `code`; `exceptions.json`
pins each known divergence by `check`, `example`, `kind` and the exact `divergence`, with a
`reason`. `kind` is `mark-model` (the permanent count divergence from ADF's mark-per-text-node
model), `unspellable` (parses but the flavour has no spelling) or `pending` (a parser gap a later
milestone may close).
JSON is editor-normal (AGENTS.md §2), two-space indent, keys sorted. `spec.json` is the vendored,
upstream machine-readable suite, byte-exact (CommonMark 0.31.2, CC-BY-SA-4.0, © John MacFarlane),
and is not re-serialized by the corpus gate.
JSON is editor-normal (AGENTS.md §2), two-space indent, keys sorted.
-436
View File
@@ -1,436 +0,0 @@
[
{
"check": "fixpoint",
"divergence": "unspellable-link",
"example": 196,
"kind": "unspellable",
"reason": "The link title holds literal newlines no escape spells."
},
{
"check": "fixpoint",
"divergence": "unspellable-link",
"example": 202,
"kind": "unspellable",
"reason": "The link destination holds a backslash the flavour cannot escape."
},
{
"check": "count",
"divergence": "ul 2/1",
"example": 301,
"kind": "pending",
"reason": "A list continuing past a marker change renders as two lists, the parser opens one."
},
{
"check": "count",
"divergence": "ol 2/1",
"example": 302,
"kind": "pending",
"reason": "A list continuing past a marker change renders as two lists, the parser opens one."
},
{
"check": "fixpoint",
"divergence": "unspellable-line-start",
"example": 330,
"kind": "unspellable",
"reason": "A paragraph opens with a code span whose backticks read back as a fence."
},
{
"check": "fixpoint",
"divergence": "unspellable-line-start",
"example": 331,
"kind": "unspellable",
"reason": "A paragraph opens with a code span whose backticks read back as a fence."
},
{
"check": "fixpoint",
"divergence": "unspellable-line-start",
"example": 340,
"kind": "unspellable",
"reason": "A paragraph opens with a code span whose backticks read back as a fence."
},
{
"check": "count",
"divergence": "em 2/1",
"example": 369,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "em 2/1",
"example": 373,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "strong 2/1",
"example": 389,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "em 1/3",
"example": 393,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 1/5",
"example": 394,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 1/3",
"example": 395,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/3",
"example": 399,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/2",
"example": 404,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/3",
"example": 406,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 2/1",
"example": 407,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "em 2/1",
"example": 408,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "em 2/1",
"example": 409,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "em 1/3",
"example": 410,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/3",
"example": 411,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/2",
"example": 413,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/2",
"example": 414,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/2",
"example": 415,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 3/1",
"example": 417,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "em 2/5 strong 1/3",
"example": 418,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 1/2",
"example": 422,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 1/3",
"example": 424,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 2/1",
"example": 425,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "strong 2/1",
"example": 426,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "strong 2/1",
"example": 427,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "strong 1/3",
"example": 428,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 1/3",
"example": 429,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 1/2",
"example": 430,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 1/2",
"example": 431,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/3 strong 2/5",
"example": 432,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "strong 1/2",
"example": 433,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 2/1",
"example": 461,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "em 2/1",
"example": 463,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "strong 2/1",
"example": 464,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "strong 2/1",
"example": 465,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "strong 3/1",
"example": 466,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "strong 2/1",
"example": 468,
"kind": "mark-model",
"reason": "CommonMark nests same-kind elements; the single mark collapses them to one."
},
{
"check": "count",
"divergence": "em 1/3",
"example": 470,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/2",
"example": 478,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/2",
"example": 479,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "a 1/0",
"example": 484,
"kind": "pending",
"reason": "An empty link text stays literal text; CommonMark renders an empty link."
},
{
"check": "text",
"divergence": "\"\" against \"[](./target.md)\"",
"example": 484,
"kind": "pending",
"reason": "An empty link text stays literal text; CommonMark renders an empty link."
},
{
"check": "count",
"divergence": "a 1/0",
"example": 487,
"kind": "pending",
"reason": "An empty link text stays literal text; CommonMark renders an empty link."
},
{
"check": "text",
"divergence": "\"\" against \"[]()\"",
"example": 487,
"kind": "pending",
"reason": "An empty link text stays literal text; CommonMark renders an empty link."
},
{
"check": "fixpoint",
"divergence": "unspellable-link",
"example": 502,
"kind": "unspellable",
"reason": "The link destination holds a backslash the flavour cannot escape."
},
{
"check": "count",
"divergence": "a 1/5 em 1/4",
"example": 516,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/3",
"example": 519,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "a 1/5 em 1/4",
"example": 530,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/2",
"example": 533,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "a 1/0",
"example": 540,
"kind": "pending",
"reason": "The case-folding shortcut reference is unresolved; CommonMark folds case and links."
},
{
"check": "text",
"divergence": "\"ẞ\" against \"[ẞ]\"",
"example": 540,
"kind": "pending",
"reason": "A case-folding shortcut reference is unresolved; CommonMark folds case and links."
},
{
"check": "count",
"divergence": "a 1/2",
"example": 554,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "a 1/2",
"example": 558,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "a 1/2",
"example": 559,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/2",
"example": 638,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
},
{
"check": "count",
"divergence": "em 1/2",
"example": 639,
"kind": "mark-model",
"reason": "The mark spans adjacent text nodes, counted once per node where CommonMark nests one element."
}
]
-346
View File
@@ -1,346 +0,0 @@
[
{
"code": "unmappable-html",
"example": 21
},
{
"code": "unmappable-html",
"example": 31
},
{
"code": "unmappable-html",
"example": 148
},
{
"code": "unmappable-html",
"example": 149
},
{
"code": "unmappable-html",
"example": 150
},
{
"code": "unmappable-html",
"example": 151
},
{
"code": "unmappable-html",
"example": 152
},
{
"code": "unmappable-html",
"example": 153
},
{
"code": "unmappable-html",
"example": 154
},
{
"code": "unmappable-html",
"example": 155
},
{
"code": "unmappable-html",
"example": 156
},
{
"code": "unmappable-html",
"example": 157
},
{
"code": "unmappable-html",
"example": 158
},
{
"code": "unmappable-html",
"example": 159
},
{
"code": "unmappable-html",
"example": 160
},
{
"code": "unmappable-html",
"example": 161
},
{
"code": "unmappable-html",
"example": 162
},
{
"code": "unmappable-html",
"example": 163
},
{
"code": "unmappable-html",
"example": 164
},
{
"code": "unmappable-html",
"example": 165
},
{
"code": "unmappable-html",
"example": 166
},
{
"code": "unmappable-html",
"example": 167
},
{
"code": "unmappable-html",
"example": 168
},
{
"code": "unmappable-html",
"example": 169
},
{
"code": "unmappable-html",
"example": 170
},
{
"code": "unmappable-html",
"example": 171
},
{
"code": "unmappable-html",
"example": 172
},
{
"code": "unmappable-html",
"example": 173
},
{
"code": "unmappable-html",
"example": 174
},
{
"code": "unmappable-html",
"example": 175
},
{
"code": "unmappable-html",
"example": 176
},
{
"code": "unmappable-html",
"example": 177
},
{
"code": "unmappable-html",
"example": 178
},
{
"code": "unmappable-html",
"example": 179
},
{
"code": "unmappable-html",
"example": 180
},
{
"code": "unmappable-html",
"example": 181
},
{
"code": "unmappable-html",
"example": 182
},
{
"code": "unmappable-html",
"example": 183
},
{
"code": "unmappable-html",
"example": 184
},
{
"code": "unmappable-html",
"example": 185
},
{
"code": "unmappable-html",
"example": 186
},
{
"code": "unmappable-html",
"example": 187
},
{
"code": "unmappable-html",
"example": 188
},
{
"code": "unmappable-html",
"example": 189
},
{
"code": "unmappable-html",
"example": 190
},
{
"code": "unmappable-html",
"example": 191
},
{
"code": "unmappable-html",
"example": 201
},
{
"code": "unmappable-html",
"example": 308
},
{
"code": "unmappable-html",
"example": 309
},
{
"code": "unmappable-html",
"example": 344
},
{
"code": "unmappable-html",
"example": 475
},
{
"code": "unmappable-html",
"example": 476
},
{
"code": "unmappable-html",
"example": 477
},
{
"code": "unmappable-html",
"example": 491
},
{
"code": "unmappable-html",
"example": 494
},
{
"code": "unmappable-image",
"example": 517
},
{
"code": "unmappable-html",
"example": 524
},
{
"code": "unmappable-image",
"example": 531
},
{
"code": "unmappable-html",
"example": 536
},
{
"code": "unmappable-image",
"example": 572
},
{
"code": "unmappable-image",
"example": 573
},
{
"code": "unmappable-image",
"example": 576
},
{
"code": "unmappable-image",
"example": 577
},
{
"code": "unmappable-image",
"example": 579
},
{
"code": "unmappable-image",
"example": 584
},
{
"code": "unmappable-image",
"example": 585
},
{
"code": "unmappable-image",
"example": 586
},
{
"code": "unmappable-image",
"example": 587
},
{
"code": "unmappable-image",
"example": 588
},
{
"code": "unmappable-image",
"example": 589
},
{
"code": "unmappable-image",
"example": 591
},
{
"code": "unmappable-html",
"example": 613
},
{
"code": "unmappable-html",
"example": 614
},
{
"code": "unmappable-html",
"example": 615
},
{
"code": "unmappable-html",
"example": 616
},
{
"code": "unmappable-html",
"example": 617
},
{
"code": "unmappable-html",
"example": 623
},
{
"code": "unmappable-html",
"example": 625
},
{
"code": "unmappable-html",
"example": 626
},
{
"code": "unmappable-html",
"example": 627
},
{
"code": "unmappable-html",
"example": 628
},
{
"code": "unmappable-html",
"example": 629
},
{
"code": "unmappable-html",
"example": 630
},
{
"code": "unmappable-html",
"example": 631
},
{
"code": "unmappable-html",
"example": 642
},
{
"code": "unmappable-html",
"example": 643
}
]
File diff suppressed because it is too large Load Diff
-340
View File
@@ -1,340 +0,0 @@
import assert from 'node:assert/strict'
import { createHash } from 'node:crypto'
import { readFileSync } from 'node:fs'
import { dirname, join } from 'node:path'
import test from 'node:test'
import { fileURLToPath } from 'node:url'
import type { AdfDocument, AdfNode } from './adf/document.ts'
import { adfToMarkdown } from './markdown/emit/adf-to-markdown.ts'
import { markdownToAdf } from './markdown/parse/markdown-to-adf.ts'
const root = join(dirname(fileURLToPath(import.meta.url)), '..', 'corpus', 'commonmark-spec')
const checks = ['count', 'fixpoint', 'text'] as const
type Check = (typeof checks)[number]
type ExceptionKind = 'mark-model' | 'pending' | 'unspellable'
type SpecExample = { example: number; html: string; markdown: string; section: string }
type Exception = { check: Check; divergence: string; example: number; kind: ExceptionKind; reason: string }
type Refusal = { code: string; example: number }
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value)
}
function isCheck(value: unknown): value is Check {
return checks.some((check) => check === value)
}
function isKind(value: unknown): value is ExceptionKind {
return value === 'mark-model' || value === 'pending' || value === 'unspellable'
}
function isSpecExample(value: unknown): value is SpecExample {
if (!isRecord(value)) return false
return typeof value['example'] === 'number' && typeof value['html'] === 'string' && typeof value['markdown'] === 'string' && typeof value['section'] === 'string'
}
function isException(value: unknown): value is Exception {
if (!isRecord(value)) return false
return (
isCheck(value['check']) &&
typeof value['divergence'] === 'string' &&
value['divergence'].length > 0 &&
typeof value['example'] === 'number' &&
isKind(value['kind']) &&
typeof value['reason'] === 'string' &&
value['reason'].length > 0
)
}
function isRefusal(value: unknown): value is Refusal {
if (!isRecord(value)) return false
return typeof value['code'] === 'string' && value['code'].length > 0 && typeof value['example'] === 'number'
}
function readJson<T>(name: string, guard: (value: unknown) => value is T, shape: string): T[] {
const parsed: unknown = JSON.parse(readFileSync(join(root, name), 'utf8'))
assert.ok(Array.isArray(parsed), `${name} is not an array`)
return parsed.map((value, index) => {
assert.ok(guard(value), `${name} holds a ${shape} with the wrong shape at ${index}`)
return value
})
}
const spec = readJson('spec.json', isSpecExample, 'spec example')
const exceptions = readJson('exceptions.json', isException, 'exception')
const refusals = readJson('refusals.json', isRefusal, 'refusal')
const exampleToRefusal = new Map(refusals.map((refusal) => [refusal.example, refusal.code]))
const exceptionIndex = new Map(exceptions.map((entry) => [`${entry.example}:${entry.check}`, entry]))
test('the CommonMark spec suite is 0.31.2, vendored byte-exact', () => {
const digest = createHash('sha256').update(readFileSync(join(root, 'spec.json'))).digest('hex')
assert.equal(digest, 'd431b29d97b6f73e69d547109cf5081578fac931e72afe95639ebe766c1b2a20')
})
test('every exception is unique, names a parsing example, and files a fixpoint only as unspellable', () => {
assert.equal(exceptionIndex.size, exceptions.length, 'one exception repeats an example and check another holds')
for (const entry of exceptions) {
assert.ok(spec.some((candidate) => candidate.example === entry.example), `exception ${entry.example} names no example in the suite`)
assert.equal(exampleToRefusal.get(entry.example), undefined, `exception ${entry.example} is on the refusal list, not an exception`)
if (entry.check === 'fixpoint') assert.equal(entry.kind, 'unspellable', `exception ${entry.example} files a fixpoint divergence as ${entry.kind}; a fixable hole is given the spelling instead`)
}
})
test('the refusal list is unique per example and names real examples', () => {
assert.equal(exampleToRefusal.size, refusals.length, 'one refusal repeats an example another holds')
for (const example of exampleToRefusal.keys()) assert.ok(spec.some((entry) => entry.example === example), `refusal ${example} names no example in the suite`)
})
// A mark is counted once per text node it touches (AGENTS.md §14).
const countKeys = ['a', 'blockquote', 'br', 'code', 'em', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'hr', 'img', 'li', 'ol', 'pre', 'strong', 'ul']
const nodeElement: Record<string, string> = {
blockquote: 'blockquote',
bulletList: 'ul',
codeBlock: 'pre',
hardBreak: 'br',
listItem: 'li',
media: 'img',
mediaInline: 'img',
orderedList: 'ol',
rule: 'hr',
}
const markElement: Record<string, string> = { code: 'code', em: 'em', link: 'a', strong: 'strong' }
const blockTags = new Set(['blockquote', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'hr', 'li', 'ol', 'p', 'pre', 'ul'])
function tagName(tag: string): string {
return tag.slice(1).replace(/^\//, '').split(/[\s/>]/)[0] ?? ''
}
function emptyCounts(): Record<string, number> {
return Object.fromEntries(countKeys.map((key) => [key, 0]))
}
function referenceCounts(html: string): Record<string, number> {
const counts = emptyCounts()
let inPre = false
for (let index = 0; index < html.length; index += 1) {
if (html[index] !== '<') continue
const close = html.indexOf('>', index)
if (close === -1) break
const tag = html.slice(index, close + 1)
if (tag.startsWith('</')) {
if (tagName(tag) === 'pre') inPre = false
index = close
continue
}
const name = tagName(tag)
if (name === 'pre') {
inPre = true
counts['pre'] = (counts['pre'] ?? 0) + 1
} else if (name === 'code' && inPre) {
// A code block's `<code>` is the `<pre>`'s body, already counted.
} else if (countKeys.includes(name)) {
counts[name] = (counts[name] ?? 0) + 1
}
index = close
}
return counts
}
function nodeCounts(document: AdfNode): Record<string, number> {
const counts = emptyCounts()
const pending: AdfNode[] = [document]
while (pending.length > 0) {
const node = pending.pop()
if (node === undefined) continue
if (node.text !== undefined) {
const seen = new Set<string>()
for (const mark of node.marks ?? []) {
const element = markElement[mark.type]
if (element !== undefined) seen.add(element)
}
for (const element of seen) counts[element] = (counts[element] ?? 0) + 1
continue
}
if (node.type === 'heading') {
const level = node.attrs?.['level']
if (typeof level === 'number') counts[`h${level}`] = (counts[`h${level}`] ?? 0) + 1
pending.push(...(node.content ?? []))
continue
}
const element = nodeElement[node.type]
if (element !== undefined) counts[element] = (counts[element] ?? 0) + 1
pending.push(...(node.content ?? []))
}
return counts
}
const namedEntity: Record<string, string> = { amp: '&', gt: '>', lt: '<', ouml: 'ö', quot: '"' }
function decodeHtmlEntity(text: string, index: number): { length: number; text: string } | undefined {
if (text[index] !== '&') return undefined
const end = text.indexOf(';', index)
if (end === -1 || end - index > 8) return undefined
const reference = text.slice(index, end + 1)
const named = namedEntity[reference.slice(1, -1)]
return named === undefined ? undefined : { length: reference.length, text: named }
}
test('the oracle decodes every entity the reference HTML holds', () => {
for (const example of spec) {
for (const [reference] of example.html.matchAll(/&#?[0-9A-Za-z]+;/g)) {
assert.ok(decodeHtmlEntity(reference, 0) !== undefined, `example ${example.example} holds ${reference}, which the oracle would leave literal`)
}
}
})
function referenceText(html: string): string {
const parts: string[] = []
let preDepth = 0
let atBoundary = true
let skipNewline = false
for (let index = 0; index < html.length; index += 1) {
const character = html.charAt(index)
if (character === '<') {
const close = html.indexOf('>', index)
if (close === -1) break
const tag = html.slice(index, close + 1)
const name = tagName(tag)
if (name === 'br') {
parts.push(' ')
atBoundary = false
skipNewline = true
index = close
continue
}
if (name === 'pre') {
if (tag.startsWith('</')) {
preDepth -= 1
trimTrailingNewline(parts)
} else {
preDepth += 1
}
atBoundary = true
} else {
atBoundary = blockTags.has(name)
}
index = close
continue
}
if (character === '\n') {
if (skipNewline) {
skipNewline = false
continue
}
if (preDepth > 0) {
parts.push('\n')
continue
}
if (!atBoundary && !followedByBlock(html, index + 1)) parts.push(' ')
continue
}
const reference = decodeHtmlEntity(html, index)
if (reference !== undefined) {
parts.push(reference.text)
atBoundary = false
index += reference.length - 1
continue
}
parts.push(character)
atBoundary = false
}
return parts.join('')
}
function trimTrailingNewline(parts: string[]): void {
const last = parts[parts.length - 1]
if (last === undefined) return
parts[parts.length - 1] = last.endsWith('\n') ? last.slice(0, -1) : last
}
// A newline beside a block open or close is a boundary rather than a soft break, so it spells no space.
function followedByBlock(html: string, index: number): boolean {
let next = index
while (next < html.length && (html[next] === '\n' || html[next] === ' ' || html[next] === '\t')) next += 1
if (next >= html.length) return true
if (html[next] !== '<') return false
const close = html.indexOf('>', next)
return close !== -1 && blockTags.has(tagName(html.slice(next, close + 1)))
}
function concatenatedText(document: AdfNode): string {
const parts: string[] = []
const pending: { inCode: boolean; node: AdfNode }[] = [{ inCode: false, node: document }]
while (pending.length > 0) {
const frame = pending.pop()
if (frame === undefined) continue
const { inCode, node } = frame
if (node.text !== undefined) {
parts.push(inCode ? node.text : node.text.replace(/\n/g, ' '))
continue
}
if (node.type === 'hardBreak') {
parts.push(' ')
continue
}
const childInCode = inCode || node.type === 'codeBlock'
const content = node.content ?? []
for (let index = content.length - 1; index >= 0; index -= 1) {
const child = content[index]
if (child !== undefined) pending.push({ inCode: childInCode, node: child })
}
}
return parts.join('')
}
function fixpointRefused(example: SpecExample, document: AdfDocument): string | undefined {
const emitted = adfToMarkdown(document)
if (!emitted.ok) return emitted.error.code
const again = markdownToAdf(emitted.value)
assert.ok(again.ok, `example ${example.example} emits markdown it cannot read back`)
assert.deepEqual(again.value, document, `example ${example.example} does not hold its own round-trip`)
return undefined
}
function textMismatch(example: SpecExample, document: AdfDocument): string | undefined {
const expected = referenceText(example.html)
const actual = concatenatedText(document)
return expected === actual ? undefined : `${JSON.stringify(expected)} against ${JSON.stringify(actual)}`
}
function countMismatch(example: SpecExample, document: AdfDocument): string | undefined {
const expected = referenceCounts(example.html)
const actual = nodeCounts(document)
const names = countKeys.filter((key) => expected[key] !== actual[key])
return names.length === 0 ? undefined : names.map((name) => `${name} ${expected[name]}/${actual[name]}`).join(' ')
}
for (const example of spec) {
test(`CommonMark example ${example.example} => ${example.section}`, () => {
const parse = markdownToAdf(example.markdown)
const refused = exampleToRefusal.get(example.example)
if (refused !== undefined) {
assert.ok(!parse.ok, `example ${example.example} was expected to refuse with ${refused} but parsed`)
assert.equal(parse.error.code, refused, `example ${example.example} refused with a different code`)
return
}
if (!parse.ok) assert.fail(`example ${example.example} was expected to parse but refused with ${parse.error.code}`)
const divergences: Record<Check, string | undefined> = {
count: countMismatch(example, parse.value),
fixpoint: fixpointRefused(example, parse.value),
text: textMismatch(example, parse.value),
}
for (const check of checks) {
const entry = exceptionIndex.get(`${example.example}:${check}`)
const divergence = divergences[check]
if (divergence === undefined) {
assert.equal(entry, undefined, `example ${example.example} passes its ${check} check but files an exception`)
} else {
assert.ok(entry !== undefined, `example ${example.example} ${check} check fails: ${divergence}`)
assert.equal(entry.divergence, divergence, `example ${example.example} ${check} diverged differently than filed`)
}
}
})
}
+2 -3
View File
@@ -1,6 +1,6 @@
import assert from 'node:assert/strict'
import { readFileSync, readdirSync } from 'node:fs'
import { basename, dirname, join, sep } from 'node:path'
import { basename, dirname, join } from 'node:path'
import test from 'node:test'
import { fileURLToPath } from 'node:url'
@@ -53,13 +53,12 @@ function pairedNames(root: string, first: string, second: string): string[] {
function corpusJsonPaths(): string[] {
return readdirSync(corpusRoot, { encoding: 'utf8', recursive: true })
.filter((name) => name.endsWith('.json'))
.filter((name) => name !== `commonmark-spec${sep}spec.json`)
.map((name) => join(corpusRoot, name))
.sort()
}
test('every corpus directory is a kind the runner reads', () => {
assert.deepEqual(directoryNames(corpusRoot), ['commonmark-spec', 'errors', 'normalization', 'round-trip'])
assert.deepEqual(directoryNames(corpusRoot), ['errors', 'normalization', 'round-trip'])
})
test('every round-trip directory is a kind the runner reads', () => {
+11 -62
View File
@@ -5,8 +5,8 @@ milestone. A done item shrinks to its title here; its full text moves to `todo-h
## Milestones
Shipping order: 3h, 3i, 3j, 5a, 5b, 5c, 5d, 5 → `0.1.0` (shipped 2026-09-05); 5e before 2027-01; 3k, 4, 4b, 4c, 5g, 10, 11, 12`0.2.0`; 4d, 5f`0.2.1`;
6, 7 → `0.3.0`; 9 → TBD.
Shipping order: 3h, 3i, 3j, 5a, 5b, 5c, 5d, 5 → `0.1.0` (shipped 2026-09-05); 5e before 2027-01; 4b, 4c and 4d`0.1.1`; 4, 3k`0.2.0`;
6, 7 → `0.3.0`; 9, 10, 11 → TBD.
The numbering is the order the work was planned in, not the order it ships.
- [x] **0 — Scaffold.**
@@ -53,7 +53,7 @@ The numbering is the order the work was planned in, not the order it ships.
- [x] **3i — The inline nodes and the marks.**
- [x] **3j — The carry and the combinations.**
- [ ] **3k — The CommonMark spec suite (`0.2.0`).** Checked in at `corpus/commonmark-spec/`,
pinned to the version it ships — the one `commonmark-grammar.ts` names for its start
pinned to the version it ships — the one `html-blocks.ts` names for its start
conditions — `corpus/README.md` gaining the kind.
**Settled** (the maintainer, 2026-08-27): three checks an example must pass, the reference
HTML each ships read as corpus data — which adds no format and no direction (§1). §2's
@@ -67,11 +67,9 @@ The numbering is the order the work was planned in, not the order it ships.
counts close both. The exception list stays the maintainer's, and one entry is owed
already: 3h continues a list across the marker change CommonMark splits on, so an example
the reference HTML gives two `<ul>` counts one `bulletList`. One outcome is no
exception and must not be filed as one: a fixable §2 hole — valid CommonMark parsing to a
document `adfToMarkdown` refuses which is what `corpus/unspellable/` held until 3c, 3e
and 3h landed their answers and emptied it. The permanent ones — a link destination or
title no escape spells, a paragraph opening with a code span — are the exceptions, named
by AGENTS.md §2.
exception and must not be filed as one: valid CommonMark parsing to a document
`adfToMarkdown` refuses is a §2 hole, which is what `corpus/unspellable/` held until 3c,
3e and 3h landed their answers and emptied it.
- [ ] **4 — Round-trip property tests (`0.2.0`)**, widening 3j's corpus round-trip past the
documents a human wrote — the thing that proves 2 and 3 beyond them. Editor-normal (§2) is
finished here, on 3i's merging — `toEditorNormal(doc)` and the equality the round-trip
@@ -86,7 +84,7 @@ The numbering is the order the work was planned in, not the order it ships.
a document that round-trips proves no other document shares its spelling — so decide here
whether that gate stays as the parser-free, faster-failing signal or goes; the half holding
no fixture duplicates is hygiene rather than a round-trip claim, and stays either way.
- [ ] **4b — The block walk's retry (`0.2.0`).** `emitBlock` walks a subtree twice wherever
- [ ] **4b — The block walk's retry (`0.1.1`).** `emitBlock` walks a subtree twice wherever
`readableBlock` reads it whole and then gives up — a list item whose first line reads back
as a thematic break — and the walk below does the same, so the cost doubles per level:
3.4kB of nested lists takes half a second, depth 20 about eight, depth 24 minutes. It
@@ -102,7 +100,7 @@ The numbering is the order the work was planned in, not the order it ships.
export persona runs in bulk walks the document twice. Both walks are linear, so this is a
constant factor rather than 4b's class change, and the parting is what gives depth its own
code (§8) — measure before joining them back.
- [ ] **4c — The scanning rule's remaining sites (`0.2.0`).** A trailing-anchored regex re-walks
- [ ] **4c — The scanning rule's remaining sites (`0.1.1`).** A trailing-anchored regex re-walks
its run from every start position, so an interior whitespace run costs quadratic time rather
than linear — 3h measured 80k spaces inside an ATX heading at 11.3s, and 3ms once the walk
replaced the regex. Three sites the same sweep did not reach: `normalizeLabel` in
@@ -114,7 +112,7 @@ The numbering is the order the work was planned in, not the order it ships.
cost, which 3i's slot parse doubles rather than changes in class, bounded by the 500-level
guard. §11's scanning rule is the whole argument; the pipeline persona feeds documents
nobody typed.
- [ ] **4d — What the gate says while it runs (`0.2.1`).** `ci.sh` runs nine legs and announces
- [ ] **4d — What the gate says while it runs (`0.1.1`).** `ci.sh` runs nine legs and announces
none of them, so five minutes of a Gitea run read as silence and a hang cannot be told from
a slow pull — the maintainer hit exactly this on the `0.1.0` release. Three causes, each its
own fix. The legs need markers: `plainpages`' `ci.sh` prints a `step()` header per leg and
@@ -138,25 +136,6 @@ The numbering is the order the work was planned in, not the order it ships.
the deadline: whether npm has added Gitea or self-hosted OIDC, and otherwise whether the
release moves to a human-approved staged publish — which fits badly with publish-on-merge,
and is the trade to weigh rather than discover on a red release run.
- [ ] **5f — Publish the bundle size (`0.2.1`).** Measure the shipped artifact and put the number in the
README, kept honest by the release pipeline rather than by a human re-reading it. The
quantity is what a consumer downloads and loads: the tarball `npm pack` produces, its
unpacked `dist`, and the built JavaScript minified + gzipped — the figure the competitors
advertise (marklassian's "12kb") and the only apple-to-apple one, since ours ships tsc's
unminified output and no minifier yet (decide here whether to minify for the build or report
the unminified gzip). A publish/pipeline leg measures it and fails when the README figure
drifts, so the number can't rot; the figure lands in README §The package beside the
"no runtime dependencies" claim. Measured today, unminified: tarball 60.4 kB, unpacked
221.5 kB, JS gzipped 45.6 kB.
- [ ] **5g — Reweight the README for the reader (`0.2.0`).** It opens with the pre-launch rationale —
Atlassian's REST APIs, `pf-editor-service/convert` being decommissioned, a link to
JRACLOUD-77436 — where a shipped package should answer what it is, what it does and for whom
first, then the shortest runnable example; the reader's top seconds go to "why this exists"
instead of "what I can do with it". Demote the Jira/endpoint background to a later "why
losslessness" note or drop it — the internal references (the `jira.atlassian.com` URL,
`pf-editor-service/convert`) don't belong in published text at all, no ticket IDs or internal
URLs. The `0.3.0` HTML future should read as an aside, not the lede: the package reads as a
shipped `0.1.0`, not a work-in-progress.
- [x] **5a — Rename to `@larvit/adf-codec`.**
- [x] **5b — The consumer's error surface.**
- [x] **5b1 — The error's source position.**
@@ -171,38 +150,8 @@ The numbering is the order the work was planned in, not the order it ships.
`htmlToMarkdown`. CommonMark spec suite runs against `markdownToHtml` from here (§10).
- [ ] **8 — CLI.** A later goal, shaped around the personas once the library exists.
- [ ] **9 — The online sandbox.** A web page with two textboxes converting back and forth between ADF and markdown, powered by the library's browser build.
- [ ] **10 — Lossy conversion (`0.2.0`).** A direction that only converts what Markdown actually supports, keeping the ADF's data while dropping what markdown cannot hold — format, design and the richer nodes.
- [ ] **11 — Evaluate `@atlaskit/adf-schema` (`0.2.0`).** Whether to add `@atlaskit/adf-schema` as a dev dependency to use as truth for the ADF schema.
- [ ] **12 — The `!adf:` re-spelling (`0.2.0`).** Replace the colon directive grammar with the
namespaced prefix, a breaking change to the emitted contract (shipped `0.1.0`, so §8 makes it
`0.2.0`). Forms: block container `!adf:name arg {attrs}``!adf:/name` — the `/` parts open
from close, nestable without a fence-length discipline, so the `::::`/`:::::` runs and their
length rule go and every container opens the constant `!adf:`; block leaf `!adf:name arg
{attrs}` with no closer; inline node `!adf:name[content]{attrs}`; directive marks
`!adf:border`/`subsup`/`textColor`/`underline` `[content]{attrs}`. Attributes and their
escaping stay `{key=value}`; the literal escape is `\!adf:`; a line opening `!adf:` claims as
today's colon-run does. Leaf vs container is decided by the node's content model rather than
syntax — the `::`/`:::` split and §4's name-set-independent recognition go, a simplification
the carry makes safe (an unknown *block* node already rides the fence, not the directive).
The carry's reserved name becomes `carry`, both spellings — the block fence info string
`` `carry` `` and the inline `!adf:carry{json="…"}` — named for what it does: it carries a node
verbatim, never "unknown-node", since a known node no section spells where it stands rides it
too. A spelling change, not a semantic one: no `ConvertErrorCode` is added, removed or renamed,
the round-trip guarantee and the carry both hold through it. Mechanical surface: the grammar in
`spec/flavour.md`, `src/adf/block-directives.ts` + `inline-directives.ts`, `src/markdown/`'s
`directive-syntax.ts`, `opaque-carry.ts` and the `emit/` + `parse/` readers, every corpus
fixture (round-trip, normalization and `errors/`), `spec.test.ts`'s prose reader, and the
README's examples.
- [ ] **12a — The spec and the decision.** Rewrite `spec/flavour.md` to the `!adf:` grammar, and
record the departures in `AGENTS.md` §4 (leaf/container by content model, carry renamed
`carry`).
- [ ] **12b — The emit side.** `adfToMarkdown` spells `!adf:` / `!adf:/name` / `!adf:carry`; its
fixtures re-spelled, green.
- [ ] **12c — The parse side and the round-trip.** `markdownToAdf` reads it back; the round-trip
corpus, the `errors/` fixtures and the CommonMark spec suite re-spelled,
`markdownToAdf(adfToMarkdown(doc))` still equals `doc`.
- [ ] **12d — The README and the sweep.** The README's examples follow; sweep docs and fixtures
for any stale `::`/`:name` spelling.
- [ ] **10 — Lossy conversion.** A direction that only converts what Markdown supports. Data from the ADF is preserved but format, design and other non-markdown constructs are dropped.
- [ ] **11 — Evaluate `@atlaskit/adf-schema`.** Determine whether `@atlaskit/adf-schema` should be added as a dev dependency to serve as truth for the ADF schema.
## The ADF inventory to cover