diff --git a/CHANGELOG.md b/CHANGELOG.md index e9faf8b..9e1caa0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -83,13 +83,14 @@ - Fix a cyclic document, in which an object holds itself: `adfToLosslessMarkdown` and `adfToPortableMarkdown` refuse it as `not-an-adf-document` and `isAdfDocument` returns `false`, where all three hung. -- Fix a document whose markdown would be longer than 536,870,888 characters, V8's longest string: - `adfToLosslessMarkdown` and `adfToPortableMarkdown` refuse it as `unsupported-node-shape` on - every engine, where they threw or ran out of memory. -- Fix a directive that, written out as plain CommonMark, would be that long: `losslessMarkdownToAdf` - refuses it as `unsupported-node-shape`, where a few megabytes of nested list lines ran it out of - memory. The markdown readers return `unsupported-node-shape` where they threw past the longest - string or array the JavaScript engine builds. +- Fix a document whose markdown would be longer than 536870888 characters, V8's longest string: + `adfToLosslessMarkdown` and `adfToPortableMarkdown` refuse it as `unsupported-node-shape` on every + engine, where they threw or ran out of memory. +- Fix a directive that, written out as plain CommonMark, would be longer than 536870888 characters: + `losslessMarkdownToAdf` refuses it as `unsupported-node-shape`, where a few megabytes of nested + list lines ran the call out of memory. +- Fix the markdown readers past the longest string or array the JavaScript engine builds: they + return `unsupported-node-shape` where they threw. - Fix deeply nested compact JSON in an `adf:` fence: `losslessMarkdownToAdf` refuses it as `unsupported-node-shape`, as it refuses any compact JSON there, where about a megabyte of it threw. - Add `adfToPortableMarkdown` and `portableMarkdownToAdf`, a lossy pair converting ADF to and from diff --git a/README.md b/README.md index c596ee3..25c536c 100644 --- a/README.md +++ b/README.md @@ -410,9 +410,9 @@ typed on `ConvertError` takes both. A call reports the first refusal in document order and stops, so a document with several surfaces them one per call. Every refusal is deterministic — there is no I/O anywhere — so a retry returns -the identical error: fix the input, or set the document aside. The one exception is JavaScriptCore -(Bun, Safari), which reports a heap running short as it reports a string past its longest: there, -retry once a call's `unsupported-node-shape` at the empty path. +the identical error: fix the input, or set the document aside. On JavaScriptCore (Bun, Safari), +retry once any call that returns `unsupported-node-shape` at the empty path: that engine reports a +heap running short the same way as a string too long to build. ### Parsing @@ -445,7 +445,7 @@ The emitter raises these, and so does a parse, which asks the emitter how to spe | Code | Fires when | What you can do | | --- | --- | --- | | `unsupported-nesting-depth` | blocks, marks, an attribute's JSON or a carried node's JSON nest past 500 levels | keep the ADF and pass the document over, or show it read-only; flatten the input where you are the one who wrote it | -| `unsupported-node-shape` | parsing, in `losslessMarkdownToAdf`: markdown spells a node with an attribute, value, argument or body its type does not take, or without one it needs, wraps a directive mark around nothing or only hard breaks, writes as a directive a node or mark the lossless flavour spells as CommonMark, or puts a reserved directive out of place: `!adf:textBreak{}` parting nothing, `!adf:doc` anywhere but as the whole document, or writes as a directive a node that, written out as plain CommonMark, would be longer than 536870888 characters. Emitting, in `adfToLosslessMarkdown`: a node not riding the carry that carries one mark type twice, a block or a non-text inline node holding `text`, or a leaf block or a non-text inline node holding `content`. Emitting, in `adfToPortableMarkdown`: a line or task list the portable writer cannot spell, a bug in this library. Emitting, in both writers: a document whose markdown would be longer than 536870888 characters, V8's longest string, on every engine; in `adfToPortableMarkdown`, also a document whose text alone is that long, edge whitespace included. In every call: a conversion that passes the longest string or array the JavaScript engine builds | fix what the message names; [`spec/flavour.md`](https://github.com/larvit/adf-codec/blob/main/spec/flavour.md) lists every type's attributes and body. For a portable line or task list, report it as a bug. Past 536870888 characters or the engine's limit, writing: set the document aside, keeping the ADF and passing it over or showing it read-only; reading: shorten the markdown, or nest its lists and quotes less deeply | +| `unsupported-node-shape` | parsing, in `losslessMarkdownToAdf`: markdown spells a node with an attribute, value, argument or body its type does not take, or without one it needs, wraps a directive mark around nothing or only hard breaks, writes as a directive a node or mark the lossless flavour spells as CommonMark, or one that, written out as plain CommonMark, would be longer than 536870888 characters, or puts a reserved directive out of place: `!adf:textBreak{}` parting nothing, `!adf:doc` anywhere but as the whole document. Emitting, in `adfToLosslessMarkdown`: a node not riding the carry that carries one mark type twice, a block or a non-text inline node holding `text`, or a leaf block or a non-text inline node holding `content`. Emitting, in `adfToPortableMarkdown`: a line or task list the portable writer cannot spell, a bug in this library. Emitting, in both writers: a document whose markdown would be longer than 536870888 characters, V8's longest string, on every engine; in `adfToPortableMarkdown`, also a document whose text alone is that long, edge whitespace included. In every call: a conversion that passes the longest string or array the JavaScript engine builds | fix what the message names; [`spec/flavour.md`](https://github.com/larvit/adf-codec/blob/main/spec/flavour.md) lists every type's attributes and body. For a portable line or task list, report it as a bug. Past 536870888 characters or the engine's limit, writing: set the document aside, keeping the ADF and passing it over or showing it read-only; reading: shorten the markdown, or nest its lists and quotes less deeply | ## The guarantees @@ -459,7 +459,8 @@ Serves [Goals](#goals) 1, 4 and 5. - Markdown this library reads, and markdown it writes, means what the CommonMark spec says. The bullets below name every exception. - CommonMark is valid input to `commonMarkToAdf`, apart from the raw HTML `unmappable-html` names, - nesting past 500 levels, and input past what the JavaScript engine holds. + nesting past 500 levels, and input whose conversion passes the longest string or array the + JavaScript engine builds. - The other readers claim shapes CommonMark reads as text or code. `losslessMarkdownToAdf` claims four: literal text matching directive, pipe-table or strikethrough syntax, and a code fence whose info string opens `adf:`. `portableMarkdownToAdf` claims what [Portable diff --git a/docs/decisions.md b/docs/decisions.md index 9be0e21..25b43f2 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -617,9 +617,9 @@ builder as a gain. The scores at be23835 were Junior 6.5, Mid 6, Senior 6 and Ar 6.13. `todo.md` item 102 is to place that path without the replay. Item 106, 2026-10-13, shipped after three rounds under a worse ruling, the maintainer's call: both paired seats read the overall as the same, but Locality fell to a mean of 5.13, laid on the spelling memo the chunk splits between -the CommonMark ask, which fills it, and the walk, which drops each entry as it reads it. The scores -at 1f4551a were Junior 5.5, Mid 5.5, Senior 6 and Architect 5.5. `todo.md` item 110 is to fill and -read the memo in one place. +the CommonMark ask, which fills it, and the walk, which drops each entry as it reads it. The overall +scores at 1f4551a were Junior 5.5, Mid 5.5, Senior 6 and Architect 5.5, mean 5.63. +`todo.md` item 110 is to fill and read the memo in one place. ## Properties on a fixed seed @@ -669,22 +669,20 @@ or pins it. 2026-10-11, the maintainer. Goals 1 and 6. Valid while V8's longest string, 2^29 − 24 characters, is the shortest of the engines the library claims. -Every writer, and the reader's CommonMark ask, refuses markdown longer than 2^29 − 24 characters -(`longestMarkdown`) as `unsupported-node-shape` at the document's path, so the same document -converts alike everywhere. A writer first refuses a document whose text alone passes the bound: the -lossless writer spells every character of it, and the portable reduction refuses on the caller's -text, edge whitespace and all, since merging that text first could build past the bound. The -builders join plain strings, and each block, the writer's result and the CommonMark ask's spelling -are held to the bound as they are built. V8 throws building the same string past the bound, -SpiderMonkey past its own longest string, and `withinEngineLimits` refuses both throws as the same -`tooLong`, so every engine stops at the same block with the same answer. The carry's two-space JSON, -which indents every value its depth over, stops at the bound unbuilt (`joinedWithin`). A deep list -or quote re-indents its whole text at each level before the bound refuses it (`todo.md` item 109). +Every writer, and `losslessMarkdownToAdf`'s `commonMarkSpelling` ask, refuses markdown longer than +2^29 − 24 characters (`longestMarkdown`), so the same document converts alike everywhere. A writer +first refuses a document whose text alone passes the bound: the lossless writer spells every +character of it, and the portable reduction refuses on the caller's text, edge whitespace and all, +since merging that text first could build past the bound. Each block, the writer's result and the +CommonMark ask's spelling are held to the bound as they are built. V8 throws building the same +string past the bound, SpiderMonkey past its own longest string, and `withinEngineLimits` refuses +both throws as the same `tooLong`, so every engine stops at the same block with the same answer. The +carry's two-space JSON, which indents every value its depth over, stops at the bound unbuilt +(`joinedWithin`). -JavaScriptCore builds a joined block up to its own longest string, 2^31 − 1, before the check. A -throw past that, past any engine's longest array, or JavaScriptCore's `Out of memory`, which also -reports a heap running short, is refused with the engine's message. `todo.md` item 108 bounds the -line's own memory. +JavaScriptCore builds a joined block up to its own longest string, 2^31 − 1, before the check. Past +that, JavaScriptCore throws `Out of memory`, as it does when its heap runs short. That throw, and +any engine's throw past its longest array, is refused with the engine's message. ## Nothing recurses unbounded @@ -750,15 +748,15 @@ two cannot disagree — which is what makes the kept value a memo rather than a 2026-09-19, the maintainer. Goal 9. Valid while the `commonMarkSpelling` ask spells a node once per level above it otherwise. -The parse and the portable reduction keep the readable spelling of each node they put the -`commonMarkSpelling` ask to in a memo, so the ask stops spelling a node once per level above it. A -walk that reads an entry drops it, and a walk fills none for the nodes below the asked one: the walk -above spells them into its own text, and kept, every level of a deep subtree would hold its whole -text (2026-10-11). `text`, `alternateText` and `spelling` carry no depth and `headroom` is affine in -it, so a read at or above the depth that filled the entry rebases; a read below re-spells, because a -hit skips the depth guards the walk it replaces runs and an ordered list past the marker cap gives -way, spending two emitter levels where the parser spent one. Only what succeeded is kept, so no path -minted at another position is ever read. +When the parse or the portable reduction asks `commonMarkSpelling` about a node, the node's readable +spelling goes into a memo, so the ask stops spelling a node once per level above it. Reading an +entry drops it, and no entry is kept for the nodes below the asked one: the walk above spells those +nodes into its own text, so keeping them would make every level of a deep subtree hold the subtree's +whole text (2026-10-11). `text`, `alternateText` and `spelling` carry no depth and `headroom` is +affine in it, so a read at or above the depth that filled the entry rebases; a read below re-spells, +because a hit skips the depth guards the walk it replaces runs and an ordered list past the marker +cap gives way, spending two emitter levels where the parser spent one. Only what succeeded is kept, +so no path minted at another position is ever read. ## Cost fixes are measured, never timed diff --git a/src/markdown/emit/adf-to-markdown.test.ts b/src/markdown/emit/adf-to-markdown.test.ts index e976019..2150f70 100644 --- a/src/markdown/emit/adf-to-markdown.test.ts +++ b/src/markdown/emit/adf-to-markdown.test.ts @@ -44,7 +44,7 @@ test('refuses a value that is not an ADF document, naming the check it failed', assert.equal(markdown(adfToLosslessMarkdown({ type: 'doc', version: Number.NaN })), 'not-an-adf-document: an ADF document holds a version number: found NaN') }) -const tooLong = 'unsupported-node-shape: spelled as markdown, this would be longer than 536870888 characters, the longest markdown this library writes' +const tooLong = 'unsupported-node-shape: the markdown for this document, or for a directive\'s plain CommonMark, would be longer than 536870888 characters, the longest this library writes' // A code block of 300 lines; two of these together fall some 1000 characters short of V8's longest string, so only what their spelling adds passes it. function nearlyHalfTheBound(): AdfNode { @@ -59,7 +59,7 @@ test('refuses a carry whose JSON would run past V8\'s longest string, on every e assert.equal(markdown(result), tooLong) }) -test('refuses lines their quotes would prefix past V8\'s longest string', () => { +test('refuses a quote whose prefixed lines would pass V8\'s longest string', () => { const code = nearlyHalfTheBound() assert.equal(markdown(adfToLosslessMarkdown(document({ content: [code, code], type: 'blockquote' }))), tooLong) }) diff --git a/src/markdown/emit/adf-to-markdown.ts b/src/markdown/emit/adf-to-markdown.ts index d747818..1c2b616 100644 --- a/src/markdown/emit/adf-to-markdown.ts +++ b/src/markdown/emit/adf-to-markdown.ts @@ -62,7 +62,7 @@ function walkBlocks(nodes: readonly AdfNode[], path: ConvertErrorPath, depth: nu for (const [index, node] of nodes.entries()) { const block = emitBlock(node, [...path, 'content', index], depth, writing) if (!block.ok) return block - // V8 throws building a block past the bound; an engine holding longer strings stops at the same block. + // Refused here, so an engine holding longer strings stops at the block where V8 throws. const fits = withinLongest(block.value.text) if (!fits.ok) return fits headroom = Math.min(headroom, block.value.headroom) @@ -136,7 +136,7 @@ function readableBlock(node: AdfNode, path: ConvertErrorPath, depth: number, wri return spellReadableBlock(node, path, depth, writing) } -// The CommonMark ask fills an entry; the walk above reads it once and spells it into its own text, so kept, every level of a deep subtree would hold its whole text. +// docs/decisions.md §The spelling memo function takeKeptSpelling(memo: SpellingMemo | undefined, node: AdfNode): KeptSpelling | undefined { const kept = memo?.get(node) memo?.delete(node) diff --git a/src/markdown/length-bound.test.ts b/src/markdown/length-bound.test.ts index a1b571d..cd8b501 100644 --- a/src/markdown/length-bound.test.ts +++ b/src/markdown/length-bound.test.ts @@ -27,7 +27,7 @@ test('refuses a string past the bound as too long on every engine, and names any assert.deepEqual(withinEngineLimits(thrown('RangeError', 'Invalid string length')), tooLong()) assert.deepEqual(withinEngineLimits(thrown('InternalError', 'allocation size overflow')), tooLong()) assert.deepEqual(withinEngineLimits(thrown('RangeError', 'Out of memory')), { - error: { code: 'unsupported-node-shape', message: 'the output would be longer than the longest string or array this JavaScript engine builds (Out of memory)', path: [] }, + error: { code: 'unsupported-node-shape', message: 'the conversion passed this JavaScript engine\'s longest string or array, or its heap (Out of memory)', path: [] }, ok: false, }) assert.equal(withinEngineLimits(thrown('RangeError', 'Invalid array length')).ok, false) diff --git a/src/markdown/length-bound.ts b/src/markdown/length-bound.ts index b014528..62cc2f5 100644 --- a/src/markdown/length-bound.ts +++ b/src/markdown/length-bound.ts @@ -1,14 +1,14 @@ import { failure, success, type Result } from '../result.ts' -// V8's longest string, held on every engine so a document converts alike everywhere (docs/decisions.md §Markdown past V8's longest string is refused on every engine). +// V8's longest string (docs/decisions.md §Markdown past V8's longest string is refused on every engine). export const longestMarkdown = 2 ** 29 - 24 -// Measured on Firefox 155, Node 24 and Bun 1.4. SpiderMonkey's overflow and V8's string overflow both mean a string past longestMarkdown; the other two name the engine's own limit. +// Measured on Firefox 155, Node 24 and Bun 1.4. SpiderMonkey's overflow and V8's string overflow both mean a string past longestMarkdown; the other two name an engine's own limit, and JavaScriptCore's also a heap running short. const pastLongestString = ['InternalError: allocation size overflow', 'RangeError: Invalid string length'] const pastEngineLimit = ['RangeError: Invalid array length', 'RangeError: Out of memory'] export function tooLong(): Result { - return failure('unsupported-node-shape', `spelled as markdown, this would be longer than ${longestMarkdown} characters, the longest markdown this library writes`, []) + return failure('unsupported-node-shape', `the markdown for this document, or for a directive's plain CommonMark, would be longer than ${longestMarkdown} characters, the longest this library writes`, []) } export function withinLongest(text: string): Result { @@ -31,11 +31,11 @@ export function withinEngineLimits(convert: () => Result): Result { try { return convert() } catch (error) { - // A stack overflow is a bug (docs/decisions.md §Nothing recurses unbounded), so it stays a throw. if (!(error instanceof Error)) throw error const thrown = `${error.name}: ${error.message}` if (pastLongestString.includes(thrown)) return tooLong() + // A stack overflow is a bug (docs/decisions.md §Nothing recurses unbounded), so it stays a throw. if (!pastEngineLimit.includes(thrown)) throw error - return failure('unsupported-node-shape', `the output would be longer than the longest string or array this JavaScript engine builds (${error.message})`, []) + return failure('unsupported-node-shape', `the conversion passed this JavaScript engine's longest string or array, or its heap (${error.message})`, []) } } diff --git a/src/markdown/opaque-carry.ts b/src/markdown/opaque-carry.ts index 481c3a3..27dc267 100644 --- a/src/markdown/opaque-carry.ts +++ b/src/markdown/opaque-carry.ts @@ -59,7 +59,7 @@ function carriedJson(node: AdfNode, spelled: object, spelling: JsonSpelling, pat if (!isJsonValue(spelled) || headroom < 0) { return failure('unsupported-nesting-depth', `a carried node's JSON nests deeper than the ${levels} levels its position leaves`, path) } - // Two-space JSON indents every value its depth over, so it stops at the bound unbuilt. + // Two-space JSON indents each value by its depth, so a short compact value can spell past the bound: join only up to it. const json = joinedWithin(canonicalJsonPieces(spelled, spelling)) return json.ok ? success({ headroom, json: json.value }) : json } diff --git a/src/markdown/parse/markdown-to-adf.ts b/src/markdown/parse/markdown-to-adf.ts index 80f00df..a7a6bdf 100644 --- a/src/markdown/parse/markdown-to-adf.ts +++ b/src/markdown/parse/markdown-to-adf.ts @@ -262,7 +262,7 @@ function directiveNode(block: DirectiveBlock, reading: Reading, path: ConvertErr if (!read.ok) return read const built = directiveBody(read.value, block.blocks, reading, path, depth) if (!built.ok) return built - // Guarded here, so V8's throw past the bound refuses at the directive as another engine's check does. + // Refused here so V8's throw past the bound is placed at this directive, like the bound check on other engines. const readable = withinEngineLimits(() => commonMarkSpelling(built.value, path, depth, { flavour: 'lossless', memo: reading.memo }) ?? success(undefined)) if (!readable.ok) return readable if (readable.value === undefined) return built diff --git a/src/markdown/portable/adf-to-portable-markdown.ts b/src/markdown/portable/adf-to-portable-markdown.ts index 5ad643f..78043f7 100644 --- a/src/markdown/portable/adf-to-portable-markdown.ts +++ b/src/markdown/portable/adf-to-portable-markdown.ts @@ -60,7 +60,7 @@ export function reduceToPortable(document: AdfDocument): Result { const fault = adfDocumentFault(document) if (fault !== undefined) return faulted(fault, []) if (document.version !== 1) return failure('unsupported-document-version', `no markdown spelling carries ADF version ${document.version}`, []) - // Refused on the caller's text, edge whitespace and all: merging it first would build past the bound. + // Measured on the caller's text, edge whitespace and all: `toEditorNormal`'s merge could build past the bound. if (heldTextLength(nodeContent(document)) > longestMarkdown) return tooLong() // Two documents the editor holds equal write the same portable markdown (docs/decisions.md §Equality is deep). const blocks = reduceBlocks(nodeContent(toEditorNormal(document)), { depth: 0, memo: new Map(), path: [] }) diff --git a/todo.md b/todo.md index d07a7c1..ef873c4 100644 --- a/todo.md +++ b/todo.md @@ -143,9 +143,10 @@ audience kept the captioned image's `!adf:caption` directive, 3 of 3, reading a Item 106 made the CommonMark ask fill the memo and the walk above drop each entry as it reads it, so a deep subtree holds no level's text twice. Every comprehension seat read that split, across `commonMarkSpelling`, `readableBlock` and their three callers, as order-dependent state, and the -chunk shipped under a worse ruling on Locality. Keep main's single reader and writer instead: when -`readableBlock` keeps a node's spelling, it drops its children's and grandchildren's entries, whose -text the node's now holds. Found by the comprehension check of item 106, 2026-10-13. +chunk shipped under a worse ruling on Locality. Give the memo one reader and writer again: when +`readableBlock` keeps a node's spelling, it drops the entries of the node's children and +grandchildren, since the node's own spelling now holds their text. Found by the comprehension check +of item 106, 2026-10-13. ### 60. Collect the questions `parse/` asks `emit/` into one named module. @@ -223,10 +224,10 @@ the reason the project read worse. Each level of a nested list or quote re-indents its whole text, so the work is the depth times the text. `losslessMarkdownToAdf` on 3 MB of lazy lines in a list nested 490 levels inside -`!adf:blockquote` spends about 5 seconds re-spelling it. The same 2 MB split over two items of a -list nested 480 levels took 11 minutes to refuse on Node 24: a deep subtree past the length bound -refuses only once every level below has re-indented its text. The writers do the same on such a -document. Found while shipping item 106, 2026-10-11. +`!adf:blockquote` spends about 5 seconds re-spelling it. 2 MB of the same lines, split over two +items of a list nested 480 levels, took 11 minutes to refuse on Node 24: a deep subtree past the +length bound refuses only once every level below has re-indented its text. The writers do the same +on such a document. Found while shipping item 106, 2026-10-11. ### 7. Ship HTML: `adfToHtml`, `htmlToAdf`, and markdown to and from HTML composed through ADF, each call named by its flavour.