From 25a17e1ed38dcc8718eb8412100f5fed8bc7612f Mon Sep 17 00:00:00 2001 From: Lilleman auf Larv Date: Sun, 11 Oct 2026 23:26:20 +0200 Subject: [PATCH] Refuse markdown past V8's longest string on every engine, before building it --- AGENTS.md | 1 + CHANGELOG.md | 9 +-- README.md | 9 +-- docs/decisions.md | 40 +++++++++---- src/adf/document.ts | 4 ++ src/canonical-json.test.ts | 7 ++- src/canonical-json.ts | 12 ++++ src/markdown/emit/adf-to-markdown.test.ts | 33 +++++++++-- src/markdown/emit/adf-to-markdown.ts | 51 +++++++++++----- src/markdown/emit/inline-line.ts | 59 +++++++++++-------- src/markdown/opaque-carry.ts | 6 +- src/markdown/parse/markdown-to-adf.test.ts | 7 ++- src/markdown/parse/markdown-to-adf.ts | 2 +- .../portable/adf-to-portable-markdown.test.ts | 5 +- .../portable/adf-to-portable-markdown.ts | 2 +- src/markdown/portable/editor-normal.ts | 5 +- src/markdown/portable/inline-reduction.ts | 4 +- src/output-length.ts | 8 +++ todo.md | 29 ++++----- 19 files changed, 203 insertions(+), 90 deletions(-) create mode 100644 src/output-length.ts diff --git a/AGENTS.md b/AGENTS.md index 3749487..b08b31b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,6 +59,7 @@ In `docs/decisions.md`: - The CommonMark suite checks three ways - The flavour spec is read as a source - The node tables answer to Atlassian's schema +- Markdown past V8's longest string is refused on every engine - Nothing recurses unbounded - JSON text is read by `parseJsonText` - No value `JSON.parse` or `structuredClone` builds loops a call diff --git a/CHANGELOG.md b/CHANGELOG.md index 88f6263..f80d7a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -83,10 +83,11 @@ - Fix a cyclic document, in which an object holds itself: `adfToLosslessMarkdown` and `adfToPortableMarkdown` refuse it as `not-an-adf-document` and `isAdfDocument` returns `false`, where all three hung. -- Fix a document whose markdown would be longer than the longest string or array the JavaScript - engine builds, on V8 about 537 million characters, or about 134 million in one text node: - `adfToLosslessMarkdown` and `adfToPortableMarkdown` refuse it as `unsupported-node-shape`, where - they threw. +- Fix a document whose markdown would be longer than 536,870,888 characters, V8's longest string: + `adfToLosslessMarkdown` and `adfToPortableMarkdown` refuse it as `unsupported-node-shape` on + every engine, before building it, where they threw or ran out of memory. +- Fix a directive whose CommonMark spelling would be that long: `losslessMarkdownToAdf` refuses it + as `unsupported-node-shape`, where a few megabytes of nested list lines ran it out of memory. - Fix deeply nested compact JSON in an `adf:` fence: `losslessMarkdownToAdf` refuses it as `unsupported-node-shape`, as it refuses any compact JSON there, where about a megabyte of it threw. - Add `adfToPortableMarkdown` and `portableMarkdownToAdf`, a lossy pair converting ADF to and from diff --git a/README.md b/README.md index eab8f97..6e42863 100644 --- a/README.md +++ b/README.md @@ -409,9 +409,10 @@ typed on `ConvertError` takes both. A call reports the first refusal in document order and stops, so a document with several surfaces them one per call. Every refusal is deterministic on one engine — there is no I/O anywhere — so a -retry returns the identical error: fix the input, or set the document aside. Only the engine-limit -refusal under `unsupported-node-shape` differs between engines; on JavaScriptCore (Bun, Safari), -retry it once, since that engine reports a heap running short the same way. +retry returns the identical error: fix the input, or set the document aside. Only the refusal naming +the longest string or array this JavaScript engine builds differs between engines; on +JavaScriptCore (Bun, Safari), retry it once, since that engine reports a heap running short the +same way. ### Parsing @@ -444,7 +445,7 @@ The emitter raises these, and so does a parse, which asks the emitter how to spe | Code | Fires when | What you can do | | --- | --- | --- | | `unsupported-nesting-depth` | blocks, marks, an attribute's JSON or a carried node's JSON nest past 500 levels | keep the ADF and pass the document over, or show it read-only; flatten the input where you are the one who wrote it | -| `unsupported-node-shape` | parsing, in `losslessMarkdownToAdf`: markdown spells a node with an attribute, value, argument or body its type does not take, or without one it needs, wraps a directive mark around nothing or only hard breaks, writes as a directive a node or mark the lossless flavour spells as CommonMark, or puts a reserved directive out of place: `!adf:textBreak{}` parting nothing, `!adf:doc` anywhere but as the whole document. Emitting, in `adfToLosslessMarkdown`: a node not riding the carry that carries one mark type twice, a block or a non-text inline node holding `text`, or a leaf block or a non-text inline node holding `content`. Emitting, in `adfToPortableMarkdown`: a line or task list the portable writer cannot spell, a bug in this library. Emitting, in both writers: a document whose markdown would be longer than the longest string or array the JavaScript engine builds — on V8, about 537 million characters of markdown, or about 134 million in one text node | fix what the message names; [`spec/flavour.md`](https://github.com/larvit/adf-codec/blob/main/spec/flavour.md) lists every type's attributes and body. For a portable line or task list, report it as a bug. Past the engine's longest string or array, keep the ADF and pass the document over, or show it read-only | +| `unsupported-node-shape` | parsing, in `losslessMarkdownToAdf`: markdown spells a node with an attribute, value, argument or body its type does not take, or without one it needs, wraps a directive mark around nothing or only hard breaks, writes as a directive a node or mark the lossless flavour spells as CommonMark, or puts a reserved directive out of place: `!adf:textBreak{}` parting nothing, `!adf:doc` anywhere but as the whole document, or writes as a directive a node whose CommonMark spelling would be longer than 536,870,888 characters. Emitting, in `adfToLosslessMarkdown`: a node not riding the carry that carries one mark type twice, a block or a non-text inline node holding `text`, or a leaf block or a non-text inline node holding `content`. Emitting, in `adfToPortableMarkdown`: a line or task list the portable writer cannot spell, a bug in this library. Emitting, in both writers: a document whose markdown would be longer than 536,870,888 characters, V8's longest string, on every engine | fix what the message names; [`spec/flavour.md`](https://github.com/larvit/adf-codec/blob/main/spec/flavour.md) lists every type's attributes and body. For a portable line or task list, report it as a bug. Past 536,870,888 characters, keep the ADF and pass the document over, or show it read-only | ## The guarantees diff --git a/docs/decisions.md b/docs/decisions.md index 19ebd7a..892b548 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -461,11 +461,12 @@ handles one cause alike whichever node, attribute or direction raised it. knows which direction it called and parting them after `0.1.0` is MAJOR (2026-09-23). - A non-finite number takes two codes: `unsupported-node-shape` parsing, `not-an-adf-document` emitting — no document holds one, so no round-trip crosses them (2026-09-23). -- A document whose markdown would be longer than the longest string or array the JavaScript engine - builds is `unsupported-node-shape` at the document's path, in either writer: the document does - not convert, and no spelling in the input causes the bound (§The code list). A carried node - passes it on about 1 MB of deeply nested JSON, since the block carry indents two spaces a level; - `todo.md` item 107 respells the carry in a length linear in its JSON (2026-10-11). +- A document whose markdown would be longer than V8's longest string is `unsupported-node-shape` + at the block that passes it, in either writer, and so is a directive `losslessMarkdownToAdf` + reads whose CommonMark spelling would be: the document does not convert, and no spelling in the + input causes the bound (§The code list). A carried node passes it on about 1 MB of deeply nested + JSON, since the block carry indents two spaces a level; `todo.md` item 107 respells the carry in + a length linear in its JSON (2026-10-11). - The markdown readers and `htmlToAdf` refuse a value that is not a string as `not-a-string`, at position `{ line: 1, offset: 0 }` with an empty path: no existing code reads true of it, and `not-an-adf-document` names the emitters' input (2026-10-05). @@ -658,6 +659,19 @@ any value round-trips. What the schema holds and the tables do not spell is pinn attribute as a gap, a type as carried — so a re-pin adding either goes red until someone spells it or pins it. +## Markdown past V8's longest string is refused on every engine + +2026-10-11, the maintainer. Goals 1 and 6. Valid while V8's longest string, 2^29 − 24 characters, +is the shortest of the engines the library claims. + +Every writer, and the reader's CommonMark ask, refuses markdown longer than 2^29 − 24 characters +(`longestOutput`) before building it, so the same document converts alike everywhere and no +amplification — an indent on every line of a deep subtree, the carry's two-space JSON, text one +string shared by many nodes — builds a string no engine was asked for. Each block counts its lines +once with the indent its containers will prefix them with, so the bound trips at the block that +passes it, before any container copies it. The engines' own throws past their longest string or +array stay refused as `unsupported-node-shape` by `withinEngineLimits`, behind this bound. + ## Nothing recurses unbounded 2026-08-25, the directive form's count 2026-09-18, the maintainer. Goal 1. Valid while an engine's @@ -722,13 +736,15 @@ two cannot disagree — which is what makes the kept value a memo rather than a 2026-09-19, the maintainer. Goal 9. Valid while the `commonMarkSpelling` ask spells a node once per level above it otherwise. -The parse and the portable reduction keep each node's readable spelling in a memo, so the -`commonMarkSpelling` ask stops spelling a node once per level above it. `text`, `alternateText` and -`spelling` carry no depth and `headroom` is affine in it, so a read at or above the depth that -filled the entry rebases; a read below re-spells, because a hit skips the depth guards the walk it -replaces runs and an ordered list past the marker cap gives way, spending two emitter levels where -the parser spent one. Only what succeeded is kept, so no path minted at another position is ever -read. +The parse and the portable reduction keep the readable spelling of each node they put the +`commonMarkSpelling` ask to in a memo, so the ask stops spelling a node once per level above it. A +walk that reads an entry drops it, and a walk fills none for the nodes below the asked one: the walk +above spells them into its own text, and kept, every level of a deep subtree would hold its whole +text (2026-10-11). `text`, `alternateText` and `spelling` carry no depth and `headroom` is affine in +it, so a read at or above the depth that filled the entry rebases; a read below re-spells, because a +hit skips the depth guards the walk it replaces runs and an ordered list past the marker cap gives +way, spending two emitter levels where the parser spent one. Only what succeeded is kept, so no path +minted at another position is ever read. ## Cost fixes are measured, never timed diff --git a/src/adf/document.ts b/src/adf/document.ts index c759b6c..043a15a 100644 --- a/src/adf/document.ts +++ b/src/adf/document.ts @@ -82,6 +82,10 @@ export function identicalMarks(left: readonly AdfMark[], right: readonly AdfMark return marksKey(left) === marksKey(right) } +export function textLength(node: AdfNode): number { + return node.text?.length ?? 0 +} + export function mergeAdjacentText(nodes: readonly AdfNode[], joins: (previous: AdfNode, node: AdfNode) => boolean): AdfNode[] { const merged: AdfNode[] = [] for (const node of nodes) { diff --git a/src/canonical-json.test.ts b/src/canonical-json.test.ts index a06af18..c480ca2 100644 --- a/src/canonical-json.test.ts +++ b/src/canonical-json.test.ts @@ -2,7 +2,7 @@ import assert from 'node:assert/strict' import test from 'node:test' import type { JsonValue } from './json-value.ts' -import { serializeCanonicalJson, spellsCanonicalJson } from './canonical-json.ts' +import { serializeCanonicalJson, serializeCanonicalJsonWithin, spellsCanonicalJson } from './canonical-json.ts' test('sorts object keys recursively', () => { const value = { b: 1, a: { d: 2, c: 3 } } @@ -62,3 +62,8 @@ test('holds a text to the canonical spelling, whole', () => { assert.ok(!spellsCanonicalJson(value, 'two-space', '{"a":[1,2]}')) assert.ok(!spellsCanonicalJson(1e21, 'compact', '1e21')) }) + +test('spells canonical JSON up to a length, and nothing past it', () => { + assert.equal(serializeCanonicalJsonWithin({ a: [1, 2] }, 'compact', 11), '{"a":[1,2]}') + assert.equal(serializeCanonicalJsonWithin({ a: [1, 2] }, 'compact', 10), undefined) +}) diff --git a/src/canonical-json.ts b/src/canonical-json.ts index b67f8f3..985754b 100644 --- a/src/canonical-json.ts +++ b/src/canonical-json.ts @@ -15,6 +15,18 @@ export function serializeCanonicalJson(value: JsonValue, spelling: JsonSpelling) return text.join('') } +// `undefined` where the spelling runs past `longest` characters, where it stops building. +export function serializeCanonicalJsonWithin(value: JsonValue, spelling: JsonSpelling, longest: number): string | undefined { + const text: string[] = [] + let length = 0 + const whole = writeCanonicalJson(value, spelling, (piece) => { + length += piece.length + text.push(piece) + return length <= longest + }) + return whole ? text.join('') : undefined +} + // Stops at the first piece that does not match `text`: re-spelled with two-space indentation, a short compact text can pass the engine's longest string. export function spellsCanonicalJson(value: JsonValue, spelling: JsonSpelling, text: string): boolean { let offset = 0 diff --git a/src/markdown/emit/adf-to-markdown.test.ts b/src/markdown/emit/adf-to-markdown.test.ts index 584ba7a..ea7e899 100644 --- a/src/markdown/emit/adf-to-markdown.test.ts +++ b/src/markdown/emit/adf-to-markdown.test.ts @@ -44,13 +44,36 @@ test('refuses a value that is not an ADF document, naming the check it failed', assert.equal(markdown(adfToLosslessMarkdown({ type: 'doc', version: Number.NaN })), 'not-an-adf-document: an ADF document holds a version number: found NaN') }) -// About 2.2 billion characters of carry JSON, past V8's, JavaScriptCore's and SpiderMonkey's longest string. -test('refuses a document whose markdown runs past the engine\'s longest string', () => { - let value: JsonValue = new Array(2_200_000).fill(0) +const tooLong = 'unsupported-node-shape: the markdown would be longer than 536870888 characters, the longest string V8 holds' + +// About 556 million characters of carry JSON: past V8's longest string, short of JavaScriptCore's and SpiderMonkey's. +test('refuses a carry whose JSON would run past V8\'s longest string, on every engine', () => { + let value: JsonValue = new Array(560_000).fill(0) for (let level = 0; level < 495; level += 1) value = [value] const result = adfToLosslessMarkdown(document({ attrs: { value }, type: 'unknownBlock' })) - assert.match(markdown(result), /^unsupported-node-shape: the output would be longer than the longest string or array this JavaScript engine builds \(/) - assert.deepEqual(path(result), []) + assert.equal(markdown(result), tooLong) + assert.deepEqual(path(result), ['content', 0]) +}) + +test('refuses lines their containers would prefix past V8\'s longest string, at the block holding them', () => { + let quote: AdfNode = { attrs: {}, content: [{ text: 'a\n'.repeat(1_100_000), type: 'text' }], type: 'codeBlock' } + for (let level = 0; level < 245; level += 1) quote = { content: [quote], type: 'blockquote' } + const result = adfToLosslessMarkdown(document(quote)) + assert.equal(markdown(result), tooLong) + assert.equal(path(result).length, 2 * 246) +}) + +test('refuses a directive whose opener and body together would pass V8\'s longest string', () => { + const long = 'a'.repeat(2 ** 28) + const expand: AdfNode = { attrs: { title: long }, content: [{ content: [{ text: long, type: 'text' }], type: 'codeBlock' }], type: 'expand' } + assert.equal(markdown(adfToLosslessMarkdown(document(expand))), tooLong) +}) + +test('refuses text past V8\'s longest string before spelling it', () => { + const long = 'a'.repeat(2 ** 28) + const result = adfToLosslessMarkdown(document(paragraph({ text: long, type: 'text' }, { text: long, type: 'text' }, { text: long, type: 'text' }))) + assert.equal(markdown(result), tooLong) + assert.deepEqual(path(result), ['content', 0]) }) test('refuses a document version the markdown cannot carry', () => { diff --git a/src/markdown/emit/adf-to-markdown.ts b/src/markdown/emit/adf-to-markdown.ts index 6506743..64ac1a2 100644 --- a/src/markdown/emit/adf-to-markdown.ts +++ b/src/markdown/emit/adf-to-markdown.ts @@ -12,6 +12,7 @@ import { fencedCodeBlock } from '../commonmark/backtick-runs.ts' import { holdsNullCharacter, isBlankLine, isThematicBreak, markerInterruptsParagraph } from '../commonmark/grammar.ts' import { languageSlot, type LanguageSlot } from '../code-language.ts' import { largestNesting } from '../../nesting.ts' +import { longestOutput, tooLong } from '../../output-length.ts' import { spellBlockDirectiveOpener } from './block-directive-spelling.ts' import { spellDirectiveCloser, spellDirectiveOpener } from '../directive-syntax.ts' import { tryImage } from './image.ts' @@ -29,9 +30,11 @@ type SpelledBlock = { spelling: 'commonmark' | 'directive'; text: string } | { a // Keyed by reference: only a caller building one object per position (the parse, the portable reduction) passes one; a consumer's document may share a node. export type SpellingMemo = Map type WalkedItem = { node: AdfNode; walk: PlacedBlocks } -export type Writing = { flavour: WrittenFlavour; memo: SpellingMemo | undefined } +// indent: the characters the containers around a block prefix each of its lines with. +export type Writing = { flavour: WrittenFlavour; indent: number; memo: SpellingMemo | undefined } const bulletDelimiters: Delimiters = ['-', '*'] +const bulletWidth = 2 export const largestListMarker = 999999999 // Bare because tryList admits no item carrying attributes, marks or text. const listItemOpener = spellDirectiveOpener('listItem', undefined, '') @@ -46,7 +49,7 @@ export function writeMarkdown(document: AdfDocument, flavour: WrittenFlavour): R if (fault !== undefined) return faulted(fault, []) if (document.version !== 1) return failure('unsupported-document-version', `no markdown spelling carries ADF version ${document.version}`, []) if (document.content === undefined) return success(`${documentSpelling}\n`) - const walk = walkBlocks(document.content, [], 0, { flavour, memo: undefined }) + const walk = walkBlocks(document.content, [], 0, { flavour, indent: 0, memo: undefined }) if (!walk.ok) return walk const text = joinBlocks(walk.value.blocks, 'document') return success(text === '' ? '' : `${text}\n`) @@ -56,15 +59,29 @@ function walkBlocks(nodes: readonly AdfNode[], path: ConvertErrorPath, depth: nu let headroom = largestNesting - depth if (headroom < 0) return tooDeep(path) const blocks: PlacedBlock[] = [] + // Each block's length once its containers prefix its lines, and the two characters parting it from the next. + let projected = 0 for (const [index, node] of nodes.entries()) { const block = emitBlock(node, [...path, 'content', index], depth, writing) if (!block.ok) return block + projected += prefixedLength(block.value.text, writing.indent) + 2 + if (projected > longestOutput) return tooLong([...path, 'content', index]) headroom = Math.min(headroom, block.value.headroom) blocks.push({ ...block.value, node }) } return success({ blocks, headroom }) } +function prefixedLength(text: string, indent: number): number { + let lines = 1 + for (let at = text.indexOf('\n'); at !== -1; at = text.indexOf('\n', at + 1)) lines += 1 + return text.length + lines * indent +} + +function indented(writing: Writing, width: number): Writing { + return { ...writing, indent: writing.indent + width } +} + function tooDeep(path: ConvertErrorPath): Result { return failure('unsupported-nesting-depth', `the document nests deeper than the ${largestNesting} levels the emitter carries`, path) } @@ -107,25 +124,25 @@ function emitBlock(node: AdfNode, path: ConvertErrorPath, depth: number, writing return emitDirectiveBlock(node, model, path, depth, () => walkBlocks(nodeContent(node), path, depth + 1, writing)) } -export function commonMarkSpelling(node: AdfNode, path: ConvertErrorPath, depth: number, writing: Writing): Result | undefined { +export function commonMarkSpelling(node: AdfNode, path: ConvertErrorPath, depth: number, writing: Writing & { memo: SpellingMemo }): Result | undefined { const readable = readableBlock(node, path, depth, writing) + if (readable === undefined) writing.memo.set(node, { block: undefined, depth }) + else if (readable.ok) writing.memo.set(node, { block: readable.value, depth }) if (readable === undefined) return undefined if (!readable.ok) return readable return readable.value.spelling === 'directive' ? undefined : success(null) } function readableBlock(node: AdfNode, path: ConvertErrorPath, depth: number, writing: Writing): Result | undefined { - const { memo } = writing - const kept = memo?.get(node) + const kept = writing.memo?.get(node) + // The walk reading an entry spells the node into its own text, so the entry is dropped: kept, every level of a deep subtree would hold its whole text. + writing.memo?.delete(node) if (kept !== undefined) { if (kept.block === undefined) return undefined // An entry holds the spelling at its fill depth. A read at or above it rebases headroom; a read below re-spells, because a hit skips the walk's depth guards. if (depth <= kept.depth) return success({ ...kept.block, headroom: kept.block.headroom + kept.depth - depth }) } - const spelled = spellReadableBlock(node, path, depth, writing) - if (spelled === undefined) memo?.set(node, { block: undefined, depth }) - else if (spelled.ok) memo?.set(node, { block: spelled.value, depth }) - return spelled + return spellReadableBlock(node, path, depth, writing) } function spellReadableBlock(node: AdfNode, path: ConvertErrorPath, depth: number, writing: Writing): Result | undefined { @@ -154,12 +171,14 @@ function spellPortableBlock(node: AdfNode, path: ConvertErrorPath, depth: number } function quotedUnder(head: string, node: AdfNode, path: ConvertErrorPath, depth: number, writing: Writing): Result { - const inner = walkBlocks(nodeContent(node), path, depth + 1, writing) + const inner = walkBlocks(nodeContent(node), path, depth + 1, indented(writing, quoteWidth)) if (!inner.ok) return inner const body = joinBlocks(inner.value.blocks, 'document') return success(commonMarkText(quoted(body === '' ? head : `${head}\n\n${body}`), inner.value.headroom)) } +const quoteWidth = 2 + function quoted(text: string): string { return text .split('\n') @@ -173,7 +192,8 @@ function tryTaskList(node: AdfNode, path: ConvertErrorPath, depth: number, writi // A child other than a task nests in the task before it. for (const [index, child] of nodeContent(node).entries()) { const task = isTaskItem(child) - const walk = task ? taskBlocks(child, [...path, 'content', index], depth + 1, writing) : placedBlock(child, [...path, 'content', index], depth + 1, writing) + const itemWriting = indented(writing, bulletWidth) + const walk = task ? taskBlocks(child, [...path, 'content', index], depth + 1, itemWriting) : placedBlock(child, [...path, 'content', index], depth + 1, itemWriting) if (!walk.ok) return walk headroom = Math.min(headroom, walk.value.headroom) const previous = items.at(-1) @@ -248,12 +268,14 @@ function emitDirectiveBody(node: AdfNode, model: BlockNodeModel, opener: string, } const walk = walkBody() if (!walk.ok) return walk - return success(directivePair(node, opener, joinBlocks(walk.value.blocks, 'directive'), walk.value.headroom)) + const body = joinBlocks(walk.value.blocks, 'directive') + if (opener.length + body.length > longestOutput) return tooLong(path) + return success(directivePair(node, opener, body, walk.value.headroom)) } function tryBlockquote(node: AdfNode, path: ConvertErrorPath, depth: number, writing: Writing): Result | undefined { if (!holdsOnlyAttributes(node, [])) return undefined - const inner = walkBlocks(nodeContent(node), path, depth + 1, writing) + const inner = walkBlocks(nodeContent(node), path, depth + 1, indented(writing, quoteWidth)) if (!inner.ok) return inner const text = joinBlocks(inner.value.blocks, 'document') const alert = flavourClaims[writing.flavour].alerts && leadingMarker(text, readAlertMarker) !== undefined @@ -313,8 +335,9 @@ function tryList(node: AdfNode, path: ConvertErrorPath, depth: number, writing: if (items.some((item) => item.type !== 'listItem' || !holdsOnlyAttributes(item, []))) return undefined const walked: WalkedItem[] = [] let headroom = Number.POSITIVE_INFINITY + const itemWriting = indented(writing, ordered ? `${start + items.length - 1}. `.length : bulletWidth) for (const [offset, item] of items.entries()) { - const walk = walkBlocks(nodeContent(item), [...path, 'content', offset], depth + 1, writing) + const walk = walkBlocks(nodeContent(item), [...path, 'content', offset], depth + 1, itemWriting) if (!walk.ok) return walk headroom = Math.min(headroom, walk.value.headroom) walked.push({ node: item, walk: walk.value }) diff --git a/src/markdown/emit/inline-line.ts b/src/markdown/emit/inline-line.ts index 5b1eb6e..eaa4ee3 100644 --- a/src/markdown/emit/inline-line.ts +++ b/src/markdown/emit/inline-line.ts @@ -10,11 +10,12 @@ import { escapeUnbalanced, spellDestination, spellLinkTarget } from '../commonma import { failure, success, type ConvertErrorPath, type Result } from '../../result.ts' import { flavourClaims, highlightDelimiter } from '../portable/conventions.ts' import { holdsNullCharacter, trimTrailingSpace } from '../commonmark/grammar.ts' -import { identicalMark, isSpellableText, nodeAttrs, nodeContent, nodeMarks } from '../../adf/document.ts' +import { identicalMark, isSpellableText, nodeAttrs, nodeContent, nodeMarks, textLength } from '../../adf/document.ts' import { inlineNodeModel } from '../../adf/inline-nodes.ts' import { joinsWhenRead, textBreakSpelling } from '../adjacent-text.ts' import { largestNesting } from '../../nesting.ts' import { longestBacktickRun } from '../commonmark/backtick-runs.ts' +import { longestOutput, tooLong } from '../../output-length.ts' import { slotFault, spellInlineDirectiveOpener, spellInlineLeafDirective } from '../directive-syntax.ts' import { spellInlineNodeAttributes } from './inline-directive-spelling.ts' import { spellTextDirective } from '../text-directive.ts' @@ -80,6 +81,7 @@ export function tryImageLine(alt: string | undefined, url: string, link?: LinkDe } function emitLine(nodes: readonly AdfNode[], container: LineContainer, path: ConvertErrorPath, flavour: WrittenFlavour): Result { + if (heldTextLength(nodes) > longestOutput) return tooLong(path) const fallbacks: LineFallbacks = { carried: new Set(), flavour, openingLinkAsDirective: false } // Terminates because takeFallback refuses a pass that took no new fallback. for (;;) { @@ -99,6 +101,17 @@ function emitLine(nodes: readonly AdfNode[], container: LineContainer, path: Con } } +// The line spells every character of its text at least once, so this much text cannot fit. +export function heldTextLength(nodes: readonly AdfNode[]): number { + let length = 0 + const pending = [...nodes] + for (let node = pending.pop(); node !== undefined; node = pending.pop()) { + length += textLength(node) + for (const child of nodeContent(node)) pending.push(child) + } + return length +} + function takeFallback(fallbacks: LineFallbacks, fallback: NodeRange | 'opening-link', path: ConvertErrorPath): Result { if (fallback === 'opening-link') { if (fallbacks.openingLinkAsDirective) return failure('unsupported-node-shape', 'an opening link spelled as a directive still reads as a link definition, so the line has no spelling left', path) @@ -172,13 +185,9 @@ function bracketedLinkTarget(target: string): InlineSegment { return { escaping: 'bracketed-link-target', text: escapeUnbalanced(target, '[', ']') } } -function refuseContentAndText(node: AdfNode, path: ConvertErrorPath): Result { - const holdsContent = nodeContent(node).length > 0 - if (holdsContent || node.text !== undefined) { - const held = holdsContent ? 'content' : 'text' - return failure('unsupported-node-shape', `a ${node.type} node holds neither content nor text: this one holds ${held}`, path) - } - return success(null) +function heldContentOrText(node: AdfNode): 'content' | 'text' | undefined { + if (nodeContent(node).length > 0) return 'content' + return node.text === undefined ? undefined : 'text' } function emitRun(nodes: readonly AdfNode[], depth: number, firstIndex: number, context: InlineContext): Result { @@ -235,40 +244,38 @@ function carries(node: AdfNode, carried: ReadonlySet, index: number): bo return typeof slot === 'string' && slotFault(node.type, slot) !== undefined && nodeContent(node).length === 0 && node.text === undefined } +// Builds the node's path only to refuse it: a path copies every level above, and a line may hold a million leaves. function emitLeaf(node: AdfNode, context: InlineContext, index: number): Result { - const path = nodePath(context, index) if (carries(node, context.carried, index)) { - const carried = carriedInline(node, path) + const carried = carriedInline(node, nodePath(context, index)) if (!carried.ok) return carried return success({ segments: [syntax(carried.value)] }) } const types = nodeMarks(node).map((mark) => mark.type) - if (new Set(types).size !== types.length) return failure('unsupported-node-shape', `a ${node.type} node carries one mark type twice`, path) + if (new Set(types).size !== types.length) return failure('unsupported-node-shape', `a ${node.type} node carries one mark type twice`, nodePath(context, index)) const model = inlineNodeModel(node.type) if (model === undefined) return success(emitText(node, context, index)) - if (node.type === 'hardBreak') return emitHardBreak(node, model, context, index, path) - return emitInlineDirective(node, model, index, path) + const held = heldContentOrText(node) + if (held !== undefined) return failure('unsupported-node-shape', `a ${node.type} node holds neither content nor text: this one holds ${held}`, nodePath(context, index)) + if (node.type === 'hardBreak') return success(emitHardBreak(node, model, context, index)) + return success(emitInlineDirective(node, model, index)) } -function emitHardBreak(node: AdfNode, model: InlineNodeModel, context: InlineContext, index: number, path: ConvertErrorPath): Result { - const empty = refuseContentAndText(node, path) - if (!empty.ok) return empty +function emitHardBreak(node: AdfNode, model: InlineNodeModel, context: InlineContext, index: number): Emission { const attributes = spellInlineNodeAttributes(node, model) - if (attributes === undefined) return success({ carry: { first: index, last: index } }) - if (attributes === '' && context.spansLines && !context.atBlockEnd) return success({ segments: [syntax('\\\n')] }) - return success({ segments: [syntax(spellInlineLeafDirective('hardBreak', attributes))] }) + if (attributes === undefined) return { carry: { first: index, last: index } } + if (attributes === '' && context.spansLines && !context.atBlockEnd) return { segments: [syntax('\\\n')] } + return { segments: [syntax(spellInlineLeafDirective('hardBreak', attributes))] } } -function emitInlineDirective(node: AdfNode, model: InlineNodeModel, index: number, path: ConvertErrorPath): Result { - const empty = refuseContentAndText(node, path) - if (!empty.ok) return empty +function emitInlineDirective(node: AdfNode, model: InlineNodeModel, index: number): Emission { const attributes = spellInlineNodeAttributes(node, model) - if (attributes === undefined) return success({ carry: { first: index, last: index } }) + if (attributes === undefined) return { carry: { first: index, last: index } } const slot = model.textAttribute === undefined ? undefined : nodeAttrs(node)[model.textAttribute] - if (slot === undefined) return success({ segments: [syntax(spellInlineLeafDirective(node.type, attributes))] }) - if (typeof slot !== 'string' || slotFault(node.type, slot) !== undefined) return success({ carry: { first: index, last: index } }) + if (slot === undefined) return { segments: [syntax(spellInlineLeafDirective(node.type, attributes))] } + if (typeof slot !== 'string' || slotFault(node.type, slot) !== undefined) return { carry: { first: index, last: index } } const content: InlineSegment[] = slot === '' ? [] : [{ escaping: 'bracketed', text: slot }] - return success({ segments: [syntax(spellInlineDirectiveOpener(node.type)), ...content, syntax(`]${attributes}`)] }) + return { segments: [syntax(spellInlineDirectiveOpener(node.type)), ...content, syntax(`]${attributes}`)] } } // CommonMark reads a raw carriage return as a line ending and a null character as U+FFFD, so neither is written raw. diff --git a/src/markdown/opaque-carry.ts b/src/markdown/opaque-carry.ts index e7673d5..af41103 100644 --- a/src/markdown/opaque-carry.ts +++ b/src/markdown/opaque-carry.ts @@ -9,8 +9,9 @@ import { infoStringCarries } from './commonmark/grammar.ts' import { isAdfNode } from '../adf/document.ts' import { isJsonValue, nestingDepth, overNested } from '../json-value.ts' import { largestNesting } from '../nesting.ts' +import { longestOutput, tooLong } from '../output-length.ts' import { parseJsonText } from '../json-text.ts' -import { serializeCanonicalJson, spellsCanonicalJson } from '../canonical-json.ts' +import { serializeCanonicalJsonWithin, spellsCanonicalJson } from '../canonical-json.ts' export const carryFencePrefix = 'adf:' @@ -58,7 +59,8 @@ function carriedJson(node: AdfNode, spelled: object, spelling: JsonSpelling, pat if (!isJsonValue(spelled) || headroom < 0) { return failure('unsupported-nesting-depth', `a carried node's JSON nests deeper than the ${levels} levels its position leaves`, path) } - return success({ headroom, json: serializeCanonicalJson(spelled, spelling) }) + const json = serializeCanonicalJsonWithin(spelled, spelling, longestOutput) + return json === undefined ? tooLong(path) : success({ headroom, json }) } // `type` is the one the fence's info string names, `undefined` for the inline carry. diff --git a/src/markdown/parse/markdown-to-adf.test.ts b/src/markdown/parse/markdown-to-adf.test.ts index dd42512..e4ac53f 100644 --- a/src/markdown/parse/markdown-to-adf.test.ts +++ b/src/markdown/parse/markdown-to-adf.test.ts @@ -449,13 +449,18 @@ test('names the canonical spelling a carried JSON reads alone', () => { }) test('refuses compact JSON in the carry fence without building its two-space spelling', () => { - const json = `{"attrs":{"value":${'['.repeat(495)}${new Array(2_200_000).fill(0).join(',')}${']'.repeat(495)}}}` + const json = `{"attrs":{"value":${'['.repeat(495)}${new Array(560_000).fill(0).join(',')}${']'.repeat(495)}}}` assert.equal( content(losslessMarkdownToAdf(`\`\`\`adf:unknownBlock\n${json}\n\`\`\`\n`)), "unsupported-node-shape: the opaque carry spells its node's JSON canonically: two-space indent, keys sorted; !adf:codeBlock {language=\"adf:unknownBlock\"} around a bare fence keeps it a code block", ) }) +test('refuses a directive whose CommonMark spelling would pass V8\'s longest string', () => { + const markdown = `!adf:blockquote\n${'- '.repeat(490)}a\\\n${'b\\\n'.repeat(560_000)}c\n!adf:/blockquote\n` + assert.equal(content(losslessMarkdownToAdf(markdown)), 'unsupported-node-shape: the markdown would be longer than 536870888 characters, the longest string V8 holds') +}) + test('names the node JSON an opaque carry restores alone', () => { const node = "unsupported-node-shape: the opaque carry holds one ADF node's JSON: this JSON is no ADF node; " assert.equal(content(losslessMarkdownToAdf('```adf:x\n[]\n```\n')), `${node}!adf:codeBlock {language="adf:x"} around a bare fence keeps it a code block`) diff --git a/src/markdown/parse/markdown-to-adf.ts b/src/markdown/parse/markdown-to-adf.ts index b66b796..fbf0aa4 100644 --- a/src/markdown/parse/markdown-to-adf.ts +++ b/src/markdown/parse/markdown-to-adf.ts @@ -254,7 +254,7 @@ function directiveNode(block: DirectiveBlock, reading: Reading, path: ConvertErr if (!read.ok) return read const built = directiveBody(read.value, block.blocks, reading, path, depth) if (!built.ok) return built - const readable = commonMarkSpelling(built.value, path, depth, { flavour: 'lossless', memo: reading.memo }) + const readable = commonMarkSpelling(built.value, path, depth, { flavour: 'lossless', indent: 0, memo: reading.memo }) if (readable === undefined) return built if (!readable.ok) return readable return failure('unsupported-node-shape', `${built.value.type} takes the CommonMark spelling, not the directive form`, path) diff --git a/src/markdown/portable/adf-to-portable-markdown.test.ts b/src/markdown/portable/adf-to-portable-markdown.test.ts index 96b02f4..8dd30ae 100644 --- a/src/markdown/portable/adf-to-portable-markdown.test.ts +++ b/src/markdown/portable/adf-to-portable-markdown.test.ts @@ -70,11 +70,10 @@ test('refuses what the document guard refuses, and nothing else', () => { assert.match(portableDocument(document(paragraph(deepInline))), /^unsupported-nesting-depth at /) }) -// The merge throws at its concatenation, before any text is scanned, so no engine holds more than the one string. -test('refuses a paragraph whose merged text passes the engine\'s longest string', () => { +test('refuses a paragraph whose text would pass V8\'s longest string, joining none of it', () => { const long = 'a'.repeat(2 ** 28) const merged = paragraph(...Array.from({ length: 9 }, () => text(long))) - assert.equal(portableDocument(document(merged)), 'unsupported-node-shape at /') + assert.equal(portableDocument(document(merged)), 'unsupported-node-shape at /content/0') }) test('refuses nesting past 500 levels wherever the reduction walks', () => { diff --git a/src/markdown/portable/adf-to-portable-markdown.ts b/src/markdown/portable/adf-to-portable-markdown.ts index ee485d2..f5ee4da 100644 --- a/src/markdown/portable/adf-to-portable-markdown.ts +++ b/src/markdown/portable/adf-to-portable-markdown.ts @@ -124,7 +124,7 @@ function portableSequence(blocks: readonly AdfNode[], reduction: Reduction): Res for (let next = pending.pop(); next !== undefined; next = pending.pop()) { const block = numberedPastMarkers(next) const listed = block.type === 'bulletList' || block.type === 'orderedList' - if (!listed || (block === next && commonMarkSpelling(block, reduction.path, reduction.depth, { flavour: 'portable', memo: reduction.memo })?.ok === true)) { + if (!listed || (block === next && commonMarkSpelling(block, reduction.path, reduction.depth, { flavour: 'portable', indent: 0, memo: reduction.memo })?.ok === true)) { sequence.push(block) continue } diff --git a/src/markdown/portable/editor-normal.ts b/src/markdown/portable/editor-normal.ts index f9ad390..6775436 100644 --- a/src/markdown/portable/editor-normal.ts +++ b/src/markdown/portable/editor-normal.ts @@ -1,7 +1,8 @@ import type { AdfAttributes, AdfDocument, AdfMark, AdfNode } from '../../adf/document.ts' import type { JsonValue } from '../../json-value.ts' -import { identicalMark, mergeAdjacentText, nodeAttrs, nodeContent, nodeMarks } from '../../adf/document.ts' +import { identicalMark, mergeAdjacentText, nodeAttrs, nodeContent, nodeMarks, textLength } from '../../adf/document.ts' import { joinsWhenRead } from '../adjacent-text.ts' +import { longestOutput } from '../../output-length.ts' type JsonContainer = JsonValue[] | { [key: string]: JsonValue } @@ -10,6 +11,8 @@ type NodeHolder = { content?: AdfNode[] } // The editor joins two text nodes where the reader would join their editor-normal forms. export function joinsWhenEditorNormal(previous: AdfNode, node: AdfNode): boolean { if (previous.type !== 'text' || node.type !== 'text') return false + // Text this long is refused as markdown either way; joining it first would build the string. + if (textLength(previous) + textLength(node) > longestOutput) return false return joinsWhenRead(normalNode(previous), normalNode(node)) } diff --git a/src/markdown/portable/inline-reduction.ts b/src/markdown/portable/inline-reduction.ts index d0b1b3a..d51611c 100644 --- a/src/markdown/portable/inline-reduction.ts +++ b/src/markdown/portable/inline-reduction.ts @@ -5,8 +5,9 @@ import { blockNodeModel } from '../../adf/block-nodes.ts' import { failure, success, type ConvertErrorPath, type Result } from '../../result.ts' import { joinsWhenEditorNormal, sameMarkWhenEditorNormal } from './editor-normal.ts' import { largestNesting } from '../../nesting.ts' +import { longestOutput, tooLong } from '../../output-length.ts' import { mergeAdjacentText, nodeAttrs, nodeContent, nodeMarks } from '../../adf/document.ts' -import { portableLineFallback, type PortableLineFallback } from '../emit/inline-line.ts' +import { heldTextLength, portableLineFallback, type PortableLineFallback } from '../emit/inline-line.ts' import { spellDestination, spellLinkTarget } from '../commonmark/link-syntax.ts' const highlight = 'backgroundColor' @@ -17,6 +18,7 @@ const keptMarks: readonly string[] = [...edgeStrippingMarks, 'code', 'link'] export function reduceInline(nodes: readonly AdfNode[], container: LineContainer, path: ConvertErrorPath, depth: number): Result { const leaves = inlineLeaves(nodes, container, path, depth) if (!leaves.ok) return leaves + if (heldTextLength(leaves.value) > longestOutput) return tooLong(path) return spellableLine(trimmedEdges(highlightsInsideSharedMarks(trimmedEdges(leaves.value))), container, path) } diff --git a/src/output-length.ts b/src/output-length.ts new file mode 100644 index 0000000..04a2ba3 --- /dev/null +++ b/src/output-length.ts @@ -0,0 +1,8 @@ +import { failure, type ConvertErrorPath, type Result } from './result.ts' + +// V8's longest string, held on every engine so a document converts alike everywhere (docs/decisions.md §Markdown past V8's longest string is refused on every engine). +export const longestOutput = 2 ** 29 - 24 + +export function tooLong(path: ConvertErrorPath): Result { + return failure('unsupported-node-shape', `the markdown would be longer than ${longestOutput} characters, the longest string V8 holds`, path) +} diff --git a/todo.md b/todo.md index 679fc0c..aa1e4e7 100644 --- a/todo.md +++ b/todo.md @@ -6,7 +6,7 @@ `Bar = 9` -`Next ID = 108` +`Next ID = 110` | Goal | W | |---|---| @@ -25,7 +25,7 @@ | ID | Release | Exempt | Item | R | S | A | G | Goals | Score | |---|---|---|---|---|---|---|---|---|---| -| 106 | 0.2.0 | defect | **Refuse a document or markdown whose output would pass a stated length before building it, so no call exhausts the heap.** | 4 | 6 | 5 | 8 | 1, 9 | 20.5 | +| 108 | 0.2.0 | defect | **Spell a line in memory linear in its length, so a paragraph shorter than V8's longest string converts within Node's default heap.** | 4 | 6 | 5 | 8 | 1, 9 | 20.5 | | 103 | 0.2.0 | | **Fold `unspellable-whitespace` into `unsupported-node-shape`, so a line ending and a null character in a content slot take one code.** | 2 | 2 | 5 | 6 | 1 | 19.5 | | 84 | 0.2.0 | | **Read GFM's pipe table without leading pipes, and with alignment colons, in `portableMarkdownToAdf`, dropping the alignment.** | 3 | 4 | 7 | 7 | 4, 7 | 19.4 | | 52 | 0.2.0 | | **Spell `colwidth` as a comma list, `colwidth="340,420"`.** | 3 | 3 | 5 | 6 | 6 | 11.5 | @@ -42,6 +42,7 @@ | 33 | 0.2.1 | defect | **Emit a line in time linear in its mark runs, in `adfToLosslessMarkdown` and `adfToPortableMarkdown`.** | 4 | 5 | 6 | 9 | 9 | 8.7 | | 98 | 0.2.1 | defect | **Read blank lines inside deeply nested list items in time linear in the markdown.** | 3 | 3 | 5 | 9 | 9 | 8.2 | | 102 | 0.2.1 | decision | **Place a block a portable task list refuses at its path in the returned document without replaying `splitTaskList` on stand-in nodes.** | 3 | 4 | 2 | 4 | 1 | 8.0 | +| 109 | 0.2.1 | defect | **Spell nested lists and quotes in time linear in the markdown.** | 3 | 5 | 4 | 7 | 9 | 5.3 | | 7 | 0.3.0 | decision | **Ship HTML: `adfToHtml`, `htmlToAdf`, and markdown to and from HTML composed through ADF, each call named by its flavour.** | 6 | 9 | 9 | 9 | 2, 4 | 25.8 | | 45 | 0.3.0 | | **Replace `isAdfDocument` with a reader returning `Result`.** | 2 | 3 | 6 | 8 | 1 | 25.2 | | 6 | 0.3.0 | decision | **Specify the HTML dialect.** | 2 | 6 | 7 | 8 | 2, 4 | 24.7 | @@ -73,19 +74,13 @@ ## Details -### 106. Refuse a document or markdown whose output would pass a stated length before building it, so no call exhausts the heap. +### 108. Spell a line in memory linear in its length, so a paragraph shorter than V8's longest string converts within Node's default heap. -The writers return `unsupported-node-shape` where the engine throws past its longest string, but -some documents exhaust the heap first, which no catch survives. `losslessMarkdownToAdf` on 2.1 MB of -markdown, a `!adf:blockquote` holding a list nested 490 levels whose paragraph runs on in 700,000 -lazy lines `b\`, crashed Node 24 out of heap: `directiveNode` re-spells the subtree, indenting each -line to its depth. `adfToLosslessMarkdown` on nine text nodes sharing one 2^28-character string got -Bun OOM-killed under 8 GB: `escape` in `line-escaping.ts` pushes one array element per character. -Where the heap holds out, the readers' re-spelling can still throw a `RangeError`. JavaScriptCore's -`Out of memory`, which the writers refuse, may also mean a failed allocation that a retry survives. -The candidate length to refuse past is V8's 2^29 − 24 on every engine, so the same document converts -alike everywhere; no entry states it yet (AGENTS.md §Stated numbers). Found by the principles audit -of item 83, 2026-10-11. +The line escaper keeps an array element per character twice (`escapings`, `placements` in `escape`, +`line-escaping.ts`) and a `Set` entry per escaped character, so `adfToLosslessMarkdown` holds about +50 bytes a character: one 20-million-character text node took 1.2 GB on Node 24, and one of 80 +million exhausted its 2.2 GB default heap. Twenty million `*` throw `RangeError: Set maximum size +exceeded`, which `withinEngineLimits` lets through. Found while shipping item 106, 2026-10-11. ### 103. Fold `unspellable-whitespace` into `unsupported-node-shape`, so a line ending and a null character in a content slot take one code. @@ -211,6 +206,12 @@ the fix. `taskOfParagraphs` keeps the original child objects. Both paired-ruling seats on 2026-10-11 named it the reason the project read worse. +### 109. Spell nested lists and quotes in time linear in the markdown. + +Each level of a nested list or quote re-indents its whole text, so `losslessMarkdownToAdf` on 3 MB +of lazy lines in a list nested 490 levels inside `!adf:blockquote` spends about 5 seconds +re-spelling it, the writers the same on its document. Found while shipping item 106, 2026-10-11. + ### 7. Ship HTML: `adfToHtml`, `htmlToAdf`, and markdown to and from HTML composed through ADF, each call named by its flavour. Lands after items 6, 59, 60 and 62. Which flavours get a composed call is this item's to settle;