diff --git a/CHANGELOG.md b/CHANGELOG.md index 605fd51..e9ea8cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -83,6 +83,10 @@ - Fix a cyclic document, in which an object holds itself: `adfToLosslessMarkdown` and `adfToPortableMarkdown` refuse it as `not-an-adf-document` and `isAdfDocument` returns `false`, where all three hung. +- Fix a document whose markdown runs past the longest string the JavaScript engine holds, about 537 + million characters on V8: `adfToLosslessMarkdown` and `adfToPortableMarkdown` refuse it as + `unsupported-node-shape`, where they threw. `losslessMarkdownToAdf` refuses compact JSON in an + `adf:` fence in time linear in it, where a megabyte of deeply nested JSON threw. - Add `adfToPortableMarkdown` and `portableMarkdownToAdf`, a lossy pair converting ADF to and from markdown GitHub, GitLab and Obsidian render: alerts, callouts, task lists, `==highlights==` and pipe tables. A task item keeps its paragraphs, then its nested task lists; the rest of the item, diff --git a/README.md b/README.md index c630ae8..b2f09f7 100644 --- a/README.md +++ b/README.md @@ -211,7 +211,7 @@ Each call's name says which markdown it reads or writes: | Call | Markdown | Refuses with | | --- | --- | --- | | `adfToLosslessMarkdown` | writes the lossless flavour: CommonMark plus GFM's pipe tables and `~~`, with `!adf:` directives and `adf:` fences for everything else | `not-an-adf-document`, `unsupported-document-version`, `unsupported-nesting-depth`, `unsupported-node-shape` | -| `adfToPortableMarkdown` | writes markdown GitHub, GitLab and Obsidian render; drops what they cannot show, and writes no directive | `not-an-adf-document`, `unsupported-document-version`, `unsupported-nesting-depth` | +| `adfToPortableMarkdown` | writes markdown GitHub, GitLab and Obsidian render; drops what they cannot show, and writes no directive | `not-an-adf-document`, `unsupported-document-version`, `unsupported-nesting-depth`, `unsupported-node-shape` | | `commonMarkToAdf` | reads strict CommonMark as its spec says: a directive, a pipe table, `~~`, `==` and an `adf:` fence read as text and code | `not-a-string`, `unmappable-html`, `unsupported-nesting-depth` | | `losslessMarkdownToAdf` | reads the lossless flavour, markdown `adfToLosslessMarkdown` wrote, edited or not | every code in [The errors](#the-errors) except `not-an-adf-document` and `unsupported-document-version` | | `portableMarkdownToAdf` | reads CommonMark, GFM's pipe tables and `~~`, alerts and task markers, Obsidian's callouts and `==`; a directive and an `adf:` fence read as text and code | `malformed-pipe-table`, `not-a-string`, `unmappable-html`, `unsupported-nesting-depth` | @@ -442,7 +442,7 @@ The emitter raises these, and so does a parse, which asks the emitter how to spe | Code | Fires when | What you can do | | --- | --- | --- | | `unsupported-nesting-depth` | blocks, marks, an attribute's JSON or a carried node's JSON nest past 500 levels | keep the ADF and pass the document over, or show it read-only; flatten the input where you are the one who wrote it | -| `unsupported-node-shape` | parsing, in `losslessMarkdownToAdf`: markdown spells a node with an attribute, value, argument or body its type does not take, or without one it needs, wraps a directive mark around nothing or only hard breaks, writes as a directive a node or mark the lossless flavour spells as CommonMark, or puts a reserved directive out of place: `!adf:textBreak{}` parting nothing, `!adf:doc` anywhere but as the whole document. Emitting, in `adfToLosslessMarkdown`: a node not riding the carry that carries one mark type twice, a block or a non-text inline node holding `text`, or a leaf block or a non-text inline node holding `content` | fix what the message names; [`spec/flavour.md`](https://github.com/larvit/adf-codec/blob/main/spec/flavour.md) lists every type's attributes and body | +| `unsupported-node-shape` | parsing, in `losslessMarkdownToAdf`: markdown spells a node with an attribute, value, argument or body its type does not take, or without one it needs, wraps a directive mark around nothing or only hard breaks, writes as a directive a node or mark the lossless flavour spells as CommonMark, or puts a reserved directive out of place: `!adf:textBreak{}` parting nothing, `!adf:doc` anywhere but as the whole document. Emitting, in `adfToLosslessMarkdown`: a node not riding the carry that carries one mark type twice, a block or a non-text inline node holding `text`, or a leaf block or a non-text inline node holding `content`. Emitting, in both writers: a document whose markdown runs past the longest string the JavaScript engine holds, about 537 million characters on V8 | fix what the message names; [`spec/flavour.md`](https://github.com/larvit/adf-codec/blob/main/spec/flavour.md) lists every type's attributes and body | ## The guarantees diff --git a/docs/decisions.md b/docs/decisions.md index b262414..2092611 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -461,6 +461,9 @@ handles one cause alike whichever node, attribute or direction raised it. knows which direction it called and parting them after `0.1.0` is MAJOR (2026-09-23). - A non-finite number takes two codes: `unsupported-node-shape` parsing, `not-an-adf-document` emitting — no document holds one, so no round-trip crosses them (2026-09-23). +- A document whose markdown runs past the longest string the JavaScript engine holds is + `unsupported-node-shape` at the document's path, in either writer: the document does not convert, + and the bound is the engine's own (2026-10-11). - The markdown readers and `htmlToAdf` refuse a value that is not a string as `not-a-string`, at position `{ line: 1, offset: 0 }` with an empty path: no existing code reads true of it, and `not-an-adf-document` names the emitters' input (2026-10-05). diff --git a/src/canonical-json.test.ts b/src/canonical-json.test.ts index 26cdf8f..a06af18 100644 --- a/src/canonical-json.test.ts +++ b/src/canonical-json.test.ts @@ -2,7 +2,7 @@ import assert from 'node:assert/strict' import test from 'node:test' import type { JsonValue } from './json-value.ts' -import { serializeCanonicalJson } from './canonical-json.ts' +import { serializeCanonicalJson, spellsCanonicalJson } from './canonical-json.ts' test('sorts object keys recursively', () => { const value = { b: 1, a: { d: 2, c: 3 } } @@ -52,3 +52,13 @@ test('spells a value nesting far past the levels a recursive walk survives', () assert.equal(serializeCanonicalJson(array, 'compact'), `${'['.repeat(levels)}1${']'.repeat(levels)}`) assert.equal(serializeCanonicalJson(object, 'compact'), `${'{"a":'.repeat(levels)}1${'}'.repeat(levels)}`) }) + +test('holds a text to the canonical spelling, whole', () => { + const value = { a: [1, 2] } + assert.ok(spellsCanonicalJson(value, 'compact', '{"a":[1,2]}')) + assert.ok(spellsCanonicalJson(value, 'two-space', serializeCanonicalJson(value, 'two-space'))) + assert.ok(!spellsCanonicalJson(value, 'compact', '{"a":[1,2]} ')) + assert.ok(!spellsCanonicalJson(value, 'compact', '{"a":[1,2]')) + assert.ok(!spellsCanonicalJson(value, 'two-space', '{"a":[1,2]}')) + assert.ok(!spellsCanonicalJson(1e21, 'compact', '1e21')) +}) diff --git a/src/canonical-json.ts b/src/canonical-json.ts index a1fd6ec..00e4d7f 100644 --- a/src/canonical-json.ts +++ b/src/canonical-json.ts @@ -7,20 +7,37 @@ type Member = { label: string; value: JsonValue } type Pending = string | { depth: number; value: JsonValue } export function serializeCanonicalJson(value: JsonValue, spelling: JsonSpelling): string { - const indent = spelling === 'compact' ? '' : ' ' const text: string[] = [] + writeCanonicalJson(value, spelling, (piece) => text.push(piece) > 0) + return text.join('') +} + +// Stops at the first piece `text` does not hold: two-space indentation can spell a short compact text past the engine's longest string. +export function spellsCanonicalJson(value: JsonValue, spelling: JsonSpelling, text: string): boolean { + let offset = 0 + const whole = writeCanonicalJson(value, spelling, (piece) => { + if (!text.startsWith(piece, offset)) return false + offset += piece.length + return true + }) + return whole && offset === text.length +} + +// `write` returns false to stop the walk, and the walk then returns false. +function writeCanonicalJson(value: JsonValue, spelling: JsonSpelling, write: (piece: string) => boolean): boolean { + const indent = spelling === 'compact' ? '' : ' ' const pending: Pending[] = [{ depth: 0, value }] for (let next = pending.pop(); next !== undefined; next = pending.pop()) { if (typeof next === 'string') { - text.push(next) + if (!write(next)) return false continue } const { depth, value: held } = next if (Array.isArray(held)) schedule(pending, '[', held.map((item) => ({ label: '', value: item })), ']', indent, depth) else if (held !== null && typeof held === 'object') schedule(pending, '{', objectMembers(held, indent), '}', indent, depth) - else text.push(Object.is(held, -0) ? '-0' : JSON.stringify(held)) + else if (!write(Object.is(held, -0) ? '-0' : JSON.stringify(held))) return false } - return text.join('') + return true } function objectMembers(value: { [key: string]: JsonValue }, indent: string): Member[] { diff --git a/src/markdown/emit/adf-to-markdown.test.ts b/src/markdown/emit/adf-to-markdown.test.ts index 05198af..ec283fe 100644 --- a/src/markdown/emit/adf-to-markdown.test.ts +++ b/src/markdown/emit/adf-to-markdown.test.ts @@ -44,6 +44,15 @@ test('refuses a value that is not an ADF document, naming the check it failed', assert.equal(markdown(adfToLosslessMarkdown({ type: 'doc', version: Number.NaN })), 'not-an-adf-document: an ADF document holds a version number: found NaN') }) +// About 2.2 billion characters of carry JSON, past V8's, JavaScriptCore's and SpiderMonkey's longest string. +test('refuses a document whose markdown runs past the engine\'s longest string', () => { + let value: JsonValue = new Array(2_200_000).fill(0) + for (let level = 0; level < 495; level += 1) value = [value] + const result = adfToLosslessMarkdown(document({ attrs: { value }, type: 'unknownBlock' })) + assert.match(markdown(result), /^unsupported-node-shape: the output runs past what this JavaScript engine holds: /) + assert.deepEqual(path(result), []) +}) + test('refuses a document version the markdown cannot carry', () => { assert.equal(code(adfToLosslessMarkdown({ type: 'doc', version: 2 })), 'unsupported-document-version') }) diff --git a/src/markdown/emit/adf-to-markdown.ts b/src/markdown/emit/adf-to-markdown.ts index b775500..6506743 100644 --- a/src/markdown/emit/adf-to-markdown.ts +++ b/src/markdown/emit/adf-to-markdown.ts @@ -7,7 +7,7 @@ import { blockDirectiveForm, documentSpelling } from '../block-directive.ts' import { blockNodeModel, blockNodes, isTaskItem } from '../../adf/block-nodes.ts' import { carriedBlock } from '../opaque-carry.ts' import { emitInlineLine } from './inline-line.ts' -import { failure, faulted, success, type ConvertErrorPath, type Result } from '../../result.ts' +import { failure, faulted, success, withinEngineLimits, type ConvertErrorPath, type Result } from '../../result.ts' import { fencedCodeBlock } from '../commonmark/backtick-runs.ts' import { holdsNullCharacter, isBlankLine, isThematicBreak, markerInterruptsParagraph } from '../commonmark/grammar.ts' import { languageSlot, type LanguageSlot } from '../code-language.ts' @@ -38,7 +38,7 @@ const listItemOpener = spellDirectiveOpener('listItem', undefined, '') const orderedDelimiters: Delimiters = ['.', ')'] export function adfToLosslessMarkdown(document: AdfDocument): Result { - return writeMarkdown(document, 'lossless') + return withinEngineLimits(() => writeMarkdown(document, 'lossless')) } export function writeMarkdown(document: AdfDocument, flavour: WrittenFlavour): Result { diff --git a/src/markdown/opaque-carry.ts b/src/markdown/opaque-carry.ts index ef32c59..e7673d5 100644 --- a/src/markdown/opaque-carry.ts +++ b/src/markdown/opaque-carry.ts @@ -10,7 +10,7 @@ import { isAdfNode } from '../adf/document.ts' import { isJsonValue, nestingDepth, overNested } from '../json-value.ts' import { largestNesting } from '../nesting.ts' import { parseJsonText } from '../json-text.ts' -import { serializeCanonicalJson } from '../canonical-json.ts' +import { serializeCanonicalJson, spellsCanonicalJson } from '../canonical-json.ts' export const carryFencePrefix = 'adf:' @@ -75,7 +75,7 @@ function readCarriedJson(raw: string, spelling: JsonSpelling, levels: number, ty if (overNested(held, levels)) { return { fault: { code: 'unsupported-nesting-depth', message: `a carried node's JSON nests deeper than the ${levels} levels its position leaves` } } } - if (serializeCanonicalJson(value, spelling) !== raw) { + if (!spellsCanonicalJson(value, spelling, raw)) { const shape = spelling === 'compact' ? 'compact, keys sorted' : 'two-space indent, keys sorted' return { fault: unsupportedNodeShape(`the opaque carry spells its node's JSON canonically: ${shape}; ${wayOut}`) } } diff --git a/src/markdown/parse/markdown-to-adf.test.ts b/src/markdown/parse/markdown-to-adf.test.ts index 22d337b..23090f6 100644 --- a/src/markdown/parse/markdown-to-adf.test.ts +++ b/src/markdown/parse/markdown-to-adf.test.ts @@ -448,6 +448,14 @@ test('names the canonical spelling a carried JSON reads alone', () => { assert.equal(content(losslessMarkdownToAdf('!adf:carry{json="{\\"type\\":\\"blockCard\\",\\"attrs\\":{}}"}\n')), `${canonically}compact, keys sorted${literal}`) }) +test('refuses compact JSON in the carry fence without spelling it two-space past the engine\'s longest string', () => { + const json = `{"attrs":{"value":${'['.repeat(495)}${new Array(2_200_000).fill(0).join(',')}${']'.repeat(495)}}}` + assert.equal( + content(losslessMarkdownToAdf(`\`\`\`adf:unknownBlock\n${json}\n\`\`\`\n`)), + "unsupported-node-shape: the opaque carry spells its node's JSON canonically: two-space indent, keys sorted; !adf:codeBlock {language=\"adf:unknownBlock\"} around a bare fence keeps it a code block", + ) +}) + test('names the node JSON an opaque carry restores alone', () => { const node = "unsupported-node-shape: the opaque carry holds one ADF node's JSON: this JSON is no ADF node; " assert.equal(content(losslessMarkdownToAdf('```adf:x\n[]\n```\n')), `${node}!adf:codeBlock {language="adf:x"} around a bare fence keeps it a code block`) diff --git a/src/markdown/portable/adf-to-portable-markdown.test.ts b/src/markdown/portable/adf-to-portable-markdown.test.ts index de56314..e6571ab 100644 --- a/src/markdown/portable/adf-to-portable-markdown.test.ts +++ b/src/markdown/portable/adf-to-portable-markdown.test.ts @@ -70,6 +70,12 @@ test('refuses what the document guard refuses, and nothing else', () => { assert.match(portableDocument(document(paragraph(deepInline))), /^unsupported-nesting-depth at /) }) +test('refuses a document whose text merges past the engine\'s longest string', () => { + const long = 'a'.repeat(2 ** 28) + const merged = paragraph(...Array.from({ length: 9 }, () => text(long))) + assert.equal(portableDocument(document(merged)), 'unsupported-node-shape at /') +}) + test('refuses nesting past 500 levels wherever the reduction walks', () => { const lowest = (bottom: AdfNode): string => { let deep = bottom diff --git a/src/markdown/portable/adf-to-portable-markdown.ts b/src/markdown/portable/adf-to-portable-markdown.ts index 3c953e5..ee485d2 100644 --- a/src/markdown/portable/adf-to-portable-markdown.ts +++ b/src/markdown/portable/adf-to-portable-markdown.ts @@ -3,7 +3,7 @@ import { adfDocumentFault, nodeAttrs, nodeContent, nodeMarks } from '../../adf/d import { blockNodeModel, isTaskItem } from '../../adf/block-nodes.ts' import { centeredImage, externalMedia } from '../external-image.ts' import { commonMarkSpelling, largestListMarker, writeMarkdown, type SpellingMemo } from '../emit/adf-to-markdown.ts' -import { failure, faulted, success, type ConvertErrorPath, type Result } from '../../result.ts' +import { failure, faulted, success, withinEngineLimits, type ConvertErrorPath, type Result } from '../../result.ts' import { flavourClaims, taskMarker } from './conventions.ts' import { inlineLeaves, isBlockNodeType, oneLine, portableLink, reduceInline, writableHref } from './inline-reduction.ts' import { inlineNodeModel } from '../../adf/inline-nodes.ts' @@ -46,8 +46,10 @@ const blockReducers: Readonly> = { } export function adfToPortableMarkdown(document: AdfDocument): Result { - const reduced = reduceToPortable(document) - return reduced.ok ? writeMarkdown(reduced.value, 'portable') : reduced + return withinEngineLimits(() => { + const reduced = reduceToPortable(document) + return reduced.ok ? writeMarkdown(reduced.value, 'portable') : reduced + }) } export function reduceToPortable(document: AdfDocument): Result { diff --git a/src/result.test.ts b/src/result.test.ts index 8503d13..27ee428 100644 --- a/src/result.test.ts +++ b/src/result.test.ts @@ -4,6 +4,8 @@ import { fileURLToPath } from 'node:url' import { readFileSync, readdirSync } from 'node:fs' import test from 'node:test' +import { success, withinEngineLimits } from './result.ts' + const sourceRoot = dirname(fileURLToPath(import.meta.url)) const union = /export type ConvertErrorCode =\n((?:\s+\| '[a-z-]+'\n)+)/ const declared = /'([a-z-]+)'/g @@ -29,3 +31,23 @@ function calledCodes(): string[] { test('every ConvertErrorCode is the code of a production call site, and every call site names a declared one', () => { assert.deepEqual(calledCodes(), declaredCodes()) }) + +test('refuses where SpiderMonkey runs out of string, and lets any other throw through', () => { + const overflow = new Error('allocation size overflow') + overflow.name = 'InternalError' + assert.deepEqual( + withinEngineLimits(() => { + throw overflow + }), + { error: { code: 'unsupported-node-shape', message: 'the output runs past what this JavaScript engine holds: allocation size overflow', path: [] }, ok: false }, + ) + assert.throws(() => + withinEngineLimits(() => { + throw new TypeError('a bug') + }), + ) + assert.deepEqual( + withinEngineLimits(() => success('x')), + { ok: true, value: 'x' }, + ) +}) diff --git a/src/result.ts b/src/result.ts index 90166f7..8d859c2 100644 --- a/src/result.ts +++ b/src/result.ts @@ -41,3 +41,13 @@ export function positioned(result: Result, position: SourcePosition): Resu export function success(value: T): { ok: true; value: T } { return { ok: true, value } } + +// Any string concatenation throws past the engine's longest string: RangeError in V8 and JavaScriptCore, InternalError in SpiderMonkey. +export function withinEngineLimits(convert: () => Result): Result { + try { + return convert() + } catch (error) { + if (!(error instanceof RangeError || (error instanceof Error && error.name === 'InternalError'))) throw error + return failure('unsupported-node-shape', `the output runs past what this JavaScript engine holds: ${error.message}`, []) + } +} diff --git a/todo.md b/todo.md index 4518da2..878e03f 100644 --- a/todo.md +++ b/todo.md @@ -25,12 +25,10 @@ | ID | Release | Exempt | Item | R | S | A | G | Goals | Score | |---|---|---|---|---|---|---|---|---|---| -| 83 | 0.2.0 | defect | **Refuse a document whose markdown would pass the engine's string limit with a result, never a `RangeError`.** | 3 | 4 | 5 | 8 | 1, 9 | 22.0 | | 103 | 0.2.0 | | **Fold `unspellable-whitespace` into `unsupported-node-shape`, so a line ending and a null character in a content slot take one code.** | 2 | 2 | 5 | 6 | 1 | 19.5 | | 84 | 0.2.0 | | **Read GFM's pipe table without leading pipes, and with alignment colons, in `portableMarkdownToAdf`, dropping the alignment.** | 3 | 4 | 7 | 7 | 4, 7 | 19.4 | | 52 | 0.2.0 | | **Spell `colwidth` as a comma list, `colwidth="340,420"`.** | 3 | 3 | 5 | 6 | 6 | 11.5 | | 53 | 0.2.0 | | **Put a block's `marks` spelling to the writer panel and adopt its pick.** | 4 | 5 | 5 | 6 | 6 | 10.0 | -| 105 | 0.2.1 | | **Prove with a test that `adfToPortableMarkdown` never returns `unsupported-node-shape`, or list the code in the README's table of calls.** | 1 | 2 | 3 | 5 | 1 | 14.5 | | 89 | 0.2.1 | | **Build a `date`, `emoji`, `inlineCard`, `mention` or `status` holding no mark Atlassian's schema withholds from it, in every reader.** | 4 | 4 | 5 | 6 | 4 | 13.0 | | 99 | 0.2.1 | defect | **Write a linked external image standing inline or in a table cell in the spelling a reader panel picks, in `adfToPortableMarkdown`.** | 3 | 3 | 6 | 7 | 7 | 12.9 | | 100 | 0.2.1 | decision | **Put the reading of an image's title as its caption back to a reader panel, now that a writer panel reads a title as hover text.** | 3 | 3 | 4 | 5 | 4, 6 | 10.9 | @@ -73,14 +71,6 @@ ## Details -### 83. Refuse a document whose markdown would pass the engine's string limit with a result, never a `RangeError`. - -The block carry indents its JSON two spaces a level, so one unknown block holding 497 nested arrays -around 540,000 numbers, about 1.1 MB as JSON, spells about 540 million characters, past V8's 2^29 − -24. `serializeCanonicalJson`'s `join` throws, and `adfToLosslessMarkdown` throws with it. Any output -past the limit does the same. Refuse it with a code from the closed list, and add a fixture. Found -by the README-goals audit, 2026-10-08. - ### 103. Fold `unspellable-whitespace` into `unsupported-node-shape`, so a line ending and a null character in a content slot take one code. `slotFault` in `directive-syntax.ts` gives a newline in an `emoji`, `mention` or `status` content @@ -112,12 +102,6 @@ Today `marks="[{\"attrs\":{\"mode\":\"wide\"},\"type\":\"breakout\"}]"`, the mar escaped JSON. Breaking where the panel picks another spelling: `MIGRATION.md`'s Spellings table gains its row. -### 105. Prove with a test that `adfToPortableMarkdown` never returns `unsupported-node-shape`, or list the code in the README's table of calls. - -`spellableLine` (`src/markdown/portable/inline-reduction.ts`) can return `unsupported-node-shape`, and -no test reaches it; the README lists three codes for the call, and an app switching on them would -meet a fourth. - ### 89. Build a `date`, `emoji`, `inlineCard`, `mention` or `status` holding no mark Atlassian's schema withholds from it, in every reader. `*!adf:mention[@a]{id=…}*` and `[!adf:status[Done]{…}](/u)` build a node holding `em` or `link`.