diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e1c5cc..da2047d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,14 @@ `content` or `marks` is spelled `{attrs=empty}`, `{content=empty}` or `{marks=empty}`, `-0` is spelled `-0`, and a `codeBlock` of several text nodes is a fence per node. A `codeBlock` holding other than plain text nodes rides the block carry, where it was refused. +- **Breaking:** `unspellable-character` and `unspellable-line-start` leave `ConvertErrorCode`, and + `adfToMarkdown` no longer refuses `unspellable-whitespace`. A carriage return in text is spelled + ` ` and a null character `!adf:text{text="\u0000"}`; a code block holding either, a text node + holding no text or holding `content`, a paragraph line opening with a code span whose backticks + read back as a fence, and an `emoji`, `mention` or `status` whose `text` holds a line ending or a + null character ride the carry. See `MIGRATION.md`. +- Fix a link `title` holding a null character: it is written as `!adf:link[text]{attrs}`, where it + read back as U+FFFD. - **Breaking:** `unspellable-link` leaves `ConvertErrorCode`; a link whose `href` or `title` no CommonMark escape spells is written as `!adf:link[text]{attrs}`. - **Breaking:** some directive refusals carry `malformed-directive` where they carried diff --git a/MIGRATION.md b/MIGRATION.md index a5fb98d..ccdb7d6 100644 --- a/MIGRATION.md +++ b/MIGRATION.md @@ -62,12 +62,18 @@ Markdown the spelling table leaves alone, which `0.2.0` reads as a different doc ### Error codes -`unspellable-link` leaves `ConvertErrorCode`: a `switch` naming it stops compiling, and the link -it named converts. +`unspellable-character`, `unspellable-line-start` and `unspellable-link` leave `ConvertErrorCode`: +a `switch` naming one stops compiling, and the document it named converts. | Input | `0.1.0` | `0.2.0` | | --- | --- | --- | | a link whose `href` or `title` no CommonMark escape spells, on emit | `unspellable-link` | spells `!adf:link[text]{attrs}` | +| a carriage return in text, on emit | `unspellable-character` | spells ` ` | +| a null character in text, on emit | `unspellable-character` | spells `!adf:text{text="\u0000"}` | +| a carriage return or null character in a code block or code span, on emit | `unspellable-character` | rides the carry | +| a paragraph line opening with a code span whose backticks read back as a fence, on emit | `unspellable-line-start` | rides the inline carry | +| an `emoji`, `mention` or `status` whose `text` holds a line ending or a null character, on emit | `unspellable-whitespace`, `unspellable-character` | rides the inline carry | +| a text node holding no text, or holding `content`, on emit | `unsupported-node-shape` | rides the carry | | a `codeBlock` holding other than plain text nodes, on emit | `unsupported-node-shape` | rides the block carry | | a leaf node given a body (`media`, `listBreak`) | `unsupported-node-shape` | `malformed-directive` | | a node with a block body written as a leaf (`panel`) | `unsupported-node-shape` | `malformed-directive` | diff --git a/README.md b/README.md index 2b2c6d2..3a9e3ac 100644 --- a/README.md +++ b/README.md @@ -180,6 +180,7 @@ Parsing — `markdownToAdf` and `plainMarkdownToAdf`, and `htmlToAdf` at `0.2.0` | `unknown-directive-name` | a directive whose name is no node or mark this version spells | check the name in `spec/flavour.md`, or escape the prefix as `\!adf:`; the spelling itself is well formed, so a later minor may give the name meaning | | `unmappable-html` | the input holds an HTML construct the documented element set does not map, a comment and a processing instruction among them — at this version that is every raw HTML construct in markdown, the element set landing at `0.2.0` | remove the construct, or write what it holds in the lossless flavour | | `unmappable-image` | an image sits inside other content that is not another image's description, or carries a title | give the image a paragraph of its own and drop the title | +| `unspellable-whitespace` | an `emoji`, `mention` or `status` directive's content slot spells a newline or a carriage return — ` `, ` `, `!adf:text{text="\n"}` | replace it with a space, or write the node as `!adf:carry` — an inline directive never spans lines | Emitting — `adfToMarkdown`, and `adfToHtml` at `0.2.0`: @@ -194,9 +195,6 @@ emit refuses: | Code | Fires when | What you can do | | --- | --- | --- | -| `unspellable-character` | text or a code block holds a carriage return or a null character, which CommonMark rewrites wherever it sits | strip or replace the character; no escape carries it through the round-trip | -| `unspellable-line-start` | a paragraph line begins with a code span whose backticks would read back as a code fence | put any text before the code span | -| `unspellable-whitespace` | an `emoji`, `mention` or `status` holds a newline in the text its inline directive spells in the content slot | replace it with a space — an inline directive never spans lines | | `unsupported-nesting-depth` | blocks, marks, an attribute's JSON or a carried node's JSON nest past 500 levels | keep the ADF and pass the document over, or show it read-only; flatten the input where you are the one who wrote it | | `unsupported-node-shape` | a node carries an attribute, value, argument or body its type does not take, or lacks one it needs — or markdown writes as a directive a node or mark the lossless flavour spells as CommonMark, or a reserved directive stands out of place: `!adf:textBreak{}` or `!adf:listBreak` parting nothing, `!adf:doc` anywhere but as the whole document | fix what the message names; `spec/flavour.md` lists every type's attributes and body | @@ -217,9 +215,7 @@ Serves Goals 1, 3 and 4. `spec/flavour.md`) — and one gap: a CommonMark image fits only as its own title-less paragraph; mid-text and titled images are error results, save an image inside another's description, which flattens into the alt text. Converting back yields the library's canonical spelling, which - round-trips byte-identically — where it converts back at all: a parse succeeding is no promise of - that, so keep the source until the way back succeeds. ``` ` `` ` ``` reads cleanly and then - refuses. + round-trips byte-identically. - Four CommonMark spellings parse without an error and build a document the reference implementation renders differently: `[](/url)` and `[]()` stay literal text against CommonMark's empty link, a list continuing past a marker change stays one list against CommonMark's two, a @@ -228,10 +224,9 @@ Serves Goals 1, 3 and 4. text, which the spec requires and the reference itself breaks, nesting one `` in the other. The first three are pinned `pending` in `corpus/commonmark-spec/exceptions.json`; the suite holds no example of the fourth. -- Not every document converts back: `adfToMarkdown` is partial on valid ADF — a text node holding - a carriage return, or a paragraph line beginning with a code span whose backticks read back as a - fence. Show the refusal and keep the document read-only; saving markdown you could not produce - is the loss the round-trip exists to stop. +- Not every document converts back: `adfToMarkdown` refuses with `unsupported-nesting-depth` or + `unsupported-node-shape` as the tables above say. Show the refusal and keep the document + read-only; saving markdown you could not produce is the loss the round-trip exists to stop. - The pipe table narrows GFM's twice: every row opens with a pipe, so GFM's bare form is an error result rather than the prose it reads as, and an alignment colon in the delimiter row is an error too — ADF holds no column alignment. The trailing pipe is canonical output, optional in diff --git a/corpus/commonmark-spec/exceptions.json b/corpus/commonmark-spec/exceptions.json index 7f0b710..1d8c723 100644 --- a/corpus/commonmark-spec/exceptions.json +++ b/corpus/commonmark-spec/exceptions.json @@ -13,27 +13,6 @@ "kind": "pending", "reason": "A list continuing past a marker change renders as two lists, the parser opens one." }, - { - "check": "fixpoint", - "divergence": "unspellable-line-start", - "example": 330, - "kind": "unspellable", - "reason": "A paragraph opens with a code span whose backticks read back as a fence." - }, - { - "check": "fixpoint", - "divergence": "unspellable-line-start", - "example": 331, - "kind": "unspellable", - "reason": "A paragraph opens with a code span whose backticks read back as a fence." - }, - { - "check": "fixpoint", - "divergence": "unspellable-line-start", - "example": 340, - "kind": "unspellable", - "reason": "A paragraph opens with a code span whose backticks read back as a fence." - }, { "check": "count", "divergence": "em 2/1", diff --git a/docs/decisions.md b/docs/decisions.md index c5c18a2..0cc7f2a 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -23,13 +23,11 @@ sanitized and a mention keeps the test user's real account id. ## Markdown in is a canonical fixpoint -2026-08-23, the maintainer. Goals 1 and 4. Valid while markdown input may be written by hand. +2026-08-23, the maintainer; the way back made total 2026-10-04. Goals 1 and 4. Valid while +markdown input may be written by hand. The other direction is a canonical fixpoint, not byte-identity: human markdown normalizes, the way -back yields the library's canonical spelling, and that spelling round-trips byte-identically — -where there is a way back. CommonMark spells some things the flavour has no escape for — a -paragraph opening with a code span whose backticks read back as a fence — so a parse succeeding -does not imply a spellable document; `corpus/commonmark-spec/exceptions.json` names those. +back yields the library's canonical spelling, and that spelling round-trips byte-identically. ## Equality is deep @@ -338,8 +336,9 @@ switches on `code` with no `default`. `unsupported-node-shape` — since a code no input reaches is one no consumer can switch on (2026-09-20). - A refusal no spelling recovers from is a gap in the flavour rather than a code: give the flavour - the spelling and the code goes (`unspellable-link`, 2026-09-13). A cause the carry answers gets - no code: a mark no spelling writes rides the carry with its node. + the spelling and the code goes (`unspellable-link`, 2026-09-13; `unspellable-character` and + `unspellable-line-start`, 2026-10-04). A cause the carry answers gets no code: a mark no + spelling writes rides the carry with its node. ## Which code a cause takes @@ -348,11 +347,10 @@ handles one cause alike whichever node, attribute or direction raised it. - A code names the cause; where one cause recurs across node types, across one mark's attributes or across directions, one code covers them all and `path` and `message` say which — - `unsupported-nesting-depth` is the 500-level guard whichever direction hits it, - `unspellable-character` the text node and the code block alike. Where two codes stay apart, the - line between them is what they name: `unspellable-character` is a character CommonMark rewrites - wherever text holds it, `unspellable-whitespace` the newline no inline directive's content slot - spans, in either direction. + `unsupported-nesting-depth` is the 500-level guard whichever direction hits it. Where two codes + stay apart, the line between them is what they name: `unspellable-whitespace` is the newline an + inline directive's content slot spells in input, where the emitter carries the node + (2026-10-04). - A claim code names the spelling claimed, never the node that spelling would have built: a malformed `!adf:table` is a `malformed-directive`, and an alignment colon a `malformed-pipe-table` — the flavour's own delimiter row is `-` runs, so the grammar refuses the @@ -525,8 +523,8 @@ Each example is a named error or markdown that parses and emits to itself byte f reference HTML's text, tags stripped and entities decoded, equals the parsed document's; and its elements count the marks and nodes they map to. The fixpoint alone passes a parser returning the empty document, the text alone one dropping every emphasis. An exception is the maintainer's to -add, and valid CommonMark parsing to a document `adfToMarkdown` refuses where a spelling could exist -is a bug to fix, never an exception. +add, and valid CommonMark parsing to a document `adfToMarkdown` refuses is a bug to fix, never an +exception. ## The flavour spec is read as a source diff --git a/spec/flavour.md b/spec/flavour.md index c29681d..d0e3e1b 100644 --- a/spec/flavour.md +++ b/spec/flavour.md @@ -245,7 +245,8 @@ form. leaves empty, or in both, is a named error, and so are fences whose info strings differ and an empty fence beside another. Each fence is an ordinary code block, and its info string decodes escapes and entity references as any other does. A node holding a child no fence holds — any but - a text node carrying no marks, `attrs` or `content` — rides the block carry. + a text node holding text free of carriage returns and null characters, and carrying no marks, + `attrs` or `content` — rides the block carry. - `heading` — container, inline body. Attributes: `level` (number), `localId` (string). `level` is the `#` count, so a heading carrying none, or one that is no whole number from 1 to 6, has no CommonMark spelling. @@ -449,6 +450,7 @@ the nodes below, `emoji`, `mention` and `status` spell their `text` attribute in plain text: `[]` is the empty string, absent content is the absent attribute, non-empty content parsing to anything but one text node carrying neither marks, attributes nor content is a named error, and so is a `text` key in `{attrs}`. An enclosing mark spelling does not reach into the slot. +A node whose `text` holds a line ending or a null character rides the inline carry. The rest take no content, `!adf:text` included; content on a node that takes none is a named error. - `date` — Attributes: `localId` (string), `timestamp` (string, epoch milliseconds). @@ -471,14 +473,15 @@ The rest take no content, `!adf:text` included; content on a node that takes non Shipped !adf:emoji[🎉]{shortName=":tada:"} on !adf:date{timestamp=1756080000000}. ``` -**Whitespace CommonMark cannot hold.** A newline inside a text node, and a space or tab where -CommonMark strips or refuses one — a block's inline content edges, either side of a line break, an -em, strong or strike spelling's inner edges, a pipe cell's edges — is spelled `!adf:text{text="…"}`, -the reserved key carrying the node's text, escaped by the attribute grammar and never literal: pipe -cells trim and pad. The emitter wraps the whitespace run alone and leaves the rest plain text, which -the spelled run joins on reading. Input reads that spelling alone: the value is one run of spaces -and tabs, or one run of newlines, and anything else — a mixed run, or text CommonMark carries -plainly — is a named error. +**Characters CommonMark cannot hold.** A carriage return in text is ` `, which CommonMark reads +back as the character. A newline inside a text node, a null character, which CommonMark reads as +U+FFFD, and a space or tab where CommonMark strips or refuses one — a block's inline content edges, +either side of a line break, an em, strong or strike spelling's inner edges, a pipe cell's edges — +is spelled `!adf:text{text="…"}`, the reserved key carrying the node's text, escaped by the +attribute grammar and never literal: pipe cells trim and pad. The emitter wraps the run alone and +leaves the rest plain text, which the spelled run joins on reading. Input reads that spelling +alone: the value is one run of spaces and tabs, one run of newlines, or one run of null characters, +and anything else — a mixed run, or text CommonMark carries plainly — is a named error. ``` !adf:text{text=" "}Two leading spaces held, and one text node split!adf:text{text="\n"}over two lines. @@ -489,8 +492,8 @@ carried, neither holds `attrs` or an empty key, and their marks are identical, a The reserved leaf `!adf:textBreak{}` parts such a pair, inside every mark spelling the two share; a code span holds no directive, so it closes and reopens. It builds no node and reads only between two such nodes: elsewhere, or with `[content]` or `{attrs}`, it is a named error (`docs/decisions.md` -§`!adf:textBreak{}` parts text CommonMark would join). A text node holding `attrs` or an empty key -rides the inline carry. +§`!adf:textBreak{}` parts text CommonMark would join). A text node holding `attrs`, `content` or an +empty key, or holding no text, rides the inline carry. ``` Hello, !adf:textBreak{}world — **Hello, !adf:textBreak{}world** — `a`!adf:textBreak{}`b` @@ -531,7 +534,8 @@ breaks at every node the emitter carries, so no emitted carry sits inside a mark An inline node whose marks no nesting spells — a mark type not listed here, an attrs key its spelling does not list, a value that is not the spelling's type, an attribute the spelling needs and the mark lacks, an empty `attrs` on a mark CommonMark spells, an order putting a code span outside -another mark, `code` over anything but a text node or over text holding a newline, or a spelling +another mark, `code` over anything but a text node holding text, over text holding a line ending or +a null character, or opening a paragraph line its backticks would read back as a fence, or a spelling CommonMark's flanking rules cannot open or close where the run sits (`un**-real**istic`), or one CommonMark's matching pairs elsewhere — the intra-word `*` runs together with a neighbouring `**`, and the multiple-of-3 rule can leave the merged run's pairing to another delimiter — rides the diff --git a/src/conformance/commonmark-spec.test.ts b/src/conformance/commonmark-spec.test.ts index d9dde4c..1059258 100644 --- a/src/conformance/commonmark-spec.test.ts +++ b/src/conformance/commonmark-spec.test.ts @@ -13,7 +13,7 @@ const root = join(dirname(fileURLToPath(import.meta.url)), '..', '..', 'corpus', const checks = ['count', 'fixpoint', 'text'] as const type Check = (typeof checks)[number] -type ExceptionKind = 'mark-model' | 'pending' | 'unspellable' +type ExceptionKind = 'mark-model' | 'pending' type SpecExample = { example: number; html: string; markdown: string; section: string } @@ -30,7 +30,7 @@ function isCheck(value: unknown): value is Check { } function isKind(value: unknown): value is ExceptionKind { - return value === 'mark-model' || value === 'pending' || value === 'unspellable' + return value === 'mark-model' || value === 'pending' } function isSpecExample(value: unknown): value is SpecExample { @@ -77,12 +77,12 @@ test('the CommonMark spec suite is 0.31.2, vendored byte-exact', () => { assert.equal(digest, 'd431b29d97b6f73e69d547109cf5081578fac931e72afe95639ebe766c1b2a20') }) -test('every exception is unique, names a parsing example, and files a fixpoint only as unspellable', () => { +test('every exception is unique, names a parsing example, and files no fixpoint divergence', () => { assert.equal(exceptionIndex.size, exceptions.length, 'one exception repeats an example and check another holds') for (const entry of exceptions) { assert.ok(spec.some((candidate) => candidate.example === entry.example), `exception ${entry.example} names no example in the suite`) assert.equal(exampleToRefusal.get(entry.example), undefined, `exception ${entry.example} is on the refusal list, not an exception`) - if (entry.check === 'fixpoint') assert.equal(entry.kind, 'unspellable', `exception ${entry.example} files a fixpoint divergence as ${entry.kind}; a fixable hole is given the spelling instead`) + assert.notEqual(entry.check, 'fixpoint', `exception ${entry.example} files a fixpoint divergence; the emitter spells every document the parser builds`) } }) diff --git a/src/conformance/markdown-property.test.ts b/src/conformance/markdown-property.test.ts index 64a435f..d998377 100644 --- a/src/conformance/markdown-property.test.ts +++ b/src/conformance/markdown-property.test.ts @@ -418,7 +418,7 @@ function holdsDirectiveShape(document: AdfDocument): boolean { return false } -test('generated markdown refuses, or what it parses to refuses to emit, or its spelling reads back and spells itself', { timeout: propertyTimeout }, () => { +test('generated markdown refuses, or its spelling reads back and spells itself', { timeout: propertyTimeout }, () => { const parameters = propertyRuns(gateRuns) let directiveShaped = 0 let fixpoints = 0 @@ -427,7 +427,7 @@ test('generated markdown refuses, or what it parses to refuses to emit, or its s const parsed = markdownToAdf(input) if (!parsed.ok) return const emitted = adfToMarkdown(parsed.value) - if (!emitted.ok) return + assert.ok(emitted.ok, emitted.ok ? '' : `${emitted.error.code}: ${emitted.error.message} — spelling ${JSON.stringify(input)}`) fixpoints += 1 if (holdsDirectiveShape(parsed.value)) directiveShaped += 1 const read = markdownToAdf(emitted.value) diff --git a/src/conformance/property-harness.ts b/src/conformance/property-harness.ts index 0f3cc41..8e9efe6 100644 --- a/src/conformance/property-harness.ts +++ b/src/conformance/property-harness.ts @@ -24,7 +24,7 @@ export const propertyTimeout = 600000 const depthIdentifier = fc.createDepthIdentifier() const emptyCell: AdfNode = { content: [{ type: 'paragraph' }], type: 'tableCell' } const flatCommonMarkShapeWeight = 4 -export const markdownPieces = fc.constantFrom(...'aZ09 \t\n!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~é\xa0🎉日ー한𠀀', '==', '[!NOTE]', '[x]', 'ab:', 'http://', directivePrefix, `${directivePrefix}a[`, `${directivePrefix}a{`) +export const markdownPieces = fc.constantFrom(...'aZ09 \t\n\r\0!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~é\xa0🎉日ー한𠀀', '==', '[!NOTE]', '[x]', 'ab:', 'http://', directivePrefix, `${directivePrefix}a[`, `${directivePrefix}a{`) const nestingCommonMarkShapeWeight = 21 const spelledTypes = new Set(['text', ...Object.keys(blockNodes), ...Object.keys(inlineNodes), ...Object.keys(markAttributes)]) @@ -106,7 +106,7 @@ const mark: Arbitrary = occasionallyEmpty(fc.oneof( )) const marks = fc.uniqueArray(mark, { maxLength: 3, selector: (held) => held.type }) -const textNode = occasionallyEmpty(fc.record({ marks, text }).map((held): AdfNode => ({ ...held, type: 'text' }))) +const textNode = occasionallyEmpty(fc.record({ marks, text: fc.oneof({ arbitrary: text, weight: 19 }, { arbitrary: fc.constant(''), weight: 1 }) }).map((held): AdfNode => ({ ...held, type: 'text' }))) const backtickRunNode = fc .record({ marks: fc.oneof(fc.constant([]), fc.constant([{ type: 'code' }]), marks), text: fc.string({ maxLength: 6, minLength: 1, unit: fc.constantFrom('`', '``', ' ', 'a') }) }) diff --git a/src/markdown/commonmark/link-syntax.ts b/src/markdown/commonmark/link-syntax.ts index f07df24..d33fac5 100644 --- a/src/markdown/commonmark/link-syntax.ts +++ b/src/markdown/commonmark/link-syntax.ts @@ -117,5 +117,5 @@ export function escapeUnbalanced(spelling: string, opener: string, closer: strin } function spellTitle(title: string): string | undefined { - return /[\n\r\\]/.test(title) || holdsEntityReference(title) ? undefined : ` "${title.replaceAll('"', '\\"')}"` + return /[\n\r\0\\]/.test(title) || holdsEntityReference(title) ? undefined : ` "${title.replaceAll('"', '\\"')}"` } diff --git a/src/markdown/directive-syntax.ts b/src/markdown/directive-syntax.ts index c4f4cfd..e42ea98 100644 --- a/src/markdown/directive-syntax.ts +++ b/src/markdown/directive-syntax.ts @@ -100,7 +100,7 @@ export function readSoleStringAttribute(span: DirectiveSpan, key: string): Read< return { value: spelled.decoded } } -// Both directions answer alike: an inline directive never spans lines, so no content slot holds a line ending. +// An inline directive never spans lines, so no content slot holds a line ending. export function slotLineEndingFault(type: string, text: string): ConvertFault | undefined { if (!/[\n\r]/.test(text)) return undefined return { code: 'unspellable-whitespace', message: `the ${type} content slot holds a newline no inline directive spans` } diff --git a/src/markdown/emit/adf-to-markdown.test.ts b/src/markdown/emit/adf-to-markdown.test.ts index b92e06c..ca0d451 100644 --- a/src/markdown/emit/adf-to-markdown.test.ts +++ b/src/markdown/emit/adf-to-markdown.test.ts @@ -35,7 +35,7 @@ test('names the node a refusal came from, and no source the emitter never read', const unspellable: AdfNode = { text: 'x', type: 'paragraph' } const list: AdfNode = { content: [{ content: [paragraph({ text: 'x', type: 'text' })], type: 'listItem' }, { content: [unspellable], type: 'listItem' }], type: 'bulletList' } assert.deepEqual(path(adfToMarkdown(document(paragraph({ text: 'x', type: 'text' }), list))), ['content', 1, 'content', 1, 'content', 0]) - assert.deepEqual(path(adfToMarkdown(document(paragraph({ text: 'x', type: 'text' }, { type: 'text' })))), ['content', 0, 'content', 1]) + assert.deepEqual(path(adfToMarkdown(document(paragraph({ text: 'x', type: 'text' }, { text: 'x', type: 'status' })))), ['content', 0, 'content', 1]) assert.deepEqual(path(adfToMarkdown({ type: 'doc', version: 2 })), []) assert.equal(position(adfToMarkdown(document(paragraph({ text: 'x', type: 'text' }), list))), undefined) }) @@ -147,8 +147,17 @@ test('carries a mark the canonical spellings cannot nest', () => { ) }) -test('refuses a line whose start block parsing would claim', () => { - assert.equal(code(adfToMarkdown(document(paragraph({ marks: [{ type: 'code' }], text: '```', type: 'text' })))), 'unspellable-line-start') +function readsBack(doc: AdfDocument): string { + const spelled = markdown(adfToMarkdown(doc)) + assert.deepEqual(markdownToAdf(spelled), { ok: true, value: doc }, `reading ${JSON.stringify(spelled)}`) + return spelled +} + +test('carries a code span whose backticks would open its line as a fence', () => { + const fence: AdfNode = { marks: [{ type: 'code' }], text: '```', type: 'text' } + const carried = '!adf:carry{json="{\\"marks\\":[{\\"type\\":\\"code\\"}],\\"text\\":\\"\\u0060\\u0060\\u0060\\",\\"type\\":\\"text\\"}"}' + assert.equal(readsBack(document(paragraph(fence))), `${carried}\n`) + assert.equal(readsBack(document(paragraph({ text: 'a', type: 'text' }, { type: 'hardBreak' }, fence, fence))), `a\\\n${carried}\`\`\`\` \`\`\` \`\`\`\`\n`) }) test('escapes the delimiter row a hard break leaves opening a pipe table with no leading pipe', () => { @@ -456,28 +465,26 @@ test('spells a list item whose marker completes a thematic break as a directive' ) }) -test('refuses the characters CommonMark rewrites', () => { - assert.equal( - markdown(adfToMarkdown(document({ content: [{ text: 'a\rb', type: 'text' }], type: 'codeBlock' }))), - 'unspellable-character: a codeBlock holds no carriage return CommonMark keeps: this text holds one', - ) - assert.equal( - markdown(adfToMarkdown(document(paragraph({ text: 'a\rb', type: 'text' })))), - 'unspellable-character: a text node holds a carriage return CommonMark rewrites', - ) - assert.equal(code(adfToMarkdown(document(paragraph({ text: 'a\u0000b', type: 'text' })))), 'unspellable-character') - assert.equal(code(adfToMarkdown(document({ content: [{ text: 'a\u0000b', type: 'text' }], type: 'codeBlock' }))), 'unspellable-character') - assert.equal(markdown(adfToMarkdown(document({ content: [{ text: '', type: 'text' }], type: 'codeBlock' }))), 'unsupported-node-shape: a text node holds text: this one has none') +test('spells the characters CommonMark rewrites', () => { + assert.equal(readsBack(document(paragraph({ text: 'a\r\r\nb', type: 'text' }))), 'a !adf:text{text="\\n"}b\n') + assert.equal(readsBack(document(paragraph({ marks: [{ type: 'strong' }], text: '\u0000\u0000a\u0000', type: 'text' }))), '**!adf:text{text="\\u0000\\u0000"}a!adf:text{text="\\u0000"}**\n') + assert.equal(readsBack(document(paragraph({ marks: [{ attrs: { href: 'u' }, type: 'link' }], text: 'a\rb\u0000', type: 'text' }))), '[a b!adf:text{text="\\u0000"}](u)\n') + assert.equal(readsBack(document({ attrs: { level: 1 }, content: [{ text: '\r', type: 'text' }], type: 'heading' })), '# \n') + assert.equal(readsBack(document({ content: [{ text: 'a\rb', type: 'text' }], type: 'codeBlock' })), '```adf:codeBlock\n{\n "content": [\n {\n "text": "a\\rb",\n "type": "text"\n }\n ]\n}\n```\n') + readsBack(document({ content: [{ text: 'a\u0000b', type: 'text' }], type: 'codeBlock' })) + readsBack(document(paragraph({ marks: [{ type: 'code' }], text: 'a\u0000b', type: 'text' }))) }) -test('refuses a text node the spelling would empty out', () => { +test('carries a text node holding no text, or holding content', () => { const nested: AdfNode[] = [{ text: 'lost', type: 'text' }] - const empty = 'unsupported-node-shape: a text node holds text: this one has none' - const holding = 'unsupported-node-shape: a text node holds no content: this one holds some' - assert.equal(markdown(adfToMarkdown(document(paragraph({ text: '', type: 'text' })))), empty) - assert.equal(markdown(adfToMarkdown(document(paragraph({ marks: [{ type: 'code' }], text: '', type: 'text' })))), empty) - assert.equal(markdown(adfToMarkdown(document(paragraph({ content: nested, text: 'x', type: 'text' })))), holding) - assert.equal(markdown(adfToMarkdown(document(paragraph({ content: nested, marks: [{ type: 'code' }], text: 'x', type: 'text' })))), holding) + assert.equal(readsBack(document(paragraph({ text: '', type: 'text' }))), '!adf:carry{json="{\\"text\\":\\"\\",\\"type\\":\\"text\\"}"}\n') + assert.equal(readsBack(document(paragraph({ type: 'text' }, { text: 'a', type: 'text' }))), '!adf:carry{json="{\\"type\\":\\"text\\"}"}a\n') + readsBack(document(paragraph({ marks: [{ type: 'code' }], text: '', type: 'text' }))) + readsBack(document(paragraph({ marks: [{ type: 'em' }], text: 'a', type: 'text' }, { marks: [{ type: 'em' }], text: '', type: 'text' }))) + readsBack(document(paragraph({ content: nested, text: 'x', type: 'text' }))) + readsBack(document(paragraph({ content: nested, marks: [{ type: 'code' }], text: 'x', type: 'text' }))) + readsBack(document({ content: [{ text: '', type: 'text' }], type: 'codeBlock' })) + readsBack(document({ content: [{ text: 'a', type: 'text' }, { type: 'text' }], type: 'codeBlock' })) }) test('carries a mark run whose edge holds whitespace CommonMark flanking counts', () => { @@ -688,8 +695,12 @@ test('refuses the content and slot an inline directive has no room for', () => { assert.equal(refused({ text: 'x', type: 'status' }), neither('status', 'text')) assert.equal(refused({ content: [{ text: 'x', type: 'text' }], type: 'hardBreak' }), neither('hardBreak', 'content')) assert.equal(refused({ text: 'x', type: 'hardBreak' }), neither('hardBreak', 'text')) - assert.equal(refused({ attrs: { text: 'a\nb' }, type: 'status' }), 'unspellable-whitespace: the status content slot holds a newline no inline directive spans') - assert.equal(refused({ attrs: { text: 'a\u0000b' }, type: 'status' }), 'unspellable-character: a status content slot holds a null character CommonMark replaces') +}) + +test('carries an inline node whose content slot holds a line ending or a null character', () => { + assert.equal(readsBack(document(paragraph({ attrs: { text: 'a\nb' }, type: 'status' }))), '!adf:carry{json="{\\"attrs\\":{\\"text\\":\\"a\\\\nb\\"},\\"type\\":\\"status\\"}"}\n') + readsBack(document(paragraph({ attrs: { text: 'a\rb' }, type: 'emoji' }))) + readsBack(document(paragraph({ attrs: { id: 'x', text: 'a\u0000b' }, type: 'mention' }))) }) test('spells the directive marks around the longest run they cover', () => { diff --git a/src/markdown/emit/adf-to-markdown.ts b/src/markdown/emit/adf-to-markdown.ts index 5c17534..cac3017 100644 --- a/src/markdown/emit/adf-to-markdown.ts +++ b/src/markdown/emit/adf-to-markdown.ts @@ -128,9 +128,9 @@ function spellReadableBlock(node: AdfNode, path: ConvertErrorPath, depth: number if (plain !== undefined) return plain if (node.type === 'blockquote') return tryBlockquote(node, path, depth, writing) if (node.type === 'bulletList' || node.type === 'orderedList') return tryList(node, path, depth, writing) - if (node.type === 'codeBlock') return tryCodeBlock(node, path) + if (node.type === 'codeBlock') return readableText(tryCodeBlock(node)) if (node.type === 'heading') return tryHeading(node, path, writing.flavour) - if (node.type === 'mediaSingle') return readableText(tryImage(node, path)) + if (node.type === 'mediaSingle') return readableText(tryImage(node)) if (node.type === 'paragraph') return tryParagraph(node, path, writing.flavour) if (node.type === 'rule') return readableText(tryRule(node)) if (node.type === 'table') return readableText(tryPipeTable(node, path, writing.flavour)) @@ -251,43 +251,36 @@ function tryBlockquote(node: AdfNode, path: ConvertErrorPath, depth: number, wri return success(commonMarkText(quoted(alert ? `\\${text}` : text), inner.value.headroom)) } -function tryCodeBlock(node: AdfNode, path: ConvertErrorPath): Result | undefined { +function tryCodeBlock(node: AdfNode): string | undefined { if (!holdsOnlyAttributes(node, ['language'])) return undefined const slot = languageSlot(nodeAttrs(node)['language']) if (slot.kind === 'attribute') return undefined - const texts = fencedTexts(node, path) - if (!texts.ok) return texts - const [only, ...others] = texts.value ?? [] + const [only, ...others] = fencedTexts(node) ?? [] if (only === undefined || others.length > 0) return undefined - return success(commonMarkText(fencedCodeBlock(slot.kind === 'fence' ? slot.info : '', only))) + return fencedCodeBlock(slot.kind === 'fence' ? slot.info : '', only) } // spec/flavour.md, The CommonMark blocks: one fence per text node, the language on each; with no fence to carry it, the attribute does. function emitCodeDirective(node: AdfNode, model: BlockNodeModel, path: ConvertErrorPath, depth: number): Result { - const texts = fencedTexts(node, path) - if (!texts.ok) return texts - if (texts.value === undefined) return commonMarkLine(carriedBlock(node, path, depth)) - const slot: LanguageSlot = texts.value.length === 0 ? { kind: 'attribute' } : languageSlot(nodeAttrs(node)['language']) + const texts = fencedTexts(node) + if (texts === undefined) return commonMarkLine(carriedBlock(node, path, depth)) + const slot: LanguageSlot = texts.length === 0 ? { kind: 'attribute' } : languageSlot(nodeAttrs(node)['language']) const opener = spellBlockDirectiveOpener(node, model, path, slot.kind === 'attribute' ? [] : ['language']) if (opener === undefined) return commonMarkLine(carriedBlock(node, path, depth)) if (!opener.ok) return opener const info = slot.kind === 'fence' ? slot.info : '' - return success(directivePair(node, opener.value, texts.value.map((text) => fencedCodeBlock(info, text)).join('\n'))) + return success(directivePair(node, opener.value, texts.map((text) => fencedCodeBlock(info, text)).join('\n'))) } -// The text each fence holds, or `undefined` where a child is no plain text node, which the carry holds instead. -function fencedTexts(node: AdfNode, path: ConvertErrorPath): Result { - if (node.content === undefined) return success(['']) - if (node.content.some((child) => !isUnmarkedBareText(child))) return success(undefined) +// The text each fence holds, or `undefined` where a child is no text a fence keeps, which the carry holds instead. +function fencedTexts(node: AdfNode): string[] | undefined { + if (node.content === undefined) return [''] const texts: string[] = [] - for (const [index, child] of node.content.entries()) { - const childPath = [...path, 'content', index] - if (typeof child.text !== 'string' || child.text === '') return failure('unsupported-node-shape', 'a text node holds text: this one has none', childPath) - if (/\r/.test(child.text)) return failure('unspellable-character', 'a codeBlock holds no carriage return CommonMark keeps: this text holds one', childPath) - if (holdsNullCharacter(child.text)) return failure('unspellable-character', 'a codeBlock holds a null character CommonMark replaces', childPath) + for (const child of node.content) { + if (!isUnmarkedBareText(child) || typeof child.text !== 'string' || child.text === '' || /\r/.test(child.text) || holdsNullCharacter(child.text)) return undefined texts.push(child.text) } - return success(texts) + return texts } function tryHeading(node: AdfNode, path: ConvertErrorPath, flavour: Flavour): Result | undefined { diff --git a/src/markdown/emit/image.ts b/src/markdown/emit/image.ts index e9ca734..c1e8f2f 100644 --- a/src/markdown/emit/image.ts +++ b/src/markdown/emit/image.ts @@ -1,5 +1,4 @@ import type { AdfNode } from '../../adf/document.ts' -import type { ConvertErrorPath } from '../../result.ts' import { holdsOnlyAttributes, nodeAttrs, nodeContent } from '../../adf/document.ts' import { serializeCanonicalJson } from '../../canonical-json.ts' import { tryImageLine } from './inline-line.ts' @@ -7,10 +6,10 @@ import { tryImageLine } from './inline-line.ts' const centeredMediaSingle = '{"layout":"center"}' const imageAttributes = ['alt', 'type', 'url'] -export function tryImage(node: AdfNode, path: ConvertErrorPath): string | undefined { +export function tryImage(node: AdfNode): string | undefined { const image = imageShape(node) if (image === undefined) return undefined - return tryImageLine(image.alt, image.url, [...path, 'content', 0]) + return tryImageLine(image.alt, image.url) } function imageShape(node: AdfNode): { alt: string | undefined; url: string } | undefined { diff --git a/src/markdown/emit/inline-line.ts b/src/markdown/emit/inline-line.ts index 636b120..4b8703f 100644 --- a/src/markdown/emit/inline-line.ts +++ b/src/markdown/emit/inline-line.ts @@ -8,7 +8,7 @@ import { claimsLine, holdsNullCharacter, trimTrailingSpace } from '../commonmark import { commonMarkLink, linkHref, markSpelling, spellMarkAttributes } from '../mark-spellings.ts' import { identicalMark, isBareText, nodeAttrs, nodeContent, nodeMarks } from '../../adf/document.ts' import { escapeUnbalanced, spellDestination } from '../commonmark/link-syntax.ts' -import { failure, faulted, success, type ConvertErrorPath, type Result } from '../../result.ts' +import { failure, success, type ConvertErrorPath, type Result } from '../../result.ts' import { highlightDelimiter } from '../plain/conventions.ts' import { inlineNodeModel } from '../../adf/inline-nodes.ts' import { joinsWhenRead, textBreakSpelling } from '../adjacent-text.ts' @@ -67,13 +67,12 @@ export function tryPipeCell(nodes: readonly AdfNode[], path: ConvertErrorPath, f return emitted.value.line } -export function tryImageLine(alt: string | undefined, href: string, path: ConvertErrorPath): string | undefined { +export function tryImageLine(alt: string | undefined, href: string): string | undefined { if (alt !== undefined && (/^[ \t]|[ \t]$|[\n\r]/.test(alt) || holdsNullCharacter(alt))) return undefined const destination = spellDestination(href) if (destination === undefined) return undefined const description: InlineSegment[] = alt === undefined ? [] : [{ escaping: 'bracketed', text: alt }] - const attempt = attemptLine([syntax('!['), ...description, syntax(`](${destination})`)], 'paragraph', path, 'lossless') - return attempt.ok ? attempt.value.line : undefined + return attemptLine([syntax('!['), ...description, syntax(`](${destination})`)], 'paragraph', 'lossless').line } function emitLine(nodes: readonly AdfNode[], container: LineContainer, path: ConvertErrorPath, flavour: Flavour): Result { @@ -87,12 +86,11 @@ function emitLine(nodes: readonly AdfNode[], container: LineContainer, path: Con if (!taken.ok) return taken continue } - const attempt = attemptLine(emission.value.segments, container, path, flavour) - if (!attempt.ok) return attempt - if (attempt.value.line !== undefined) { - return success({ line: attempt.value.line, openingLinkAsDirective: fallbacks.openingLinkAsDirective, segments: emission.value.segments }) + const attempt = attemptLine(emission.value.segments, container, flavour) + if (attempt.line !== undefined) { + return success({ line: attempt.line, openingLinkAsDirective: fallbacks.openingLinkAsDirective, segments: emission.value.segments }) } - const taken = takeFallback(fallbacks, attempt.value.fallback, path) + const taken = takeFallback(fallbacks, attempt.fallback, path) if (!taken.ok) return taken } } @@ -117,12 +115,23 @@ function lineSegments(nodes: readonly AdfNode[], container: LineContainer, path: return success({ segments: spellEdgeWhitespace(emission.value.segments) }) } -function attemptLine(segments: readonly InlineSegment[], container: LineContainer, path: ConvertErrorPath, flavour: Flavour): Result { +function attemptLine(segments: readonly InlineSegment[], container: LineContainer, flavour: Flavour): LineAttempt { const verdict = lineVerdict(segments, container, flavour) - if (verdict.kind === 'opening-link') return success({ fallback: 'opening-link' }) - if (verdict.kind === 'unspellable-run') return success({ fallback: verdict.runs[0] }) - if (verdict.kind === 'claimed-line') return failure('unspellable-line-start', `block parsing would claim the emitted line ${JSON.stringify(verdict.text)}`, path) - return success({ line: verdict.text }) + if (verdict.kind === 'opening-link') return { fallback: 'opening-link' } + if (verdict.kind === 'unspellable-run') return { fallback: verdict.runs[0] } + if (verdict.kind === 'line') return { line: verdict.text } + return { fallback: lineOpener(segments, verdict.line) } +} + +// A code span's backticks are what block parsing claims, and its segment names its node; a segment naming none leaves takeFallback to refuse the line. +function lineOpener(segments: readonly InlineSegment[], line: number): NodeRange { + let breaks = 0 + const opener = segments.find((segment) => { + const opens = breaks === line && segment.text !== '' + breaks += segment.text.split('\n').length - 1 + return opens + }) + return opener?.nodes ?? { first: 0, last: -1 } } // The fallbacks in the order a line takes them, or the line where it takes none. @@ -238,7 +247,7 @@ function emitLeaf(node: AdfNode, context: InlineContext, index: number): Result< const types = nodeMarks(node).map((mark) => mark.type) if (new Set(types).size !== types.length) return failure('unsupported-node-shape', `a ${node.type} node carries one mark type twice`, path) const model = inlineNodeModel(node.type) - if (model === undefined) return emitText(node, context, index, path) + if (model === undefined) return success(emitText(node, context, index)) if (node.type === 'hardBreak') return emitHardBreak(node, model, context, index, path) return emitInlineDirective(node, model, index, path) } @@ -260,38 +269,27 @@ function emitInlineDirective(node: AdfNode, model: InlineNodeModel, index: numbe const slot = model.textAttribute === undefined ? undefined : nodeAttrs(node)[model.textAttribute] if (slot === undefined) return success({ segments: [syntax(spellInlineLeafDirective(node.type, attributes))] }) if (typeof slot !== 'string') return success({ carry: { first: index, last: index } }) - const spans = slotLineEndingFault(node.type, slot) - if (spans !== undefined) return faulted(spans, path) - if (holdsNullCharacter(slot)) return failure('unspellable-character', `a ${node.type} content slot holds a null character CommonMark replaces`, path) + if (slotLineEndingFault(node.type, slot) !== undefined || holdsNullCharacter(slot)) return success({ carry: { first: index, last: index } }) const content: InlineSegment[] = slot === '' ? [] : [{ escaping: 'bracketed', text: slot }] return success({ segments: [syntax(spellInlineDirectiveOpener(node.type)), ...content, syntax(`]${attributes}`)] }) } -function textHoldingContent(node: AdfNode, path: ConvertErrorPath): Result | undefined { - return node.type === 'text' && nodeContent(node).length > 0 ? failure('unsupported-node-shape', 'a text node holds no content: this one holds some', path) : undefined -} - -function emitText(node: AdfNode, context: InlineContext, index: number, path: ConvertErrorPath): Result { - const holding = textHoldingContent(node, path) - if (holding !== undefined) return holding - if (!isBareText(node)) return success({ carry: { first: index, last: index } }) - if (typeof node.text !== 'string' || node.text === '') return failure('unsupported-node-shape', 'a text node holds text: this one has none', path) - if (/\r/.test(node.text)) return failure('unspellable-character', 'a text node holds a carriage return CommonMark rewrites', path) - if (holdsNullCharacter(node.text)) return failure('unspellable-character', 'a text node holds a null character CommonMark replaces', path) +// spec/flavour.md, Inline nodes: CommonMark reads a carriage return as a line ending, and a null character as U+FFFD. +function emitText(node: AdfNode, context: InlineContext, index: number): Emission { + if (!isBareText(node) || typeof node.text !== 'string' || node.text === '') return { carry: { first: index, last: index } } const escaping: InlineEscaping = context.bracketed ? 'bracketed' : 'backslash' - const parts = node.text.split(/(\n+)/).filter((part) => part !== '') - return success({ segments: parts.map((part) => (part.startsWith('\n') ? textDirectiveSegment(part) : { escaping, text: part })) }) + const parts = node.text.split(/(\n+|\0+|\r)/).filter((part) => part !== '') + return { segments: parts.map((part) => (part === '\r' ? syntax(' ') : /^[\n\0]/.test(part) ? textDirectiveSegment(part) : { escaping, text: part })) } } function emitMarkedRun(nodes: readonly AdfNode[], mark: AdfMark, depth: number, index: number, context: InlineContext): Result { - const path = nodePath(context, index) const range: NodeRange = { first: index, last: index + nodes.length - 1 } if (mark.type === 'backgroundColor' && context.flavour === 'plain') return emitHighlight(nodes, depth, range, context) const spelling = markSpelling(mark.type) if (spelling === undefined) return success({ carry: range }) const attributes = spellMarkAttributes(mark, spelling) if (attributes === undefined) return success({ carry: range }) - if (spelling.kind === 'code') return emitCodeSpan(nodes, depth, range, path) + if (spelling.kind === 'code') return success(emitCodeSpan(nodes, depth, range)) if (spelling.kind === 'emphasis') return emitEmphasis(nodes, spelling.spelling, depth, range, context) const link = spelling.kind === 'link' ? tryLink(nodes, mark, depth, range, context) : undefined if (link !== undefined) return link @@ -325,20 +323,17 @@ function emitHighlight(nodes: readonly AdfNode[], depth: number, range: NodeRang } // Each node its own span: CommonMark reads two adjacent text nodes in one span back as one. -function emitCodeSpan(nodes: readonly AdfNode[], depth: number, range: NodeRange, path: ConvertErrorPath): Result { - const spans: string[] = [] - for (const node of nodes) { - const holding = textHoldingContent(node, path) - if (holding !== undefined) return holding - if (!isBareText(node) || nodeMarks(node).length !== depth + 1) return success({ carry: range }) +function emitCodeSpan(nodes: readonly AdfNode[], depth: number, range: NodeRange): Emission { + const segments: InlineSegment[] = [] + for (const [offset, node] of nodes.entries()) { const { text } = node - if (typeof text !== 'string' || text === '') return failure('unsupported-node-shape', 'a text node holds text: this one has none', path) - if (/[\n\r]/.test(text)) return success({ carry: range }) - if (holdsNullCharacter(text)) return failure('unspellable-character', 'a code span holds a null character CommonMark replaces', path) + if (!isBareText(node) || nodeMarks(node).length !== depth + 1 || typeof text !== 'string' || text === '' || /[\n\r]/.test(text) || holdsNullCharacter(text)) return { carry: range } + if (offset > 0) segments.push(syntax(textBreakSpelling)) const fence = '`'.repeat(longestBacktickRun(text) + 1) - spans.push(`${fence}${needsPadding(text) ? ` ${text} ` : text}${fence}`) + const index = range.first + offset + segments.push({ escaping: 'none', nodes: { first: index, last: index }, text: `${fence}${needsPadding(text) ? ` ${text} ` : text}${fence}` }) } - return success({ segments: [syntax(spans.join(textBreakSpelling))] }) + return { segments } } function needsPadding(text: string): boolean { diff --git a/src/markdown/parse/markdown-to-adf.test.ts b/src/markdown/parse/markdown-to-adf.test.ts index b542d01..c5a1aca 100644 --- a/src/markdown/parse/markdown-to-adf.test.ts +++ b/src/markdown/parse/markdown-to-adf.test.ts @@ -304,10 +304,8 @@ test('names the directive form a node CommonMark spells refuses', () => { assert.deepEqual(content(markdownToAdf('!adf:bulletList\n!adf:listItem\n---\n!adf:/listItem\n!adf:/bulletList\n')), [bulletList(item({ type: 'rule' }))]) }) -// The spelling the emitter refuses gives the emitter's own error, never a second name for it. -test('gives back the refusal the CommonMark spelling itself raises', () => { - const lineStart = '!adf:blockquote\n` `` `\n!adf:/blockquote\n' - assert.equal(content(markdownToAdf(lineStart)), 'unspellable-line-start: block parsing would claim the emitted line "``` `` ```"') +test('refuses the directive form of a blockquote opening with a code span the carry spells', () => { + assert.equal(content(markdownToAdf('!adf:blockquote\n` `` `\n!adf:/blockquote\n')), 'unsupported-node-shape: blockquote takes the CommonMark spelling, not the directive form') }) test('names the directive name no node reads back to', () => { @@ -1010,6 +1008,11 @@ test('names the content slot no lone plain text node reads back from', () => { assert.equal(content(markdownToAdf('Part !adf:mention{id=b1c2 text=A}.\n')), 'unsupported-node-shape: mention spells its text attribute in the content slot, never in {attrs}') }) +test('reads the characters CommonMark rewrites from their spellings', () => { + assert.deepEqual(content(markdownToAdf('a b!adf:text{text="\\u0000\\u0000"}\n')), [paragraph('a\rb\u0000\u0000')]) + assert.equal(content(markdownToAdf('!adf:text{text="\\u0000a"}\n')), 'unsupported-node-shape: text spells one run of spaces and tabs, one run of newlines, or one run of null characters: this one spells none of them') +}) + test('reads the whitespace the reserved text directive carries', () => { assert.deepEqual(content(markdownToAdf('!adf:text{text=" "}a\n')), [paragraph(' a')]) assert.deepEqual(content(markdownToAdf('a!adf:text{text="\\n"}b\n')), [paragraph('a\nb')]) @@ -1018,7 +1021,7 @@ test('reads the whitespace the reserved text directive carries', () => { }) test('names the text directive spelling no whitespace run reads back from', () => { - const named = 'unsupported-node-shape: text spells one run of spaces and tabs, or one run of newlines: this one spells neither' + const named = 'unsupported-node-shape: text spells one run of spaces and tabs, one run of newlines, or one run of null characters: this one spells none of them' assert.equal(content(markdownToAdf('!adf:text{text=hi}\n')), named) assert.equal(content(markdownToAdf('!adf:text{text=" \\n"}\n')), named) assert.equal(content(markdownToAdf('!adf:text{text=""}\n')), named) diff --git a/src/markdown/text-directive.ts b/src/markdown/text-directive.ts index 05ce403..92691d7 100644 --- a/src/markdown/text-directive.ts +++ b/src/markdown/text-directive.ts @@ -2,7 +2,7 @@ import type { DirectiveSpan, Read } from './directive-syntax.ts' import { readSoleStringAttribute, spellAttributes, spellInlineLeafDirective, spellStringAttribute, unsupportedNodeShape } from './directive-syntax.ts' const name = 'text' -const whitespaceRun = /^(?:[ \t]+|\n+)$/ +const spelledRun = /^(?:[ \t]+|\n+|\0+)$/ export const textDirectiveName = name @@ -14,6 +14,6 @@ export function readTextDirective(span: DirectiveSpan): Read | undefined if (span.name !== name) return undefined const spelled = readSoleStringAttribute(span, name) if (spelled.fault !== undefined) return spelled - if (!whitespaceRun.test(spelled.value)) return { fault: unsupportedNodeShape(`${name} spells one run of spaces and tabs, or one run of newlines: this one spells neither`) } + if (!spelledRun.test(spelled.value)) return { fault: unsupportedNodeShape(`${name} spells one run of spaces and tabs, one run of newlines, or one run of null characters: this one spells none of them`) } return spelled } diff --git a/src/result.ts b/src/result.ts index 7ca127b..8c86450 100644 --- a/src/result.ts +++ b/src/result.ts @@ -5,8 +5,6 @@ export type ConvertErrorCode = | 'unknown-directive-name' | 'unmappable-html' | 'unmappable-image' - | 'unspellable-character' - | 'unspellable-line-start' | 'unspellable-whitespace' | 'unsupported-document-version' | 'unsupported-nesting-depth' diff --git a/todo.md b/todo.md index 5b1ebc7..44ef38d 100644 --- a/todo.md +++ b/todo.md @@ -25,7 +25,6 @@ | ID | Release | Exempt | Item | R | S | A | G | Goals | Score | |---|---|---|---|---|---|---|---|---|---| | 7 | 0.2.0 | | **Ship HTML: `adfToHtml`, `htmlToAdf`, and `markdownToHtml` / `htmlToMarkdown` composed through ADF.** | 6 | 9 | 9 | 9 | 2, 3 | 25.8 | -| 55 | 0.2.0 | defect | **Spell every document the emitter refuses today for a carriage return, a NUL, a code-span line start, a newline inside an emoji, mention or status, or a text node whose text is empty or missing, or which holds content.** | 5 | 5 | 7 | 9 | 1 | 25.8 | | 45 | 0.2.0 | | **Replace `isAdfDocument` with a reader returning `Result`.** | 2 | 3 | 6 | 8 | 1 | 25.2 | | 6 | 0.2.0 | decision | **Specify the HTML dialect.** | 2 | 6 | 7 | 8 | 2, 3 | 24.7 | | 64 | 0.2.0 | defect | **Read back unchanged on V8 every JSON key the emitter writes, with a fixture whose key holds `\`, `"` or a control character.** | 4 | 4 | 6 | 8 | 1, 7 | 23.0 | @@ -63,24 +62,6 @@ Lands after items 6, 59, 60, 61 and 62. The CommonMark spec suite also runs agai `markdownToHtml`. The README documents HTML as it documents markdown, and its tagline and `package.json`'s `description` regain HTML. -### 55. Spell every document the emitter refuses today for a carriage return, a NUL, a code-span line start, a newline inside an emoji, mention or status, or a text node whose text is empty or missing, or which holds content. - -Today `inline-line.ts` refuses a carriage return or NUL in text, and a NUL in an emoji, mention or -status (`unspellable-character`); and a paragraph opening with a code-span run -(`unspellable-line-start`); `fencedTexts` refuses the same characters in a code block; -`inline-line.ts` refuses a newline in an emoji, mention or status (`unspellable-whitespace`); -`inline-line.ts` and `fencedTexts` refuse a text node whose `text` is empty or missing, and -`inline-line.ts` one holding `content`. In text, a carriage return is CommonMark's ` ` (Goal 4), -and a NUL is `!adf:text{text="\u0000"}` holding just the NUL, since CommonMark reads `�` as -U+FFFD (a writer panel, 3 of 3, 2026-10-04). The carries spell every other case: the inline carry in -a paragraph, the block carry in a code block. - -The maintainer approved on 2026-10-04: §Markdown in is a canonical fixpoint drops its code-span -exception, §Which code a cause takes drops `unspellable-whitespace` as an emitter cause; -`unspellable-character` and `unspellable-line-start` retire. `unspellable-whitespace` stays, since -the reader raises it for a content slot spanning a newline. Found by the README-goals audit, -2026-10-03. - ### 45. Replace `isAdfDocument` with a reader returning `Result`. Goal 1 has every call return a result; the boolean guard is the one export that does not, and it