Refuse the pipe form where a link hides a pipe, and fall back to the directive form for every image CommonMark cannot spell

corpus/round-trip/block-nodes/container-crossing.json

@@ -34,13 +34,31 @@
                 {
                   "content": [
                     {
-                      "text": "Plain item.",
+                      "content": [
+                        {
+                          "attrs": {
+                            "title": "Deeper still"
+                          },
+                          "content": [
+                            {
+                              "content": [
+                                {
+                                  "text": "Two list levels down.",
                                   "type": "text"
                                 }
                               ],
                               "type": "paragraph"
                             }
                           ],
+                          "type": "expand"
+                        }
+                      ],
+                      "type": "listItem"
+                    }
+                  ],
+                  "type": "bulletList"
+                }
+              ],
               "type": "listItem"
             }
           ],
@@ -64,6 +82,23 @@
                 }
               ],
               "type": "expand"
+            },
+            {
+              "attrs": {
+                "panelType": "note"
+              },
+              "content": [
+                {
+                  "content": [
+                    {
+                      "text": "Superseded by the next run.",
+                      "type": "text"
+                    }
+                  ],
+                  "type": "paragraph"
+                }
+              ],
+              "type": "panel"
             }
           ],
           "type": "blockquote"

corpus/round-trip/block-nodes/container-crossing.md

@@ -2,9 +2,15 @@
 - :::panel warning
   Check the torque before signing off.
   :::
-- Plain item.
+- - :::expand {title="Deeper still"}
+    Two list levels down.
+    :::
 
 > :::expand {title="Full build log"}
 > The build ran for 11 minutes.
 > :::
+>
+> :::panel note
+> Superseded by the next run.
+> :::
 ::::

corpus/round-trip/block-nodes/image-boundary.json

@@ -82,6 +82,37 @@
         }
       ],
       "type": "mediaSingle"
+    },
+    {
+      "attrs": {
+        "layout": "center"
+      },
+      "content": [
+        {
+          "attrs": {
+            "alt": "",
+            "type": "external",
+            "url": "https://example.com/moon.png"
+          },
+          "type": "media"
+        }
+      ],
+      "type": "mediaSingle"
+    },
+    {
+      "attrs": {
+        "layout": "center"
+      },
+      "content": [
+        {
+          "attrs": {
+            "type": "external",
+            "url": "https://example.com/plan.png?a=1&b=2"
+          },
+          "type": "media"
+        }
+      ],
+      "type": "mediaSingle"
     }
   ],
   "type": "doc",

corpus/round-trip/block-nodes/image-boundary.md

@@ -16,3 +16,11 @@ Taken from Kiruna.
 :::mediaSingle {layout=center}
 ::media {alt="The assembly drawing" marks="[{\"attrs\":{\"href\":\"https://example.com/assembly\"},\"type\":\"link\"}]" type=external url="https://example.com/assembly.png"}
 :::
+
+:::mediaSingle {layout=center}
+::media {alt="" type=external url="https://example.com/moon.png"}
+:::
+
+:::mediaSingle {layout=center}
+::media {type=external url="https://example.com/plan.png?a=1&b=2"}
+:::

corpus/unspellable/media-empty-alt.error

@@ -1 +0,0 @@
-ambiguous-attribute-spelling

corpus/unspellable/media-empty-alt.json

@@ -1,22 +0,0 @@
-{
-  "content": [
-    {
-      "attrs": {
-        "layout": "center"
-      },
-      "content": [
-        {
-          "attrs": {
-            "alt": "",
-            "type": "external",
-            "url": "https://example.com/moon.png"
-          },
-          "type": "media"
-        }
-      ],
-      "type": "mediaSingle"
-    }
-  ],
-  "type": "doc",
-  "version": 1
-}

spec/flavour.md

@@ -42,8 +42,8 @@ normalizes to it through the round-trip.
   the leading delimiter of a construct that would otherwise open, re-scan from there, and
   repeat — with the opener literal the closer parses as text, so `*not emphasis*` is
   `\*not emphasis*`, one backslash.
-- Blocks separated by one blank line at document level and between CommonMark blocks; two
+- Blocks separated by one blank line at document level, inside a blockquote and between CommonMark
-  directive blocks inside a container take none. No trailing whitespace outside a code block's
+  blocks; two directive blocks inside a container take none. No trailing whitespace outside a code block's
   content, single trailing newline; a document with no blocks is the empty string.
 
 ## Directives
@@ -90,7 +90,8 @@ container is open, a named error otherwise.
 **Attributes**: `{key=value key2="two words"}`. `{attrs}` is optional in every form, and `{}` is
 valid — no attributes. A bare value matches `[A-Za-z0-9_-]+`; any other value is double-quoted
 with JSON string escaping (`\"` `\\` `\n` `\t` `\uXXXX`, …) — total over
-Unicode, and raw newlines never appear inside quotes. All values are strings at the grammar
+Unicode, and raw newlines never appear inside quotes. The closing `}` is the first one outside
+quotes, since a quoted value holds `}` unescaped. All values are strings at the grammar
 level; each node's section assigns types. Canonical form orders keys alphabetically, spells
 values bare wherever allowed, inside quotes escapes only what it must using the shortest escape
 form, and omits empty `{attrs}` except where the `{` itself claims the directive
@@ -200,7 +201,9 @@ The moon, at night.
 a `mediaSingle` with attrs exactly `{"layout":"center"}` holding an `external` `media` — `url`
 from the destination, `alt` the description's plain-text content when non-empty. `adfToMarkdown`
 emits the image form for exactly that shape — those attrs and no others, no marks on either
-node, no caption, and a `media` carrying nothing beyond `alt`, `type` and `url`. An image amid
+node, no caption, and a `media` carrying nothing beyond `alt`, `type` and `url` — and only where
+CommonMark spells the pair: a destination or a description the image form cannot hold, an empty
+`alt` included, takes the directive form instead. An image amid
 other text, or one carrying a title, is a named error: `mediaInline` carries a media
 `collection` + `id`, never a URL, and no media node carries a title.
 
@@ -210,9 +213,9 @@ One header row plus plain inline cells is a pipe table; anything richer is the d
 (AGENTS.md §4). Precisely: a table emits as a pipe table exactly when the `table`, every row
 and every cell carry no attrs and no marks, the first row is all `tableHeader` and the rest all
 `tableCell`, every row has the header's cell count, and every cell holds exactly one attr-less,
-mark-less paragraph — an empty cell holds one empty paragraph — with no `|` in a code span:
+mark-less paragraph — an empty cell holds one empty paragraph — with no `|` in a code span,
-backslash escapes are inert there, so pipe form cannot spell that pipe and the table takes the
+link destination or link title: backslash escapes are inert in everything the inline layer spells
-directive form. A pipe table parses back to exactly that shape.
+as syntax, so pipe form cannot spell that pipe and the table takes the directive form. A pipe table parses back to exactly that shape.
 
 ```
 | Part | Qty |

src/adf-to-markdown.test.ts

@@ -334,10 +334,10 @@ test('spells the image form for exactly the centered external media shape', () =
   assert.equal(markdown(adfToMarkdown(single({ alt: 'The moon', type: 'external', url }))), `![The moon](${url})\n`)
   assert.equal(markdown(adfToMarkdown(single({ type: 'external', url }))), `![](${url})\n`)
   assert.equal(markdown(adfToMarkdown(single({ alt: 'a [b] c', type: 'external', url }))), `![a \\[b\\] c](${url})\n`)
-  assert.equal(code(adfToMarkdown(single({ alt: '', type: 'external', url }))), 'ambiguous-attribute-spelling')
+  const fallback = (attrs: AdfAttributes): boolean => markdown(adfToMarkdown(single(attrs))).startsWith(':::mediaSingle {layout=center}')
-  assert.equal(code(adfToMarkdown(single({ alt: 'a\nb', type: 'external', url }))), 'unspellable-whitespace')
+  assert.ok(fallback({ alt: '', type: 'external', url }))
-  assert.equal(code(adfToMarkdown(single({ alt: 'a\u0000b', type: 'external', url }))), 'unspellable-character')
+  assert.ok(fallback({ alt: 'a\nb', type: 'external', url }) && fallback({ alt: 'a\u0000b', type: 'external', url }))
-  assert.equal(code(adfToMarkdown(single({ type: 'external', url: 'https://example.com/a b>c' }))), 'unspellable-link-destination')
+  assert.ok(fallback({ type: 'external', url: 'https://example.com/a b>c' }) && fallback({ alt: ' moon ', type: 'external', url }))
   assert.equal(code(adfToMarkdown(single({ alt: 4, type: 'external', url }))), 'unsupported-node-shape')
   assert.equal(code(adfToMarkdown(single({ type: 'external', url: 4 }))), 'unsupported-node-shape')
   assert.equal(code(adfToMarkdown(single({ type: 'external', url }, paragraph()))), 'unsupported-node-shape')
@@ -372,4 +372,8 @@ test('spells a table as a pipe table only where every row and cell is plain', ()
   assert.equal(code(adfToMarkdown(table(row(cell('tableHeader', { attrs: { localId: 'a' }, type: 'paragraph' }))))), 'unspelled-node-attribute')
   assert.ok(directive(adfToMarkdown(table(row(cell('tableHeader', { attrs: { level: 1 }, type: 'heading' }))))))
   assert.equal(code(adfToMarkdown(table(row(cell('tableHeader', { content: [{ text: ' a', type: 'text' }], type: 'paragraph' }))))), 'unspellable-whitespace')
+  const marked = (mark: AdfMark): AdfDocument => table(row(cell('tableHeader', { content: [{ marks: [mark], text: 'l', type: 'text' }], type: 'paragraph' })))
+  assert.ok(directive(adfToMarkdown(marked({ attrs: { href: 'https://example.com/?x|y' }, type: 'link' }))))
+  assert.ok(directive(adfToMarkdown(marked({ attrs: { href: 'https://example.com/', title: 'a|b' }, type: 'link' }))))
+  assert.equal(markdown(adfToMarkdown(marked({ attrs: { href: 'https://example.com/x' }, type: 'link' }))), '| [l](https://example.com/x) |\n| --- |\n')
 })

src/adf-to-markdown.ts

@@ -67,7 +67,7 @@ function separationBetween(previous: PlacedBlock, next: PlacedBlock, container:
   if (previous.spelling === 'directive' && next.spelling === 'directive') return success('\n')
   return failure(
     'unspelled-block-separation',
-    `the canonical form leaves the separation between a ${previous.node.type} and a ${next.node.type} in a container body unspelled`,
+    `the canonical form leaves the separation between a ${previous.spelling} and a ${next.spelling} block in a container body unspelled`,
     next.path,
   )
 }

src/corpus.test.ts

@@ -77,7 +77,7 @@ function fenceNestingFault(markdown: string): string | undefined {
   const open: number[] = []
   let codeFence: string | undefined
   for (const line of markdown.split('\n')) {
-    const content = line.replace(/^ {0,3}(?:(?:> ?|[-*+] |\d{1,9}[.)] ) {0,3})*/, '')
+    const content = line.replace(/^[ \t]*(?:(?:> ?|[-*+] |\d{1,9}[.)] )[ \t]*)*/, '')
     const backticks = /^(`{3,}|~{3,})/.exec(content)?.[1]
     if (codeFence !== undefined) {
       if (backticks !== undefined && backticks[0] === codeFence[0] && backticks.length >= codeFence.length) codeFence = undefined
@@ -103,7 +103,9 @@ function fenceNestingFault(markdown: string): string | undefined {
 
 test('the fence nesting check catches a fence a container cannot hold', () => {
   assert.equal(fenceNestingFault(':::panel info\n- :::panel warning\n  B\n  :::\n:::'), '"- :::panel warning" sits in a container fenced with 3 colons')
-  assert.equal(fenceNestingFault('::::panel info\n- :::panel warning\n  B\n  :::\n::::'), undefined)
+  assert.equal(fenceNestingFault(':::panel info\n- - :::panel warning\n    B\n    :::\n:::'), '"- - :::panel warning" sits in a container fenced with 3 colons')
+  assert.equal(fenceNestingFault(':::panel info\n10. :::panel warning\n    B\n    :::\n:::'), '"10. :::panel warning" sits in a container fenced with 3 colons')
+  assert.equal(fenceNestingFault('::::panel info\n- - :::panel warning\n    B\n    :::\n::::'), undefined)
   assert.equal(fenceNestingFault(':::tableCell\n```text\n:::::::panel warning\n:::\n```\n:::'), undefined)
 })
 

src/markdown-image.ts

@@ -1,6 +1,6 @@
 import type { AdfNode } from './adf-document.ts'
+import type { ConvertErrorPath, Result } from './result.ts'
 import { emitImageLine } from './markdown-inline.ts'
-import { failure, type ConvertErrorPath, type Result } from './result.ts'
 import { serializeCanonicalJson } from './canonical-json.ts'
 
 const centeredMediaSingle = '{"layout":"center"}'
@@ -9,9 +9,8 @@ const imageAttributes = ['alt', 'type', 'url']
 export function emitImage(node: AdfNode, path: ConvertErrorPath): Result<string> | undefined {
   const image = imageShape(node)
   if (image === undefined) return undefined
-  const mediaPath = [...path, 'content', 0]
+  const line = emitImageLine(image.alt, image.url, [...path, 'content', 0])
-  if (image.alt === '') return failure('ambiguous-attribute-spelling', 'an empty media alt and an absent one share one image spelling', mediaPath)
+  return line.ok ? line : undefined
-  return emitImageLine(image.alt, image.url, mediaPath)
 }
 
 function imageShape(node: AdfNode): { alt: string | undefined; url: string } | undefined {
@@ -22,7 +21,7 @@ function imageShape(node: AdfNode): { alt: string | undefined; url: string } | u
   const attrs = media.attrs ?? {}
   const alt = attrs['alt']
   const url = attrs['url']
-  if (Object.keys(attrs).some((key) => !imageAttributes.includes(key)) || attrs['type'] !== 'external') return undefined
+  if (Object.keys(attrs).some((key) => !imageAttributes.includes(key)) || attrs['type'] !== 'external' || typeof url !== 'string') return undefined
-  if (typeof url !== 'string' || (alt !== undefined && typeof alt !== 'string')) return undefined
+  if (alt !== undefined && (typeof alt !== 'string' || alt === '')) return undefined
   return { alt, url }
 }

src/markdown-inline.ts

@@ -24,7 +24,9 @@ export function emitInlineLine(nodes: readonly AdfNode[], container: LineContain
 }
 
 export function emitImageLine(alt: string | undefined, href: string, path: ConvertErrorPath): Result<string> {
-  if (alt !== undefined && /[\n\r]/.test(alt)) return failure('unspellable-whitespace', 'a media alt holds a newline no image description spells', path)
+  if (alt !== undefined && /^[ \t]|[ \t]$|[\n\r]/.test(alt)) {
+    return failure('unspellable-whitespace', 'a media alt holds whitespace no image description spells', path)
+  }
   if (alt !== undefined && holdsNullCharacter(alt)) return failure('unspellable-character', 'a media alt holds a null character CommonMark replaces', path)
   const destination = spellDestination(href, path)
   if (!destination.ok) return destination

src/markdown-pipe-table.ts

@@ -1,4 +1,5 @@
 import type { AdfNode } from './adf-document.ts'
+import type { JsonValue } from './json-value.ts'
 import { emitInlineLine } from './markdown-inline.ts'
 import { success, type ConvertErrorPath, type Result } from './result.ts'
 
@@ -48,6 +49,18 @@ function plainParagraph(cell: AdfNode): AdfNode | undefined {
   const content = cell.content ?? []
   const paragraph = content[0]
   if (paragraph === undefined || content.length !== 1 || paragraph.type !== 'paragraph' || !isPlain(paragraph)) return undefined
-  const pipedCode = (paragraph.content ?? []).some((child) => (child.marks ?? []).some((mark) => mark.type === 'code') && (child.text ?? '').includes('|'))
+  return (paragraph.content ?? []).some(spellsPipeAsSyntax) ? undefined : paragraph
-  return pipedCode ? undefined : paragraph
+}
+
+// A pipe the inline layer emits as syntax takes no backslash, so the cell has no pipe spelling.
+function spellsPipeAsSyntax(child: AdfNode): boolean {
+  return (child.marks ?? []).some((mark) => {
+    if (mark.type === 'code') return (child.text ?? '').includes('|')
+    if (mark.type !== 'link') return false
+    return holdsPipe(mark.attrs?.['href']) || holdsPipe(mark.attrs?.['title'])
+  })
+}
+
+function holdsPipe(value: JsonValue | undefined): boolean {
+  return typeof value === 'string' && value.includes('|')
 }

todo.md

@@ -28,15 +28,14 @@ detail is settled at its own milestone.
       carries a `localId` with no spelling, `codeBlock` also `hideLineNumbers`, `uniqueId` and
       `wrap`, `blockquote` also marks, and `hardBreak` `text` and `localId` with no section for
       the carry fallback to reach. Picking one (directive sections for those nodes, or the opaque
-      carry) is a permanent format decision (§8). Three collision sites wait in
+      carry) is a permanent format decision (§8). Two collision sites wait in
       `corpus/unspellable/` meanwhile, each a choice between the absent attribute and the empty
-      value: a `codeBlock` whose info string is empty and `media` with an empty `alt`, which one
+      value: a `codeBlock` whose info string is empty, and an `orderedList` starting at 1,
-      "exactly that shape" rule — as the CommonMark image already uses — could settle together,
+      independent of the totality answer since `order: 9` keeps the markdown form either way.
-      and an `orderedList` starting at 1, independent of the totality answer since `order: 9`
+      Neither has a second spelling to fall back to, which is what settled the third — a `media`
-      keeps the markdown form either way. **Also blocked**: the link rule covers destination spaces only, so two shapes
+      with an empty `alt` takes the directive form (`spec/flavour.md`, the CommonMark image). **Also blocked**: the link rule covers destination spaces only, so two shapes
-      break §2 silently — href `https://example.com/a)b` emits `[t](https://example.com/a)b)`,
+      have no spelling and are refused meanwhile — href `https://example.com/a)b` and title
-      read back as href `…/a` plus literal `b)`; title `He said "hi"` emits
+      `He said "hi"`, both in `corpus/unspellable/`. Two defensible spellings each — angle
-      `[t](u "He said "hi"")`, which holds no title. Two defensible spellings each — angle
       brackets or a backslash escape, and for titles `'…'` or `(…)` besides — so §8 leaves
       the pick here. **Also blocked**: block separation is unstated for a CommonMark block beside a
       directive block in a container body — an `expand` whose content is `paragraph` "A" then a
@@ -71,7 +70,9 @@ detail is settled at its own milestone.
         a `.json` beside the `ConvertErrorCode` it must return, the emitter half of `corpus/errors/`.
   - [ ] **2c — Inline nodes and marks.** `inline-nodes/` green. `InlineSegment.kind` splits into
         its two axes here — escapability (`attribute` for `:text{text="…"}`, `backslash`, `none`)
-        and the emphasis role — rather than gaining a third value that means one of each.
+        and the emphasis role — rather than gaining a third value that means one of each. A lone
+        surrogate in a text node emits verbatim and becomes U+FFFD on any UTF-8 encode, a §2 break
+        plain text still holds open — attribute values already escape it.
   - [ ] **2d — The opaque carry** (§3). Fixtures and emitter together, into
         `corpus/round-trip/opaque-carry/`: an unknown node in both positions, the reserved `adf`
         info string, and the `codeBlock` whose language is `adf`.