Merge pull request 'Escape the quoted attribute value, and pin the canonical form and the pipe cell's edges' (#23) from attribute-escape into main
CI / gate (push) Successful in 4s

This commit was merged in pull request #23.
This commit is contained in:
2026-08-27 00:09:23 +02:00
13 changed files with 331 additions and 57 deletions
@@ -22,5 +22,5 @@ Taken from Kiruna.
:::
:::mediaSingle {layout=center}
::media {type=external url="https://example.com/plan.png?a=1&b=2"}
::media {type=external url="https://example.com/plan.png?a=1\u0026amp;b=2"}
:::
@@ -0,0 +1,60 @@
{
"content": [
{
"attrs": {
"isNumberColumnEnabled": true,
"layout": "wide",
"width": 760.5
},
"content": [
{
"content": [
{
"attrs": {
"colspan": 2,
"colwidth": [
340,
420
]
},
"content": [
{
"content": [
{
"text": "Assembly",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableHeader"
}
],
"type": "tableRow"
}
],
"type": "table"
},
{
"attrs": {
"localId": "01a0-33_z",
"title": "a\nb\tc\"d\\e"
},
"content": [
{
"content": [
{
"text": "Notes.",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "expand"
}
],
"type": "doc",
"version": 1
}
@@ -0,0 +1,11 @@
:::::table {isNumberColumnEnabled=true layout=wide width="760.5"}
::::tableRow
:::tableHeader {colspan=2 colwidth="[340,420]"}
Assembly
:::
::::
:::::
:::expand {localId=01a0-33_z title="a\nb\tc\"d\\e"}
Notes.
:::
@@ -0,0 +1,56 @@
{
"content": [
{
"content": [
{
"text": "a `b ",
"type": "text"
},
{
"attrs": {
"url": "c`d"
},
"type": "inlineCard"
}
],
"type": "paragraph"
},
{
"content": [
{
"attrs": {
"url": "https://example.com/s?q=a&amp;b<c"
},
"type": "inlineCard"
}
],
"type": "paragraph"
},
{
"attrs": {
"extensionKey": "jira",
"parameters": {
"jql": "a|b `c` & d < e"
}
},
"type": "extension"
},
{
"content": [
{
"text": "[a",
"type": "text"
},
{
"attrs": {
"url": "b](c)"
},
"type": "inlineCard"
}
],
"type": "paragraph"
}
],
"type": "doc",
"version": 1
}
@@ -0,0 +1,7 @@
a `b :inlineCard{url="c\u0060d"}
:inlineCard{url="https://example.com/s?q=a\u0026amp;b\u003cc"}
::extension {extensionKey=jira parameters="{\"jql\":\"a\u007cb \u0060c\u0060 \u0026 d \u003c e\"}"}
[a:inlineCard{url="b](c)"}
@@ -0,0 +1,107 @@
{
"content": [
{
"content": [
{
"content": [
{
"content": [
{
"content": [
{
"text": "Cell",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableHeader"
},
{
"content": [
{
"content": [
{
"text": "Note",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableHeader"
}
],
"type": "tableRow"
},
{
"content": [
{
"content": [
{
"content": [
{
"text": " lead",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableCell"
},
{
"content": [
{
"type": "paragraph"
}
],
"type": "tableCell"
}
],
"type": "tableRow"
},
{
"content": [
{
"content": [
{
"content": [
{
"attrs": {
"style": "a|b",
"text": "ok"
},
"type": "status"
}
],
"type": "paragraph"
}
],
"type": "tableCell"
},
{
"content": [
{
"content": [
{
"text": "trail ",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableCell"
}
],
"type": "tableRow"
}
],
"type": "table"
}
],
"type": "doc",
"version": 1
}
@@ -0,0 +1,4 @@
| Cell | Note |
| --- | --- |
| :text{text=" "}lead | |
| :status[ok]{style="a\u007cb"} | trail:text{text=" "} |
+12 -11
View File
@@ -93,12 +93,14 @@ container is open, a named error otherwise.
**Attributes**: `{key=value key2="two words"}`. `{attrs}` is optional in every form, and `{}` is
valid — no attributes. A bare value matches `[A-Za-z0-9_-]+`; any other value is double-quoted
with JSON string escaping (`\"` `\\` `\n` `\t` `\uXXXX`, …) — total over
Unicode, and raw newlines never appear inside quotes. The closing `}` is the first one outside
quotes, since a quoted value holds `}` unescaped. All values are strings at the grammar
level; each node's section assigns types. Canonical form orders keys alphabetically, spells
values bare wherever allowed, inside quotes escapes only what it must using the shortest escape
form, and omits empty `{attrs}` except where the `{` itself claims the directive
(`:hardBreak{}`).
Unicode, and raw newlines never appear inside quotes. A quoted value also escapes `` ` ``, `&`,
`<` and `|` as `\u0060`, `\u0026`, `\u003c` and `\u007c`; `*`, `_`, `~`, `[`, `]` and `(` resolve
after a directive binds and stay raw. The closing `}` is the first one outside quotes, since a
quoted value holds `}` unescaped. All values are strings at the grammar level; each node's section
assigns types.
Canonical form orders keys alphabetically, spells values bare wherever allowed, escapes inside
quotes in the shortest form each escape has, and omits empty `{attrs}` except where the `{` itself
claims the directive (`:hardBreak{}`).
**Escaping**: the emitter backslash-escapes whatever literal text would otherwise parse as
directive syntax — the leading `:` of a would-be directive, `]` inside content, a `{` right
@@ -219,9 +221,8 @@ One header row plus plain inline cells is a pipe table; anything richer is the d
and every cell carry no attrs and no marks, the first row is all `tableHeader` and the rest all
`tableCell`, every row has the header's cell count, and every cell holds exactly one attr-less,
mark-less paragraph — an empty cell holds one empty paragraph — with no `|` anywhere the
inline layer spells as syntax a code span, a link destination or title, an inline directive's
attributes: backslash escapes are inert there, so pipe form cannot spell that pipe and the table
takes the directive form. A pipe table parses back to exactly that shape.
inline layer spells as syntax: a code span, an autolink, a link destination or title. A `|` there
takes the directive form instead. A pipe table parses back to exactly that shape.
```
| Part | Qty |
@@ -238,8 +239,8 @@ inline layer's ordinary CommonMark escaping yields the pipe; each cell is the in
one paragraph, trimmed; canonical form pads cells with single spaces and ends rows with `|`
(optional in input). Named errors: a delimiter or body row whose cell count differs from the
header's, and an alignment colon in the delimiter row — ADF holds no column alignment. In a
pipe cell a hard break is `:hardBreak{}`, a literal `|` is `\|`, and `|` inside a quoted
attribute value is `\u007c` (JSON has no `\|`).
pipe cell a hard break is `:hardBreak{}` and a literal `|` is `\|`; a `|` inside a quoted
attribute value is already `\u007c`, so the split never reaches it.
The directive form nests cells as containers of block content inside `tableRow` containers:
+12 -6
View File
@@ -255,7 +255,14 @@ test('escapes a literal delimiter that would merge with an emitted one', () => {
assert.equal(code(adfToMarkdown(document(paragraph({ text: 'x', type: 'text' }, marked('~a', { type: 'strike' }))))), 'unspellable-mark')
assert.equal(emitted({ text: '`', type: 'text' }, marked('x', { type: 'code' })), '\\``x`\n')
assert.equal(emitted(marked('x', { type: 'code' }), { text: '`', type: 'text' }), '`x`\\`\n')
assert.equal(emitted({ text: '`a', type: 'text' }, marked('b', { type: 'code' })), '\\`a`b`\n')
assert.equal(
emitted({ text: '<https://example.com/', type: 'text' }, { attrs: { url: 'x>' }, type: 'inlineCard' }),
'\\<https://example.com/:inlineCard{url="x>"}\n',
)
assert.equal(emitted({ text: '!', type: 'text' }, marked('x', { attrs: { href: 'https://example.com/' }, type: 'link' })), '\\![x](https://example.com/)\n')
assert.equal(emitted({ text: '[a', type: 'text' }, marked('x', { attrs: { href: 'https://example.com/' }, type: 'link' })), '[a[x](https://example.com/)\n')
assert.equal(emitted({ text: '[a](b)', type: 'text' }), '\\[a](b)\n')
assert.equal(emitted(marked('x', { type: 'underline' }), { text: '{}', type: 'text' }), ':underline[x]\\{}\n')
assert.equal(emitted({ attrs: { text: '' }, type: 'status' }, { text: '{color=red}', type: 'text' }), ':status[]\\{color=red}\n')
assert.equal(emitted(marked('x', { attrs: { href: 'https://example.com/' }, type: 'link' }), { text: '{}', type: 'text' }), '[x](https://example.com/){}\n')
@@ -402,8 +409,8 @@ test('spells a table as a pipe table only where every row and cell is plain', ()
assert.ok(directive(adfToMarkdown(table(row(cell('tableHeader', text('a'), text('b')))))))
assert.equal(code(adfToMarkdown(table(row(cell('tableHeader', { attrs: { localId: 'a' }, type: 'paragraph' }))))), 'unspelled-node-attribute')
assert.equal(
markdown(adfToMarkdown(table(row(cell('tableHeader', { content: [{ type: 'blockCard' }], type: 'paragraph' }))))),
'| :adf{json="{\\"type\\":\\"blockCard\\"}"} |\n| --- |\n',
markdown(adfToMarkdown(table(row(cell('tableHeader', { content: [{ attrs: { url: 'a|b' }, type: 'blockCard' }], type: 'paragraph' }))))),
'| :adf{json="{\\"attrs\\":{\\"url\\":\\"a\\u007cb\\"},\\"type\\":\\"blockCard\\"}"} |\n| --- |\n',
)
assert.equal(code(adfToMarkdown(table(row(cell('tableHeader', { content: [{ text: '\fa', type: 'text' }], type: 'paragraph' }))))), 'unspellable-whitespace')
assert.ok(directive(adfToMarkdown(table(row(cell('tableHeader', { attrs: { level: 1 }, type: 'heading' }))))))
@@ -411,10 +418,8 @@ test('spells a table as a pipe table only where every row and cell is plain', ()
const marked = (mark: AdfMark): AdfDocument => table(row(cell('tableHeader', { content: [{ marks: [mark], text: 'l', type: 'text' }], type: 'paragraph' })))
assert.ok(directive(adfToMarkdown(marked({ attrs: { href: 'https://example.com/?x|y' }, type: 'link' }))))
assert.ok(directive(adfToMarkdown(marked({ attrs: { href: 'https://example.com/', title: 'a|b' }, type: 'link' }))))
const fallsBack = (node: AdfNode): boolean => directive(adfToMarkdown(table(row(cell('tableHeader', { content: [node], type: 'paragraph' })))))
assert.ok(fallsBack({ marks: [{ type: 'code' }], text: 'a|b', type: 'text' }))
assert.ok(fallsBack({ attrs: { style: 'a|b' }, type: 'status' }))
assert.ok(fallsBack({ attrs: { url: 'https://example.com/?a|b' }, type: 'blockCard' }))
const codeSpan: AdfNode = { marks: [{ type: 'code' }], text: 'a|b', type: 'text' }
assert.ok(directive(adfToMarkdown(table(row(cell('tableHeader', { content: [codeSpan], type: 'paragraph' }))))))
assert.equal(markdown(adfToMarkdown(marked({ attrs: { href: 'https://example.com/x' }, type: 'link' }))), '| [l](https://example.com/x) |\n| --- |\n')
})
@@ -427,6 +432,7 @@ test('spells an inline node as a directive with its content slot and attributes'
assert.equal(emitted({ attrs: { id: '1f389', text: 'a]b' }, type: 'emoji' }), ':emoji[a\\]b]{id=1f389}\n')
assert.equal(emitted({ attrs: { data: { url: 'https://example.com/' } }, type: 'inlineCard' }), ':inlineCard{data="{\\"url\\":\\"https://example.com/\\"}"}\n')
assert.equal(emitted({ attrs: { height: 24 }, type: 'mediaInline' }), ':mediaInline{height=24}\n')
assert.equal(emitted({ attrs: { url: 'a`b&c<d|e' }, type: 'inlineCard' }), ':inlineCard{url="a\\u0060b\\u0026c\\u003cd\\u007ce"}\n')
})
test('refuses an inline node attribute no section spells', () => {
+4 -1
View File
@@ -9,6 +9,9 @@ export type SpelledPairs = { fault: AttributeFault; pairs?: undefined } | { faul
const bareToken = /^[A-Za-z0-9_-]+$/
// spec/flavour.md, Attributes.
const quotedEscapes = /[&<`|]/g
export function isBareToken(text: string): boolean {
return bareToken.test(text)
}
@@ -53,5 +56,5 @@ export function spellStringAttribute(text: string): string {
}
function quote(text: string): string {
return JSON.stringify(text)
return JSON.stringify(text).replace(quotedEscapes, (character) => `\\u${character.charCodeAt(0).toString(16).padStart(4, '0')}`)
}
+31 -8
View File
@@ -2,7 +2,7 @@ import { escapesLineClaim, isUnicodeWhitespace, opensBracketedAutolink, startsEn
export type EmphasisRole = 'close' | 'open'
export type InlineEscaping = 'attribute' | 'backslash' | 'bracketed' | 'none'
export type InlineEscaping = 'backslash' | 'bracketed' | 'none'
export type InlineSegment =
| { emphasis: EmphasisRole; escaping: 'none'; mark: string; text: string }
@@ -19,7 +19,7 @@ const delimiters = ['*', '_', '`', '~']
const asciiPunctuation = /[!"#$%&'()*+,\-./:;<=>?@[\\\]^_`{|}~]/
const htmlConstructs = [/^<[!?]/, /^<\/?[A-Za-z][A-Za-z0-9-]*(?:[\s/>]|$)/, /^<[^\s<>@]+@[^\s<>@]+>/]
const inlineDirectiveOpener = /^:[a-z][A-Za-z0-9]*[[{]/
const linkOpener = /\](?=[([:])/
const followsLinkText = /[([:]/
const unicodePunctuation = /[\p{P}\p{S}]/u
export function assembleInlineLine(segments: readonly InlineSegment[], container: LineContainer): AssembledLine {
@@ -65,7 +65,7 @@ function escape(segments: readonly InlineSegment[], container: LineContainer): A
for (let index = 0; index < scan.length; index += 1) {
const escaping = escapings[index]
const escapable = escaping === 'backslash' || escaping === 'bracketed'
if (escapable && (mergesWithSyntax(scan, escapings, index) || opensConstruct(scan, index, escaping === 'bracketed', container, escaped))) {
if (escapable && (mergesWithSyntax(scan, escapings, index) || opensConstruct(scan, escapings, index, escaping === 'bracketed', container, escaped))) {
output += '\\'
escaped.add(index)
}
@@ -128,13 +128,20 @@ function touchesSyntax(scan: string, escapings: readonly (InlineEscaping | undef
}
function isSyntax(escaping: InlineEscaping | undefined): boolean {
return escaping === 'attribute' || escaping === 'none'
return escaping === 'none'
}
function opensConstruct(scan: string, index: number, inBrackets: boolean, container: LineContainer, escaped: ReadonlySet<number>): boolean {
function opensConstruct(
scan: string,
escapings: readonly (InlineEscaping | undefined)[],
index: number,
inBrackets: boolean,
container: LineContainer,
escaped: ReadonlySet<number>,
): boolean {
if (container === 'heading' && closesHeading(scan, index)) return true
if (container === 'paragraph' && claimsLineStart(scan, index)) return true
return claimsCharacter(scan, index, inBrackets, container, escaped)
return claimsCharacter(scan, escapings, index, inBrackets, container, escaped)
}
function claimsLineStart(scan: string, index: number): boolean {
@@ -150,7 +157,14 @@ function closesHeading(scan: string, index: number): boolean {
return index === 0 || /[ \t]/.test(scan.charAt(index - 1))
}
function claimsCharacter(scan: string, index: number, inBrackets: boolean, container: LineContainer, escaped: ReadonlySet<number>): boolean {
function claimsCharacter(
scan: string,
escapings: readonly (InlineEscaping | undefined)[],
index: number,
inBrackets: boolean,
container: LineContainer,
escaped: ReadonlySet<number>,
): boolean {
const character = scan.charAt(index)
const rest = scan.slice(index)
if (inBrackets && (character === '[' || character === ']')) return true
@@ -159,12 +173,21 @@ function claimsCharacter(scan: string, index: number, inBrackets: boolean, conta
if (character === '&') return startsEntityReference(rest)
if (character === '<') return opensBracketedAutolink(rest) || htmlConstructs.some((construct) => construct.test(rest))
if (character === ':') return inlineDirectiveOpener.test(rest)
if (character === '[') return linkOpener.test(rest)
if (character === '[') return opensLink(scan, escapings, index)
if (character === '`') return opensCodeSpan(scan, index, escaped)
if (character === '*' || character === '_' || character === '~') return opensEmphasis(scan, index, escaped)
return false
}
// A `]` the emitter spelled sits inside a construct that binds before link text does.
function opensLink(scan: string, escapings: readonly (InlineEscaping | undefined)[], index: number): boolean {
for (let cursor = index + 1; cursor < scan.length; cursor += 1) {
if (scan.charAt(cursor) !== ']' || isSyntax(escapings[cursor])) continue
if (followsLinkText.test(scan.charAt(cursor + 1))) return true
}
return false
}
function opensCodeSpan(scan: string, index: number, escaped: ReadonlySet<number>): boolean {
if (!startsRun(scan, index, escaped)) return false
const length = runLength(scan, index)
+1 -1
View File
@@ -92,7 +92,7 @@ function carryEdges(segment: InlineSegment, leading: boolean, trailing: boolean)
}
function carriedText(text: string): InlineSegment {
return { escaping: 'attribute', text: spellLeafDirective('text', spellAttributes([['text', spellStringAttribute(text)]])) }
return syntax(spellLeafDirective('text', spellAttributes([['text', spellStringAttribute(text)]])))
}
function spellLeafDirective(name: string, attributes: string): string {
+25 -29
View File
@@ -72,12 +72,11 @@ detail is settled at its own milestone.
the emitter's refusals a corpus home while the directories grow: `corpus/unspellable/`,
a `.json` beside the `ConvertErrorCode` it must return, the emitter half of `corpus/errors/`.
- [x] **2c — Inline nodes and marks.** `inline-nodes/` green. `InlineSegment` splits into its
two axes — escapability (`attribute` for `:text{text="…"}`, `backslash`, `bracketed`,
`none`) and the emphasis role. A lone surrogate in a text node emits verbatim and becomes
U+FFFD on any UTF-8 encode, a §2 break plain text still holds open — attribute values
already escape it. The pipe form's fallback reads the emitted segments rather than naming
the nodes whose attribute values spell a pipe as syntax, so 2e's `\u007c` narrows it in
one place.
two axes — escapability (`backslash`, `bracketed`, `none`) and the emphasis role. A lone
surrogate in a text node emits verbatim and becomes U+FFFD on any UTF-8 encode, a §2 break
plain text still holds open — attribute values already escape it. The pipe form's fallback
reads the emitted segments rather than naming the nodes whose attribute values spell a pipe
as syntax, so 2e's `\u007c` narrows it in one place.
- [x] **2d — The opaque carry** (§3). Fixtures and emitter together, into
`corpus/round-trip/opaque-carry/`: an unknown node in both positions, the reserved `adf`
info string, and the `codeBlock` whose language is `adf` — carried whole ahead of the
@@ -96,24 +95,21 @@ detail is settled at its own milestone.
`inline-directives.ts`, which holds four of the nine marks while the rest are branch
literals in the emitter — and the parser (3) needs every name to make `:em[x]` the named
error `spec/flavour.md` promises.
- [ ] **2e3 — Attribute canonicalization and the quoted value's escape.** Attribute
canonicalization, a pipe cell's whitespace edges, and `\u007c` for a `|` inside a quoted
attribute value. The rule is wider than the pipe: a quoted value spells `` ` ``, `&` and
`<` raw today, so a backtick pair in an `inlineCard`'s `data`, a `status`'s `style` or the
inline carry's `json` reads back as a code span, an entity or raw HTML inside the value.
One escape settles all four, and the pick decides whether a directive or a code span wins
where they overlap (3). `escaping: 'attribute'` earns its keep at that rule or collapses
into `none`: nothing the escaper does tells the two apart today, since a carried segment
holds only spaces, tabs and newlines.
- [x] **2e3 — Attribute canonicalization and the quoted value's escape.**
**Settled** (the maintainer, 2026-08-26): a quoted attribute value escapes `` ` ``, `&`,
`<` and `|` as `\u0060`, `\u0026`, `\u003c` and `\u007c`, in every directive, block and
inline alike — the constructs those four open all bind at or before a directive does, and
nothing else reaches into `{attrs}`. Emitted attributes being inert leaves 3 free to keep
CommonMark's own precedence between a directive and a code span, and collapsed
`escaping: 'attribute'` into `none`. The escaper's link-opener scan skips emitted syntax
to match: a `](` inside a directive escapes no text `[`.
- [ ] **2e4 — The carry's fallback triggers.** `spec/flavour.md` carries a node its section
cannot spell — an attrs key no section lists, a value that is not the section's type, an
arg slot holding no bare token, marks no nesting spells — where the emitter still refuses,
which leaves the refusals a container's own spelling owns. The carry spells
`escaping: 'none'`, which is what makes `tryPipeCell` refuse the pipe form for a carry
whose JSON holds a pipe. The flanking trigger 2e2 added to that list is the odd one out:
`unspellableMark` finds it after assembly and names a mark type against the line's path,
so the failing run needs identifying before the carry can replace the refusal
`corpus/unspellable/mark-inside-word` pins.
which leaves the refusals a container's own spelling owns. The flanking trigger 2e2
added to that list is the odd one out: `unspellableMark` finds it after assembly and
names a mark type against the line's path, so the failing run needs identifying before
the carry can replace the refusal `corpus/unspellable/mark-inside-word` pins.
- [ ] **2e5 — Combined documents and the collision property.** Documents combining nodes rather
than isolating one, and the gate's collision property: no two corpus documents may emit
the same bytes — one spelling for two documents is a round-trip break no parser can undo,
@@ -142,14 +138,14 @@ detail is settled at its own milestone.
every raw-HTML construct in input is an error result. The CommonMark spec suite runs
against it from here (§10). The parser owes `~` the same `can_open`/`can_close` the emitter
assumes — CommonMark flanking, as for `*` — which `spec/flavour.md` does not yet pin, and
the precedence between a directive and the constructs a raw attribute value opens inside it —
a code span, an entity, raw HTML — which one directive alone already reaches until 2e's
escape lands. Whether a claimed line interrupts a paragraph is pinned for the plain case and
unstated for the lazy one: a closing fence on the line after a blockquote's open paragraph is
lazy continuation in CommonMark, which would swallow the fence and leave the container
unclosed. 2e1's `closing-fence-line` orders its blockquote away from that edge meanwhile.
`src/` gets its hierarchy at the same split — `adf/`,
`markdown/`, `html/`, the grammar module shared inside `markdown/` — while the rename is
the precedence in input between a directive and a code span, an entity or raw HTML written
raw inside its attributes — 2e3's escape keeps emitted output clear of that collision, so the
pick binds input alone. Whether a claimed line interrupts a paragraph is pinned for the plain
case and unstated for the lazy one: a closing fence on the line after a blockquote's open
paragraph is lazy continuation in CommonMark, which would swallow the fence and leave the
container unclosed. 2e1's `closing-fence-line` orders its blockquote away from that edge
meanwhile. `src/` gets its hierarchy at the same split — `adf/`, `markdown/`, `html/`, the
grammar module shared inside `markdown/` — while the rename is
still mechanical. Three files do not move whole: `block-directives.ts` and
`inline-directives.ts` each hold a node table milestones 6-7 need in `adf/` beside a
markdown spelling that belongs in `markdown/`, and `directive-attributes.ts` fuses the