Escape the quoted attribute value, and pin the canonical form and the pipe cell's edges #23

Merged
lilleman merged 4 commits from attribute-escape into main 2026-08-27 00:09:23 +02:00
13 changed files with 273 additions and 44 deletions
Showing only changes of commit ec0ae1f5db - Show all commits
@@ -22,5 +22,5 @@ Taken from Kiruna.
:::
:::mediaSingle {layout=center}
::media {type=external url="https://example.com/plan.png?a=1&b=2"}
::media {type=external url="https://example.com/plan.png?a=1\u0026amp;b=2"}
:::
@@ -0,0 +1,60 @@
{
"content": [
{
"attrs": {
"isNumberColumnEnabled": true,
"layout": "wide",
"width": 760.5
},
"content": [
{
"content": [
{
"attrs": {
"colspan": 2,
"colwidth": [
340,
420
]
},
"content": [
{
"content": [
{
"text": "Assembly",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableHeader"
}
],
"type": "tableRow"
}
],
"type": "table"
},
{
"attrs": {
"localId": "01a0-33_z",
"title": "a\nb\tc\"d\\e"
},
"content": [
{
"content": [
{
"text": "Notes.",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "expand"
}
],
"type": "doc",
"version": 1
}
@@ -0,0 +1,11 @@
:::::table {isNumberColumnEnabled=true layout=wide width="760.5"}
::::tableRow
:::tableHeader {colspan=2 colwidth="[340,420]"}
Assembly
:::
::::
:::::
:::expand {localId=01a0-33_z title="a\nb\tc\"d\\e"}
Notes.
:::
@@ -0,0 +1,41 @@
{
"content": [
{
"content": [
{
"text": "a `b ",
"type": "text"
},
{
"attrs": {
"url": "c`d"
},
"type": "inlineCard"
}
],
"type": "paragraph"
},
{
"content": [
{
"attrs": {
"url": "https://example.com/s?q=a&amp;b<c"
},
"type": "inlineCard"
}
],
"type": "paragraph"
},
{
"attrs": {
"extensionKey": "jira",
"parameters": {
"jql": "a|b `c` & d < e"
}
},
"type": "extension"
}
],
"type": "doc",
"version": 1
}
@@ -0,0 +1,5 @@
a `b :inlineCard{url="c\u0060d"}
:inlineCard{url="https://example.com/s?q=a\u0026amp;b\u003cc"}
::extension {extensionKey=jira parameters="{\"jql\":\"a\u007cb \u0060c\u0060 \u0026 d \u003c e\"}"}
@@ -0,0 +1,107 @@
{
"content": [
{
"content": [
{
"content": [
{
"content": [
{
"content": [
{
"text": "Cell",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableHeader"
},
{
"content": [
{
"content": [
{
"text": "Note",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableHeader"
}
],
"type": "tableRow"
},
{
"content": [
{
"content": [
{
"content": [
{
"text": " lead",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableCell"
},
{
"content": [
{
"type": "paragraph"
}
],
"type": "tableCell"
}
],
"type": "tableRow"
},
{
"content": [
{
"content": [
{
"content": [
{
"attrs": {
"style": "a|b",
"text": "ok"
},
"type": "status"
}
],
"type": "paragraph"
}
],
"type": "tableCell"
},
{
"content": [
{
"content": [
{
"text": "trail ",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "tableCell"
}
],
"type": "tableRow"
}
],
"type": "table"
}
],
"type": "doc",
"version": 1
}
@@ -0,0 +1,4 @@
| Cell | Note |
| --- | --- |
| :text{text=" "}lead | |
| :status[ok]{style="a\u007cb"} | trail:text{text=" "} |
+13 -10
View File
@@ -93,11 +93,14 @@ container is open, a named error otherwise.
**Attributes**: `{key=value key2="two words"}`. `{attrs}` is optional in every form, and `{}` is
valid — no attributes. A bare value matches `[A-Za-z0-9_-]+`; any other value is double-quoted
with JSON string escaping (`\"` `\\` `\n` `\t` `\uXXXX`, …) — total over
Unicode, and raw newlines never appear inside quotes. The closing `}` is the first one outside
quotes, since a quoted value holds `}` unescaped. All values are strings at the grammar
level; each node's section assigns types. Canonical form orders keys alphabetically, spells
values bare wherever allowed, inside quotes escapes only what it must using the shortest escape
form, and omits empty `{attrs}` except where the `{` itself claims the directive
Unicode, and raw newlines never appear inside quotes. A quoted value also escapes `` ` ``, `&`,
`<` and `|` as `\u0060`, `\u0026`, `\u003c` and `\u007c` — the code span, entity reference,
autolink, raw HTML and pipe cell's split all bind at or before a directive does, so a raw one
would reach back out of the value; `*`, `_`, `~`, `[` and `(` resolve after and stay raw. The
closing `}` is the first one outside quotes, since a quoted value holds `}` unescaped. All values
are strings at the grammar level; each node's section assigns types. Canonical form orders keys
alphabetically, spells values bare wherever allowed, escapes inside quotes in the shortest form
each escape has, and omits empty `{attrs}` except where the `{` itself claims the directive
(`:hardBreak{}`).
**Escaping**: the emitter backslash-escapes whatever literal text would otherwise parse as
@@ -219,9 +222,9 @@ One header row plus plain inline cells is a pipe table; anything richer is the d
and every cell carry no attrs and no marks, the first row is all `tableHeader` and the rest all
`tableCell`, every row has the header's cell count, and every cell holds exactly one attr-less,
mark-less paragraph — an empty cell holds one empty paragraph — with no `|` anywhere the
inline layer spells as syntax — a code span, a link destination or title, an inline directive's
attributes: backslash escapes are inert there, so pipe form cannot spell that pipe and the table
takes the directive form. A pipe table parses back to exactly that shape.
inline layer spells as syntax — a code span, a link destination or title: backslash escapes are
inert there, so pipe form cannot spell that pipe and the table takes the directive form. A pipe
table parses back to exactly that shape.
```
| Part | Qty |
@@ -238,8 +241,8 @@ inline layer's ordinary CommonMark escaping yields the pipe; each cell is the in
one paragraph, trimmed; canonical form pads cells with single spaces and ends rows with `|`
(optional in input). Named errors: a delimiter or body row whose cell count differs from the
header's, and an alignment colon in the delimiter row — ADF holds no column alignment. In a
pipe cell a hard break is `:hardBreak{}`, a literal `|` is `\|`, and `|` inside a quoted
attribute value is `\u007c` (JSON has no `\|`).
pipe cell a hard break is `:hardBreak{}` and a literal `|` is `\|`; a `|` inside a quoted
attribute value is already `\u007c`, so the split never reaches it.
The directive form nests cells as containers of block content inside `tableRow` containers:
+5 -6
View File
@@ -402,8 +402,8 @@ test('spells a table as a pipe table only where every row and cell is plain', ()
assert.ok(directive(adfToMarkdown(table(row(cell('tableHeader', text('a'), text('b')))))))
assert.equal(code(adfToMarkdown(table(row(cell('tableHeader', { attrs: { localId: 'a' }, type: 'paragraph' }))))), 'unspelled-node-attribute')
assert.equal(
markdown(adfToMarkdown(table(row(cell('tableHeader', { content: [{ type: 'blockCard' }], type: 'paragraph' }))))),
'| :adf{json="{\\"type\\":\\"blockCard\\"}"} |\n| --- |\n',
markdown(adfToMarkdown(table(row(cell('tableHeader', { content: [{ attrs: { url: 'a|b' }, type: 'blockCard' }], type: 'paragraph' }))))),
'| :adf{json="{\\"attrs\\":{\\"url\\":\\"a\\u007cb\\"},\\"type\\":\\"blockCard\\"}"} |\n| --- |\n',
)
assert.equal(code(adfToMarkdown(table(row(cell('tableHeader', { content: [{ text: '\fa', type: 'text' }], type: 'paragraph' }))))), 'unspellable-whitespace')
assert.ok(directive(adfToMarkdown(table(row(cell('tableHeader', { attrs: { level: 1 }, type: 'heading' }))))))
@@ -411,10 +411,8 @@ test('spells a table as a pipe table only where every row and cell is plain', ()
const marked = (mark: AdfMark): AdfDocument => table(row(cell('tableHeader', { content: [{ marks: [mark], text: 'l', type: 'text' }], type: 'paragraph' })))
assert.ok(directive(adfToMarkdown(marked({ attrs: { href: 'https://example.com/?x|y' }, type: 'link' }))))
assert.ok(directive(adfToMarkdown(marked({ attrs: { href: 'https://example.com/', title: 'a|b' }, type: 'link' }))))
const fallsBack = (node: AdfNode): boolean => directive(adfToMarkdown(table(row(cell('tableHeader', { content: [node], type: 'paragraph' })))))
assert.ok(fallsBack({ marks: [{ type: 'code' }], text: 'a|b', type: 'text' }))
assert.ok(fallsBack({ attrs: { style: 'a|b' }, type: 'status' }))
assert.ok(fallsBack({ attrs: { url: 'https://example.com/?a|b' }, type: 'blockCard' }))
const codeSpan: AdfNode = { marks: [{ type: 'code' }], text: 'a|b', type: 'text' }
assert.ok(directive(adfToMarkdown(table(row(cell('tableHeader', { content: [codeSpan], type: 'paragraph' }))))))
assert.equal(markdown(adfToMarkdown(marked({ attrs: { href: 'https://example.com/x' }, type: 'link' }))), '| [l](https://example.com/x) |\n| --- |\n')
})
@@ -427,6 +425,7 @@ test('spells an inline node as a directive with its content slot and attributes'
assert.equal(emitted({ attrs: { id: '1f389', text: 'a]b' }, type: 'emoji' }), ':emoji[a\\]b]{id=1f389}\n')
assert.equal(emitted({ attrs: { data: { url: 'https://example.com/' } }, type: 'inlineCard' }), ':inlineCard{data="{\\"url\\":\\"https://example.com/\\"}"}\n')
assert.equal(emitted({ attrs: { height: 24 }, type: 'mediaInline' }), ':mediaInline{height=24}\n')
assert.equal(emitted({ attrs: { url: 'a`b&c<d|e' }, type: 'inlineCard' }), ':inlineCard{url="a\\u0060b\\u0026c\\u003cd\\u007ce"}\n')
})
test('refuses an inline node attribute no section spells', () => {
+4 -1
View File
@@ -9,6 +9,9 @@ export type SpelledPairs = { fault: AttributeFault; pairs?: undefined } | { faul
const bareToken = /^[A-Za-z0-9_-]+$/
// spec/flavour.md, Attributes.
const quotedEscapes = /[&<`|]/g
export function isBareToken(text: string): boolean {
return bareToken.test(text)
}
@@ -53,5 +56,5 @@ export function spellStringAttribute(text: string): string {
}
function quote(text: string): string {
return JSON.stringify(text)
return JSON.stringify(text).replace(quotedEscapes, (character) => `\\u${character.charCodeAt(0).toString(16).padStart(4, '0')}`)
}
+2 -2
View File
@@ -2,7 +2,7 @@ import { escapesLineClaim, isUnicodeWhitespace, opensBracketedAutolink, startsEn
export type EmphasisRole = 'close' | 'open'
export type InlineEscaping = 'attribute' | 'backslash' | 'bracketed' | 'none'
export type InlineEscaping = 'backslash' | 'bracketed' | 'none'
export type InlineSegment =
| { emphasis: EmphasisRole; escaping: 'none'; mark: string; text: string }
@@ -128,7 +128,7 @@ function touchesSyntax(scan: string, escapings: readonly (InlineEscaping | undef
}
function isSyntax(escaping: InlineEscaping | undefined): boolean {
return escaping === 'attribute' || escaping === 'none'
return escaping === 'none'
}
function opensConstruct(scan: string, index: number, inBrackets: boolean, container: LineContainer, escaped: ReadonlySet<number>): boolean {
+1 -1
View File
@@ -92,7 +92,7 @@ function carryEdges(segment: InlineSegment, leading: boolean, trailing: boolean)
}
function carriedText(text: string): InlineSegment {
return { escaping: 'attribute', text: spellLeafDirective('text', spellAttributes([['text', spellStringAttribute(text)]])) }
return syntax(spellLeafDirective('text', spellAttributes([['text', spellStringAttribute(text)]])))
}
function spellLeafDirective(name: string, attributes: string): string {
+19 -23
View File
@@ -96,24 +96,20 @@ detail is settled at its own milestone.
`inline-directives.ts`, which holds four of the nine marks while the rest are branch
literals in the emitter — and the parser (3) needs every name to make `:em[x]` the named
error `spec/flavour.md` promises.
- [ ] **2e3 — Attribute canonicalization and the quoted value's escape.** Attribute
canonicalization, a pipe cell's whitespace edges, and `\u007c` for a `|` inside a quoted
attribute value. The rule is wider than the pipe: a quoted value spells `` ` ``, `&` and
`<` raw today, so a backtick pair in an `inlineCard`'s `data`, a `status`'s `style` or the
inline carry's `json` reads back as a code span, an entity or raw HTML inside the value.
One escape settles all four, and the pick decides whether a directive or a code span wins
where they overlap (3). `escaping: 'attribute'` earns its keep at that rule or collapses
into `none`: nothing the escaper does tells the two apart today, since a carried segment
holds only spaces, tabs and newlines.
- [x] **2e3 — Attribute canonicalization and the quoted value's escape.**
**Settled** (the maintainer, 2026-08-26): a quoted attribute value escapes `` ` ``, `&`,
`<` and `|` as `\u0060`, `\u0026`, `\u003c` and `\u007c`, in every directive, block and
inline alike — the constructs those four open all bind at or before a directive does, and
nothing else reaches into `{attrs}`. Emitted attributes being inert leaves 3 free to keep
CommonMark's own precedence between a directive and a code span, and collapsed
`escaping: 'attribute'` into `none`.
- [ ] **2e4 — The carry's fallback triggers.** `spec/flavour.md` carries a node its section
cannot spell — an attrs key no section lists, a value that is not the section's type, an
arg slot holding no bare token, marks no nesting spells — where the emitter still refuses,
which leaves the refusals a container's own spelling owns. The carry spells
`escaping: 'none'`, which is what makes `tryPipeCell` refuse the pipe form for a carry
whose JSON holds a pipe. The flanking trigger 2e2 added to that list is the odd one out:
`unspellableMark` finds it after assembly and names a mark type against the line's path,
so the failing run needs identifying before the carry can replace the refusal
`corpus/unspellable/mark-inside-word` pins.
which leaves the refusals a container's own spelling owns. The flanking trigger 2e2
added to that list is the odd one out: `unspellableMark` finds it after assembly and
names a mark type against the line's path, so the failing run needs identifying before
the carry can replace the refusal `corpus/unspellable/mark-inside-word` pins.
- [ ] **2e5 — Combined documents and the collision property.** Documents combining nodes rather
than isolating one, and the gate's collision property: no two corpus documents may emit
the same bytes — one spelling for two documents is a round-trip break no parser can undo,
@@ -142,14 +138,14 @@ detail is settled at its own milestone.
every raw-HTML construct in input is an error result. The CommonMark spec suite runs
against it from here (§10). The parser owes `~` the same `can_open`/`can_close` the emitter
assumes — CommonMark flanking, as for `*` — which `spec/flavour.md` does not yet pin, and
the precedence between a directive and the constructs a raw attribute value opens inside it —
a code span, an entity, raw HTML — which one directive alone already reaches until 2e's
escape lands. Whether a claimed line interrupts a paragraph is pinned for the plain case and
unstated for the lazy one: a closing fence on the line after a blockquote's open paragraph is
lazy continuation in CommonMark, which would swallow the fence and leave the container
unclosed. 2e1's `closing-fence-line` orders its blockquote away from that edge meanwhile.
`src/` gets its hierarchy at the same split — `adf/`,
`markdown/`, `html/`, the grammar module shared inside `markdown/` — while the rename is
the precedence in input between a directive and a code span, an entity or raw HTML written
raw inside its attributes — 2e3's escape keeps emitted output clear of that collision, so the
pick binds input alone. Whether a claimed line interrupts a paragraph is pinned for the plain
case and unstated for the lazy one: a closing fence on the line after a blockquote's open
paragraph is lazy continuation in CommonMark, which would swallow the fence and leave the
container unclosed. 2e1's `closing-fence-line` orders its blockquote away from that edge
meanwhile. `src/` gets its hierarchy at the same split — `adf/`, `markdown/`, `html/`, the
grammar module shared inside `markdown/` — while the rename is
still mechanical. Three files do not move whole: `block-directives.ts` and
`inline-directives.ts` each hold a node table milestones 6-7 need in `adf/` beside a
markdown spelling that belongs in `markdown/`, and `directive-attributes.ts` fuses the