From cd0a4cb0e8124411cdaa7f469c735f387521c15e Mon Sep 17 00:00:00 2001 From: Lilleman auf Larv Date: Tue, 1 Sep 2026 11:09:21 +0200 Subject: [PATCH] Read the node tables backwards, a directive to the node and marks it names --- AGENTS.md | 6 +- corpus/errors/directive-adf-reserved.error | 1 + corpus/errors/directive-adf-reserved.md | 3 + corpus/errors/directive-attribute-order.error | 1 + corpus/errors/directive-attribute-order.md | 1 + .../errors/directive-attribute-unknown.error | 1 + corpus/errors/directive-attribute-unknown.md | 3 + corpus/errors/directive-body-on-leaf.error | 1 + corpus/errors/directive-body-on-leaf.md | 3 + corpus/errors/directive-content-slot.error | 1 + corpus/errors/directive-content-slot.md | 1 + .../errors/directive-value-uncanonical.error | 1 + corpus/errors/directive-value-uncanonical.md | 1 + spec/flavour.md | 120 ++++++++++-------- src/adf/attribute-vocabulary.ts | 14 +- src/adf/block-directives.ts | 2 +- src/adf/document.ts | 2 +- src/adf/inline-directives.ts | 8 +- src/corpus.test.ts | 21 +++ src/markdown/block-directive-marks.ts | 23 ++++ src/markdown/directive-syntax.test.ts | 37 ++++-- src/markdown/directive-syntax.ts | 66 ++++++---- src/markdown/emit/block-directive-spelling.ts | 13 +- src/markdown/parse/blocks.test.ts | 2 +- src/markdown/parse/blocks.ts | 2 +- src/markdown/parse/directive-nodes.ts | 91 +++++++++++++ src/markdown/parse/inline-content.ts | 10 +- src/markdown/parse/markdown-to-adf.test.ts | 88 ++++++++++++- src/markdown/parse/markdown-to-adf.ts | 52 ++++++-- src/spec.test.ts | 103 +++++++++++++++ todo.md | 20 ++- 31 files changed, 558 insertions(+), 140 deletions(-) create mode 100644 corpus/errors/directive-adf-reserved.error create mode 100644 corpus/errors/directive-adf-reserved.md create mode 100644 corpus/errors/directive-attribute-order.error create mode 100644 corpus/errors/directive-attribute-order.md create mode 100644 corpus/errors/directive-attribute-unknown.error create mode 100644 corpus/errors/directive-attribute-unknown.md create mode 100644 corpus/errors/directive-body-on-leaf.error create mode 100644 corpus/errors/directive-body-on-leaf.md create mode 100644 corpus/errors/directive-content-slot.error create mode 100644 corpus/errors/directive-content-slot.md create mode 100644 corpus/errors/directive-value-uncanonical.error create mode 100644 corpus/errors/directive-value-uncanonical.md create mode 100644 src/markdown/block-directive-marks.ts create mode 100644 src/markdown/parse/directive-nodes.ts create mode 100644 src/spec.test.ts diff --git a/AGENTS.md b/AGENTS.md index 165fcc6..5e5b018 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -89,7 +89,11 @@ direction hits it. A claim code names the spelling claimed, never the node that a malformed `:::table` is a `malformed-directive`. A cause the carry answers gets no code: a mark no spelling writes rides the carry with its node. A directive whose name reads back to no node is `unknown-directive-name` rather than a claim code — the spelling is well formed, and telling that -apart from a typo is what a consumer switches on when a later MINOR gives the name meaning. A +apart from a typo is what a consumer switches on when a later MINOR gives the name meaning. What +the grammar itself refuses stays a claim code, key order among it; a well-formed directive the +node tables refuse — an attribute a node does not hold or spells elsewhere, a value outside its +kind or its canonical spelling, an argument or a body its content model does not take — is +`unsupported-node-shape`, the emitter's code for the same mismatch read the other way. A refusal found before its position is known — the block walk's, a directive reader's — is a `ConvertFault`, the code and message without the path; the node walk attaches the path as it descends, so a document reports its first error in document order. diff --git a/corpus/errors/directive-adf-reserved.error b/corpus/errors/directive-adf-reserved.error new file mode 100644 index 0000000..4f35411 --- /dev/null +++ b/corpus/errors/directive-adf-reserved.error @@ -0,0 +1 @@ +malformed-directive diff --git a/corpus/errors/directive-adf-reserved.md b/corpus/errors/directive-adf-reserved.md new file mode 100644 index 0000000..867275e --- /dev/null +++ b/corpus/errors/directive-adf-reserved.md @@ -0,0 +1,3 @@ +:::adf +{"type":"rule"} +::: diff --git a/corpus/errors/directive-attribute-order.error b/corpus/errors/directive-attribute-order.error new file mode 100644 index 0000000..4f35411 --- /dev/null +++ b/corpus/errors/directive-attribute-order.error @@ -0,0 +1 @@ +malformed-directive diff --git a/corpus/errors/directive-attribute-order.md b/corpus/errors/directive-attribute-order.md new file mode 100644 index 0000000..2c7ad5f --- /dev/null +++ b/corpus/errors/directive-attribute-order.md @@ -0,0 +1 @@ +::media {type=file id=a-1} diff --git a/corpus/errors/directive-attribute-unknown.error b/corpus/errors/directive-attribute-unknown.error new file mode 100644 index 0000000..a811539 --- /dev/null +++ b/corpus/errors/directive-attribute-unknown.error @@ -0,0 +1 @@ +unsupported-node-shape diff --git a/corpus/errors/directive-attribute-unknown.md b/corpus/errors/directive-attribute-unknown.md new file mode 100644 index 0000000..e812626 --- /dev/null +++ b/corpus/errors/directive-attribute-unknown.md @@ -0,0 +1,3 @@ +:::panel info {bogus=1} +Part. +::: diff --git a/corpus/errors/directive-body-on-leaf.error b/corpus/errors/directive-body-on-leaf.error new file mode 100644 index 0000000..a811539 --- /dev/null +++ b/corpus/errors/directive-body-on-leaf.error @@ -0,0 +1 @@ +unsupported-node-shape diff --git a/corpus/errors/directive-body-on-leaf.md b/corpus/errors/directive-body-on-leaf.md new file mode 100644 index 0000000..7f6b9ed --- /dev/null +++ b/corpus/errors/directive-body-on-leaf.md @@ -0,0 +1,3 @@ +:::rule +Part. +::: diff --git a/corpus/errors/directive-content-slot.error b/corpus/errors/directive-content-slot.error new file mode 100644 index 0000000..a811539 --- /dev/null +++ b/corpus/errors/directive-content-slot.error @@ -0,0 +1 @@ +unsupported-node-shape diff --git a/corpus/errors/directive-content-slot.md b/corpus/errors/directive-content-slot.md new file mode 100644 index 0000000..b8bce7f --- /dev/null +++ b/corpus/errors/directive-content-slot.md @@ -0,0 +1 @@ +Part :mention[@Mikael]{id=b1c2}. diff --git a/corpus/errors/directive-value-uncanonical.error b/corpus/errors/directive-value-uncanonical.error new file mode 100644 index 0000000..a811539 --- /dev/null +++ b/corpus/errors/directive-value-uncanonical.error @@ -0,0 +1 @@ +unsupported-node-shape diff --git a/corpus/errors/directive-value-uncanonical.md b/corpus/errors/directive-value-uncanonical.md new file mode 100644 index 0000000..de317da --- /dev/null +++ b/corpus/errors/directive-value-uncanonical.md @@ -0,0 +1 @@ +::rule {localId="a-1"} diff --git a/spec/flavour.md b/spec/flavour.md index 14ae16a..26a6a96 100644 --- a/spec/flavour.md +++ b/spec/flavour.md @@ -112,7 +112,9 @@ quoted value holds `}` unescaped. All values are strings at the grammar level; e assigns types. Canonical form orders keys alphabetically, spells values bare wherever allowed, escapes inside quotes in the shortest form each escape has, and omits empty `{attrs}` except where the `{` itself -claims the directive (`:hardBreak{}`). +claims the directive (`:hardBreak{}`). Input reads that spelling alone: keys out of order, a value +quoted where bare carries it, an escape longer than it need be, and a number or `json` value +outside its canonical JSON spelling are each a named error naming the spelling to write instead. **Escaping**: the emitter backslash-escapes whatever literal text would otherwise parse as directive syntax — the leading `:` of a would-be directive, `]` inside content, a `{` right @@ -179,21 +181,21 @@ CommonMark spells `blockquote`, `bulletList`, `codeBlock`, `heading`, `listItem` cannot — `localId` (string) on any of them, marks, and the values below — takes the directive form. -- `blockquote`, `bulletList`, `listItem` — containers, block body; no attributes beyond `localId`. +- `blockquote`, `bulletList`, `listItem` — containers, block body. Attributes: `localId` (string). - `codeBlock` — container, body one fenced code block whose info string is the language and whose - content is the node's. Attributes: `hideLineNumbers` (boolean), `language` (string), `localId`, - `uniqueId` (string), `wrap` (boolean). A language no info string carries back — empty, the - reserved `adf`, or holding a backtick, a backslash, a control character, edge whitespace or an - entity reference — rides the `language` attribute instead and the fence carries no info string; - writing both is a named error. -- `heading` — container, inline body. Attributes: `level` (number), `localId`. `level` is the `#` - count, so a heading carrying none, or one that is no whole number from 1 to 6, has no CommonMark - spelling. -- `orderedList` — container of `listItem`, block body. Attributes: `localId`, `order` (number). - `order` is the first marker, so a list carrying none, one that is no whole number from 0, or one - whose markers would run past 999999999, has no CommonMark spelling. -- `paragraph` — container, inline body; no attributes beyond `localId`. -- `rule` — leaf; no attributes beyond `localId`. + content is the node's. Attributes: `hideLineNumbers` (boolean), `language` (string), `localId` + (string), `uniqueId` (string), `wrap` (boolean). A language no info string carries back — empty, + the reserved `adf`, or holding a backtick, a backslash, a control character, edge whitespace or + an entity reference — rides the `language` attribute instead and the fence carries no info + string; writing both is a named error. +- `heading` — container, inline body. Attributes: `level` (number), `localId` (string). `level` is + the `#` count, so a heading carrying none, or one that is no whole number from 1 to 6, has no + CommonMark spelling. +- `orderedList` — container of `listItem`, block body. Attributes: `localId` (string), `order` + (number). `order` is the first marker, so a list carrying none, one that is no whole number + from 0, or one whose markers would run past 999999999, has no CommonMark spelling. +- `paragraph` — container, inline body. Attributes: `localId` (string). +- `rule` — leaf. Attributes: `localId` (string). ```` :::codeBlock {localId=01a03d5c-9b21-73f4-8e6a-0c47b1d9e2f8 wrap=true} @@ -205,9 +207,10 @@ fn main() {} ### Panel -Container; the arg is `panelType` (`custom` `error` `info` `note` `success` `tip` `warning`). -Attributes: `localId` (string), `panelColor` (string), `panelIcon` (string), `panelIconId` -(string), `panelIconText` (string) — the editor writes the last four for `custom` panels. +- `panel` — container; the arg is `panelType` (`custom` `error` `info` `note` `success` `tip` + `warning`). Attributes: `localId` (string), `panelColor` (string), `panelIcon` (string), + `panelIconId` (string), `panelIconText` (string) — the editor writes the last four for `custom` + panels. ``` :::panel warning @@ -217,8 +220,8 @@ Check the collation before importing. ### Expand -`expand` and `nestedExpand`: containers, no arg — same syntax, two node types, the name picks -which. Attributes: `localId` (string), `title` (string). +- `expand`, `nestedExpand` — containers, no arg; same syntax, two node types, the name picks + which. Attributes: `localId` (string), `title` (string). ``` :::expand {title="Full build log"} @@ -236,7 +239,7 @@ which. Attributes: `localId` (string), `title` (string). `layout` (`align-end` `align-start` `center` `full-width` `wide` `wrap-left` `wrap-right`), `localId` (string), `width` (number), `widthType` (`percentage` `pixel`). - `caption` — container, inline body. Attributes: `localId` (string). -- `mediaGroup` — container of `::media` leaves, no attributes. +- `mediaGroup` — container of `::media` leaves. Attributes: none. ``` ::::mediaSingle {layout=center width=50} @@ -306,11 +309,13 @@ Bolt M8 ::::: ``` -Attributes — `table`: `displayMode` (`default` `fixed`), `isNumberColumnEnabled` (boolean), -`layout` (`align-end` `align-start` `center` `default` `full-width` `wide`), `localId` -(string), `width` (number, pixels). `tableRow`: `localId` (string). `tableCell` and -`tableHeader`: `background` (string), `colspan` (number), `colwidth` (json, one pixel width per -spanned column), `localId` (string), `rowspan` (number), `valign` (`bottom` `middle` `top`). +- `table` — container of `tableRow` containers. Attributes: `displayMode` (`default` `fixed`), + `isNumberColumnEnabled` (boolean), `layout` (`align-end` `align-start` `center` `default` + `full-width` `wide`), `localId` (string), `width` (number, pixels). +- `tableRow` — container of cells. Attributes: `localId` (string). +- `tableCell`, `tableHeader` — containers, block body. Attributes: `background` (string), + `colspan` (number), `colwidth` (json, one pixel width per spanned column), `localId` (string), + `rowspan` (number), `valign` (`bottom` `middle` `top`). ### Task and decision lists @@ -318,7 +323,8 @@ spanned column), `localId` (string), `rowspan` (number), `valign` (`bottom` `mid Attributes: `localId` (string). - `taskItem` — container, inline body; the arg is the state (`DONE` `TODO`). Attributes: `localId` (string). -- `blockTaskItem` — container, block body; arg and attributes as `taskItem`. +- `blockTaskItem` — container, block body; the arg is the state as `taskItem`. Attributes: + `localId` (string). - `decisionList` — container of `decisionItem` directives. Attributes: `localId` (string). - `decisionItem` — container, inline body. Attributes: `localId` (string), `state` (string — free-form; the editor writes `DECIDED`). @@ -336,9 +342,9 @@ Ship it ### Layout -`layoutSection` — container of `layoutColumn` containers; attributes: `localId` (string). -`layoutColumn` — container, block body; attributes: `localId` (string), `valign` (`bottom` -`middle` `top`), `width` (number — percent). +- `layoutSection` — container of `layoutColumn` containers. Attributes: `localId` (string). +- `layoutColumn` — container, block body. Attributes: `localId` (string), `valign` (`bottom` + `middle` `top`), `width` (number — percent). ``` ::::layoutSection @@ -353,10 +359,11 @@ Right. ### Extensions -`extension` — leaf. `bodiedExtension` — container, block body. `multiBodiedExtension` — -container of `extensionFrame` containers; `extensionFrame` — container, block body, no -attributes. The other three share: `extensionKey` (string), `extensionType` (string), `layout` -(`default` `full-width` `wide`), `localId` (string), `parameters` (json), `text` (string). +- `extension`, `bodiedExtension`, `multiBodiedExtension` — a leaf, a container with a block body, + and a container of `extensionFrame` containers. Attributes: `extensionKey` (string), + `extensionType` (string), `layout` (`default` `full-width` `wide`), `localId` (string), + `parameters` (json), `text` (string). +- `extensionFrame` — container, block body. Attributes: none. ``` ::extension {extensionKey=toc extensionType="com.atlassian.confluence.macro.core" parameters="{\"maxLevel\":2}"} @@ -364,8 +371,8 @@ attributes. The other three share: `extensionKey` (string), `extensionType` (str ### Sync blocks -`syncBlock` — leaf. `bodiedSyncBlock` — container, block body. Attributes: `localId` (string), -`resourceId` (string). +- `syncBlock`, `bodiedSyncBlock` — a leaf and a container with a block body. Attributes: + `localId` (string), `resourceId` (string). ``` ::syncBlock {localId=0198f3a2-af74-7e91-b26c-70b15f4d9ec3 resourceId="ari:cloud:confluence:site/page/123"} @@ -381,18 +388,19 @@ is a named error, and so is a `text` key in `{attrs}`. An enclosing mark spellin into the slot. The rest take no content, `:text` included; content on a node that takes none is a named error. -- `date` — `localId` (string), `timestamp` (string, epoch milliseconds). -- `emoji` — `id` (string), `localId` (string), `shortName` (string, `:name:`), `text` (string). -- `hardBreak` — `localId` (string), `text` (string). -- `inlineCard` — `data` (json), `localId` (string), `url` (string); real payloads carry one or - the other. -- `mediaInline` — `alt` (string), `collection` (string), `data` (json), `height` (number), `id` - (string), `localId` (string), `occurrenceKey` (string), `type` (`file` `image` `link`), `width` - (number). -- `mention` — `accessLevel` (`APPLICATION` `CONTAINER` `NONE` `SITE`), `id` (string), `localId` - (string), `text` (string), `userType` (`APP` `DEFAULT` `SPECIAL`). -- `status` — `color` (`blue` `green` `neutral` `purple` `red` `yellow`), `localId` (string), - `style` (string), `text` (string). +- `date` — Attributes: `localId` (string), `timestamp` (string, epoch milliseconds). +- `emoji` — Attributes: `id` (string), `localId` (string), `shortName` (string, `:name:`), `text` + (string). +- `hardBreak` — Attributes: `localId` (string), `text` (string). +- `inlineCard` — Attributes: `data` (json), `localId` (string), `url` (string); real payloads + carry one or the other. +- `mediaInline` — Attributes: `alt` (string), `collection` (string), `data` (json), `height` + (number), `id` (string), `localId` (string), `occurrenceKey` (string), `type` (`file` `image` + `link`), `width` (number). +- `mention` — Attributes: `accessLevel` (`APPLICATION` `CONTAINER` `NONE` `SITE`), `id` (string), + `localId` (string), `text` (string), `userType` (`APP` `DEFAULT` `SPECIAL`). +- `status` — Attributes: `color` (`blue` `green` `neutral` `purple` `red` `yellow`), `localId` + (string), `style` (string), `text` (string). ``` :status[In review]{color=yellow} — :mention[@Mikael]{id=01a032c3-7a7c-775f-a730-2d79351338b4} @@ -415,14 +423,16 @@ the rest plain text; `markdownToAdf` merges adjacent text nodes carrying identic ## Marks An inline node's marks ride the spelling wrapped around them, never the block sections' reserved -`marks` key. `code`, `em`, `link`, `strike` and `strong` keep their markdown spellings, which hold -no attributes beyond `link`'s `href` and `title`, and are not directive names: `:em[x]` is a named -error. The other four are inline directives, content required non-empty: +`marks` key. `code`, `em`, `link`, `strike` and `strong` keep their markdown spellings, and are +not directive names: `:em[x]` is a named error. The other four are inline directives, content +required non-empty. -- `border` — `color` (string, `#rrggbb` or `#rrggbbaa`), `size` (number, 1–3). -- `subsup` — `type` (`sub` `sup`). -- `textColor` — `color` (string, `#rrggbb`). -- `underline` — no attributes. +- `border` — Attributes: `color` (string, `#rrggbb` or `#rrggbbaa`), `size` (number, 1–3). +- `code`, `em`, `strike`, `strong` — Attributes: none. +- `link` — Attributes: `href` (string), `title` (string). +- `subsup` — Attributes: `type` (`sub` `sup`). +- `textColor` — Attributes: `color` (string, `#rrggbb`). +- `underline` — Attributes: none. A spelling adds its mark to every inline node it wraps, and nesting is the marks array in order, outermost first: `_:underline[x]_` gives marks `[em, underline]`, `:underline[_x_]` the reverse. diff --git a/src/adf/attribute-vocabulary.ts b/src/adf/attribute-vocabulary.ts index 8c8fde5..b88a00d 100644 --- a/src/adf/attribute-vocabulary.ts +++ b/src/adf/attribute-vocabulary.ts @@ -1,15 +1,17 @@ import type { AdfAttributes } from './document.ts' import type { JsonValue } from '../json-value.ts' -type AttributeKind = 'boolean' | 'json' | 'number' | 'string' +export type AttributeKind = 'boolean' | 'json' | 'number' | 'string' export type AttributeVocabulary = Readonly> -export type VocabularyPair = - | { key: string; kind: 'boolean'; value: boolean } - | { key: string; kind: 'json'; value: JsonValue } - | { key: string; kind: 'number'; value: number } - | { key: string; kind: 'string'; value: string } +export type VocabularyValue = + | { kind: 'boolean'; value: boolean } + | { kind: 'json'; value: JsonValue } + | { kind: 'number'; value: number } + | { kind: 'string'; value: string } + +export type VocabularyPair = VocabularyValue & { key: string } export function vocabularyPairs(attrs: AdfAttributes, vocabulary: AttributeVocabulary, spelledElsewhere: readonly string[]): VocabularyPair[] | undefined { const pairs: VocabularyPair[] = [] diff --git a/src/adf/block-directives.ts b/src/adf/block-directives.ts index f750afd..5b33d9e 100644 --- a/src/adf/block-directives.ts +++ b/src/adf/block-directives.ts @@ -41,7 +41,7 @@ const mediaAttributes: AttributeVocabulary = { const syncBlockAttributes: AttributeVocabulary = { localId: 'string', resourceId: 'string' } -const blockDirectives = { +export const blockDirectives = { blockTaskItem: { attributes: localIdAttributes, contentModel: 'block' }, blockquote: { attributes: localIdAttributes, contentModel: 'block' }, bodiedExtension: { attributes: extensionAttributes, contentModel: 'block' }, diff --git a/src/adf/document.ts b/src/adf/document.ts index 7502f23..fc696c9 100644 --- a/src/adf/document.ts +++ b/src/adf/document.ts @@ -37,7 +37,7 @@ export function isAdfDocument(value: unknown): value is AdfDocument { return !('content' in value) || isNodeArray(value['content']) } -function isAdfMark(value: unknown): value is AdfMark { +export function isAdfMark(value: unknown): value is AdfMark { if (!isRecord(value) || !holdsOnly(value, markKeys)) return false if (typeof value['type'] !== 'string') return false return !('attrs' in value) || isAttributes(value['attrs']) diff --git a/src/adf/inline-directives.ts b/src/adf/inline-directives.ts index 75ab247..6636b2f 100644 --- a/src/adf/inline-directives.ts +++ b/src/adf/inline-directives.ts @@ -5,9 +5,9 @@ export type InlineDirective = { textAttribute?: string } -const inlineDirectives: Readonly> = { +export const inlineDirectives: Readonly> = { date: { attributes: { localId: 'string', timestamp: 'string' } }, - emoji: { attributes: { id: 'string', localId: 'string', shortName: 'string' }, textAttribute: 'text' }, + emoji: { attributes: { id: 'string', localId: 'string', shortName: 'string', text: 'string' }, textAttribute: 'text' }, hardBreak: { attributes: { localId: 'string', text: 'string' } }, inlineCard: { attributes: { data: 'json', localId: 'string', url: 'string' } }, mediaInline: { @@ -23,8 +23,8 @@ const inlineDirectives: Readonly> = { width: 'number', }, }, - mention: { attributes: { accessLevel: 'string', id: 'string', localId: 'string', userType: 'string' }, textAttribute: 'text' }, - status: { attributes: { color: 'string', localId: 'string', style: 'string' }, textAttribute: 'text' }, + mention: { attributes: { accessLevel: 'string', id: 'string', localId: 'string', text: 'string', userType: 'string' }, textAttribute: 'text' }, + status: { attributes: { color: 'string', localId: 'string', style: 'string', text: 'string' }, textAttribute: 'text' }, } export function inlineDirective(type: string): InlineDirective | undefined { diff --git a/src/corpus.test.ts b/src/corpus.test.ts index 9a38e98..e22652a 100644 --- a/src/corpus.test.ts +++ b/src/corpus.test.ts @@ -17,6 +17,8 @@ const roundTripRoot = join(corpusRoot, 'round-trip') const unspellableRoot = join(corpusRoot, 'unspellable') const emittingDirectories = ['block-nodes', 'combinations', 'commonmark-subset', 'inline-nodes', 'opaque-carry'] +// A directory joins once every fixture in it reads back to its document. +const parsingDirectories = ['commonmark-subset'] function directoryNames(root: string): string[] { return readdirSync(root, { withFileTypes: true }) @@ -88,6 +90,25 @@ for (const directory of emittingDirectories) { } } +test('every parsing directory emits the markdown it reads back', () => { + assert.deepEqual( + parsingDirectories.filter((directory) => emittingDirectories.includes(directory)), + parsingDirectories, + ) +}) + +for (const directory of parsingDirectories) { + for (const name of fixtureNames(directory, '.md')) { + test(`${directory}/${name} reads its markdown back to the document beside it`, () => { + const expected: unknown = JSON.parse(readFileSync(join(roundTripRoot, directory, `${name}.json`), 'utf8')) + assert.ok(isAdfDocument(expected), `${name}.json is not an ADF document`) + const result = markdownToAdf(readFileSync(join(roundTripRoot, directory, `${name}.md`), 'utf8')) + assert.ok(result.ok, result.ok ? '' : `${result.error.code}: ${result.error.message}`) + assert.deepEqual(result.value, expected) + }) + } +} + function roundTripFixtures(): { name: string; path: string }[] { return emittingDirectories.flatMap((directory) => fixtureNames(directory, '.json').map((name) => ({ name: `${directory}/${name}`, path: join(roundTripRoot, directory, `${name}.json`) })), diff --git a/src/markdown/block-directive-marks.ts b/src/markdown/block-directive-marks.ts new file mode 100644 index 0000000..dffda09 --- /dev/null +++ b/src/markdown/block-directive-marks.ts @@ -0,0 +1,23 @@ +import type { AdfMark } from '../adf/document.ts' +import type { JsonValue } from '../json-value.ts' +import { isAdfMark } from '../adf/document.ts' +import { serializeCanonicalJson } from '../canonical-json.ts' + +export const marksAttribute = 'marks' + +export function markValues(marks: readonly AdfMark[]): JsonValue { + return marks.map((mark) => { + const attrs = mark.attrs ?? {} + return Object.keys(attrs).length === 0 ? { type: mark.type } : { attrs, type: mark.type } + }) +} + +export function readMarkValues(value: JsonValue): AdfMark[] | undefined { + if (!Array.isArray(value) || value.length === 0) return undefined + const marks: AdfMark[] = [] + for (const item of value) { + if (!isAdfMark(item)) return undefined + marks.push(item) + } + return serializeCanonicalJson(markValues(marks), 'compact') === serializeCanonicalJson(value, 'compact') ? marks : undefined +} diff --git a/src/markdown/directive-syntax.test.ts b/src/markdown/directive-syntax.test.ts index d198971..da5395b 100644 --- a/src/markdown/directive-syntax.test.ts +++ b/src/markdown/directive-syntax.test.ts @@ -1,15 +1,18 @@ import assert from 'node:assert/strict' import test from 'node:test' -import type { DirectiveLine } from './directive-syntax.ts' +import type { DirectiveAttributes, DirectiveLine } from './directive-syntax.ts' import { largestNesting } from '../nesting.ts' import { readDirectiveLine, readInlineDirective } from './directive-syntax.ts' -function attributes(...pairs: [string, string][]): ReadonlyMap { - return new Map(pairs) +// A pair the input spells bare reads its own text back; a quoted one names the spelling beside it. +type Pair = [string, string, string?] + +function attributes(...pairs: Pair[]): DirectiveAttributes { + return new Map(pairs.map(([key, text, spelling]) => [key, { spelling: spelling ?? text, text }])) } -function header(colons: number, name: string, argument?: string, ...pairs: [string, string][]): { value: DirectiveLine } { +function header(colons: number, name: string, argument?: string, ...pairs: Pair[]): { value: DirectiveLine } { return { value: { argument, attributes: attributes(...pairs), colons, kind: 'header', name } } } @@ -25,7 +28,7 @@ function inline(text: string): unknown { return { attributes: read.value.attributes, content: read.value.content, length: read.value.length, name: read.value.name } } -function spans(text: string, name: string, content: string | undefined, ...pairs: [string, string][]): void { +function spans(text: string, name: string, content: string | undefined, ...pairs: Pair[]): void { assert.deepEqual(inline(text), { attributes: attributes(...pairs), content, length: text.length, name }) } @@ -48,15 +51,23 @@ test('reads the leaf and container forms, their argument and their attributes', assert.deepEqual(readDirectiveLine('::taskItem TODO'), header(2, 'taskItem', 'TODO')) assert.deepEqual(readDirectiveLine('::hardBreak {}'), header(2, 'hardBreak')) assert.deepEqual(readDirectiveLine('::media {id=a-1 type=file}'), header(2, 'media', undefined, ['id', 'a-1'], ['type', 'file'])) - assert.deepEqual(readDirectiveLine('::panel info {panelColor="#ff0000"} '), header(2, 'panel', 'info', ['panelColor', '#ff0000'])) + assert.deepEqual(readDirectiveLine('::panel info {panelColor="#ff0000"} '), header(2, 'panel', 'info', ['panelColor', '#ff0000', '"#ff0000"'])) assert.deepEqual(readDirectiveLine(':::panel info'), header(3, 'panel', 'info')) }) test('decodes a quoted attribute value, the escapes {attrs} reserves included', () => { - assert.deepEqual(readDirectiveLine('::extension {text="two words"}'), header(2, 'extension', undefined, ['text', 'two words'])) - assert.deepEqual(readDirectiveLine('::extension {text="a\\u0060b\\u0026c\\u003cd\\u007ce"}'), header(2, 'extension', undefined, ['text', 'a`b&c { + assert.equal(fault('::media {type=file id=a-1}'), 'the {attrs} keys read in alphabetical order: id before type') + assert.deepEqual(readDirectiveLine('::media {id=a-1 type=file}'), header(2, 'media', undefined, ['id', 'a-1'], ['type', 'file'])) }) test('names the directive line no spelling reads', () => { @@ -97,7 +108,7 @@ test('reads an inline directive only where a bracket or a brace follows the name assert.equal(inline(':mention @A'), 'unclaimed') spans(':mention[@A]', 'mention', '@A') spans(':date{timestamp=1756080000000}', 'date', undefined, ['timestamp', '1756080000000']) - spans(':emoji[]{shortName=":tada:"}', 'emoji', '', ['shortName', ':tada:']) + spans(':emoji[]{shortName=":tada:"}', 'emoji', '', ['shortName', ':tada:', '":tada:"']) spans(':underline[ a ]', 'underline', ' a ') }) @@ -107,10 +118,10 @@ test('binds an inline directive as a unit, its content balancing brackets like l spans(':underline[a `]` b]', 'underline', 'a `]` b') spans(':underline[a `b c]', 'underline', 'a `b c') spans(':underline[:status[x]{color=red}]', 'underline', ':status[x]{color=red}') - spans(':status[x]{color=red style="bold "}', 'status', 'x', ['color', 'red'], ['style', 'bold ']) + spans(':status[x]{color=red style="bold "}', 'status', 'x', ['color', 'red'], ['style', 'bold ', '"bold "']) assert.deepEqual(inline(':underline[a]{}(b)'), { attributes: attributes(), content: 'a', length: 15, name: 'underline' }) assert.deepEqual(inline(':underline[a] {}'), { attributes: attributes(), content: 'a', length: 13, name: 'underline' }) - assert.deepEqual(inline(':text{text=" "} and more'), { attributes: attributes(['text', ' ']), content: undefined, length: 15, name: 'text' }) + assert.deepEqual(inline(':text{text=" "} and more'), { attributes: attributes(['text', ' ', '" "']), content: undefined, length: 15, name: 'text' }) }) test('names the inline directive left unclosed at the end of its line', () => { diff --git a/src/markdown/directive-syntax.ts b/src/markdown/directive-syntax.ts index 09e5165..298dd7f 100644 --- a/src/markdown/directive-syntax.ts +++ b/src/markdown/directive-syntax.ts @@ -1,13 +1,17 @@ +import type { AttributeKind, VocabularyPair, VocabularyValue } from '../adf/attribute-vocabulary.ts' import type { ConvertFault } from '../result.ts' import type { JsonValue } from '../json-value.ts' -import type { VocabularyPair } from '../adf/attribute-vocabulary.ts' import { backslashEscape, claimsDirectiveLine } from './commonmark-grammar.ts' import { backtickRun, closingBacktickRun } from './backtick-runs.ts' +import { isJsonValue } from '../json-value.ts' import { largestNesting } from '../nesting.ts' import { runLength } from './emphasis-matching.ts' import { serializeCanonicalJson } from '../canonical-json.ts' -export type DirectiveAttributes = ReadonlyMap +// The value as the input spells it, beside the string the grammar decodes it to. +export type DirectiveValue = { spelling: string; text: string } + +export type DirectiveAttributes = ReadonlyMap export type DirectiveLine = | { argument: string | undefined; attributes: DirectiveAttributes; colons: number; kind: 'header'; name: string } @@ -19,7 +23,7 @@ export type Read = { fault: ConvertFault; value?: undefined } | { fault?: und type Attributes = { attributes: DirectiveAttributes; length: number } -type AttributePair = { end: number; key: string; value: string } +type AttributePair = { end: number; key: string; value: DirectiveValue } const bareTokenSource = '[A-Za-z0-9_-]+' const bareRun = new RegExp(bareTokenSource, 'y') @@ -34,9 +38,19 @@ const rawReserved = new RegExp(reservedSource) const noAttributes: DirectiveAttributes = new Map() const nameFault = 'a directive name reads [a-z][A-Za-z0-9]*' +const orderFault = 'the {attrs} keys read in alphabetical order' const pairFault = 'an attribute reads key=value, the value bare or double-quoted' const shapeFault = 'a directive line reads a name, one bare argument and {attrs}, one space apart' +export function attributeValue(text: string, kind: AttributeKind): VocabularyValue | undefined { + if (kind === 'string') return { kind, value: text } + if (kind === 'boolean') return text === 'true' || text === 'false' ? { kind, value: text === 'true' } : undefined + const parsed = parseJson(text) + if (parsed === undefined) return undefined + if (kind === 'json') return { kind, value: parsed } + return typeof parsed === 'number' ? { kind, value: parsed } : undefined +} + export function isBareToken(text: string): boolean { return bareToken.test(text) } @@ -77,6 +91,13 @@ export function spellStringAttribute(text: string): string { return isBareToken(text) ? text : quote(text) } +export function spellAttributeValue(value: VocabularyValue): string { + if (value.kind === 'boolean') return `${value.value}` + if (value.kind === 'json') return spellJsonAttribute(value.value) + if (value.kind === 'number') return spellStringAttribute(JSON.stringify(value.value)) + return spellStringAttribute(value.value) +} + export function spellVocabulary(pairs: readonly VocabularyPair[]): [string, string][] { return pairs.map((pair): [string, string] => [pair.key, spellAttributeValue(pair)]) } @@ -85,13 +106,6 @@ export function unknownDirectiveFault(name: string): ConvertFault { return { code: 'unknown-directive-name', message: `the directive name ${name} reads back to no node` } } -function spellAttributeValue(pair: VocabularyPair): string { - if (pair.kind === 'boolean') return `${pair.value}` - if (pair.kind === 'json') return spellJsonAttribute(pair.value) - if (pair.kind === 'number') return spellStringAttribute(JSON.stringify(pair.value)) - return spellStringAttribute(pair.value) -} - function quote(text: string): string { return JSON.stringify(text).replace(quotedEscapes, (character) => `\\u${escapeDigits(character)}`) } @@ -192,8 +206,9 @@ function readCodeSpanEnd(text: string, index: number): number | undefined { } function readAttributes(text: string, index: number): Read { - const attributes = new Map() + const attributes = new Map() let cursor = index + 1 + let previous = '' while (cursor < text.length && text.charAt(cursor) !== '}') { if (attributes.size > 0) { if (text.charAt(cursor) !== ' ') return { fault: malformedDirective(pairFault) } @@ -201,8 +216,11 @@ function readAttributes(text: string, index: number): Read { } const pair = readAttributePair(text, cursor) if (pair.fault !== undefined) return { fault: pair.fault } - if (attributes.has(pair.value.key)) return { fault: malformedDirective(`the attribute key ${pair.value.key} is spelled twice`) } - attributes.set(pair.value.key, pair.value.value) + const key = pair.value.key + if (attributes.has(key)) return { fault: malformedDirective(`the attribute key ${key} is spelled twice`) } + if (key < previous) return { fault: malformedDirective(`${orderFault}: ${key} before ${previous}`) } + previous = key + attributes.set(key, pair.value.value) cursor = pair.value.end } if (text.charAt(cursor) !== '}') return { fault: malformedDirective('the {attrs} closing brace is missing') } @@ -220,29 +238,29 @@ function readAttributePair(text: string, index: number): Read { return { value: { end: quoted.value.end, key, value: quoted.value.value } } } bareRun.lastIndex = start - const value = bareRun.exec(text)?.[0] - if (value === undefined) return { fault: malformedDirective(pairFault) } - return { value: { end: start + value.length, key, value } } + const bare = bareRun.exec(text)?.[0] + if (bare === undefined) return { fault: malformedDirective(pairFault) } + return { value: { end: start + bare.length, key, value: { spelling: bare, text: bare } } } } -function readQuotedValue(text: string, index: number): Read<{ end: number; value: string }> { +function readQuotedValue(text: string, index: number): Read<{ end: number; value: DirectiveValue }> { let cursor = index + 1 while (cursor < text.length && text.charAt(cursor) !== '"') cursor += text.charAt(cursor) === '\\' ? 2 : 1 if (text.charAt(cursor) !== '"') return { fault: malformedDirective('the {attrs} quoted value is unclosed') } - const raw = text.slice(index, cursor + 1) - const character = rawReserved.exec(raw)?.[0] + const spelling = text.slice(index, cursor + 1) + const character = rawReserved.exec(spelling)?.[0] if (character !== undefined) { return { fault: malformedDirective(`a raw ${character} inside {attrs} breaks the directive: spell it \\u${escapeDigits(character)}`) } } - const value = parseJsonString(raw) - if (value === undefined) return { fault: malformedDirective('the {attrs} quoted value is not a JSON string') } - return { value: { end: cursor + 1, value } } + const parsed = parseJson(spelling) + if (typeof parsed !== 'string') return { fault: malformedDirective('the {attrs} quoted value is not a JSON string') } + return { value: { end: cursor + 1, value: { spelling, text: parsed } } } } -function parseJsonString(raw: string): string | undefined { +function parseJson(raw: string): JsonValue | undefined { try { const value: unknown = JSON.parse(raw) - return typeof value === 'string' ? value : undefined + return isJsonValue(value) ? value : undefined } catch { return undefined } diff --git a/src/markdown/emit/block-directive-spelling.ts b/src/markdown/emit/block-directive-spelling.ts index 62bebb7..05b5a0a 100644 --- a/src/markdown/emit/block-directive-spelling.ts +++ b/src/markdown/emit/block-directive-spelling.ts @@ -1,8 +1,8 @@ -import type { AdfMark, AdfNode } from '../../adf/document.ts' +import type { AdfNode } from '../../adf/document.ts' import type { BlockDirective } from '../../adf/block-directives.ts' -import type { JsonValue } from '../../json-value.ts' import { blockArgument } from '../block-directive-arguments.ts' import { isBareToken, spellAttributes, spellJsonAttribute, spellVocabulary } from '../directive-syntax.ts' +import { markValues, marksAttribute } from '../block-directive-marks.ts' import { vocabularyPairs } from '../../adf/attribute-vocabulary.ts' export function spellDirectiveHeader(node: AdfNode, directive: BlockDirective, spelledByBody: readonly string[] = []): string | undefined { @@ -14,7 +14,7 @@ export function spellDirectiveHeader(node: AdfNode, directive: BlockDirective, s if (pairs === undefined) return undefined const spelledPairs = spellVocabulary(pairs) const marks = node.marks ?? [] - if (marks.length > 0) spelledPairs.push(['marks', spellJsonAttribute(markValues(marks))]) + if (marks.length > 0) spelledPairs.push([marksAttribute, spellJsonAttribute(markValues(marks))]) const attributes = spellAttributes(spelledPairs) return `${node.type}${argument}${attributes === '' ? '' : ` ${attributes}`}` } @@ -25,10 +25,3 @@ function spellArgument(node: AdfNode, argumentAttribute: string | undefined): st if (typeof value !== 'string' || !isBareToken(value)) return undefined return ` ${value}` } - -function markValues(marks: readonly AdfMark[]): JsonValue { - return marks.map((mark) => { - const attrs = mark.attrs ?? {} - return Object.keys(attrs).length === 0 ? { type: mark.type } : { attrs, type: mark.type } - }) -} diff --git a/src/markdown/parse/blocks.test.ts b/src/markdown/parse/blocks.test.ts index 30f4156..93ac65e 100644 --- a/src/markdown/parse/blocks.test.ts +++ b/src/markdown/parse/blocks.test.ts @@ -90,7 +90,7 @@ test('holds a directive container open until the fence that closes it', () => { assert.deepEqual(parseBlocks(':::panel info {panelColor="#ff0000"}\nPart.\n:::\n').blocks, [ { argument: 'info', - attributes: new Map([['panelColor', '#ff0000']]), + attributes: new Map([['panelColor', { spelling: '"#ff0000"', text: '#ff0000' }]]), blocks: [{ kind: 'paragraph', text: 'Part.' }], kind: 'directive', name: 'panel', diff --git a/src/markdown/parse/blocks.ts b/src/markdown/parse/blocks.ts index c5b8689..c07bd21 100644 --- a/src/markdown/parse/blocks.ts +++ b/src/markdown/parse/blocks.ts @@ -31,7 +31,7 @@ export type Block = export type ParsedBlocks = { blocks: Block[]; definitions: Map } -type DirectiveBlock = Extract +export type DirectiveBlock = Extract type ListBlock = Extract diff --git a/src/markdown/parse/directive-nodes.ts b/src/markdown/parse/directive-nodes.ts new file mode 100644 index 0000000..a271526 --- /dev/null +++ b/src/markdown/parse/directive-nodes.ts @@ -0,0 +1,91 @@ +import type { AdfAttributes, AdfMark, AdfNode } from '../../adf/document.ts' +import type { AttributeVocabulary } from '../../adf/attribute-vocabulary.ts' +import type { BlockDirective } from '../../adf/block-directives.ts' +import type { DirectiveAttributes, DirectiveSpan, DirectiveValue } from '../directive-syntax.ts' +import { attributeValue, spellAttributeValue, unknownDirectiveFault } from '../directive-syntax.ts' +import { blockArgument } from '../block-directive-arguments.ts' +import { blockDirective } from '../../adf/block-directives.ts' +import { carryName } from '../opaque-carry.ts' +import { failure, faulted, success, type ConvertErrorPath, type Result } from '../../result.ts' +import { inlineDirective } from '../../adf/inline-directives.ts' +import { marksAttribute, readMarkValues } from '../block-directive-marks.ts' + +export type BlockDirectiveNode = { contentModel: BlockDirective['contentModel']; node: AdfNode } + +type Elsewhere = { key: string; place: string } + +export function readBlockDirectiveNode( + name: string, + argument: string | undefined, + attributes: DirectiveAttributes, + path: ConvertErrorPath, +): Result { + if (name === carryName) { + return failure('malformed-directive', `the name ${carryName} is reserved for the opaque carry, whose block form is the fence`, path) + } + const directive = blockDirective(name) + if (directive === undefined) return faulted(unknownDirectiveFault(name), path) + const argumentKey = blockArgument(name) + const rest = new Map(attributes) + rest.delete(marksAttribute) + const elsewhere = argumentKey === undefined ? undefined : { key: argumentKey, place: 'as the directive argument' } + const attrs = readVocabulary(name, rest, directive.attributes, elsewhere, path) + if (!attrs.ok) return attrs + if (argument !== undefined) { + if (argumentKey === undefined) return failure('unsupported-node-shape', `a ${name} takes no argument`, path) + attrs.value[argumentKey] = argument + } + const spelled = attributes.get(marksAttribute) + const marks: Result = spelled === undefined ? success(undefined) : readMarks(name, spelled, path) + if (!marks.ok) return marks + return success({ contentModel: directive.contentModel, node: directiveNode(name, attrs.value, marks.value) }) +} + +export function readInlineDirectiveNode(span: DirectiveSpan, path: ConvertErrorPath): Result { + const directive = inlineDirective(span.name) + if (directive === undefined) return faulted(unknownDirectiveFault(span.name), path) + const slot = directive.textAttribute + if (span.content !== undefined) { + const message = slot === undefined ? `a ${span.name} takes no content` : `the content slot a ${span.name} spells its ${slot} attribute in is unsupported` + return failure('unsupported-node-shape', message, path) + } + const elsewhere = slot === undefined ? undefined : { key: slot, place: 'in the content slot' } + const attrs = readVocabulary(span.name, span.attributes, directive.attributes, elsewhere, path) + if (!attrs.ok) return attrs + return success(directiveNode(span.name, attrs.value, undefined)) +} + +function readVocabulary( + type: string, + attributes: DirectiveAttributes, + vocabulary: AttributeVocabulary, + elsewhere: Elsewhere | undefined, + path: ConvertErrorPath, +): Result { + const attrs: AdfAttributes = {} + for (const [key, spelled] of attributes) { + if (key === elsewhere?.key) return failure('unsupported-node-shape', `a ${type} spells its ${key} attribute ${elsewhere.place}`, path) + const kind = Object.hasOwn(vocabulary, key) ? vocabulary[key] : undefined + if (kind === undefined) return failure('unsupported-node-shape', `a ${type} holds no ${key} attribute`, path) + const read = attributeValue(spelled.text, kind) + if (read === undefined) return failure('unsupported-node-shape', `the ${key} attribute of a ${type} is a ${kind}`, path) + const spelling = spellAttributeValue(read) + if (spelling !== spelled.spelling) return failure('unsupported-node-shape', `a ${type} spells its ${key} attribute as ${key}=${spelling}`, path) + attrs[key] = read.value + } + return success(attrs) +} + +function readMarks(type: string, spelled: DirectiveValue, path: ConvertErrorPath): Result { + const read = attributeValue(spelled.text, 'json') + const marks = read === undefined || spellAttributeValue(read) !== spelled.spelling ? undefined : readMarkValues(read.value) + if (marks === undefined) { + return failure('unsupported-node-shape', `the ${marksAttribute} attribute of a ${type} is its marks array in canonical JSON`, path) + } + return success(marks) +} + +function directiveNode(type: string, attrs: AdfAttributes, marks: readonly AdfMark[] | undefined): AdfNode { + const named = Object.keys(attrs).length === 0 ? { type } : { attrs, type } + return marks === undefined ? named : { ...named, marks: [...marks] } +} diff --git a/src/markdown/parse/inline-content.ts b/src/markdown/parse/inline-content.ts index f22bb33..3a7b70b 100644 --- a/src/markdown/parse/inline-content.ts +++ b/src/markdown/parse/inline-content.ts @@ -7,7 +7,8 @@ import { delimiterFlags, matchEmphasis, runLength } from '../emphasis-matching.t import { failure, faulted, success, type ConvertErrorPath, type Result } from '../../result.ts' import { mergeAdjacentText } from '../../adf/editor-normal.ts' import { normalizeLabel, readInlineTarget, readLabel } from '../link-syntax.ts' -import { readInlineDirective, unknownDirectiveFault } from '../directive-syntax.ts' +import { readInlineDirective } from '../directive-syntax.ts' +import { readInlineDirectiveNode } from './directive-nodes.ts' export type InlineContent = { image: AdfNode; nodes?: undefined } | { image?: undefined; nodes: AdfNode[] } @@ -144,7 +145,12 @@ function readDirective(scan: Scan, index: number): Result { scan.pending += ':' return success(index + 1) } - return faulted(directive.fault ?? unknownDirectiveFault(directive.value.name), scan.path) + if (directive.fault !== undefined) return faulted(directive.fault, scan.path) + const node = readInlineDirectiveNode(directive.value, scan.path) + if (!node.ok) return node + flush(scan, false) + pushNode(scan, node.value) + return success(index + directive.value.length) } function flush(scan: Scan, strip: boolean): void { diff --git a/src/markdown/parse/markdown-to-adf.test.ts b/src/markdown/parse/markdown-to-adf.test.ts index acb8652..50c506b 100644 --- a/src/markdown/parse/markdown-to-adf.test.ts +++ b/src/markdown/parse/markdown-to-adf.test.ts @@ -3,6 +3,7 @@ import test from 'node:test' import type { AdfDocument, AdfMark, AdfNode } from '../../adf/document.ts' import type { Result } from '../../result.ts' +import { largestNesting } from '../../nesting.ts' import { markdownToAdf } from './markdown-to-adf.ts' const em: AdfMark = { type: 'em' } @@ -134,16 +135,91 @@ test('claims a block-level colon run with no directive to parse it', () => { assert.deepEqual(content(markdownToAdf(':: two\n')), [paragraph(':: two')]) }) -test('reads the three directive forms, and names the node none of them reads back to', () => { - assert.equal(code(markdownToAdf('::rule\n')), 'unknown-directive-name') - assert.equal(code(markdownToAdf(' :::panel info\nx\n:::\n')), 'unknown-directive-name') - assert.equal(code(markdownToAdf('Part :mention[@A]{id=b1c2}.\n')), 'unknown-directive-name') - assert.equal(content(markdownToAdf('::rule\n')), 'unknown-directive-name: the directive name rule reads back to no node') - assert.deepEqual(path(markdownToAdf('Part.\n\n::rule\n')), ['content', 1]) +test('reads the three directive forms into the nodes the tables name', () => { + assert.deepEqual(content(markdownToAdf('::rule\n')), [{ type: 'rule' }]) + assert.deepEqual(content(markdownToAdf('::paragraph\n')), [{ type: 'paragraph' }]) + assert.deepEqual(content(markdownToAdf(' :::panel info\nPart.\n:::\n')), [ + { attrs: { panelType: 'info' }, content: [paragraph('Part.')], type: 'panel' }, + ]) + assert.deepEqual(content(markdownToAdf(':::blockquote\n:::\n')), [{ type: 'blockquote' }]) + assert.deepEqual(content(markdownToAdf(':::heading {level=2}\nPart.\n:::\n')), [{ attrs: { level: 2 }, content: [text('Part.')], type: 'heading' }]) + assert.deepEqual(content(markdownToAdf('Part:hardBreak{}.\n')), [{ content: [text('Part'), hardBreak(), text('.')], type: 'paragraph' }]) +}) + +test('names the directive name no node reads back to', () => { + assert.equal(code(markdownToAdf(':::widget info\nx\n:::\n')), 'unknown-directive-name') + assert.equal(content(markdownToAdf('::widget\n')), 'unknown-directive-name: the directive name widget reads back to no node') + assert.equal(code(markdownToAdf(':widget[x]\n')), 'unknown-directive-name') + assert.deepEqual(path(markdownToAdf('Part.\n\n::widget\n')), ['content', 1]) assert.equal(content(markdownToAdf('Part.\n:::x\n')), 'malformed-directive: a container fenced with 3 colons is unclosed') assert.deepEqual(path(markdownToAdf('Part.\n:::x\n')), ['content', 1]) }) +test('names the reserved carry name a block directive spells', () => { + const reserved = 'malformed-directive: the name adf is reserved for the opaque carry, whose block form is the fence' + assert.equal(content(markdownToAdf('::adf\n')), reserved) + assert.equal(content(markdownToAdf(':::adf\nx\n:::\n')), reserved) +}) + +test('reads each attribute value as the type its section assigns', () => { + assert.deepEqual(content(markdownToAdf('::media {height=10 id=a-1 type=file url="/x y" width="20.5"}\n')), [ + { attrs: { height: 10, id: 'a-1', type: 'file', url: '/x y', width: 20.5 }, type: 'media' }, + ]) + assert.deepEqual(content(markdownToAdf(':::table {isNumberColumnEnabled=true}\n:::\n')), [{ attrs: { isNumberColumnEnabled: true }, type: 'table' }]) + assert.deepEqual(content(markdownToAdf(':::tableCell {colwidth="[340,420]"}\n:::\n')), [{ attrs: { colwidth: [340, 420] }, type: 'tableCell' }]) + assert.deepEqual(content(markdownToAdf('::rule {localId=a-1}\n')), [{ attrs: { localId: 'a-1' }, type: 'rule' }]) +}) + +test('reads the reserved marks key as the node array it spells', () => { + assert.deepEqual(content(markdownToAdf('::rule {marks="[{\\"type\\":\\"em\\"}]"}\n')), [{ marks: [em], type: 'rule' }]) + assert.deepEqual(content(markdownToAdf('::rule {localId=a-1 marks="[{\\"attrs\\":{\\"mode\\":\\"wide\\"},\\"type\\":\\"breakout\\"}]"}\n')), [ + { attrs: { localId: 'a-1' }, marks: [{ attrs: { mode: 'wide' }, type: 'breakout' }], type: 'rule' }, + ]) +}) + +test('names the marks key no marks array reads back from', () => { + const named = 'unsupported-node-shape: the marks attribute of a rule is its marks array in canonical JSON' + assert.equal(content(markdownToAdf('::rule {marks="[]"}\n')), named) + assert.equal(content(markdownToAdf('::rule {marks="[1]"}\n')), named) + assert.equal(content(markdownToAdf('::rule {marks="{}"}\n')), named) + assert.equal(content(markdownToAdf('::rule {marks=x}\n')), named) + assert.equal(content(markdownToAdf('::rule {marks="[{\\"attrs\\":{},\\"type\\":\\"em\\"}]"}\n')), named) +}) + +test('names the attribute a node holds no reading for', () => { + assert.equal(content(markdownToAdf('::rule {bogus=1}\n')), 'unsupported-node-shape: a rule holds no bogus attribute') + assert.equal(content(markdownToAdf('::media {width=wide}\n')), 'unsupported-node-shape: the width attribute of a media is a number') + assert.equal(content(markdownToAdf(':::table {isNumberColumnEnabled=yes}\n:::\n')), 'unsupported-node-shape: the isNumberColumnEnabled attribute of a table is a boolean') + assert.equal(content(markdownToAdf('::media {width=true}\n')), 'unsupported-node-shape: the width attribute of a media is a number') + assert.equal(content(markdownToAdf(':::tableCell {colwidth="[340,"}\n:::\n')), 'unsupported-node-shape: the colwidth attribute of a tableCell is a json') + const deep = `${'['.repeat(largestNesting + 2)}${']'.repeat(largestNesting + 2)}` + assert.equal(content(markdownToAdf(`:::tableCell {colwidth="${deep}"}\n:::\n`)), 'unsupported-node-shape: the colwidth attribute of a tableCell is a json') + assert.equal(content(markdownToAdf(':::panel info {panelType=note}\nx\n:::\n')), 'unsupported-node-shape: a panel spells its panelType attribute as the directive argument') + assert.equal(content(markdownToAdf('Part :mention{id=b1c2 text=A}.\n')), 'unsupported-node-shape: a mention spells its text attribute in the content slot') +}) + +test('names the attribute value spelled outside the canonical form', () => { + assert.equal(content(markdownToAdf('::rule {localId="a-1"}\n')), 'unsupported-node-shape: a rule spells its localId attribute as localId=a-1') + assert.equal(content(markdownToAdf('::media {width="20.0"}\n')), 'unsupported-node-shape: a media spells its width attribute as width=20') + assert.equal(content(markdownToAdf(':::tableCell {colwidth="[340, 420]"}\n:::\n')), 'unsupported-node-shape: a tableCell spells its colwidth attribute as colwidth="[340,420]"') +}) + +test('names the argument and the body a node takes no reading for', () => { + assert.equal(content(markdownToAdf('::rule x\n')), 'unsupported-node-shape: a rule takes no argument') + assert.equal(content(markdownToAdf(':::rule\nPart.\n:::\n')), 'unsupported-node-shape: a rule holds no content') + assert.equal(content(markdownToAdf('::bulletList\n')), 'unsupported-node-shape: a bulletList spells its body in the container form :::bulletList') + assert.equal(content(markdownToAdf(':::paragraph\n:::\n')), 'unsupported-node-shape: an empty paragraph is the leaf form ::paragraph') + assert.equal(content(markdownToAdf(':::paragraph\nOne.\n\nTwo.\n:::\n')), 'unsupported-node-shape: a paragraph takes one paragraph as its body') + assert.equal(content(markdownToAdf(':::paragraph\n---\n:::\n')), 'unsupported-node-shape: a paragraph takes one paragraph as its body') + assert.equal(content(markdownToAdf(':::codeBlock\n```\nx\n```\n:::\n')), 'unsupported-node-shape: the fenced body of a codeBlock is unsupported') + assert.equal(content(markdownToAdf(':::paragraph\n![a](/u)\n:::\n')), 'unmappable-image: no ADF node carries an image inside a paragraph') + assert.equal(content(markdownToAdf('Part :date[now]{timestamp=1}.\n')), 'unsupported-node-shape: a date takes no content') + assert.equal( + content(markdownToAdf('Part :emoji[x]{shortName=":x:"}.\n')), + 'unsupported-node-shape: the content slot a emoji spells its text attribute in is unsupported', + ) +}) + test('leaves the colon that opens no directive the text it is', () => { assert.deepEqual(content(markdownToAdf('At 10:30 :smile: today.\n')), [paragraph('At 10:30 :smile: today.')]) assert.deepEqual(content(markdownToAdf('\\:mention[@A]\n')), [paragraph(':mention[@A]')]) diff --git a/src/markdown/parse/markdown-to-adf.ts b/src/markdown/parse/markdown-to-adf.ts index 50e28a7..25aea81 100644 --- a/src/markdown/parse/markdown-to-adf.ts +++ b/src/markdown/parse/markdown-to-adf.ts @@ -1,11 +1,11 @@ import type { AdfDocument, AdfNode } from '../../adf/document.ts' -import type { Block } from './blocks.ts' +import type { Block, DirectiveBlock } from './blocks.ts' import type { LinkDefinitions } from './inline-content.ts' import { failure, faulted, success, type ConvertErrorPath, type Result } from '../../result.ts' import { largestNesting } from '../../nesting.ts' import { parseBlocks } from './blocks.ts' import { parseInlineContent } from './inline-content.ts' -import { unknownDirectiveFault } from '../directive-syntax.ts' +import { readBlockDirectiveNode } from './directive-nodes.ts' export function markdownToAdf(markdown: string): Result { const parsed = parseBlocks(markdown) @@ -34,7 +34,7 @@ function blockNode(block: Block, definitions: LinkDefinitions, path: ConvertErro case 'code': return success(codeBlockNode(block.language, block.text)) case 'directive': - return faulted(unknownDirectiveFault(block.name), path) + return directiveNode(block, definitions, path, depth) case 'fault': return faulted(block.fault, path) case 'heading': @@ -44,16 +44,42 @@ function blockNode(block: Block, definitions: LinkDefinitions, path: ConvertErro case 'orderedList': return listNode({ attrs: { order: block.start }, type: 'orderedList' }, block.items, definitions, path, depth) case 'paragraph': - return contentNode({ type: 'paragraph' }, block.text, definitions, path) + return paragraphNode(block.text, definitions, path) case 'rule': return success({ type: 'rule' }) } } +function directiveNode(block: DirectiveBlock, definitions: LinkDefinitions, path: ConvertErrorPath, depth: number): Result { + const read = readBlockDirectiveNode(block.name, block.argument, block.attributes, path) + if (!read.ok) return read + const { contentModel, node } = read.value + const blocks = block.blocks + if (blocks === undefined) { + if (contentModel === 'none' || contentModel === 'inline') return success(node) + return failure('unsupported-node-shape', `a ${node.type} spells its body in the container form :::${node.type}`, path) + } + if (contentModel === 'none') return failure('unsupported-node-shape', `a ${node.type} holds no content`, path) + if (contentModel === 'code') return failure('unsupported-node-shape', `the fenced body of a ${node.type} is unsupported`, path) + if (contentModel === 'block') return containerNode(node, blocks, definitions, path, depth) + return inlineBodyNode(node, blocks, definitions, path) +} + +function inlineBodyNode(node: AdfNode, blocks: readonly Block[], definitions: LinkDefinitions, path: ConvertErrorPath): Result { + if (blocks.length === 0) return failure('unsupported-node-shape', `an empty ${node.type} is the leaf form ::${node.type}`, path) + const only = blocks.length === 1 ? blocks[0] : undefined + if (only?.kind !== 'paragraph') return failure('unsupported-node-shape', `a ${node.type} takes one paragraph as its body`, path) + return contentNode(node, only.text, definitions, path) +} + function containerNode(node: AdfNode, blocks: readonly Block[], definitions: LinkDefinitions, path: ConvertErrorPath, depth: number): Result { const content = blockNodes(blocks, definitions, path, depth + 1) if (!content.ok) return content - return success(content.value.length === 0 ? node : { ...node, content: content.value }) + return success(withContent(node, content.value)) +} + +function withContent(node: AdfNode, content: readonly AdfNode[]): AdfNode { + return content.length === 0 ? node : { ...node, content: [...content] } } function listNode(node: AdfNode, items: readonly Block[][], definitions: LinkDefinitions, path: ConvertErrorPath, depth: number): Result { @@ -71,13 +97,17 @@ function codeBlockNode(language: string, text: string): AdfNode { return text === '' ? node : { ...node, content: [{ text, type: 'text' }] } } -// spec/flavour.md, The CommonMark image: only a paragraph gives an image the block it needs. -function contentNode(node: AdfNode, text: string, definitions: LinkDefinitions, path: ConvertErrorPath): Result { +// spec/flavour.md, The CommonMark image: only a plain paragraph gives an image the block it needs. +function paragraphNode(text: string, definitions: LinkDefinitions, path: ConvertErrorPath): Result { const content = parseInlineContent(text, definitions, path) if (!content.ok) return content const image = content.value.image - if (image !== undefined) { - return node.type === 'paragraph' ? success(image) : failure('unmappable-image', `no ADF node carries an image inside a ${node.type}`, path) - } - return success(content.value.nodes.length === 0 ? node : { ...node, content: content.value.nodes }) + return success(image === undefined ? withContent({ type: 'paragraph' }, content.value.nodes) : image) +} + +function contentNode(node: AdfNode, text: string, definitions: LinkDefinitions, path: ConvertErrorPath): Result { + const content = parseInlineContent(text, definitions, path) + if (!content.ok) return content + if (content.value.image !== undefined) return failure('unmappable-image', `no ADF node carries an image inside a ${node.type}`, path) + return success(withContent(node, content.value.nodes)) } diff --git a/src/spec.test.ts b/src/spec.test.ts new file mode 100644 index 0000000..a4af0f1 --- /dev/null +++ b/src/spec.test.ts @@ -0,0 +1,103 @@ +import assert from 'node:assert/strict' +import { readFileSync } from 'node:fs' +import { dirname, join } from 'node:path' +import test from 'node:test' +import { fileURLToPath } from 'node:url' + +import type { AttributeKind, AttributeVocabulary } from './adf/attribute-vocabulary.ts' +import { blockDirectives } from './adf/block-directives.ts' +import { inlineDirectives } from './adf/inline-directives.ts' +import { markAttributes } from './adf/mark-attributes.ts' + +type Declared = { attributes: AttributeVocabulary } + +const specPath = join(dirname(fileURLToPath(import.meta.url)), '..', 'spec', 'flavour.md') +const introducer = 'Attributes: ' +const codeFence = /^`{3,}/ +const directiveName = /`([a-z][A-Za-z0-9]*)`/g +const namedType = /^`([a-z][A-Za-z0-9]*)` \(([^)]*)\)/ +const owned = ' — ' + +// Every node bullet of one section, its continuation lines folded in and its examples dropped. +function bullets(heading: string): string[] { + const items: string[] = [] + let fence: string | undefined + let item: string | undefined + let inside = false + for (const line of readFileSync(specPath, 'utf8').split('\n')) { + if (line.startsWith('## ')) inside = line === `## ${heading}` + if (!inside) continue + const marker = codeFence.exec(line)?.[0] + if (fence !== undefined) { + if (marker !== undefined && marker.length >= fence.length) fence = undefined + continue + } + if (marker !== undefined) { + fence = marker + continue + } + if (line.startsWith('- ')) { + if (item !== undefined) items.push(item) + item = line.slice(2) + } else if (item !== undefined && line.startsWith(' ')) item += ` ${line.trim()}` + else if (item !== undefined) { + items.push(item) + item = undefined + } + } + return items +} + +function declarations(heading: string): Record { + const declared: Record = {} + for (const item of bullets(heading)) { + const cut = item.indexOf(owned) + assert.notEqual(cut, -1, `${heading}: the bullet ${item} names no node ahead of a ${owned.trim()}`) + const attributes = attributeList(heading, item.slice(cut)) + for (const [, name] of item.slice(0, cut).matchAll(directiveName)) { + assert.equal(declared[name ?? ''], undefined, `${heading}: ${name ?? ''} is declared twice`) + declared[name ?? ''] = { attributes } + } + } + return declared +} + +function attributeList(heading: string, prose: string): AttributeVocabulary { + const at = prose.indexOf(introducer) + assert.notEqual(at, -1, `${heading}: ${prose} lists no attributes`) + let rest = prose.slice(at + introducer.length) + if (rest.startsWith('none')) return {} + const attributes: Record = {} + for (;;) { + const pair = namedType.exec(rest) + assert.notEqual(pair, null, `${heading}: ${rest} reads no \`name\` (type) pair`) + attributes[pair?.[1] ?? ''] = attributeKind(heading, pair?.[2] ?? '') + rest = rest.slice(pair?.[0].length ?? 0) + if (!rest.startsWith(', ')) return attributes + rest = rest.slice(2) + } +} + +// A parenthesized value set documents what payloads hold; the type stays string. +function attributeKind(heading: string, parenthesized: string): AttributeKind { + const first = parenthesized.split(/[\s,]/)[0] ?? '' + if (first === 'boolean' || first === 'json' || first === 'number' || first === 'string') return first + assert.ok(first.startsWith('`'), `${heading}: ${first} is neither an attribute kind nor a value set`) + return 'string' +} + +function vocabularies(table: Readonly>): Record { + return Object.fromEntries(Object.entries(table).map(([type, entry]) => [type, { attributes: entry.attributes }])) +} + +test('the block node table holds the attributes spec/flavour.md gives each node', () => { + assert.deepEqual(declarations('Block nodes'), vocabularies(blockDirectives)) +}) + +test('the inline node table holds the attributes spec/flavour.md gives each node', () => { + assert.deepEqual(declarations('Inline nodes'), vocabularies(inlineDirectives)) +}) + +test('the mark table holds the attributes spec/flavour.md gives each mark', () => { + assert.deepEqual(declarations('Marks'), vocabularies(Object.fromEntries(Object.entries(markAttributes).map(([type, attributes]) => [type, { attributes }])))) +}) diff --git a/todo.md b/todo.md index ffd6790..fc42b7f 100644 --- a/todo.md +++ b/todo.md @@ -293,7 +293,7 @@ detail is settled at its own milestone. `spec/flavour.md`'s closing-fence sentence now says: a run reaching past the innermost leaves the fence it did not close a named error, which §2 prefers to closing more than the author wrote. - - [ ] **3g — The node tables read backwards.** `commonmark-subset/` reads back, the first + - [x] **3g — The node tables read backwards.** `commonmark-subset/` reads back, the first directory to. A parsed directive becomes its node: the name to the type and an unknown one to a named error, the arg to the attribute it names, each value to the type its section assigns, the body to `content`, the reserved `marks` key to the marks array. 3a's drift @@ -302,11 +302,23 @@ detail is settled at its own milestone. the reserved `adf` name in block position needs an error of its own — 3f reports it as `unknown-directive-name`, which §8 makes the signal that a later MINOR may give the name meaning, and `adf` never will. + **Settled** (the maintainer, 2026-09-01): the reserved `adf` name in block position is a + `malformed-directive` — the grammar section states the reservation, so it is that spelling + the name breaks — and a well-formed directive the tables refuse is `unsupported-node-shape`, + the emitter's code for the same mismatch read the other way; AGENTS.md §8 carries the + split. And input reads canonical `{attrs}` alone, keys in order and every value spelled as + the emitter spells it, the error naming the spelling to write instead: §8 makes loosening a + MINOR, so strict is the reversible direction, as 3f already settled for spacing. + Two refusals land here for a later chunk to lift, on the same rule: the inline `[content]` + slot, which 3i opens for `emoji`, `mention` and `status`, and the `codeBlock` content + model's fenced body, 3h's. `Read` stays where 3f left it — the node reader knows its + path and returns `Result`, so no second reader took it. The drift guard earned itself on + the way in: the spec's `text` attribute was missing from three inline table entries, which + the content slot spells and the vocabulary walk already passes over. - [ ] **3h — The block nodes.** `block-nodes/` reads back: the plain-versus-directive choice 2f settles, read from the other side; the `codeBlock` directive's fenced body and the - `language` attribute a bare fence leaves it; the media family's composition; both table - forms, the pipe table's cell split and its named errors; and the one paragraph an inline - body takes. + `language` attribute a bare fence leaves it; the media family's composition; and both + table forms, the pipe table's cell split and its named errors. **Settled** (the maintainer, 2026-08-27): 1d's last pick, the one `container-block-separation` holds — a CommonMark block and a directive block sit adjacent in a container body with no blank line between them. That reduces the three cases to one