larvit/adf-codec
3
0
Fork 0
Code Pull Requests 1 Actions Activity

4.3: the markdown property #80

Merged
lilleman merged 12 commits from markdown-property into main 2026-09-15 22:48:43 +02:00
Conversation 0 Commits 12 Files Changed 20
+937 -181
20 changed files with 937 additions and 181 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+4 -3
View File
@@ -227,9 +227,10 @@ compared against `undefined` — have a half no valid document reaches.
The corpus, all checked in: hand-built fixtures per node and combination; real ADF Atlassian's The corpus, all checked in: hand-built fixtures per node and combination; real ADF Atlassian's
editor wrote; the CommonMark spec suite against `markdownToAdf` and `markdownToHtml`. editor wrote; the CommonMark spec suite against `markdownToAdf` and `markdownToHtml`.
Beside the corpus, properties run over documents generated from the node tables, on a fixed seed in Beside the corpus, properties run over documents generated from the node tables and over generated
the gate; `PROPERTY_RUNS=<runs>` raises the runs and randomizes the seed for local digging, and a markdown, on a fixed seed in the gate; `PROPERTY_RUNS=<runs>` raises the runs and randomizes the
counterexample found becomes a round-trip fixture. seed for local digging, and a counterexample found becomes a round-trip fixture. The generators and
run parameters properties share live in `src/property-harness.ts`, outside the build and coverage.
`spec/flavour.md` is read as a source too, so the node tables cannot drift from the prose they `spec/flavour.md` is read as a source too, so the node tables cannot drift from the prose they
copy: each `- ` bullet in `## Block nodes`, `## Inline nodes` and `## Marks` declares the nodes copy: each `- ` bullet in `## Block nodes`, `## Inline nodes` and `## Marks` declares the nodes
corpus/round-trip/combinations/code-span-joined-run.json
+73
View File
@@ -0,0 +1,73 @@
{
"content": [
{
"content": [
{
"text": "`` ``",
"type": "text"
},
{
"marks": [
{
"type": "code"
}
],
"text": "a",
"type": "text"
}
],
"type": "paragraph"
},
{
"content": [
{
"marks": [
{
"type": "underline"
}
],
"text": "``",
"type": "text"
},
{
"text": "``",
"type": "text"
},
{
"marks": [
{
"type": "code"
}
],
"text": "a",
"type": "text"
}
],
"type": "paragraph"
},
{
"content": [
{
"text": "``\\`",
"type": "text"
},
{
"marks": [
{
"type": "code"
}
],
"text": "`",
"type": "text"
},
{
"text": "`",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "doc",
"version": 1
}
corpus/round-trip/combinations/code-span-joined-run.md
+5
View File
@@ -0,0 +1,5 @@
\`\` \`\``a`
:underline[\`\`]\`\``a`
\`\`\\\``` ` ``\`
corpus/round-trip/combinations/link-definition-carry.json
+45
View File
@@ -0,0 +1,45 @@
{
"content": [
{
"content": [
{
"marks": [
{
"attrs": {
"href": "/u"
},
"type": "link"
},
{
"type": "code"
}
],
"text": "]: a",
"type": "text"
}
],
"type": "paragraph"
},
{
"content": [
{
"attrs": {
"timestamp": "]:a"
},
"marks": [
{
"attrs": {
"href": "/u"
},
"type": "link"
}
],
"type": "date"
}
],
"type": "paragraph"
}
],
"type": "doc",
"version": 1
}
corpus/round-trip/combinations/link-definition-carry.md
+3
View File
@@ -0,0 +1,3 @@
:adf{json="{\"marks\":[{\"attrs\":{\"href\":\"/u\"},\"type\":\"link\"},{\"type\":\"code\"}],\"text\":\"]: a\",\"type\":\"text\"}"}
:adf{json="{\"attrs\":{\"timestamp\":\"]:a\"},\"marks\":[{\"attrs\":{\"href\":\"/u\"},\"type\":\"link\"}],\"type\":\"date\"}"}
corpus/round-trip/combinations/link-definition-line.json
+76
View File
@@ -0,0 +1,76 @@
{
"content": [
{
"content": [
{
"text": "[",
"type": "text"
},
{
"attrs": {
"timestamp": "]:a"
},
"type": "date"
}
],
"type": "paragraph"
},
{
"content": [
{
"content": [
{
"text": "[",
"type": "text"
},
{
"marks": [
{
"type": "code"
}
],
"text": "]: a",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "blockquote"
},
{
"content": [
{
"content": [
{
"content": [
{
"text": "[",
"type": "text"
},
{
"attrs": {
"timestamp": "]:a"
},
"type": "date"
},
{
"type": "hardBreak"
},
{
"text": "b",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "listItem"
}
],
"type": "bulletList"
}
],
"type": "doc",
"version": 1
}
corpus/round-trip/combinations/link-definition-line.md
+6
View File
@@ -0,0 +1,6 @@
\[:date{timestamp="]:a"}
> \[`]: a`
- \[:date{timestamp="]:a"}\
b
corpus/round-trip/commonmark-subset/text-backtick-run.json
+31
View File
@@ -0,0 +1,31 @@
{
"content": [
{
"content": [
{
"text": "`a ``b``",
"type": "text"
}
],
"type": "paragraph"
},
{
"content": [
{
"text": "`a",
"type": "text"
},
{
"type": "hardBreak"
},
{
"text": "```b``",
"type": "text"
}
],
"type": "paragraph"
}
],
"type": "doc",
"version": 1
}
corpus/round-trip/commonmark-subset/text-backtick-run.md
+4
View File
@@ -0,0 +1,4 @@
\`a \`\`b``
\`a\
\`\`\`b``
package.json
+1 -1
View File
@@ -23,7 +23,7 @@
}, },
"scripts": { "scripts": {
"build": "tsc -p tsconfig.build.json", "build": "tsc -p tsconfig.build.json",
"test": "node --test --experimental-test-coverage --test-coverage-exclude=\"src/**/*.test.ts\" --test-coverage-branches=98 --test-coverage-functions=100 --test-coverage-lines=100 \"src/**/*.test.ts\"", "test": "node --test --experimental-test-coverage --test-coverage-exclude=\"src/**/*.test.ts\" --test-coverage-exclude=src/property-harness.ts --test-coverage-branches=98 --test-coverage-functions=100 --test-coverage-lines=100 \"src/**/*.test.ts\"",
"typecheck": "tsc --noEmit && tsc --noEmit -p tsconfig.build.json" "typecheck": "tsc --noEmit && tsc --noEmit -p tsconfig.build.json"
}, },
"devDependencies": { "devDependencies": {
spec/flavour.md
+7
View File
@@ -54,6 +54,13 @@ normalizes to it through the round-trip.
- Entity references in input decode to their characters; output backslash-escapes only where text - Entity references in input decode to their characters; output backslash-escapes only where text
would otherwise parse as syntax, scanning the assembled line rather than each text node: escape would otherwise parse as syntax, scanning the assembled line rather than each text node: escape
the leading delimiter of a construct that would otherwise open, re-scan from there, and repeat. the leading delimiter of a construct that would otherwise open, re-scan from there, and repeat.
A backtick run escapes whole, and a bare one escapes wherever a later backtick string of its
length forms around an escape in the same inline content — an escaped backtick alone, one joined
to the bare run after it, or a bare run an escape splits off: CommonMark reads no escape inside a
code span, so such a string still closes one.
Where a paragraph opens with what reads as a link reference definition, which resolves before
any inline construct binds (a `]` inside a code span or `{attrs}` counts), an opening text `[`
escapes and an opening link's nodes ride the carry.
An emphasis delimiter run in text escapes where CommonMark can open **or** close with it, so An emphasis delimiter run in text escapes where CommonMark can open **or** close with it, so
`*not emphasis*` is `\*not emphasis\*` — no delimiter the emitter did not write reaches the `*not emphasis*` is `\*not emphasis\*` — no delimiter the emitter did not write reaches the
matching below, which is what lets the emitter decide its own pairings. matching below, which is what lets the emitter decide its own pairings.
src/adf-property.test.ts
+2 -162
View File
@@ -1,173 +1,13 @@
import fc from 'fast-check' import fc from 'fast-check'
import assert from 'node:assert/strict' import assert from 'node:assert/strict'
import { env } from 'node:process'
import test from 'node:test' import test from 'node:test'
import type { AdfAttributes, AdfDocument, AdfMark, AdfNode } from './adf/document.ts' import { adfDocument, propertyRuns, propertyTimeout } from './property-harness.ts'
import type { Arbitrary } from 'fast-check'
import type { AttributeKind, AttributeVocabulary } from './adf/attribute-vocabulary.ts'
import type { JsonValue } from './json-value.ts'
import { adfToMarkdown } from './markdown/emit/adf-to-markdown.ts' import { adfToMarkdown } from './markdown/emit/adf-to-markdown.ts'
import { blockArgument } from './markdown/block-directive-arguments.ts'
import { blockDirectives } from './adf/block-directives.ts'
import { inlineDirectives } from './adf/inline-directives.ts'
import { markAttributes } from './adf/mark-attributes.ts'
import { markdownToAdf } from './markdown/parse/markdown-to-adf.ts' import { markdownToAdf } from './markdown/parse/markdown-to-adf.ts'
import { toEditorNormal } from './adf/editor-normal.ts' import { toEditorNormal } from './adf/editor-normal.ts'
type Positions = { block: AdfNode; inline: AdfNode }
const deepRunsVariable = 'PROPERTY_RUNS'
const gateRuns = 1600 const gateRuns = 1600
const gateSeed = 20260914
// Bun's test runner stops a test after five seconds unless the test sets its own timeout.
const propertyTimeout = 600000
const depthIdentifier = fc.createDepthIdentifier()
const emptyCell: AdfNode = { content: [{ type: 'paragraph' }], type: 'tableCell' }
const flatCommonMarkShapeWeight = 4
const markdownPieces = fc.constantFrom(...'aZ09 \t\n!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~é\xa0🎉', ':a[', ':a{', 'ab:', 'http://')
const nestingCommonMarkShapeWeight = 21
const spelledTypes = new Set(['text', ...Object.keys(blockDirectives), ...Object.keys(inlineDirectives), ...Object.keys(markAttributes)])
function textOf(minLength: number): Arbitrary<string> {
return fc.oneof(
{ arbitrary: fc.string({ maxLength: 12, minLength, unit: markdownPieces }), weight: 4 },
{ arbitrary: fc.string({ maxLength: 6, minLength, unit: 'grapheme' }), weight: 1 },
)
}
const text = textOf(1)
const unknownType = fc.oneof(fc.stringMatching(/^[a-z][A-Za-z0-9]{0,7}$/), text).filter((type) => !spelledTypes.has(type))
const numberValue = fc.oneof({ arbitrary: fc.integer({ max: 10, min: -1 }), weight: 3 }, { arbitrary: fc.double({ noDefaultInfinity: true, noNaN: true }), weight: 1 })
// V8's JSON.parse returns a wrong key after parsing a key holding an escaped backslash (https://issues.chromium.org/issues/521080746); Bun is unaffected.
const keyPiece = fc
.oneof({ arbitrary: markdownPieces, weight: 4 }, { arbitrary: fc.string({ maxLength: 1, minLength: 1, unit: 'grapheme' }), weight: 1 })
.filter((piece) => !/[\\"\x00-\x1f]/.test(piece))
const jsonKey = fc.string({ maxLength: 8, unit: keyPiece })
const { jsonValue } = fc.letrec<{ jsonValue: JsonValue }>((tie) => ({
jsonValue: fc.oneof(
{ depthSize: 'small', maxDepth: 2 },
fc.oneof(fc.constant(null), fc.boolean(), numberValue, textOf(0)),
fc.array(tie('jsonValue'), { maxLength: 3 }),
fc.dictionary(jsonKey, tie('jsonValue'), { maxKeys: 3, noNullPrototype: true }),
),
}))
const valueByKind: Readonly<Record<AttributeKind, Arbitrary<JsonValue>>> = {
boolean: fc.boolean(),
json: jsonValue,
number: numberValue,
string: textOf(0),
}
function attributes(vocabulary: AttributeVocabulary): Arbitrary<AdfAttributes> {
const model = Object.fromEntries(
Object.entries(vocabulary).map(([key, kind]) => [key, fc.oneof({ arbitrary: fc.constant(undefined), weight: 2 }, { arbitrary: valueByKind[kind], weight: 1 })]),
)
return fc.record(model).map(heldAttributes)
}
function heldAttributes(held: Readonly<Record<string, JsonValue | undefined>>): AdfAttributes {
const attrs: AdfAttributes = {}
for (const [key, value] of Object.entries(held)) if (value !== undefined) attrs[key] = value
return attrs
}
function pipeTable({ body, header }: { body: AdfNode[][]; header: AdfNode[] }): AdfNode {
const rows = [header, ...body.map((cells) => header.map((_, index) => cells[index] ?? emptyCell))]
return { content: rows.map((content): AdfNode => ({ content, type: 'tableRow' })), type: 'table' }
}
const mark: Arbitrary<AdfMark> = fc.oneof(
{ arbitrary: fc.oneof(...Object.entries(markAttributes).map(([type, vocabulary]) => attributes(vocabulary).map((attrs) => ({ attrs, type })))), weight: 9 },
{ arbitrary: fc.record({ attrs: fc.dictionary(jsonKey, jsonValue, { maxKeys: 2, noNullPrototype: true }), type: unknownType }), weight: 1 },
)
const marks = fc.uniqueArray(mark, { maxLength: 3, selector: (held) => held.type })
const textNode = fc.record({ marks, text }).map((held): AdfNode => ({ ...held, type: 'text' }))
const autolinkTextNode = fc
.record({ href: fc.tuple(fc.constantFrom('ab:', 'http://'), textOf(0)).map(([scheme, rest]) => `${scheme}${rest}`), marks })
.map(({ href, marks: held }): AdfNode => ({ marks: [...held.filter((outer) => outer.type !== 'link'), { attrs: { href }, type: 'link' }], text: href, type: 'text' }))
const inlineNodes = Object.entries(inlineDirectives).map(([type, directive]) =>
fc.record({ attrs: attributes(directive.attributes), marks }).map((held): AdfNode => ({ ...held, type })),
)
function weighted(arbitraries: readonly Arbitrary<AdfNode>[], weight: number): { arbitrary: Arbitrary<AdfNode>; weight: number }[] {
return arbitraries.map((arbitrary) => ({ arbitrary, weight }))
}
const positions = fc.letrec<Positions>((tie) => {
const blockContent = fc.array(tie('block'), { depthIdentifier, maxLength: 3 })
const inlineContent = fc.array(tie('inline'), { depthIdentifier, maxLength: 4 })
const contentByModel = {
block: blockContent,
code: fc.array(text.map((held): AdfNode => ({ text: held, type: 'text' })), { maxLength: 2 }),
inline: inlineContent,
none: fc.constant<AdfNode[]>([]),
}
const blockMarks = fc.oneof({ arbitrary: fc.constant<AdfMark[]>([]), weight: 4 }, { arbitrary: marks, weight: 1 })
const blockNodes = Object.entries(blockDirectives).map(([type, directive]) => {
const argument = blockArgument(type)
const vocabulary: AttributeVocabulary = argument === undefined ? directive.attributes : { ...directive.attributes, [argument]: 'string' }
const node = fc.record({ attrs: attributes(vocabulary), content: contentByModel[directive.contentModel], marks: blockMarks }).map((held): AdfNode => ({ ...held, type }))
return { leaf: directive.contentModel === 'code' || directive.contentModel === 'none', node }
})
const unknownNode = fc
.record({ attrs: fc.dictionary(jsonKey, jsonValue, { maxKeys: 2, noNullPrototype: true }), content: fc.array(tie('inline'), { depthIdentifier, maxLength: 2 }), marks, type: unknownType })
.map((held): AdfNode => held)
const leafBlocks = blockNodes.filter((entry) => entry.leaf).map((entry) => entry.node)
const containerBlocks = blockNodes.filter((entry) => !entry.leaf).map((entry) => entry.node)
const misplacedWeight = 7
const paragraph = inlineContent.map((content): AdfNode => ({ content, type: 'paragraph' }))
const cell = (type: string) => paragraph.map((held): AdfNode => ({ content: [held], type }))
const listItems = fc.array(
blockContent.map((content): AdfNode => ({ content, type: 'listItem' })),
{ depthIdentifier, maxLength: 3, minLength: 1 },
)
const flatCommonMarkShapes = [
fc.record({ content: inlineContent, level: fc.integer({ max: 6, min: 1 }) }).map(({ content, level }): AdfNode => ({ attrs: { level }, content, type: 'heading' })),
paragraph,
fc.record({ body: fc.array(fc.array(cell('tableCell'), { maxLength: 3 }), { maxLength: 2 }), header: fc.array(cell('tableHeader'), { maxLength: 3, minLength: 1 }) }).map(pipeTable),
]
const nestingCommonMarkShapes = [
blockContent.map((content): AdfNode => ({ content, type: 'blockquote' })),
listItems.map((content): AdfNode => ({ content, type: 'bulletList' })),
fc
.record({ content: listItems, order: fc.oneof({ arbitrary: fc.integer({ max: 3, min: 0 }), weight: 4 }, { arbitrary: fc.integer({ max: 999999999, min: 0 }), weight: 1 }) })
.map(({ content, order }): AdfNode => ({ attrs: { order }, content, type: 'orderedList' })),
]
const flatBlocks = [...weighted(leafBlocks, 2), ...weighted(flatCommonMarkShapes, flatCommonMarkShapeWeight)]
return {
block: fc.oneof(
{ depthIdentifier, depthSize: 'small', maxDepth: 4 },
{ arbitrary: fc.oneof(...flatBlocks), weight: flatBlocks.reduce((sum, entry) => sum + entry.weight, 0) },
{ arbitrary: fc.oneof(...containerBlocks), weight: containerBlocks.length * 2 },
{ arbitrary: fc.oneof(textNode, ...inlineNodes, unknownNode), weight: misplacedWeight },
{ arbitrary: fc.oneof(...nestingCommonMarkShapes), weight: nestingCommonMarkShapes.length * nestingCommonMarkShapeWeight },
),
inline: fc.oneof(
{ depthIdentifier, depthSize: 'small', maxDepth: 4 },
{ arbitrary: textNode, weight: 12 },
{ arbitrary: autolinkTextNode, weight: 2 },
{ arbitrary: fc.oneof(...inlineNodes), weight: 7 },
{ arbitrary: fc.oneof(...blockNodes.map((entry) => entry.node), unknownNode), weight: 2 },
),
}
})
const adfDocument = fc.array(positions.block, { depthIdentifier, maxLength: 4, minLength: 1 }).map((content): AdfDocument => toEditorNormal({ content, type: 'doc', version: 1 }))
function runParameters(): { numRuns: number; seed?: number } {
const deepRuns = env[deepRunsVariable]
if (deepRuns === undefined) return { numRuns: gateRuns, seed: gateSeed }
assert.ok(/^[1-9]\d*$/.test(deepRuns), `${deepRunsVariable} is a run count in digits, such as ${deepRunsVariable}=10000: found ${JSON.stringify(deepRuns)}`)
return { numRuns: Number(deepRuns) }
}
test('a generated document refuses to emit, or its markdown reads back to it', { timeout: propertyTimeout }, () => { test('a generated document refuses to emit, or its markdown reads back to it', { timeout: propertyTimeout }, () => {
fc.assert( fc.assert(
@@ -178,6 +18,6 @@ test('a generated document refuses to emit, or its markdown reads back to it', {
assert.ok(read.ok, read.ok ? '' : `${read.error.code}: ${read.error.message} — reading ${JSON.stringify(emitted.value)}`) assert.ok(read.ok, read.ok ? '' : `${read.error.code}: ${read.error.message} — reading ${JSON.stringify(emitted.value)}`)
assert.deepEqual(toEditorNormal(read.value), document, `reading ${JSON.stringify(emitted.value)}`) assert.deepEqual(toEditorNormal(read.value), document, `reading ${JSON.stringify(emitted.value)}`)
}), }),
runParameters(), propertyRuns(gateRuns),
) )
}) })
src/markdown-property.test.ts
+432
View File
@@ -0,0 +1,432 @@
import fc from 'fast-check'
import assert from 'node:assert/strict'
import test from 'node:test'
import type { AdfDocument } from './adf/document.ts'
import type { Arbitrary, DepthIdentifier } from 'fast-check'
import type { AttributeVocabulary } from './adf/attribute-vocabulary.ts'
import type { JsonValue } from './json-value.ts'
import type { Result } from './result.ts'
import { adfDocument, attributes, jsonKey, jsonValue, markdownPieces, propertyRuns, propertyTimeout, textOf } from './property-harness.ts'
import { adfToMarkdown } from './markdown/emit/adf-to-markdown.ts'
import { blockArgument } from './markdown/block-directive-arguments.ts'
import { blockDirectives } from './adf/block-directives.ts'
import { carryName } from './markdown/opaque-carry.ts'
import { fencedCodeBlock } from './markdown/backtick-runs.ts'
import { inlineDirectives } from './adf/inline-directives.ts'
import { listBreakName } from './markdown/list-break.ts'
import { markAttributes } from './adf/mark-attributes.ts'
import { markSpelling } from './markdown/mark-spellings.ts'
import { markdownToAdf } from './markdown/parse/markdown-to-adf.ts'
import { marksAttribute } from './markdown/block-directive-marks.ts'
import { nodeContent, nodeMarks } from './adf/document.ts'
import { serializeCanonicalJson } from './canonical-json.ts'
import { spellAttributes, spellJsonAttribute, spellLeafDirective, spellStringAttribute, spellVocabulary } from './markdown/directive-syntax.ts'
import { textDirectiveName } from './markdown/text-directive.ts'
import { toEditorNormal } from './adf/editor-normal.ts'
import { vocabularyPairs } from './adf/attribute-vocabulary.ts'
type Choice = { arbitrary: Arbitrary<string>; hostile?: true; weight: number }
type Edit = [at: number, removed: number, inserted: string]
type InlineMarkdown = { destination: Arbitrary<string>; inlines: Arbitrary<string>; label: Arbitrary<string>; oneLine: Arbitrary<string>; text: Arbitrary<string>; word: Arbitrary<string> }
type LeafMarkdown = { fencedCode: Arbitrary<string>; leafBlock: Arbitrary<string> }
const commonMarkTypes = new Set(['blockquote', 'bulletList', 'codeBlock', 'hardBreak', 'heading', 'listItem', 'orderedList', 'paragraph', 'rule', 'text'])
const directiveShapedFloor = 330
const fixpointFloor = 600
const gateRuns = 1000
const markdownMarkTypes = new Set(Object.keys(markAttributes).filter((type) => markSpelling(type)?.kind !== 'directive'))
const vocabularies = [...Object.values(blockDirectives).map((directive) => directive.attributes), ...Object.values(inlineDirectives).map((directive) => directive.attributes), ...Object.values(markAttributes)]
const attributeKeys = [
...new Set([...vocabularies.flatMap((vocabulary) => Object.keys(vocabulary)), ...Object.keys(blockDirectives).flatMap((type) => blockArgument(type) ?? []), marksAttribute, 'json', textDirectiveName]),
]
const directiveNames = [...Object.keys(blockDirectives), ...Object.keys(inlineDirectives), ...Object.keys(markAttributes), carryName, listBreakName, textDirectiveName]
// Hostile generation reaches refusals; clean generation holds none a single piece would trip, so a whole document reaches the emitter.
function choose(hostile: boolean, choices: readonly Choice[], depth?: { depthIdentifier: DepthIdentifier; maxDepth: number }): Arbitrary<string> {
const held = choices.filter((choice) => hostile || choice.hostile !== true).map(({ arbitrary, weight }) => ({ arbitrary, weight }))
return depth === undefined ? fc.oneof(...held) : fc.oneof({ ...depth, depthSize: 'small' }, ...held)
}
const bareToken = fc.stringMatching(/^[A-Za-z0-9_-]{1,8}$/)
const prose = fc.stringMatching(/^[A-Za-z][a-z]{0,6}(?: [a-z]{1,6}){0,3}$/)
const cleanText = fc.string({ maxLength: 12, unit: fc.constantFrom(...'aZ09 \t!"#$%&\'()*+,-./;=?@[\\]^_`{}~é\xa0🎉') })
const syntaxTokens = fc.constantFrom(
...[...String.fromCodePoint(0x0, 0xb, 0xc, 0x85, 0xa0, 0x200b, 0x2028, 0x3000, 0xfeff)],
'\n',
'\r\n',
'\r',
'\t',
' ',
' \n',
'\\\n',
'**',
'__',
'~~',
'***',
'```',
'~~~',
'![',
'](',
'{}',
'::',
':::',
'> ',
'- ',
'* ',
'1. ',
'2) ',
'# ',
'---',
'===',
'| ',
' |',
'<!--',
'-->',
'&amp;',
'&#',
)
const piece = fc.oneof(markdownPieces, syntaxTokens)
const namedEntity = fc.constantFrom('&amp;', '&lt;', '&quot;', '&copy;', '&nbsp;', '&ouml;', '&bogus;', '&amp', '&#;', '&AMP;', '&ngE;')
const numericEntity = fc
.tuple(fc.oneof(fc.integer({ max: 0x7f, min: 0 }), fc.integer({ max: 0xffff, min: 0 }), fc.integer({ max: 0x110000, min: 0 })), fc.boolean())
.map(([code, hex]) => (hex ? `&#x${code.toString(16)};` : `&#${code};`))
const escape = fc.constantFrom(...'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~a \n').map((escaped) => `\\${escaped}`)
const backticks = fc.integer({ max: 3, min: 1 }).map((count) => '`'.repeat(count))
const codeSpan = fc
.tuple(backticks, fc.oneof(prose, textOf(0)), fc.oneof({ arbitrary: fc.constant(undefined), weight: 4 }, { arbitrary: backticks, weight: 1 }))
.map(([opener, body, closer]) => `${opener}${body}${closer ?? opener}`)
const autolink = fc.oneof(
fc.tuple(fc.constantFrom('http://', 'https://', 'mailto:', 'ab:', 'x+y.z-:'), prose).map(([scheme, rest]) => `<${scheme}${rest.replaceAll(' ', '/')}>`),
fc.stringMatching(/^<[a-z.+]{1,6}@[a-z-]{1,6}(?:\.[a-z]{1,4})?>$/),
)
const hostileAutolink = fc.tuple(fc.constantFrom('http://', 'ab:', 'a:'), textOf(0)).map(([scheme, rest]) => `<${scheme}${rest}>`)
const inlineHtml = fc.constantFrom('<span>', '</span>', '<a href="x">', "<b class='y'/>", '<!-- c -->', '<!---->', '<?x?>', '<![CDATA[x]]>', '<!X y>', '<br/>', '<b', '<3', '< a>')
const hardBreak = fc.constantFrom('\\\n', ' \n', '\n', ':hardBreak{}')
const textDirective = fc.constantFrom(' ', ' ', '\t', '\n', '\n\n').map((held) => `:${textDirectiveName}{${textDirectiveName}=${spellStringAttribute(held)}}`)
const hostileTextDirective = fc.constantFrom(' \n', 'a', '').map((held) => `:${textDirectiveName}{${textDirectiveName}=${spellStringAttribute(held)}}`)
const url = fc.stringMatching(/^https?:\/\/[a-z]{1,6}\.[a-z]{2,3}(?:\/[a-z0-9()]{0,5})?$/)
const title = (text: Arbitrary<string>) => fc.oneof(fc.constant(''), text.map((held) => ` "${held}"`), text.map((held) => ` '${held}'`), text.map((held) => ` (${held})`))
const attributeValue = fc.oneof(
{ arbitrary: bareToken, weight: 3 },
{ arbitrary: fc.constantFrom('true', 'false', '0', '1', '3', '-1', '1.5', '1e2', '"1"', '01', 'null', '"[]"', '"{}"', '"#deebff"'), weight: 2 },
{ arbitrary: textOf(0).map(spellStringAttribute), weight: 3 },
{ arbitrary: jsonValue.map(spellJsonAttribute), weight: 2 },
{ arbitrary: textOf(0).map((held) => JSON.stringify(held)), weight: 1 },
{ arbitrary: textOf(0).map((held) => `"${held}`), weight: 1 },
)
const attributePairs = fc.uniqueArray(fc.tuple(fc.oneof({ arbitrary: fc.constantFrom(...attributeKeys), weight: 4 }, { arbitrary: bareToken, weight: 1 }), attributeValue), {
maxLength: 3,
minLength: 1,
selector: ([key]) => key,
})
const hostileAttributes = fc.oneof(
{ arbitrary: fc.constant(''), weight: 3 },
{
arbitrary: fc.tuple(attributePairs, fc.boolean()).map(([pairs, sorted]) => {
const ordered = sorted ? pairs.toSorted(([left], [right]) => (left < right ? -1 : 1)) : pairs
return `{${ordered.map(([key, value]) => `${key}=${value}`).join(' ')}}`
}),
weight: 6,
},
{ arbitrary: fc.constantFrom('{}', '{ }', '{a}', '{a=}', '{=b}', '{a=b', '{a=b c=d}', '{a="}"}'), weight: 1 },
)
function tableAttributes(vocabulary: AttributeVocabulary, slot?: string): Arbitrary<string> {
return attributes(vocabulary).map((attrs) => spellAttributes(spellVocabulary(vocabularyPairs(attrs, vocabulary, slot === undefined ? [] : [slot]) ?? [])))
}
const directiveName = fc.oneof({ arbitrary: fc.constantFrom(...directiveNames), weight: 8 }, { arbitrary: fc.stringMatching(/^[A-Za-z0-9-]{1,6}$/), weight: 1 })
const hostileArgument = fc.oneof(
{ arbitrary: fc.constant(''), weight: 3 },
{ arbitrary: fc.constantFrom(' info', ' warning', ' custom', ' DONE', ' TODO'), weight: 2 },
{ arbitrary: fc.oneof(bareToken.map((held) => ` ${held}`), fc.constantFrom(' info', ' a b', ' "a"')), weight: 1 },
)
function directiveHeader(colons: number, name: string, argument: string, attrs: string): string {
return `${':'.repeat(colons)}${name}${argument}${attrs === '' ? '' : ` ${attrs}`}`
}
function container(header: (colons: number) => string, body: string, closer: number | null): string {
const colons = Math.max(2, ...[...body.matchAll(/:{2,}/g)].map(([run]) => run.length)) + 1
return [header(colons), ...(body === '' ? [] : [body]), ':'.repeat(closer ?? colons)].join('\n')
}
const carriedNode = fc.oneof(
fc.tuple(fc.constantFrom('mention', 'paragraph', 'status', 'widget'), fc.dictionary(jsonKey, jsonValue, { maxKeys: 2, noNullPrototype: true })).map(([type, attrs]): JsonValue => ({ attrs, type })),
textOf(1).map((held): JsonValue => ({ text: held, type: 'text' })),
)
const inlineCarry = carriedNode.map((node) => `:${carryName}{json=${spellStringAttribute(serializeCanonicalJson(node, 'compact'))}}`)
const hostileInlineCarry = fc.oneof(carriedNode, jsonValue).map((node) => `:${carryName}{json=${spellStringAttribute(JSON.stringify(node, null, 1))}}`)
const blockCarry = carriedNode.map((node) => fencedCodeBlock(carryName, serializeCanonicalJson(node, 'two-space')))
const hostileBlockCarry = fc.oneof(carriedNode, jsonValue).map((node) => fencedCodeBlock(carryName, JSON.stringify(node)))
function prefixLines(body: string, first: string, rest: (index: number) => string): string {
return body
.split('\n')
.map((line, index) => (index === 0 ? `${first}${line}` : line === '' ? rest(index).trimEnd() : `${rest(index)}${line}`))
.join('\n')
}
const separator = fc.oneof({ arbitrary: fc.constant('\n\n'), weight: 4 }, { arbitrary: fc.constant('\n'), weight: 3 }, { arbitrary: fc.constantFrom('\n\n\n', '\n \n', '\n\t\n'), weight: 1 })
const quotePrefix = fc.oneof({ arbitrary: fc.constant('> '), weight: 6 }, { arbitrary: fc.constantFrom('>', ' > ', '> ', '>\t', ''), weight: 1 })
const listMarker = fc.oneof(
{ arbitrary: fc.constantFrom('-', '*', '+'), weight: 3 },
{
arbitrary: fc
.tuple(fc.oneof({ arbitrary: fc.integer({ max: 3, min: 0 }), weight: 4 }, { arbitrary: fc.integer({ max: 1000000000, min: 0 }), weight: 1 }), fc.constantFrom('.', ')'))
.map(([start, delimiter]) => `${start}${delimiter}`),
weight: 2,
},
)
const indentDrift = fc.oneof({ arbitrary: fc.constant(0), weight: 6 }, { arbitrary: fc.integer({ max: 2, min: -2 }), weight: 1 })
const closerDrift = fc.option(fc.integer({ max: 5, min: 2 }), { freq: 6 })
function markdownOf(hostile: boolean): Arbitrary<string> {
const inline = inlineMarkdown(hostile)
return blockMarkdown(hostile, inline, leafBlocks(hostile, inline))
}
function inlineMarkdown(hostile: boolean): InlineMarkdown {
const inlineDepth = fc.createDepthIdentifier()
const text = hostile ? fc.oneof(cleanText, textOf(0)) : cleanText
const word = fc.oneof({ arbitrary: prose, weight: 3 }, { arbitrary: text.filter((held) => held !== ''), weight: 2 })
const destination = fc.oneof(url, text, fc.stringMatching(/^<[0-9.#][a-z0-9 ]{0,6}>$/), fc.constant(''), ...(hostile ? [text.map((held) => `<${held}>`)] : []))
const label = fc.oneof(prose, word)
const { inlines } = fc.letrec<{ inline: string; inlines: string }>((tie) => ({
inline: choose(
hostile,
[
{ arbitrary: word, weight: 12 },
{ arbitrary: fc.oneof(codeSpan, autolink, namedEntity, numericEntity, escape, hardBreak, textDirective, inlineCarry), weight: 8 },
{ arbitrary: fc.oneof(hostileAutolink, hostileInlineCarry, hostileTextDirective, inlineHtml, piece), hostile: true, weight: 4 },
{
arbitrary: fc
.tuple(fc.constantFrom('*', '_', '**', '__', '***', '~~', '~'), fc.constantFrom('', '', ' '), tie('inlines'), fc.constantFrom('', '', ' '), fc.option(fc.constantFrom('*', '_', '**', '~~'), { freq: 4 }))
.map(([opener, inside, body, closing, closer]) => `${opener}${inside}${body}${closing}${closer ?? opener}`),
weight: 4,
},
{
arbitrary: fc
.tuple(fc.constantFrom('', '', hostile ? '!' : ''), tie('inlines'), fc.oneof(fc.tuple(destination, title(text)).map(([target, titled]) => `(${target}${titled})`), label.map((held) => `[${held}]`), fc.constantFrom('', '[]')))
.map(([image, content, target]) => `${image}[${content}]${target}`),
weight: 3,
},
{
arbitrary: fc.oneof(
...Object.entries(inlineDirectives).map(([name, directive]) =>
fc
.tuple(directive.textAttribute === undefined ? fc.constant(null) : fc.option(hostile ? word : prose), tableAttributes(directive.attributes, directive.textAttribute))
.map(([slot, attrs]) => (slot === null ? spellLeafDirective(name, attrs) : `:${name}[${slot}]${attrs}`)),
),
...Object.entries(markAttributes)
.filter(([name]) => hostile || markSpelling(name)?.kind === 'directive')
.map(([name, vocabulary]) => fc.tuple(tie('inlines'), tableAttributes(vocabulary)).map(([content, attrs]) => `:${name}[${content}]${attrs}`)),
),
weight: 2,
},
{
arbitrary: fc.tuple(directiveName, fc.option(tie('inlines'), { freq: 3 }), hostileAttributes).map(([name, content, attrs]) => `:${name}${content === null ? '' : `[${content}]`}${attrs}`),
hostile: true,
weight: 2,
},
],
{ depthIdentifier: inlineDepth, maxDepth: 3 },
),
inlines: fc.array(tie('inline'), { depthIdentifier: inlineDepth, maxLength: 4, minLength: 1 }).map((parts) => parts.join('')),
}))
return { destination, inlines, label, oneLine: inlines.map((held) => held.replace(/[\n\r]/g, ' ')), text, word }
}
function leafBlocks(hostile: boolean, { destination, inlines, label, oneLine, text, word }: InlineMarkdown): LeafMarkdown {
const fencedCode = fc
.tuple(
fc.constantFrom('```', '```', '~~~', '````', '``'),
fc.oneof(fc.constant(''), bareToken, text),
fc.array(fc.oneof(prose, text, fc.constantFrom('```', '~~~', ':::', ' x')), { maxLength: 3 }),
fc.constantFrom('', '', '`', '~', 'none'),
)
.map(([fence, info, lines, closer]) => [`${fence}${info}`, ...lines, ...(closer === 'none' ? [] : [`${fence}${closer}`])].join('\n'))
const pipeTable = fc
.record({
body: fc.array(fc.array(oneLine, { maxLength: 3 }), { maxLength: 2 }),
delimiter: fc.array(fc.constantFrom('---', '-', ':--', '--:', ':-:', '', '==='), { maxLength: 3, minLength: 1 }),
header: fc.array(oneLine, { maxLength: 3, minLength: 1 }),
leading: fc.boolean(),
regular: hostile ? fc.boolean() : fc.constant(true),
trailing: fc.boolean(),
})
.map(({ body, delimiter, header, leading, regular, trailing }) => {
const row = (cells: readonly string[]) => (regular || leading ? `| ${cells.join(' | ')}` : cells.join(' | ')) + (regular || trailing ? ' |' : '')
const width = (cells: readonly string[]) => (regular ? header.map((_, index) => cells[index] ?? '') : cells)
return [row(header), row(regular ? header.map(() => '---') : delimiter), ...body.map((cells) => row(width(cells)))].join('\n')
})
const leafBlock = choose(hostile, [
{
arbitrary: fc
.tuple(fc.integer({ max: 7, min: 1 }), fc.constantFrom(' ', ' ', '', '\t'), oneLine, fc.constantFrom('', '', ' #', '#', ' ## '))
.map(([level, gap, content, closer]) => `${'#'.repeat(level)}${gap}${content}${closer}`),
weight: 3,
},
{
arbitrary: fc.tuple(inlines, fc.constantFrom('=', '-'), fc.integer({ max: 4, min: 1 }), fc.constantFrom('', ' ')).map(([content, underline, length, trailing]) => `${content}\n${underline.repeat(length)}${trailing}`),
weight: 2,
},
{ arbitrary: fc.constantFrom('---', '***', '___', '- - -', ' * * *', '_____', '--', '*-*'), weight: 1 },
{ arbitrary: fencedCode, weight: 2 },
{ arbitrary: blockCarry, weight: 1 },
{
arbitrary: fc.tuple(fc.constantFrom(' ', '\t', ' '), fc.array(fc.oneof(prose, text), { maxLength: 3, minLength: 1 })).map(([indent, lines]) => lines.map((line) => `${indent}${line}`).join('\n')),
weight: 1,
},
{ arbitrary: pipeTable, weight: 2 },
{ arbitrary: fc.tuple(label, destination, title(text)).map(([name, target, titled]) => `[${name}]: ${target}${titled}`), weight: 1 },
{ arbitrary: fc.tuple(word, fc.oneof(url, fc.constant(''))).map(([alt, target]) => `![${alt}](${target})`), weight: 1 },
{ arbitrary: hostileBlockCarry, hostile: true, weight: 1 },
{
arbitrary: fc.constantFrom('<div>\ntext\n</div>', '<!-- comment -->', '<pre>\nx\n</pre>', '<?php echo 1; ?>', '<!DOCTYPE html>', '<table>', '<custom-tag attr="1">', '</div>', '<![CDATA[\nx\n]]>'),
hostile: true,
weight: 1,
},
{
arbitrary: fc.tuple(fc.constantFrom(2, 2, 3, 1, 4), directiveName, hostileArgument, hostileAttributes).map(([colons, name, argument, attrs]) => directiveHeader(colons, name, argument, attrs)),
hostile: true,
weight: 1,
},
{ arbitrary: fc.constantFrom(':::', '::', '::::', ':::panel', '::: panel', ':::panel info extra'), hostile: true, weight: 1 },
])
return { fencedCode, leafBlock }
}
function blockMarkdown(hostile: boolean, { inlines, oneLine }: InlineMarkdown, { fencedCode, leafBlock }: LeafMarkdown): Arbitrary<string> {
const blockDepth = fc.createDepthIdentifier()
const { blocks } = fc.letrec<{ block: string; blocks: string }>((tie) => {
const bodyByModel = { block: fc.oneof(tie('blocks'), fc.constant('')), code: fencedCode, inline: fc.oneof(oneLine, fc.constant('')) }
const tableDirectives = Object.entries(blockDirectives).map(([name, directive]) => {
const argument =
blockArgument(name) === undefined
? fc.constant('')
: fc.oneof({ arbitrary: fc.constantFrom(' DONE', ' TODO', ' custom', ' info', ' warning'), weight: 3 }, { arbitrary: bareToken.map((held) => ` ${held}`), weight: 1 })
const attrs = hostile ? fc.oneof({ arbitrary: tableAttributes(directive.attributes), weight: 4 }, { arbitrary: hostileAttributes, weight: 1 }) : tableAttributes(directive.attributes)
if (directive.contentModel === 'none') return fc.tuple(argument, attrs).map(([held, spelled]) => directiveHeader(2, name, held, spelled))
return fc
.tuple(argument, attrs, bodyByModel[directive.contentModel], hostile ? closerDrift : fc.constant(null))
.map(([held, spelled, body, closer]) => container((colons) => directiveHeader(colons, name, held, spelled), body, closer))
})
return {
block: choose(
hostile,
[
{ arbitrary: fc.array(inlines, { maxLength: 3, minLength: 1 }).map((lines) => lines.join('\n')), weight: 10 },
{ arbitrary: leafBlock, weight: 10 },
{
arbitrary: fc.tuple(tie('blocks'), fc.array(quotePrefix, { maxLength: 3, minLength: 1 })).map(([body, prefixes]) => prefixLines(body, '> ', (index) => prefixes[index % prefixes.length] ?? '')),
weight: 3,
},
{
arbitrary: fc
.tuple(listMarker, fc.array(fc.tuple(fc.option(listMarker, { freq: 4 }), fc.constantFrom(' ', ' ', ' ', ' ', '\t', '', ' '), tie('blocks'), indentDrift), { maxLength: 3, minLength: 1 }), separator)
.map(([listed, items, gap]) =>
items
.map(([own, space, body, drift]) => {
const marker = own ?? listed
return prefixLines(body, `${marker}${space}`, () => ' '.repeat(Math.max(0, marker.length + space.length + drift)))
})
.join(gap === '\n\n' ? '\n\n' : '\n'),
),
weight: 4,
},
{ arbitrary: fc.oneof(...tableDirectives), weight: 3 },
{
arbitrary: fc
.tuple(directiveName, hostileArgument, hostileAttributes, tie('blocks'), fc.option(fc.integer({ max: 5, min: 3 }), { freq: 2 }), closerDrift)
.map(([name, argument, attrs, body, opener, closer]) => container((colons) => directiveHeader(opener ?? colons, name, argument, attrs), body, closer)),
hostile: true,
weight: 1,
},
],
{ depthIdentifier: blockDepth, maxDepth: 3 },
),
blocks: fc
.array(fc.tuple(separator, tie('block')), { depthIdentifier: blockDepth, maxLength: 3, minLength: 1 })
.map((entries) => entries.map(([gap, block], index) => (index === 0 ? block : `${gap}${block}`)).join('')),
}
})
return blocks
}
const cleanMarkdown = markdownOf(false)
const hostileMarkdown = markdownOf(true)
const canonical = adfDocument
.map((document) => adfToMarkdown(document))
.filter((emitted): emitted is Extract<Result<string>, { ok: true }> => emitted.ok)
.map((emitted) => emitted.value)
function edited(markdown: string, edits: readonly Edit[]): string {
let text = markdown
for (const [at, removed, inserted] of edits) {
const index = at % (text.length + 1)
text = `${text.slice(0, index)}${inserted}${text.slice(index + removed)}`
}
return text
}
const edit: Arbitrary<Edit> = fc.tuple(fc.nat(), fc.nat({ max: 3 }), fc.oneof(piece, fc.constant('')))
const markdown = fc.oneof(
{ arbitrary: cleanMarkdown, weight: 4 },
{ arbitrary: hostileMarkdown, weight: 2 },
{ arbitrary: canonical, weight: 2 },
{ arbitrary: fc.tuple(fc.oneof(cleanMarkdown, canonical), fc.array(edit, { maxLength: 3, minLength: 1 })).map(([held, edits]) => edited(held, edits)), weight: 3 },
{ arbitrary: fc.string({ maxLength: 40, unit: piece }), weight: 1 },
)
const document = fc
.tuple(markdown, fc.oneof({ arbitrary: fc.constant('\n'), weight: 8 }, { arbitrary: fc.constantFrom('\r\n', '\r'), weight: 1 }), fc.constantFrom('', '', '\n', ' \n'))
.map(([held, ending, trailing]) => `${held}${trailing}`.replaceAll('\n', ending))
function holdsDirectiveShape(document: AdfDocument): boolean {
const pending = [...nodeContent(document)]
for (let node = pending.pop(); node !== undefined; node = pending.pop()) {
if (!commonMarkTypes.has(node.type) || nodeMarks(node).some((mark) => !markdownMarkTypes.has(mark.type))) return true
pending.push(...nodeContent(node))
}
return false
}
test('generated markdown refuses, or what it parses to refuses to emit, or its spelling reads back and spells itself', { timeout: propertyTimeout }, () => {
const parameters = propertyRuns(gateRuns)
let directiveShaped = 0
let fixpoints = 0
fc.assert(
fc.property(document, (input) => {
const parsed = markdownToAdf(input)
if (!parsed.ok) return
const emitted = adfToMarkdown(parsed.value)
if (!emitted.ok) return
fixpoints += 1
if (holdsDirectiveShape(parsed.value)) directiveShaped += 1
const read = markdownToAdf(emitted.value)
assert.ok(read.ok, read.ok ? '' : `${read.error.code}: ${read.error.message} — reading ${JSON.stringify(emitted.value)}`)
assert.deepEqual(toEditorNormal(read.value), toEditorNormal(parsed.value), `reading ${JSON.stringify(emitted.value)}`)
const respelled = adfToMarkdown(read.value)
assert.ok(respelled.ok, respelled.ok ? '' : `${respelled.error.code}: ${respelled.error.message} — spelling ${JSON.stringify(emitted.value)} again`)
assert.equal(respelled.value, emitted.value)
}),
parameters,
)
if (!parameters.gate) return
assert.ok(fixpoints >= fixpointFloor, `${fixpoints} of ${gateRuns} runs reached the fixpoint, under the floor of ${fixpointFloor}`)
assert.ok(directiveShaped >= directiveShapedFloor, `${directiveShaped} runs reaching the fixpoint held a node or mark outside CommonMark's own types, under the floor of ${directiveShapedFloor}`)
})
src/markdown/emit/inline-line.ts
+1 -1
View File
@@ -292,6 +292,6 @@ function emitLink(nodes: readonly AdfNode[], mark: AdfMark, depth: number, range
if (!inner.ok) return inner if (!inner.ok) return inner
if (inner.value.carry !== undefined) return inner if (inner.value.carry !== undefined) return inner
const spelledTarget: InlineSegment = context.bracketed ? { escaping: 'bracketed-link-target', text: escapeUnbalanced(target.value, '[', ']') } : syntax(target.value) const spelledTarget: InlineSegment = context.bracketed ? { escaping: 'bracketed-link-target', text: escapeUnbalanced(target.value, '[', ']') } : syntax(target.value)
return success({ segments: [syntax('['), ...inner.value.segments, syntax(']('), spelledTarget, syntax(')')] }) return success({ segments: [{ escaping: 'none', nodes: range, text: '[' }, ...inner.value.segments, syntax(']('), spelledTarget, syntax(')')] })
} }
src/markdown/emit/line-escaping.ts
+50 -8
View File
@@ -3,6 +3,7 @@ import { delimiterFlags, isWordCharacter, matchEmphasis, runLength } from '../em
import { backslashEscape, escapesLineClaim, inlineHtmlConstruct, opensBracketedAutolink, opensEmailAutolink, type LinePosition } from '../commonmark-grammar.ts' import { backslashEscape, escapesLineClaim, inlineHtmlConstruct, opensBracketedAutolink, opensEmailAutolink, type LinePosition } from '../commonmark-grammar.ts'
import { isBareDelimiterRow } from '../pipe-table-syntax.ts' import { isBareDelimiterRow } from '../pipe-table-syntax.ts'
import { opensInlineDirective } from '../directive-syntax.ts' import { opensInlineDirective } from '../directive-syntax.ts'
import { opensLinkDefinition } from '../link-reference-definitions.ts'
import { readEntityReference } from '../entity-references.ts' import { readEntityReference } from '../entity-references.ts'
export type EmphasisRole = 'close' | 'open' export type EmphasisRole = 'close' | 'open'
@@ -13,7 +14,8 @@ export type NodeRange = { first: number; last: number }
export type InlineSegment = export type InlineSegment =
| { emphasis: EmphasisRole; escaping: 'none'; nodes: NodeRange; text: string } | { emphasis: EmphasisRole; escaping: 'none'; nodes: NodeRange; text: string }
| { emphasis?: undefined; escaping: InlineEscaping; text: string } | { emphasis?: undefined; escaping: 'none'; nodes: NodeRange; text: string }
| { emphasis?: undefined; escaping: InlineEscaping; nodes?: undefined; text: string }
export type AssembledLine = { line: string; unspellableRun: NodeRange | undefined } export type AssembledLine = { line: string; unspellableRun: NodeRange | undefined }
@@ -27,8 +29,7 @@ type EmittedRun = { canClose: boolean; canOpen: boolean; character: string; deli
const delimiters = ['*', '_', '`', '~'] const delimiters = ['*', '_', '`', '~']
// The `:` keeps a `[label]: url` line escaped: unescaped, the parser swallows it as a link reference definition. const followsLinkText = /[([]/
const followsLinkText = /[([:]/
export function assembleInlineLine(segments: readonly InlineSegment[], container: LineContainer): AssembledLine { export function assembleInlineLine(segments: readonly InlineSegment[], container: LineContainer): AssembledLine {
return escape(resolveEmphasis(segments), container) return escape(resolveEmphasis(segments), container)
@@ -67,9 +68,24 @@ function escape(segments: readonly InlineSegment[], container: LineContainer): A
const scan = segments.map((segment) => segment.text).join('') const scan = segments.map((segment) => segment.text).join('')
const escapings: InlineEscaping[] = [] const escapings: InlineEscaping[] = []
for (const segment of segments) for (let index = 0; index < segment.text.length; index += 1) escapings.push(segment.escaping) for (const segment of segments) for (let index = 0; index < segment.text.length; index += 1) escapings.push(segment.escaping)
const escaped = new Set<number>() const escaped = escapedIndexes(scan, escapings, container)
const placements: number[] = [] const placements: number[] = []
let output = '' let output = ''
for (let index = 0; index < scan.length; index += 1) {
if (escaped.has(index)) output += '\\'
placements.push(output.length)
output += scan.charAt(index)
}
if (container === 'paragraph' && opensLinkDefinition(output)) {
const opener = segments[0]?.nodes
if (opener !== undefined) return { line: output, unspellableRun: opener }
return { line: `\\${output}`, unspellableRun: unspellableRun(segments, output, placements) }
}
return { line: output, unspellableRun: unspellableRun(segments, output, placements) }
}
function escapedIndexes(scan: string, escapings: readonly InlineEscaping[], container: LineContainer): Set<number> {
const escaped = new Set<number>()
const linkClose = lastLinkClose(scan, escapings) const linkClose = lastLinkClose(scan, escapings)
let line = scanLine(scan, 0) let line = scanLine(scan, 0)
for (let index = 0; index < scan.length; index += 1) { for (let index = 0; index < scan.length; index += 1) {
@@ -84,13 +100,37 @@ function escape(segments: readonly InlineSegment[], container: LineContainer): A
(escaping === 'bracketed-link-target' && (escaping === 'bracketed-link-target' &&
((scan.charAt(index) === '`' && opensCodeSpan(scan, index, escaped)) || (scan.charAt(index) === ':' && opensInlineDirective(scan, index)))) ((scan.charAt(index) === '`' && opensCodeSpan(scan, index, escaped)) || (scan.charAt(index) === ':' && opensInlineDirective(scan, index))))
) { ) {
output += '\\'
escaped.add(index) escaped.add(index)
} }
placements.push(output.length)
output += scan.charAt(index)
} }
return { line: output, unspellableRun: unspellableRun(segments, output, placements) } escapeClosedRuns(scan, escapings, escaped)
return escaped
}
// CommonMark reads no escape inside a code span, so a backtick string an escape forms or splits off still closes one an earlier bare run opens.
function escapeClosedRuns(scan: string, escapings: readonly InlineEscaping[], escaped: Set<number>): void {
const formed = new Set<number>()
let end = scan.length - 1
while (end >= 0) {
if (scan.charAt(end) !== '`') {
end -= 1
continue
}
let start = end
while (scan.charAt(start - 1) === '`') start -= 1
let segmentEnd = end
for (let index = end; index > start; index -= 1) {
if (!escaped.has(index)) continue
formed.add(segmentEnd - index + 1)
segmentEnd = index - 1
}
if (segmentEnd !== end || escaped.has(start)) formed.add(segmentEnd - start + 1)
else if (escapings[start] !== 'none' && formed.has(end - start + 1)) {
for (let index = start; index <= end; index += 1) escaped.add(index)
formed.add(1)
}
end = start - 1
}
} }
function unspellableRun(segments: readonly InlineSegment[], output: string, placements: readonly number[]): NodeRange | undefined { function unspellableRun(segments: readonly InlineSegment[], output: string, placements: readonly number[]): NodeRange | undefined {
@@ -247,6 +287,8 @@ function lastLinkClose(scan: string, escapings: readonly (InlineEscaping | undef
} }
function opensCodeSpan(scan: string, index: number, escaped: ReadonlySet<number>): boolean { function opensCodeSpan(scan: string, index: number, escaped: ReadonlySet<number>): boolean {
// A run escapes whole: a rest left bare would be a raw run of another length for a closer.
if (scan.charAt(index - 1) === '`' && escaped.has(index - 1)) return true
if (!startsRun(scan, index, escaped)) return false if (!startsRun(scan, index, escaped)) return false
const opener = backtickRun(scan, index) const opener = backtickRun(scan, index)
return closingBacktickRun(scan, index + opener, opener) !== undefined return closingBacktickRun(scan, index + opener, opener) !== undefined
src/markdown/parse/link-reference-definitions.ts → src/markdown/link-reference-definitions.ts
+6 -2
View File
@@ -1,10 +1,14 @@
import type { LinkDefinition, LinkPart } from '../link-syntax.ts' import type { LinkDefinition, LinkPart } from './link-syntax.ts'
import { normalizeLabel, readDestination, readLabel, readTitle, skipLinkWhitespace } from '../link-syntax.ts' import { normalizeLabel, readDestination, readLabel, readTitle, skipLinkWhitespace } from './link-syntax.ts'
type ReadDefinition = { definition: LinkDefinition; label: string; length: number } type ReadDefinition = { definition: LinkDefinition; label: string; length: number }
const restOfLine = /^[ \t]*(?:\n|$)/ const restOfLine = /^[ \t]*(?:\n|$)/
export function opensLinkDefinition(text: string): boolean {
return readDefinition(text) !== undefined
}
export function readLinkDefinitions(definitions: Map<string, LinkDefinition>, text: string): string { export function readLinkDefinitions(definitions: Map<string, LinkDefinition>, text: string): string {
let rest = text let rest = text
let read = readDefinition(rest) let read = readDefinition(rest)
src/markdown/parse/blocks.ts
+1 -1
View File
@@ -18,7 +18,7 @@ import {
} from '../commonmark-grammar.ts' } from '../commonmark-grammar.ts'
import { directiveLineEscape, malformedDirective, readDirectiveLine } from '../directive-syntax.ts' import { directiveLineEscape, malformedDirective, readDirectiveLine } from '../directive-syntax.ts'
import { barePipeCells, isDelimiterRow, isPipeAlignment, isPipeDelimiter, malformedPipeTable, pipeCells } from '../pipe-table-syntax.ts' import { barePipeCells, isDelimiterRow, isPipeAlignment, isPipeDelimiter, malformedPipeTable, pipeCells } from '../pipe-table-syntax.ts'
import { readLinkDefinitions } from './link-reference-definitions.ts' import { readLinkDefinitions } from '../link-reference-definitions.ts'
export type Block = { position: SourcePosition } & ( export type Block = { position: SourcePosition } & (
| { argument: string | undefined; attributes: DirectiveAttributes; blocks: Block[] | undefined; kind: 'directive'; name: string } | { argument: string | undefined; attributes: DirectiveAttributes; blocks: Block[] | undefined; kind: 'directive'; name: string }
src/property-harness.ts
+171
View File
@@ -0,0 +1,171 @@
import fc from 'fast-check'
import assert from 'node:assert/strict'
import { env } from 'node:process'
import type { AdfAttributes, AdfDocument, AdfMark, AdfNode } from './adf/document.ts'
import type { Arbitrary } from 'fast-check'
import type { AttributeKind, AttributeVocabulary } from './adf/attribute-vocabulary.ts'
import type { JsonValue } from './json-value.ts'
import { blockArgument } from './markdown/block-directive-arguments.ts'
import { blockDirectives } from './adf/block-directives.ts'
import { inlineDirectives } from './adf/inline-directives.ts'
import { markAttributes } from './adf/mark-attributes.ts'
import { toEditorNormal } from './adf/editor-normal.ts'
type Positions = { block: AdfNode; inline: AdfNode }
const deepRunsVariable = 'PROPERTY_RUNS'
const gateSeed = 20260914
// Bun's test runner stops a test after five seconds unless the test sets its own timeout.
export const propertyTimeout = 600000
const depthIdentifier = fc.createDepthIdentifier()
const emptyCell: AdfNode = { content: [{ type: 'paragraph' }], type: 'tableCell' }
const flatCommonMarkShapeWeight = 4
export const markdownPieces = fc.constantFrom(...'aZ09 \t\n!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~é\xa0🎉', ':a[', ':a{', 'ab:', 'http://')
const nestingCommonMarkShapeWeight = 21
const spelledTypes = new Set(['text', ...Object.keys(blockDirectives), ...Object.keys(inlineDirectives), ...Object.keys(markAttributes)])
export function textOf(minLength: number): Arbitrary<string> {
return fc.oneof(
{ arbitrary: fc.string({ maxLength: 12, minLength, unit: markdownPieces }), weight: 4 },
{ arbitrary: fc.string({ maxLength: 6, minLength, unit: 'grapheme' }), weight: 1 },
)
}
const text = textOf(1)
const unknownType = fc.oneof(fc.stringMatching(/^[a-z][A-Za-z0-9]{0,7}$/), text).filter((type) => !spelledTypes.has(type))
const numberValue = fc.oneof({ arbitrary: fc.integer({ max: 10, min: -1 }), weight: 3 }, { arbitrary: fc.double({ noDefaultInfinity: true, noNaN: true }), weight: 1 })
// V8's JSON.parse returns a wrong key after parsing a key holding an escaped backslash (https://issues.chromium.org/issues/521080746); Bun is unaffected.
const keyPiece = fc
.oneof({ arbitrary: markdownPieces, weight: 4 }, { arbitrary: fc.string({ maxLength: 1, minLength: 1, unit: 'grapheme' }), weight: 1 })
.filter((piece) => !/[\\"\x00-\x1f]/.test(piece))
export const jsonKey = fc.string({ maxLength: 8, unit: keyPiece })
export const { jsonValue } = fc.letrec<{ jsonValue: JsonValue }>((tie) => ({
jsonValue: fc.oneof(
{ depthSize: 'small', maxDepth: 2 },
fc.oneof(fc.constant(null), fc.boolean(), numberValue, textOf(0)),
fc.array(tie('jsonValue'), { maxLength: 3 }),
fc.dictionary(jsonKey, tie('jsonValue'), { maxKeys: 3, noNullPrototype: true }),
),
}))
const valueByKind: Readonly<Record<AttributeKind, Arbitrary<JsonValue>>> = {
boolean: fc.boolean(),
json: jsonValue,
number: numberValue,
string: textOf(0),
}
export function attributes(vocabulary: AttributeVocabulary): Arbitrary<AdfAttributes> {
const model = Object.fromEntries(
Object.entries(vocabulary).map(([key, kind]) => [key, fc.oneof({ arbitrary: fc.constant(undefined), weight: 2 }, { arbitrary: valueByKind[kind], weight: 1 })]),
)
return fc.record(model).map(heldAttributes)
}
function heldAttributes(held: Readonly<Record<string, JsonValue | undefined>>): AdfAttributes {
const attrs: AdfAttributes = {}
for (const [key, value] of Object.entries(held)) if (value !== undefined) attrs[key] = value
return attrs
}
function pipeTable({ body, header }: { body: AdfNode[][]; header: AdfNode[] }): AdfNode {
const rows = [header, ...body.map((cells) => header.map((_, index) => cells[index] ?? emptyCell))]
return { content: rows.map((content): AdfNode => ({ content, type: 'tableRow' })), type: 'table' }
}
const mark: Arbitrary<AdfMark> = fc.oneof(
{ arbitrary: fc.oneof(...Object.entries(markAttributes).map(([type, vocabulary]) => attributes(vocabulary).map((attrs) => ({ attrs, type })))), weight: 9 },
{ arbitrary: fc.record({ attrs: fc.dictionary(jsonKey, jsonValue, { maxKeys: 2, noNullPrototype: true }), type: unknownType }), weight: 1 },
)
const marks = fc.uniqueArray(mark, { maxLength: 3, selector: (held) => held.type })
const textNode = fc.record({ marks, text }).map((held): AdfNode => ({ ...held, type: 'text' }))
const backtickRunNode = fc
.record({ marks: fc.oneof(fc.constant<AdfMark[]>([]), fc.constant<AdfMark[]>([{ type: 'code' }]), marks), text: fc.string({ maxLength: 6, minLength: 1, unit: fc.constantFrom('`', '``', ' ', 'a') }) })
.map((held): AdfNode => ({ ...held, type: 'text' }))
const autolinkTextNode = fc
.record({ href: fc.tuple(fc.constantFrom('ab:', 'http://'), textOf(0)).map(([scheme, rest]) => `${scheme}${rest}`), marks })
.map(({ href, marks: held }): AdfNode => ({ marks: [...held.filter((outer) => outer.type !== 'link'), { attrs: { href }, type: 'link' }], text: href, type: 'text' }))
const inlineNodes = Object.entries(inlineDirectives).map(([type, directive]) =>
fc.record({ attrs: attributes(directive.attributes), marks }).map((held): AdfNode => ({ ...held, type })),
)
function weighted(arbitraries: readonly Arbitrary<AdfNode>[], weight: number): { arbitrary: Arbitrary<AdfNode>; weight: number }[] {
return arbitraries.map((arbitrary) => ({ arbitrary, weight }))
}
const positions = fc.letrec<Positions>((tie) => {
const blockContent = fc.array(tie('block'), { depthIdentifier, maxLength: 3 })
const inlineContent = fc.array(tie('inline'), { depthIdentifier, maxLength: 4 })
const contentByModel = {
block: blockContent,
code: fc.array(text.map((held): AdfNode => ({ text: held, type: 'text' })), { maxLength: 2 }),
inline: inlineContent,
none: fc.constant<AdfNode[]>([]),
}
const blockMarks = fc.oneof({ arbitrary: fc.constant<AdfMark[]>([]), weight: 4 }, { arbitrary: marks, weight: 1 })
const blockNodes = Object.entries(blockDirectives).map(([type, directive]) => {
const argument = blockArgument(type)
const vocabulary: AttributeVocabulary = argument === undefined ? directive.attributes : { ...directive.attributes, [argument]: 'string' }
const node = fc.record({ attrs: attributes(vocabulary), content: contentByModel[directive.contentModel], marks: blockMarks }).map((held): AdfNode => ({ ...held, type }))
return { leaf: directive.contentModel === 'code' || directive.contentModel === 'none', node }
})
const unknownNode = fc
.record({ attrs: fc.dictionary(jsonKey, jsonValue, { maxKeys: 2, noNullPrototype: true }), content: fc.array(tie('inline'), { depthIdentifier, maxLength: 2 }), marks, type: unknownType })
.map((held): AdfNode => held)
const leafBlocks = blockNodes.filter((entry) => entry.leaf).map((entry) => entry.node)
const containerBlocks = blockNodes.filter((entry) => !entry.leaf).map((entry) => entry.node)
const misplacedWeight = 7
const paragraph = fc.oneof({ arbitrary: inlineContent, weight: 3 }, { arbitrary: fc.array(backtickRunNode, { maxLength: 4, minLength: 2 }), weight: 1 }).map((content): AdfNode => ({ content, type: 'paragraph' }))
const cell = (type: string) => paragraph.map((held): AdfNode => ({ content: [held], type }))
const listItems = fc.array(
blockContent.map((content): AdfNode => ({ content, type: 'listItem' })),
{ depthIdentifier, maxLength: 3, minLength: 1 },
)
const flatCommonMarkShapes = [
fc.record({ content: inlineContent, level: fc.integer({ max: 6, min: 1 }) }).map(({ content, level }): AdfNode => ({ attrs: { level }, content, type: 'heading' })),
paragraph,
fc.record({ body: fc.array(fc.array(cell('tableCell'), { maxLength: 3 }), { maxLength: 2 }), header: fc.array(cell('tableHeader'), { maxLength: 3, minLength: 1 }) }).map(pipeTable),
]
const nestingCommonMarkShapes = [
blockContent.map((content): AdfNode => ({ content, type: 'blockquote' })),
listItems.map((content): AdfNode => ({ content, type: 'bulletList' })),
fc
.record({ content: listItems, order: fc.oneof({ arbitrary: fc.integer({ max: 3, min: 0 }), weight: 4 }, { arbitrary: fc.integer({ max: 999999999, min: 0 }), weight: 1 }) })
.map(({ content, order }): AdfNode => ({ attrs: { order }, content, type: 'orderedList' })),
]
const flatBlocks = [...weighted(leafBlocks, 2), ...weighted(flatCommonMarkShapes, flatCommonMarkShapeWeight)]
return {
block: fc.oneof(
{ depthIdentifier, depthSize: 'small', maxDepth: 4 },
{ arbitrary: fc.oneof(...flatBlocks), weight: flatBlocks.reduce((sum, entry) => sum + entry.weight, 0) },
{ arbitrary: fc.oneof(...containerBlocks), weight: containerBlocks.length * 2 },
{ arbitrary: fc.oneof(textNode, ...inlineNodes, unknownNode), weight: misplacedWeight },
{ arbitrary: fc.oneof(...nestingCommonMarkShapes), weight: nestingCommonMarkShapes.length * nestingCommonMarkShapeWeight },
),
inline: fc.oneof(
{ depthIdentifier, depthSize: 'small', maxDepth: 4 },
{ arbitrary: textNode, weight: 12 },
{ arbitrary: autolinkTextNode, weight: 2 },
{ arbitrary: backtickRunNode, weight: 3 },
{ arbitrary: fc.oneof(...inlineNodes), weight: 7 },
{ arbitrary: fc.oneof(...blockNodes.map((entry) => entry.node), unknownNode), weight: 2 },
),
}
})
export const adfDocument = fc.array(positions.block, { depthIdentifier, maxLength: 4, minLength: 1 }).map((content): AdfDocument => toEditorNormal({ content, type: 'doc', version: 1 }))
export function propertyRuns(gateRuns: number): { gate: boolean; numRuns: number; seed?: number } {
const deepRuns = env[deepRunsVariable]
if (deepRuns === undefined) return { gate: true, numRuns: gateRuns, seed: gateSeed }
assert.ok(/^[1-9]\d*$/.test(deepRuns), `${deepRunsVariable} is a run count in digits, such as ${deepRunsVariable}=10000: found ${JSON.stringify(deepRuns)}`)
return { gate: false, numRuns: Number(deepRuns) }
}
todo.md
+18 -2
View File
@@ -59,6 +59,20 @@ proves 12, 13 spells 11's gaps in 12's grammar, and 12 rewrites code 4b and 4c c
- [ ] **4.3 — The markdown property.** Generated markdown through `markdownToAdf` never throws, - [ ] **4.3 — The markdown property.** Generated markdown through `markdownToAdf` never throws,
and the runs fit the budget; where it parses and `adfToMarkdown` spells the result, that and the runs fit the budget; where it parses and `adfToMarkdown` spells the result, that
spelling parses and emits to itself byte for byte (§2). spelling parses and emits to itself byte for byte (§2).
**Settled** (the maintainer, 2026-09-15): the generators and run parameters 4.2 and 4.3
share live in one test-only module in `src/`, kept out of the build and coverage, with
10c's properties as its third user. Under the gate seed the property asserts floors on the
runs reaching the fixpoint and on the directive-shaped ones. It lands when hunts of several
hundred thousand runs per engine pass clean, since the breaks hit once per ~150,000 runs,
past a 10,000-run bar. Two breaks, both shipped in `0.1.0`, are fixed inside it. A backtick
string an escape formed closed an earlier bare run's code span, since CommonMark reads no
escape inside one: an escaped backtick alone or, as the review found, one joined to the bare
run after it. A backtick run now escapes whole, and a bare run escapes wherever a later
string of its length forms around an escape in the same inline content: the generalized pass
the maintainer chose (2026-09-15). A paragraph's opening read as a link
reference definition across a `]` the emitter spelled: the emitter now escapes the opening
`[` exactly when the parser's own definition reader accepts the paragraph, and a link
opening it rides the carry until 13b.
- [x] **4.4 — The real payloads.** - [x] **4.4 — The real payloads.**
- [ ] **4b — The block walk's retry (`0.2.0`).** `emitBlock` walks a subtree twice wherever - [ ] **4b — The block walk's retry (`0.2.0`).** `emitBlock` walks a subtree twice wherever
`readableBlock` reads it whole and then gives up — a list item whose first line reads back `readableBlock` reads it whole and then gives up — a list item whose first line reads back
@@ -237,7 +251,7 @@ proves 12, 13 spells 11's gaps in 12's grammar, and 12 rewrites code 4b and 4c c
`src/adf/block-directives.ts` + `inline-directives.ts`, `src/markdown/`'s `src/adf/block-directives.ts` + `inline-directives.ts`, `src/markdown/`'s
`directive-syntax.ts`, `opaque-carry.ts` and the `emit/` + `parse/` readers, every corpus `directive-syntax.ts`, `opaque-carry.ts` and the `emit/` + `parse/` readers, every corpus
fixture (round-trip, normalization and `errors/`), the prose reader over `spec/flavour.md`, fixture (round-trip, normalization and `errors/`), the prose reader over `spec/flavour.md`,
and the README's examples. the markdown property's generator, and the README's examples.
**Settled** (the maintainer, 2026-09-13): **Settled** (the maintainer, 2026-09-13):
- A line opening `!adf:name` is a block line when a space or the line's end follows the name, - A line opening `!adf:name` is a block line when a space or the line's end follows the name,
and a paragraph when `[` or `{` does. Claiming stays syntactic and structure comes from the and a paragraph when `[` or `{` does. Claiming stays syntactic and structure comes from the
@@ -292,7 +306,9 @@ proves 12, 13 spells 11's gaps in 12's grammar, and 12 rewrites code 4b and 4c c
exceptions it cures re-derived, and the README's code table and its "not every document exceptions it cures re-derived, and the README's code table and its "not every document
converts back" guarantee following; the gap list is empty. A round-trip fixture holds the converts back" guarantee following; the gap list is empty. A round-trip fixture holds the
shape 4.2's review left refused until then: an autolink-shaped link under a directive mark shape 4.2's review left refused until then: an autolink-shaped link under a directive mark
whose href holds `\:name{`. whose href holds `\:name{`. A link opening a paragraph whose opening reads as a link
reference definition, which 4.3 leaves riding the carry, takes the directive link too, with
its round-trip fixture (the maintainer, 2026-09-15).
## The ADF inventory to cover ## The ADF inventory to cover
tsconfig.build.json
+1 -1
View File
@@ -27,6 +27,6 @@
"forceConsistentCasingInFileNames": true, "forceConsistentCasingInFileNames": true,
"skipLibCheck": true "skipLibCheck": true
}, },
"exclude": ["src/**/*.test.ts"], "exclude": ["src/**/*.test.ts", "src/property-harness.ts"],
"include": ["src"] "include": ["src"]
} }