4b - the block walk spells a list from one walk, reporting its headroom so the list limit stays 500 #96

Merged
lilleman merged 4 commits from 4b into main 2026-09-18 00:52:09 +02:00
8 changed files with 154 additions and 56 deletions
+11 -2
View File
@@ -284,10 +284,19 @@ someone spells it or pins it.
where the general form fails on the same node. Refusing there refuses a document the general
form spells, so a refusal the general form does not share belongs in the general form or
nowhere — save the nested list a tight spelling would swallow, whose refusal the
tight-versus-blank answer owns (`todo.md` 2b).
tight-versus-blank answer owns (`todo.md` 2b). A readable spelling that must spell its subtree
before it can give way — the list, whose thematic-break first line and blank lines exist only
spelled — hands that one walk to the general form instead: giving way after the walk walks
again at every level, doubling per level (4b).
- Nothing recurses unbounded: the guards walk iteratively, and blocks, marks and JSON values — an
attribute's and a carried node's alike — are all held to 500 levels, so a deep document is a
`Result` rather than the stack overflow that waits near 2000.
`Result` rather than the stack overflow that waits near 2000. A level is one block-list
recursion in either direction: a readable list's items sit one below it, its directive
spelling's two. So a list giving way after its walk owes the directive form a level the walk
did not count, and the walk reports its headroom — the least slack any depth guard below it
has — for the fallback to refuse at zero rather than walk again; counting every list twice
halved the list limit, counting the directive form once doubled the parser's frames per level
(the maintainer, 2026-09-18).
- A reader takes the text and an index — a sticky regex whose `lastIndex` the caller sets on the
line before it reads, `indexOf` — never a fresh slice per character, and a per-character walk
hoists the scan that does not vary with the character. The pipeline persona feeds documents
+9 -4
View File
@@ -19,15 +19,20 @@ export function isJsonValue(value: unknown): value is JsonValue {
return true
}
export function overNested(value: JsonValue, levels: number = largestNesting): boolean {
const pending: { depth: number; item: JsonValue }[] = [{ depth: 0, item: value }]
export function nestingDepth(value: unknown): number {
const pending: { depth: number; item: unknown }[] = [{ depth: 0, item: value }]
let deepest = 0
while (pending.length > 0) {
const entry = pending.pop()
if (entry === undefined) continue
const { depth, item } = entry
if (depth > levels) return true
deepest = Math.max(deepest, depth)
if (Array.isArray(item)) for (const child of item) pending.push({ depth: depth + 1, item: child })
else if (item !== null && typeof item === 'object') for (const child of Object.values(item)) pending.push({ depth: depth + 1, item: child })
}
return false
return deepest
}
export function overNested(value: unknown, levels: number = largestNesting): boolean {
return nestingDepth(value) > levels
}
+29
View File
@@ -440,6 +440,15 @@ test('spells a list item whose marker completes a thematic break as a directive'
const nested: AdfNode = { content: [item({ content: [item()], type: 'bulletList' })], type: 'bulletList' }
assert.equal(markdown(adfToMarkdown(document(nested))), '- -\n')
assert.equal(markdown(adfToMarkdown(document({ content: [item(nested)], type: 'bulletList' }))), '!adf:bulletList\n!adf:listItem\n- -\n!adf:/listItem\n!adf:/bulletList\n')
assert.equal(
markdown(adfToMarkdown(document({ content: [item(paragraph({ text: 'a', type: 'text' }), nested), item({ type: 'rule' })], type: 'bulletList' }))),
'!adf:bulletList\n!adf:listItem\na\n\n- -\n!adf:/listItem\n!adf:listItem\n---\n!adf:/listItem\n!adf:/bulletList\n',
)
const spaced: AdfNode = { content: [{ text: 'a\n \nb', type: 'text' }], type: 'codeBlock' }
assert.equal(
markdown(adfToMarkdown(document({ attrs: { order: 3 }, content: [item(spaced)], type: 'orderedList' }))),
'!adf:orderedList {order=3}\n!adf:listItem\n```\na\n \nb\n```\n!adf:/listItem\n!adf:/orderedList\n',
)
})
test('refuses the characters CommonMark rewrites', () => {
@@ -500,6 +509,26 @@ test('refuses a document nested deeper than the emitter carries', () => {
let carried: AdfNode = paragraph({ text: 'x', type: 'text' })
for (let depth = 0; depth < 500; depth += 1) carried = { content: [carried], type: 'blockquote' }
assert.ok(adfToMarkdown(document(carried)).ok)
const listed = (levels: number, first: readonly AdfNode[]): AdfNode => {
let list: AdfNode = { content: [{ content: [...first], type: 'listItem' }], type: 'bulletList' }
for (let level = 1; level < levels; level += 1) list = { content: [{ content: [...first, list], type: 'listItem' }], type: 'bulletList' }
return list
}
assert.ok(adfToMarkdown(document(listed(largestNesting, [paragraph({ text: 'x', type: 'text' })]))).ok)
assert.equal(code(adfToMarkdown(document(listed(largestNesting + 1, [paragraph({ text: 'x', type: 'text' })])))), 'unsupported-nesting-depth')
const directiveLists = largestNesting / 2
const deep = document(listed(directiveLists, [{ type: 'rule' }]))
assert.deepEqual(markdownToAdf(markdown(adfToMarkdown(deep))), { ok: true, value: deep })
assert.equal(code(adfToMarkdown(document(listed(directiveLists + 1, [{ type: 'rule' }])))), 'unsupported-nesting-depth')
assert.equal(code(adfToMarkdown(document({ content: [listed(directiveLists, [{ type: 'rule' }])], type: 'panel' }))), 'unsupported-nesting-depth')
const chain = (levels: number): AdfNode => {
let card: AdfNode = { type: 'blockCard' }
for (let level = 0; level < levels; level += 1) card = { content: [card], type: 'blockCard' }
return card
}
const carriedInItem = (levels: number): AdfDocument => document(listed(1, [{ type: 'rule' }, chain(levels)]))
assert.deepEqual(markdownToAdf(markdown(adfToMarkdown(carriedInItem(248)))), { ok: true, value: carriedInItem(248) })
assert.equal(code(adfToMarkdown(carriedInItem(249))), 'unsupported-nesting-depth')
})
test('emits an empty list item without trailing whitespace', () => {
+76 -41
View File
@@ -1,7 +1,7 @@
import type { AdfDocument, AdfNode } from '../../adf/document.ts'
import type { BlockDirective } from '../../adf/block-directives.ts'
import { adfDocumentFault, carriesOnly, nodeAttrs, nodeContent, nodeMarks } from '../../adf/document.ts'
import { blockDirective } from '../../adf/block-directives.ts'
import { blockDirective, blockDirectives } from '../../adf/block-directives.ts'
import { blockDirectiveForm } from '../block-directive-forms.ts'
import { carriedBlock } from '../opaque-carry.ts'
import { emitInlineLine } from './inline-line.ts'
@@ -12,41 +12,57 @@ import { languageSlot } from '../code-language.ts'
import { largestNesting } from '../../nesting.ts'
import { listBreakSpelling } from '../list-break.ts'
import { spellBlockDirectiveOpener } from './block-directive-spelling.ts'
import { spellDirectiveCloser } from '../directive-syntax.ts'
import { spellDirectiveCloser, spellDirectiveOpener } from '../directive-syntax.ts'
import { tryImage } from './image.ts'
import { tryPipeTable } from './pipe-table.ts'
type BlockContainer = 'directive' | 'document' | 'list-item'
type BlockSpelling = 'commonmark' | 'directive' | 'list'
type EmittedBlock = { spelling: BlockSpelling; text: string }
type EmittedBlock = { headroom: number; spelling: BlockSpelling; text: string }
type PlacedBlock = EmittedBlock & { node: AdfNode }
type Walk = { blocks: readonly PlacedBlock[]; headroom: number }
type WalkedItem = { node: AdfNode; walk: Walk }
const largestListMarker = 999999999
// Bare because emitList admits no item carrying attributes, marks or text.
const listItemOpener = spellDirectiveOpener('listItem', undefined, '')
export function adfToMarkdown(document: AdfDocument): Result<string> {
const fault = adfDocumentFault(document)
if (fault !== undefined) return faulted(fault, [])
if (document.version !== 1) return failure('unsupported-document-version', `no markdown spelling carries ADF version ${document.version}`, [])
const blocks = emitBlocks(nodeContent(document), 'document', [], 0)
if (!blocks.ok) return blocks
return success(blocks.value === '' ? '' : `${blocks.value}\n`)
const walk = walkBlocks(nodeContent(document), [], 0)
if (!walk.ok) return walk
const text = joinBlocks(walk.value.blocks, 'document')
return success(text === '' ? '' : `${text}\n`)
}
function emitBlocks(nodes: readonly AdfNode[], container: BlockContainer, path: ConvertErrorPath, depth: number): Result<string> {
if (depth > largestNesting) return failure('unsupported-nesting-depth', `the document nests deeper than the ${largestNesting} levels the emitter carries`, path)
// headroom: the least slack any depth guard below the walk has.
function walkBlocks(nodes: readonly AdfNode[], path: ConvertErrorPath, depth: number): Result<Walk> {
let headroom = largestNesting - depth
if (headroom < 0) return tooDeep(path)
const blocks: PlacedBlock[] = []
for (const [index, node] of nodes.entries()) {
const block = emitBlock(node, [...path, 'content', index], depth)
if (!block.ok) return block
headroom = Math.min(headroom, block.value.headroom)
blocks.push({ ...block.value, node })
}
return success({ blocks, headroom })
}
function tooDeep(path: ConvertErrorPath): Result<never> {
return failure('unsupported-nesting-depth', `the document nests deeper than the ${largestNesting} levels the emitter carries`, path)
}
function joinBlocks(blocks: readonly PlacedBlock[], container: BlockContainer): string {
let text = ''
for (const [index, block] of blocks.entries()) {
const previous = blocks[index - 1]
if (previous !== undefined) text += separationBetween(previous, block, container)
text += block.text
}
return success(text)
return text
}
function separationBetween(previous: PlacedBlock, next: PlacedBlock, container: BlockContainer): string {
@@ -73,13 +89,14 @@ function emitBlock(node: AdfNode, path: ConvertErrorPath, depth: number): Result
if (directive === undefined) return commonMarkLine(carriedBlock(node, path, depth))
const readable = readableBlock(node, path, depth)
if (readable !== undefined) return readable
return emitDirectiveBlock(node, directive, path, depth)
return emitDirectiveBlock(node, directive, path, depth, () => walkBlocks(nodeContent(node), path, depth + 1))
}
export function commonMarkSpelling(node: AdfNode, path: ConvertErrorPath, depth: number): Result<null> | undefined {
const readable = readableBlock(node, path, depth)
if (readable === undefined) return undefined
return readable.ok ? success(null) : readable
if (!readable.ok) return readable
return readable.value.spelling === 'directive' ? undefined : success(null)
}
function readableBlock(node: AdfNode, path: ConvertErrorPath, depth: number): Result<EmittedBlock> | undefined {
@@ -98,45 +115,49 @@ function readableText(text: string | undefined): Result<EmittedBlock> | undefine
return text === undefined ? undefined : success(commonMarkText(text))
}
function commonMarkLine(text: Result<string>): Result<EmittedBlock> {
if (!text.ok) return text
return success(commonMarkText(text.value))
function commonMarkLine(carried: Result<{ headroom: number; text: string }>): Result<EmittedBlock> {
if (!carried.ok) return carried
return success({ ...carried.value, spelling: 'commonmark' })
}
function commonMarkText(text: string): EmittedBlock {
return { spelling: 'commonmark', text }
function commonMarkText(text: string, headroom: number = Number.POSITIVE_INFINITY): EmittedBlock {
return { headroom, spelling: 'commonmark', text }
}
function directivePair(node: AdfNode, opener: string, body: string): EmittedBlock {
return { spelling: 'directive', text: `${opener}\n${body === '' ? '' : `${body}\n`}${spellDirectiveCloser(node.type)}` }
function directivePair(node: AdfNode, opener: string, body: string, headroom: number = Number.POSITIVE_INFINITY): EmittedBlock {
return { headroom, spelling: 'directive', text: `${opener}\n${body === '' ? '' : `${body}\n`}${spellDirectiveCloser(node.type)}` }
}
function emitDirectiveBlock(node: AdfNode, directive: BlockDirective, path: ConvertErrorPath, depth: number): Result<EmittedBlock> {
function emitDirectiveBlock(node: AdfNode, directive: BlockDirective, path: ConvertErrorPath, depth: number, walkBody: () => Result<Walk>): Result<EmittedBlock> {
if (node.text !== undefined) return failure('unsupported-node-shape', `a ${node.type} carries no text: this one holds text`, path)
if (blockDirectiveForm(node.type) === 'leaf' && nodeContent(node).length > 0) return failure('unsupported-node-shape', `a ${node.type} holds no content: this one holds some`, path)
if (directive.contentModel === 'code') return emitCodeDirective(node, directive, path, depth)
const opener = spellBlockDirectiveOpener(node, directive)
if (opener === undefined) return commonMarkLine(carriedBlock(node, path, depth))
return emitDirectiveBody(node, directive, opener, path, depth)
return emitDirectiveBody(node, directive, opener, path, walkBody)
}
function emitDirectiveBody(node: AdfNode, directive: BlockDirective, opener: string, path: ConvertErrorPath, depth: number): Result<EmittedBlock> {
if (blockDirectiveForm(node.type) === 'leaf') return success({ spelling: 'directive', text: opener })
const content = nodeContent(node)
const body = directive.contentModel === 'inline' ? emitInlineLine(content, 'paragraph', path) : emitBlocks(content, 'directive', path, depth + 1)
if (!body.ok) return body
return success(directivePair(node, opener, body.value))
function emitDirectiveBody(node: AdfNode, directive: BlockDirective, opener: string, path: ConvertErrorPath, walkBody: () => Result<Walk>): Result<EmittedBlock> {
if (blockDirectiveForm(node.type) === 'leaf') return success({ headroom: Number.POSITIVE_INFINITY, spelling: 'directive', text: opener })
if (directive.contentModel === 'inline') {
const line = emitInlineLine(nodeContent(node), 'paragraph', path)
if (!line.ok) return line
return success(directivePair(node, opener, line.value))
}
const walk = walkBody()
if (!walk.ok) return walk
return success(directivePair(node, opener, joinBlocks(walk.value.blocks, 'directive'), walk.value.headroom))
}
function emitBlockquote(node: AdfNode, path: ConvertErrorPath, depth: number): Result<EmittedBlock> | undefined {
if (!carriesOnly(node, [])) return undefined
const inner = emitBlocks(nodeContent(node), 'document', path, depth + 1)
const inner = walkBlocks(nodeContent(node), path, depth + 1)
if (!inner.ok) return inner
const text = inner.value
const text = joinBlocks(inner.value.blocks, 'document')
.split('\n')
.map((line) => (line === '' ? '>' : `> ${line}`))
.join('\n')
return success(commonMarkText(text))
return success(commonMarkText(text, inner.value.headroom))
}
function emitCodeBlock(node: AdfNode, path: ConvertErrorPath): Result<EmittedBlock> | undefined {
@@ -197,14 +218,29 @@ function emitList(node: AdfNode, path: ConvertErrorPath, depth: number): Result<
const start = listStart(node, items.length)
if (start === undefined || items.length === 0) return undefined
if (items.some((item) => item.type !== 'listItem' || !carriesOnly(item, []))) return undefined
const lines: string[] = []
const walked: WalkedItem[] = []
let headroom = Number.POSITIVE_INFINITY
for (const [offset, item] of items.entries()) {
const emitted = emitListItem(item, ordered ? `${start + offset}. ` : '- ', [...path, 'content', offset], depth)
if (emitted === undefined) return undefined
if (!emitted.ok) return emitted
lines.push(emitted.value)
const walk = walkBlocks(nodeContent(item), [...path, 'content', offset], depth + 1)
if (!walk.ok) return walk
headroom = Math.min(headroom, walk.value.headroom)
walked.push({ node: item, walk: walk.value })
}
return success({ spelling: 'list', text: lines.join('\n') })
const lines: string[] = []
for (const [offset, item] of walked.entries()) {
const line = listItemLines(item.walk.blocks, ordered ? `${start + offset}. ` : '- ')
if (line === undefined) {
if (headroom < 1) return tooDeep(path)
return emitDirectiveBlock(node, ordered ? blockDirectives.orderedList : blockDirectives.bulletList, path, depth, () => success({ blocks: directiveItems(walked), headroom: headroom - 1 }))
}
lines.push(line)
}
return success({ headroom, spelling: 'list', text: lines.join('\n') })
}
// The directive form sinks each item's blocks a level below where the walk read them.
function directiveItems(items: readonly WalkedItem[]): PlacedBlock[] {
return items.map((item) => ({ ...directivePair(item.node, listItemOpener, joinBlocks(item.walk.blocks, 'directive'), item.walk.headroom - 1), node: item.node }))
}
function listStart(node: AdfNode, items: number): number | undefined {
@@ -214,16 +250,15 @@ function listStart(node: AdfNode, items: number): number | undefined {
return start + items - 1 > largestListMarker ? undefined : start
}
function emitListItem(item: AdfNode, marker: string, path: ConvertErrorPath, depth: number): Result<string> | undefined {
const inner = emitBlocks(nodeContent(item), 'list-item', path, depth + 1)
if (!inner.ok) return inner
if (inner.value === '') return success(marker.trimEnd())
const body = inner.value.split('\n')
function listItemLines(blocks: readonly PlacedBlock[], marker: string): string | undefined {
const inner = joinBlocks(blocks, 'list-item')
if (inner === '') return marker.trimEnd()
const body = inner.split('\n')
if (body.some((line) => line !== '' && isBlankLine(line))) return undefined
const indent = ' '.repeat(marker.length)
const lines = body.map((line, index) => (index === 0 ? `${marker}${line}` : line === '' ? '' : `${indent}${line}`))
if (isThematicBreak(lines[0] ?? '')) return undefined
return success(lines.join('\n'))
return lines.join('\n')
}
function emitParagraph(node: AdfNode, path: ConvertErrorPath): Result<EmittedBlock> | undefined {
+8 -7
View File
@@ -3,7 +3,7 @@ import type { DirectiveSpan, Read } from './directive-syntax.ts'
import type { JsonSpelling } from '../canonical-json.ts'
import { failure, success, type ConvertErrorPath, type Result } from '../result.ts'
import { isAdfNode } from '../adf/document.ts'
import { isJsonValue, overNested } from '../json-value.ts'
import { isJsonValue, nestingDepth, overNested } from '../json-value.ts'
import { fencedCodeBlock } from './backtick-runs.ts'
import { largestNesting } from '../nesting.ts'
import { malformedDirective, readSoleStringAttribute, spellAttributes, spellInlineLeafDirective, spellStringAttribute, unsupportedNodeShape } from './directive-syntax.ts'
@@ -13,16 +13,16 @@ export const carryName = 'carry'
const jsonAttribute = 'json'
export function carriedBlock(node: AdfNode, path: ConvertErrorPath, depth: number): Result<string> {
export function carriedBlock(node: AdfNode, path: ConvertErrorPath, depth: number): Result<{ headroom: number; text: string }> {
const json = carriedJson(node, 'two-space', path, largestNesting - depth)
if (!json.ok) return json
return success(fencedCodeBlock(carryName, json.value))
return success({ headroom: json.value.headroom, text: fencedCodeBlock(carryName, json.value.json) })
}
export function carriedInline(node: AdfNode, path: ConvertErrorPath): Result<string> {
const json = carriedJson(node, 'compact', path, largestNesting)
if (!json.ok) return json
return success(spellInlineLeafDirective(carryName, spellAttributes([[jsonAttribute, spellStringAttribute(json.value)]])))
return success(spellInlineLeafDirective(carryName, spellAttributes([[jsonAttribute, spellStringAttribute(json.value.json)]])))
}
export function readCarriedBlock(body: string, depth: number): Read<AdfNode> {
@@ -36,11 +36,12 @@ export function readCarriedInline(span: DirectiveSpan): Read<AdfNode> | undefine
return readCarriedJson(spelled.value, 'compact', largestNesting)
}
function carriedJson(node: AdfNode, spelling: JsonSpelling, path: ConvertErrorPath, levels: number): Result<string> {
if (!isJsonValue(node) || overNested(node, levels)) {
function carriedJson(node: AdfNode, spelling: JsonSpelling, path: ConvertErrorPath, levels: number): Result<{ headroom: number; json: string }> {
const headroom = levels - nestingDepth(node)
if (!isJsonValue(node) || headroom < 0) {
return failure('unsupported-nesting-depth', `a carried node's JSON nests deeper than the ${levels} levels its position leaves`, path)
}
return success(serializeCanonicalJson(node, spelling))
return success({ headroom, json: serializeCanonicalJson(node, spelling) })
}
function readCarriedJson(raw: string, spelling: JsonSpelling, levels: number): Read<AdfNode> {
@@ -675,6 +675,12 @@ test('refuses input nested deeper than the parser carries', () => {
assert.ok(markdownToAdf(nest(repeated('panel'), '!adf:paragraph {localId=a-1}\nPart.\n!adf:/paragraph\n')).ok)
assert.deepEqual(position(markdownToAdf(nest(['expand', ...repeated('panel'), 'expand'], 'Part.\n'))), { line: 501, offset: 5501 })
assert.equal(code(markdownToAdf(nest(['panel', ...repeated('expand'), 'panel'], 'Part.\n'))), 'unsupported-nesting-depth')
const listed = (levels: number): string => `${'!adf:bulletList\n!adf:listItem\n---\n'.repeat(levels)}${'!adf:/listItem\n!adf:/bulletList\n'.repeat(levels)}`
const directiveLists = largestNesting / 2
assert.ok(markdownToAdf(listed(directiveLists)).ok)
assert.equal(code(markdownToAdf(listed(directiveLists + 1))), 'unsupported-nesting-depth')
assert.ok(markdownToAdf(`${'- '.repeat(largestNesting)}a\n`).ok)
assert.equal(code(markdownToAdf(`${'- '.repeat(largestNesting + 1)}a\n`)), 'unsupported-nesting-depth')
})
test('decodes the backslash escapes CommonMark spells, and keeps the rest literal', () => {
+1 -1
View File
@@ -1,2 +1,2 @@
// One level per block-list recursion in either direction — a list and its items count once — or the two guards disagree.
// Levels count per AGENTS.md §11.
export const largestNesting = 500
+14 -1
View File
@@ -64,6 +64,11 @@ proves 12, 13 spells 11's gaps in 12's grammar, and 12 rewrites code 4b and 4c c
export persona runs in bulk walks the document twice. Both walks are linear, so this is a
constant factor rather than 4b's class change, and the parting is what gives depth its own
code (§8) — measure before joining them back.
**Settled** (the maintainer, 2026-09-18): the limit stays 500 readable lists, the walk
reporting its headroom (§11). Counting every list twice was rejected for halving the limit,
counting the directive form once for doubling the parser's frames per level.
**Measured** (2026-09-18): `adfDocumentFault` walks a 9 MB document in 52 ms against 314 ms
for the emit, so its two walks stay parted.
- [ ] **4c — The scanning rule's remaining sites (`0.2.0`).** A trailing-anchored regex re-walks
its run from every start position, so an interior whitespace run costs quadratic time rather
than linear — 3h measured 80k spaces inside an ATX heading at 11.3s, and 3ms once the walk
@@ -81,7 +86,15 @@ proves 12, 13 spells 11's gaps in 12's grammar, and 12 rewrites code 4b and 4c c
(the stability-reviewer, 2026-09-16). §11's scanning rule is
the whole argument; the pipeline persona feeds documents nobody typed.
`readDirectiveContent`'s scan splits into named steps with that fix rather than keeping its
complexity (the maintainer, 2026-09-16).
complexity (the maintainer, 2026-09-16). A sixth 4b leaves behind: the parser asks
`commonMarkSpelling` at every directive-spelled list it reads, and the answer spells the whole
subtree below, itself quadratic in the depth left, so nested directive lists cost about the
cube of their depth — 250 rule-first levels parse in 1.3 s at 16.5 kB, 1.5 MB of that shape
at 250 levels in 0.6 s — bounded by the depth guard like `readNestedDirective` (the
maintainer, 2026-09-18). A seventh is a throw rather than a cost: `adfDocumentFault` pushes a
node's content with a spread, so past about 125k sibling nodes the guard throws a
`RangeError` where §11 owes a `Result` — a loop over the content closes it (the
stability-reviewer and the maintainer, 2026-09-18).
- [ ] **4d — What the gate says while it runs (`0.2.1`).** `ci.sh` runs nine legs and announces
none of them, so five minutes of a Gitea run read as silence and a hang cannot be told from
a slow pull — the maintainer hit exactly this on the `0.1.0` release. Three causes, each its