Answer architecture round 2: the escape walk owns its escapes and names every token kind, a directive read takes the claimed prefix, the reader computes one end, imports sort, and item 96 covers unclosed backtick runs

docs/comprehension-history.md

@@ -335,3 +335,27 @@ Inherited architect, decided by:
 - Locality: `src/markdown/emit/adf-to-markdown.ts:72` `joinBlocks`: a block's spelling depends on the run before it, where the base's separation was a pure function of a pair
 - Shape: `src/markdown/emit/adf-to-markdown.ts:334` `spelledList`: one exit to the directive form, though bullets pass `() => ''` as the number
 - Self-sufficiency: `src/markdown/emit/adf-to-markdown.ts:369` `tryListItemLines`: the `*` branch needs the spec open, cancelling the spec pointers removed
+
+## 2026-10-09T14:50:52Z, PR #27 at 2378782, against a9fedb3
+
+Ruling: better
+
+| Seat | Navigation | Locality | Shape | Self-sufficiency | Overall |
+|---|---|---|---|---|---|
+| Mid A | better | better | better | better | better |
+| Inherited architect | better | better | better | same | better |
+
+Mid A, decided by:
+
+- Locality, Self-sufficiency: `src/markdown/inline-tokens.ts:20` `readInlineToken`: the parser and the escaper ask one function what the reader takes at a character, where each held its own copy of the rules
+- Locality, Shape: `src/markdown/emit/line-escaping.ts:102` `escapeClaims`: the order-dependent `afterEscape` and `pairsEquals` flags are gone; runs go whole to `escapeRun` and fixed inputs ride one `EscapeWalk` record
+- Shape, Navigation: `src/markdown/parse/inline-content.ts:110` `scanInline`: a two-level character dispatch and eight helpers became one switch over token kinds
+- Shape: `src/markdown/emit/line-escaping.ts:158` `claimsCharacter`: switches on `token.kind` in place of a chain of character tests
+
+Inherited architect, decided by:
+
+- Navigation, Shape: `src/markdown/inline-tokens.ts:20` `readInlineToken`: one tokenizer answers what the reader sees at an index, where two parallel implementations had to agree
+- Locality: `src/markdown/emit/line-escaping.ts:112` `escapeClaims`: each step reads on its own, no flag carried from the previous character
+- Shape: `src/markdown/emit/line-escaping.ts:158` `claimsCharacter`: switches on the token kind and takes the shared record; the heading closer is worked out once in `closingHashes`
+- Shape, Navigation: `src/markdown/parse/inline-content.ts:121` `readToken`: one exhaustive switch over `InlineToken['kind']` replaces a two-level dispatch
+- Locality: `src/markdown/emit/line-escaping.ts:173` `claimsCharacter`: `default: return true` claims every kind it does not name, so a new kind is escaped without its author deciding

src/markdown/directive-syntax.test.ts

@@ -2,8 +2,8 @@ import assert from 'node:assert/strict'
 import test from 'node:test'
 
 import type { DirectiveAttributes, DirectiveLine } from './directive-syntax.ts'
+import { claimDirectivePrefix, readDirectiveLine, readInlineDirective } from './directive-syntax.ts'
 import { largestNesting } from '../nesting.ts'
-import { claimsDirectivePrefix, readDirectiveLine, readInlineDirective } from './directive-syntax.ts'
 
 // A pair the input spells bare decodes to itself; a quoted one names its spelling beside the decoding.
 type Pair = [string, string, string?]
@@ -22,8 +22,9 @@ function fault(line: string): string {
 }
 
 function inline(text: string): unknown {
-  if (!claimsDirectivePrefix(text, 0)) return 'unclaimed'
-  const read = readInlineDirective(text, 0)
+  const prefix = claimDirectivePrefix(text, 0)
+  if (prefix === undefined) return 'unclaimed'
+  const read = readInlineDirective(text, prefix)
   if (read.fault !== undefined) return read.fault.message
   return { attributes: read.value.attributes, content: read.value.content, length: read.value.length, name: read.value.name }
 }

src/markdown/directive-syntax.ts

@@ -10,6 +10,9 @@ import { serializeCanonicalJson } from '../canonical-json.ts'
 
 export type AttributeReading = { refusal: 'kind' | 'nesting'; value?: undefined } | { refusal?: undefined; value: VocabularyValue }
 
+// An `!adf:` standing at `index`, which only `claimDirectivePrefix` builds.
+export type ClaimedPrefix = { readonly [claimedPrefix]: true; readonly index: number }
+
 export type DirectiveValue = { decoded: string; spelling: string }
 
 export type DirectiveAttributes = ReadonlyMap<string, DirectiveValue>
@@ -35,6 +38,7 @@ type DirectiveContent = { end: number; spans: NestedSpans }
 
 export const directivePrefix = '!adf:'
 
+const claimedPrefix: unique symbol = Symbol('claimed prefix')
 const bareTokenSource = '[A-Za-z0-9_-]+'
 const bareRun = new RegExp(bareTokenSource, 'y')
 const bareToken = new RegExp(`^${bareTokenSource}$`)
@@ -67,6 +71,10 @@ export function attributeValue(text: string, kind: AttributeKind): AttributeRead
   return overNested(parsed) ? { refusal: 'nesting' } : { value: { kind, value: parsed } }
 }
 
+export function claimDirectivePrefix(text: string, index: number): ClaimedPrefix | undefined {
+  return claimsDirectivePrefix(text, index) ? { [claimedPrefix]: true, index } : undefined
+}
+
 export function claimsDirectivePrefix(text: string, index: number): boolean {
   return text.startsWith(directivePrefix, index)
 }
@@ -88,9 +96,8 @@ export function readDirectiveLine(line: string): Read<DirectiveLine> | undefined
   return line.charAt(directivePrefix.length) === '/' ? readCloserLine(line) : readOpenerLine(line)
 }
 
-// The caller has checked `claimsDirectivePrefix` at `index`.
-export function readInlineDirective(text: string, index: number): Read<DirectiveSpan> {
-  return readClaimedDirective(text, index, 1)
+export function readInlineDirective(text: string, prefix: ClaimedPrefix): Read<DirectiveSpan> {
+  return readClaimedDirective(text, prefix.index, 1)
 }
 
 export function readSoleStringAttribute(span: DirectiveSpan, key: string): Read<string> {

src/markdown/emit/line-escaping.ts

@@ -32,15 +32,14 @@ type EmittedDelimiter = { closes: boolean; offset: number; pair: number; width:
 
 type EmittedRun = { canClose: boolean; canOpen: boolean; character: string; delimiters: EmittedDelimiter[]; length: number; start: number }
 
-// What stays fixed while one line's escapes are decided; `escaped` collects them.
+// What stays fixed while one line's escapes are decided.
 type EscapeWalk = {
-  claims: Claims
-  container: LineContainer
-  escaped: Set<number>
-  escapings: readonly InlineEscaping[]
-  headingCloser: number | undefined
-  linkClose: number
-  scan: string
+  readonly claims: Claims
+  readonly container: LineContainer
+  readonly escapings: readonly InlineEscaping[]
+  readonly headingCloser: number | undefined
+  readonly linkClose: number
+  readonly scan: string
 }
 
 const followsLinkText = /[([]/
@@ -73,7 +72,7 @@ function resolveEmphasis(segments: readonly InlineSegment[]): InlineSegment[] {
     const openOffset = offsets[openerIndex] ?? 0
     const closeOffset = offsets[index] ?? 0
     if (opener.text !== '_') continue
-    if (!isWordCharacter(charAt(scan, openOffset - 1)) && !isWordCharacter(charAt(scan, closeOffset + 1))) continue
+    if (!isWordCharacter(scan.charAt(openOffset - 1)) && !isWordCharacter(scan.charAt(closeOffset + 1))) continue
     opener.text = '*'
     segment.text = '*'
   }
@@ -103,47 +102,47 @@ function escapeClaims(scan: string, escapings: readonly InlineEscaping[], contai
   const walk: EscapeWalk = {
     claims,
     container,
-    escaped: new Set(),
     escapings,
     headingCloser: container === 'heading' ? closingHashes(scan) : undefined,
     linkClose: lastLinkClose(scan, escapings),
     scan,
   }
+  const escaped = new Set<number>()
   let line = scanLine(scan, 0)
   let index = 0
   while (index < scan.length) {
     while (index > line.start + line.text.length) line = scanLine(scan, line.start + line.text.length + 1)
     if (runCharacters.includes(scan.charAt(index))) {
-      index = escapeRun(walk, index, line)
+      const end = index + runLength(scan, index)
+      for (const claimed of runEscapes(walk, index, end, line)) escaped.add(claimed)
+      index = end
       continue
     }
-    const escaping = escapings[index]
     const token = readInlineToken(scan, index, claims)
-    const claimed = isEscapable(escaping)
-      ? index === walk.headingCloser || claimsLineStart(line, index, container) || claimsCharacter(walk, index, token)
-      : escaping === 'bracketed-link-target' && token.kind === 'directive'
-    if (claimed) walk.escaped.add(index)
+    const claimed = claimsCharacter(walk, index, token, line)
+    if (claimed) escaped.add(index)
     // The reader takes a `==` whole, so its second `=` starts nothing unless the first is escaped.
     index += !claimed && token.kind === 'highlight' ? token.width : 1
   }
-  return walk.escaped
+  return escaped
 }
 
 // A run joining the emitter's own delimiter escapes its text; a run of text escapes from its start while the reader claims what the escapes leave of it.
-function escapeRun(walk: EscapeWalk, start: number, line: ScanLine): number {
-  const { escaped, escapings, scan } = walk
-  const end = start + runLength(scan, start)
+function runEscapes(walk: EscapeWalk, start: number, end: number, line: ScanLine): number[] {
+  const { escapings, scan } = walk
+  const indexes: number[] = []
   const run = escapings.slice(start, end)
   if (run.some(isSyntax) && run.some(isEscapable)) {
-    for (let index = start; index < end; index += 1) if (isEscapable(escapings[index])) escaped.add(index)
-    return end
+    for (let index = start; index < end; index += 1) if (isEscapable(escapings[index])) indexes.push(index)
+    return indexes
   }
   for (let index = start; index < end; index += 1) {
     // A backtick run escapes whole: a rest left bare would be a raw run of another length for a closer.
-    if (!(index > start && scan.charAt(index) === '`') && !claimsRunRest(walk, index, end, line)) break
-    escaped.add(index)
+    const claimed = (index > start && scan.charAt(index) === '`') || claimsRunRest(walk, index, end, line)
+    if (!claimed) break
+    indexes.push(index)
   }
-  return end
+  return indexes
 }
 
 function claimsRunRest({ claims, container, escapings, scan }: EscapeWalk, index: number, end: number, line: ScanLine): boolean {
@@ -155,22 +154,35 @@ function claimsRunRest({ claims, container, escapings, scan }: EscapeWalk, index
   return token.kind !== 'text'
 }
 
-function claimsCharacter({ container, escapings, linkClose, scan }: EscapeWalk, index: number, token: InlineToken): boolean {
-  const character = scan.charAt(index)
-  const inBrackets = escapings[index] === 'bracketed'
+function claimsCharacter({ container, escapings, headingCloser, linkClose, scan }: EscapeWalk, index: number, token: InlineToken, line: ScanLine): boolean {
+  const escaping = escapings[index]
+  if (escaping === 'bracketed-link-target') return token.kind === 'directive'
+  if (!isEscapable(escaping)) return false
+  if (index === headingCloser || claimsLineStart(line, index, container)) return true
   switch (token.kind) {
+    case 'autolink':
+    case 'code-span':
+    case 'delimiter-run':
+    case 'directive':
+    case 'entity':
+    case 'escape':
+    case 'hard-break':
+    case 'html':
+      return true
     case 'bracket':
       if (token.image) return isSyntax(escapings[index + 1])
-      return inBrackets || index < linkClose
+      return escaping === 'bracketed' || index < linkClose
     case 'bracket-close':
-      return inBrackets
+      return escaping === 'bracketed'
     case 'highlight':
       return token.opens || token.closes
-    case 'text':
+    // A backslash before a line ending reads as a hard break, so no line ending escapes.
+    case 'line-ending':
+    case 'text': {
+      const character = scan.charAt(index)
       if (character === '|') return container === 'table-cell'
       return character === '{' && scan.charAt(index - 1) === ']' && isSyntax(escapings[index - 1])
-    default:
-      return true
+    }
   }
 }
 
@@ -289,7 +301,7 @@ function emittedRuns(segments: readonly InlineSegment[], placements: readonly nu
     })
   }
   for (const run of runs) {
-    const flags = delimiterFlags(run.character, charAt(output, run.start - 1), output.charAt(run.start + run.length))
+    const flags = delimiterFlags(run.character, output.charAt(run.start - 1), output.charAt(run.start + run.length))
     run.canClose = flags.canClose
     run.canOpen = flags.canOpen
   }
@@ -332,7 +344,3 @@ function lastLinkClose(scan: string, escapings: readonly (InlineEscaping | undef
   }
   return -1
 }
-
-function charAt(text: string, index: number): string {
-  return index < 0 ? '' : text.charAt(index)
-}

src/markdown/inline-tokens.ts

@@ -1,7 +1,8 @@
+import type { ClaimedPrefix } from './directive-syntax.ts'
 import type { Claims } from './portable/conventions.ts'
 import { backslashEscape, inlineHtmlConstruct, readBracketedAutolink, readEmailAutolink } from './commonmark/grammar.ts'
 import { backtickRun, closingBacktickRun } from './commonmark/backtick-runs.ts'
-import { claimsDirectivePrefix } from './directive-syntax.ts'
+import { claimDirectivePrefix } from './directive-syntax.ts'
 import { delimiterFlags, runLength } from './commonmark/emphasis-matching.ts'
 import { highlightDelimiter, highlightFlanking } from './portable/conventions.ts'
 import { readEntityReference } from './commonmark/entity-references.ts'
@@ -13,7 +14,7 @@ export type InlineToken =
   | { kind: 'bracket-close' | 'entity' | 'escape' | 'hard-break' | 'line-ending' | 'text'; width: number }
   | { kind: 'code-span'; opener: number; width: number }
   | { canClose: boolean; canOpen: boolean; kind: 'delimiter-run'; width: number }
-  | { kind: 'directive' }
+  | { kind: 'directive'; prefix: ClaimedPrefix }
   | { closes: boolean; kind: 'highlight'; opens: boolean; width: number }
   | { construct: string; kind: 'html' }
 
@@ -21,9 +22,11 @@ export function readInlineToken(source: string, index: number, claims: Claims):
   switch (source.charAt(index)) {
     case '<':
       return angleToken(source, index)
-    case '!':
-      if (claims.directives && claimsDirectivePrefix(source, index)) return { kind: 'directive' }
+    case '!': {
+      const prefix = claims.directives ? claimDirectivePrefix(source, index) : undefined
+      if (prefix !== undefined) return { kind: 'directive', prefix }
       return source.charAt(index + 1) === '[' ? { image: true, kind: 'bracket', width: 2 } : text(1)
+    }
     case '[':
       return { image: false, kind: 'bracket', width: 1 }
     case ']':

src/markdown/parse/inline-content.ts

@@ -1,21 +1,21 @@
 import type { AdfMark, AdfNode } from '../../adf/document.ts'
+import type { ClaimedPrefix, DirectiveSpan, NestedSpans } from '../directive-syntax.ts'
 import type { Claims } from '../portable/conventions.ts'
-import type { DirectiveSpan, NestedSpans } from '../directive-syntax.ts'
 import type { EmphasisPairing } from '../commonmark/emphasis-matching.ts'
 import type { InlineToken } from '../inline-tokens.ts'
 import type { LineContainer } from '../line-container.ts'
 import type { LinkDefinition } from '../commonmark/link-syntax.ts'
-import { decodeTextEscapes, trimTrailingSpace } from '../commonmark/grammar.ts'
 import { centeredImage, externalMedia } from '../external-image.ts'
 import { commonMarkLink, linkHref } from '../mark-spellings.ts'
-import { matchEmphasis } from '../commonmark/emphasis-matching.ts'
+import { decodeTextEscapes, trimTrailingSpace } from '../commonmark/grammar.ts'
 import { failure, faulted, success, type ConvertErrorPath, type Result } from '../../result.ts'
 import { highlightDelimiter } from '../portable/conventions.ts'
 import { identicalMarks, mergeAdjacentText, nodeAttrs, nodeMarks } from '../../adf/document.ts'
 import { inlineNodeModel } from '../../adf/inline-nodes.ts'
 import { joinsWhenRead, textBreakName, textBreakSpelling } from '../adjacent-text.ts'
-import { noSpans, readInlineDirective } from '../directive-syntax.ts'
+import { matchEmphasis } from '../commonmark/emphasis-matching.ts'
 import { normalizeLabel, readInlineTarget, readLabel } from '../commonmark/link-syntax.ts'
+import { noSpans, readInlineDirective } from '../directive-syntax.ts'
 import { openingLinkTakesDirective } from '../emit/inline-line.ts'
 import { readCarriedInline } from '../opaque-carry.ts'
 import { readDirectiveMark } from './directive-marks.ts'
@@ -119,47 +119,42 @@ function scanInline(scan: Scan): Result<Scanned> {
 }
 
 function readToken(scan: Scan, index: number, token: InlineToken): Result<number> {
+  if (token.kind === 'directive') return readDirective(scan, index, token.prefix)
+  if (token.kind === 'html') return failure('unmappable-html', `no raw HTML converts at this version: ${token.construct}`, scan.path)
+  const end = index + token.width
   switch (token.kind) {
-    case 'autolink': {
-      const end = index + token.width
+    case 'autolink':
       flush(scan, false)
       pushNode(scan, linkedText(scan.source.slice(index + 1, end - 1), token.email ? 'mailto:' : ''))
       return success(end)
-    }
     case 'bracket':
       flush(scan, false)
-      scan.pieces.push({ active: true, image: token.image, kind: 'open', start: index + token.width })
-      return success(index + token.width)
+      scan.pieces.push({ active: true, image: token.image, kind: 'open', start: end })
+      return success(end)
     case 'bracket-close':
       return closeBracket(scan, index)
-    case 'code-span': {
-      const end = index + token.width
+    case 'code-span':
       flush(scan, false)
       pushNode(scan, { marks: [{ type: 'code' }], text: codeSpanText(scan.source.slice(index + token.opener, end - token.opener)), type: 'text' })
       return success(end)
-    }
     case 'delimiter-run':
       flush(scan, false)
       scan.pieces.push({ canClose: token.canClose, canOpen: token.canOpen, character: scan.source.charAt(index), kind: 'run', length: token.width })
-      return success(index + token.width)
-    case 'directive':
-      return readDirective(scan, index)
+      return success(end)
     case 'entity':
     case 'escape':
     case 'text':
-      scan.pending += scan.source.slice(index, index + token.width)
-      return success(index + token.width)
+      scan.pending += scan.source.slice(index, end)
+      return success(end)
     case 'hard-break':
       // CommonMark strips the spaces the two-space break is spelled with, and keeps those before a backslash.
       flush(scan, false)
       pushNode(scan, { type: 'hardBreak' })
-      return success(index + token.width)
+      return success(end)
     case 'highlight':
       flush(scan, false)
       scan.pieces.push({ closes: token.closes, kind: 'highlight', opens: token.opens })
-      return success(index + token.width)
-    case 'html':
-      return failure('unmappable-html', `no raw HTML converts at this version: ${token.construct}`, scan.path)
+      return success(end)
     case 'line-ending':
       return success(readLineEnding(scan, index))
   }
@@ -173,10 +168,10 @@ function readLineEnding(scan: Scan, index: number): number {
   return index + 1
 }
 
-function readDirective(scan: Scan, index: number): Result<number> {
+function readDirective(scan: Scan, index: number, prefix: ClaimedPrefix): Result<number> {
   const held = scan.spans.get(index)
   if (held !== undefined) return pushDirective(scan, held, index)
-  const directive = readInlineDirective(scan.source, index)
+  const directive = readInlineDirective(scan.source, prefix)
   if (directive.fault !== undefined) return faulted(directive.fault, scan.path)
   return pushDirective(scan, directive.value, index)
 }

todo.md

@@ -34,7 +34,7 @@
 | 62 | 0.2.0 | decision | **Move the portable flavour's reading out of `parse/` and its writing out of `emit/` into `markdown/portable/`, so `emit/` no longer imports `portable/`.** | 4 | 5 | 2 | 6 | 2 | 9.4 |
 | 92 | 0.2.0 |  | **Lower the reading cost of the files a chunk reads most before its first commit.** | 2 | 4 | 2 | 4 | 1 | 9.0 |
 | 89 | 0.2.1 |  | **Build a `date`, `emoji`, `inlineCard`, `mention` or `status` holding no mark Atlassian's schema withholds from it, in every reader.** | 4 | 4 | 5 | 6 | 4 | 13.0 |
-| 96 | 0.2.1 | defect | **Read and write a text holding many `<!--`, `<?`, `<![CDATA[` or `<!` openers without a terminator in time linear in the text.** | 2 | 3 | 5 | 9 | 9 | 9.2 |
+| 96 | 0.2.1 | defect | **Read and write a text holding many unclosed backtick runs, or many `<!--`, `<?`, `<![CDATA[` or `<!` openers without a terminator, in time linear in the text.** | 2 | 3 | 5 | 9 | 9 | 9.2 |
 | 33 | 0.2.1 | defect | **Emit a line in time linear in its mark runs, in `adfToLosslessMarkdown` and `adfToPortableMarkdown`.** | 4 | 5 | 6 | 9 | 9 | 8.7 |
 | 7 | 0.3.0 | decision | **Ship HTML: `adfToHtml`, `htmlToAdf`, and markdown to and from HTML composed through ADF, each call named by its flavour.** | 6 | 9 | 9 | 9 | 2, 4 | 25.8 |
 | 45 | 0.3.0 |  | **Replace `isAdfDocument` with a reader returning `Result<AdfDocument>`.** | 2 | 3 | 6 | 8 | 1 | 25.2 |
@@ -135,11 +135,13 @@ them 68188. Read most over the last five, opens and searches: `README.md` 32, `d
 may reject the document. Replace `InlineNodeModel.takesMarks` (`src/adf/inline-nodes.ts`) with the
 mark types a node takes, and have the emitter carry a node holding another.
 
-### 96. Read and write a text holding many `<!--`, `<?`, `<![CDATA[` or `<!` openers without a terminator in time linear in the text.
+### 96. Read and write a text holding many unclosed backtick runs, or many `<!--`, `<?`, `<![CDATA[` or `<!` openers without a terminator, in time linear in the text.
 
-`inlineHtmlConstruct` (`src/markdown/commonmark/grammar.ts`) searches the rest of the text for the
-opener's terminator at every opener, so `<!--` repeated with no `-->` is quadratic in both
-directions: the reader and the emitter's escape walk both ask it through `readInlineToken`.
+`readInlineToken` (`src/markdown/inline-tokens.ts`) searches the rest of the text at every opener:
+`closingBacktickRun` for a run of the same length, and `inlineHtmlConstruct` for the terminator. So
+backtick runs of lengths 1 to k with no closers cost about k·n, and `<!--` repeated with no `-->`
+is quadratic, in the reader and the emitter's escape walk alike. commonmark.js keeps a cache of
+backtick runs by length for the first.
 
 ### 33. Emit a line in time linear in its mark runs, in `adfToLosslessMarkdown` and `adfToPortableMarkdown`.