Convert ADF to markdown and back through adf-codec's lossless pair #561

Merged
lilleman merged 7 commits from adf-lossless into main 2026-11-16 13:35:34 +01:00
5 changed files with 116 additions and 20 deletions
+1
View File
@@ -2,6 +2,7 @@
## Unreleased
- `adf_codec.convert` can now convert ADF to markdown and back without losing a block's `localId` or the empty paragraphs Jira uses for spacing, so a Jira or Confluence document read and saved back unedited comes back identical: `markdown_to_adf_lossless`, `adf_to_markdown_lossless` and `adfs_to_markdown_lossless`. No tool calls them yet.
- A config whose `plan` names a Jira epic must now set `jira-score-formula` and `jira-score-bar`, and fails to load without them. `todo-item add` scores the item by them, refuses one under the bar unless it is exempt, and ranks the new issue among the issues the loop filed: questions first, then security findings, then by score. A person's issues keep their rank.
- `todo-item edit` and `move` work on a plan kept in a Jira epic. They change only an issue the loop filed, and never a link. Of an issue a person filed, `edit` sets only the scores, and posts the new `--comment` on the issue to tell the team why; a plan in a file takes the flag and keeps no comment.
- A release someone else cut while it still held plan items no longer stops the loop: those items count as the next release's, `chunk-preflight` tells the worker to move them there, and `loop-claude` routes on them. Where the plan names no later release and `todo.md` named none since the cut, the loop still asks which release they belong to.
+40 -10
View File
@@ -1,6 +1,10 @@
"""Converts markdown to ADF (Atlassian Document Format, the rich text of Jira Cloud and Confluence) and back.
It runs @larvit/adf-codec, a Node package, in a throwaway container that holds node_shim/shim.mjs; the shim's header says what its exit status means.
adf-codec has two pairs of calls, and each function here names the one it runs:
- The lossless pair, `*_lossless`, for a document that already exists in Jira or Confluence and is edited and saved back. It keeps each block's localId and the empty paragraphs Jira uses for spacing, so a document read and saved unedited comes back identical.
- The portable pair, with no suffix, for new text written from scratch, and for markdown shown to a person. It drops the localId and the empty paragraphs, so saving a document that exists changes it.
"""
import contextlib
import dataclasses
@@ -31,7 +35,7 @@ class Ran:
err: str
# Runs the shim in the given mode (`to-adf` or `to-markdown`) on a text, or fails to run it.
# Runs the shim in the given mode (`to-adf`, `to-markdown`, `to-adf-lossless` or `to-markdown-lossless`) on a text, or fails to run it.
Run = typing.Callable[[str, str], Ran | Failure]
# Runs a docker command line on a text, in at most the given seconds, or fails to run it.
Docker = typing.Callable[[list[str], str, int], Ran | Failure]
@@ -149,9 +153,8 @@ def refusal(out: str) -> Refused | Failure:
return Failure(f"the adf-codec shim refused with output that is not an error: {out.strip()}")
def markdown_to_adf(markdown: str, run: Run = container_run) -> dict[str, typing.Any] | Failure:
"""The ADF document for the markdown."""
out = answer("to-adf", markdown, run)
def to_adf(mode: str, markdown: str, run: Run) -> dict[str, typing.Any] | Failure:
out = answer(mode, markdown, run)
if isinstance(out, Refused):
return out.failure
if isinstance(out, Failure):
@@ -163,11 +166,10 @@ def markdown_to_adf(markdown: str, run: Run = container_run) -> dict[str, typing
return document
def adfs_to_markdown(documents: list[dict[str, typing.Any]], run: Run = container_run) -> list[str] | Refused | Failure:
"""The markdown for each ADF document, in their order, from one container run, so a caller with many documents pays one container start."""
def to_markdowns(mode: str, documents: list[dict[str, typing.Any]], run: Run) -> list[str] | Refused | Failure:
if not documents:
return []
out = answer("to-markdown", json.dumps(documents), run)
out = answer(mode, json.dumps(documents), run)
if isinstance(out, Refused) and not 0 <= out.index < len(documents):
return Failure(f"the adf-codec shim refused a document at index {out.index}, past the {len(documents)} it was handed")
if isinstance(out, Refused | Failure):
@@ -181,9 +183,37 @@ def adfs_to_markdown(documents: list[dict[str, typing.Any]], run: Run = containe
return texts
def adf_to_markdown(document: dict[str, typing.Any], run: Run = container_run) -> str | Failure:
"""The markdown for the ADF document."""
texts = adfs_to_markdown([document], run)
def one_markdown(texts: list[str] | Refused | Failure) -> str | Failure:
if isinstance(texts, Refused):
return texts.failure
return texts if isinstance(texts, Failure) else texts[0]
def markdown_to_adf(markdown: str, run: Run = container_run) -> dict[str, typing.Any] | Failure:
"""The ADF document for the markdown, by the portable pair."""
return to_adf("to-adf", markdown, run)
def markdown_to_adf_lossless(markdown: str, run: Run = container_run) -> dict[str, typing.Any] | Failure:
"""The ADF document for the markdown, by the lossless pair."""
return to_adf("to-adf-lossless", markdown, run)
def adfs_to_markdown(documents: list[dict[str, typing.Any]], run: Run = container_run) -> list[str] | Refused | Failure:
"""The markdown for each ADF document, in their order, by the portable pair, from one container run, so a caller with many documents pays one container start."""
return to_markdowns("to-markdown", documents, run)
def adfs_to_markdown_lossless(documents: list[dict[str, typing.Any]], run: Run = container_run) -> list[str] | Refused | Failure:
"""The markdown for each ADF document, in their order, by the lossless pair, from one container run."""
return to_markdowns("to-markdown-lossless", documents, run)
def adf_to_markdown(document: dict[str, typing.Any], run: Run = container_run) -> str | Failure:
"""The markdown for the ADF document, by the portable pair."""
return one_markdown(adfs_to_markdown([document], run))
def adf_to_markdown_lossless(document: dict[str, typing.Any], run: Run = container_run) -> str | Failure:
"""The markdown for the ADF document, by the lossless pair."""
return one_markdown(adfs_to_markdown_lossless([document], run))
+12 -9
View File
@@ -1,10 +1,11 @@
// Converts between markdown and ADF with @larvit/adf-codec's portable calls. The mode is the one argument:
// Converts between markdown and ADF with @larvit/adf-codec. The mode is the one argument:
// to-adf stdin is markdown, stdout is the ADF document as JSON
// to-markdown stdin is a JSON array of ADF documents, stdout a JSON array of their markdown texts, in their order
// to-adf-lossless, to-markdown-lossless the same, through adf-codec's lossless pair in place of its portable pair
// Exit 0: stdout holds the answer. Exit 2: adf-codec refused the input, and stdout holds {"error": {code, message, path, position?}},
// with "index", the refused document's place in the array, beside "error" in to-markdown.
// Exit 1: the shim could not run, and stderr says why.
import { adfToPortableMarkdown, portableMarkdownToAdf } from '@larvit/adf-codec'
import { adfToLosslessMarkdown, adfToPortableMarkdown, losslessMarkdownToAdf, portableMarkdownToAdf } from '@larvit/adf-codec'
const REFUSED = 2
@@ -19,17 +20,17 @@ function refuse(refusal) {
return REFUSED
}
function toAdf(markdown) {
const result = portableMarkdownToAdf(markdown)
function toAdf(convert, markdown) {
const result = convert(markdown)
if (!result.ok) return refuse({ error: result.error })
process.stdout.write(JSON.stringify(result.value))
return 0
}
function toMarkdown(documents) {
function toMarkdown(convert, documents) {
const texts = []
for (const [index, document] of documents.entries()) {
const result = adfToPortableMarkdown(document)
const result = convert(document)
if (!result.ok) return refuse({ index, error: result.error })
texts.push(result.value)
}
@@ -39,9 +40,11 @@ function toMarkdown(documents) {
async function main(mode) {
const input = await readStdin()
if (mode === 'to-adf') return toAdf(input)
if (mode === 'to-markdown') return toMarkdown(JSON.parse(input))
throw new Error(`mode must be to-adf or to-markdown, not ${mode}`)
if (mode === 'to-adf') return toAdf(portableMarkdownToAdf, input)
if (mode === 'to-adf-lossless') return toAdf(losslessMarkdownToAdf, input)
if (mode === 'to-markdown') return toMarkdown(adfToPortableMarkdown, JSON.parse(input))
if (mode === 'to-markdown-lossless') return toMarkdown(adfToLosslessMarkdown, JSON.parse(input))
throw new Error(`mode must be to-adf, to-adf-lossless, to-markdown or to-markdown-lossless, not ${mode}`)
}
main(process.argv[2]).then(
+55
View File
@@ -66,6 +66,36 @@ def test_adf_to_markdown_returns_the_text_of_its_one_document():
assert run.calls == [("to-markdown", json.dumps([DOC]))]
def test_markdown_to_adf_lossless_runs_the_lossless_mode():
run = fake(0, json.dumps(DOC))
assert convert.markdown_to_adf_lossless("Hi\n", run) == DOC
assert run.calls == [("to-adf-lossless", "Hi\n")]
def test_adfs_to_markdown_lossless_sends_every_document_in_one_lossless_run():
run = fake(0, json.dumps(["a\n", "b\n"]))
assert convert.adfs_to_markdown_lossless([DOC, DOC], run) == ["a\n", "b\n"]
assert run.calls == [("to-markdown-lossless", json.dumps([DOC, DOC]))]
def test_adf_to_markdown_lossless_returns_the_text_of_its_one_document():
run = fake(0, json.dumps(["a\n"]))
assert convert.adf_to_markdown_lossless(DOC, run) == "a\n"
assert run.calls == [("to-markdown-lossless", json.dumps([DOC]))]
def test_adfs_to_markdown_lossless_names_a_refused_documents_index():
refusal = {"index": 1, "error": {"code": "not-an-adf-document", "message": "no type", "path": []}}
refused = convert.adfs_to_markdown_lossless([DOC, DOC], fake(2, json.dumps(refusal)))
assert refused == convert.Refused(1, Failure("adf-codec refused the input with not-an-adf-document: no type (at the document)"))
def test_adfs_to_markdown_lossless_of_no_documents_runs_no_container():
run = fake(1)
assert convert.adfs_to_markdown_lossless([], run) == []
assert run.calls == []
def test_a_refusal_becomes_a_failure_with_adf_codecs_code_and_message():
refusal = {"error": {"code": "malformed-pipe-table", "message": "row has too many cells", "path": [], "position": {"line": 3, "offset": 9}}}
failure = convert.markdown_to_adf("| a |\n", fake(2, json.dumps(refusal)))
@@ -279,3 +309,28 @@ def test_round_trip_through_the_real_container():
assert isinstance(second, convert.Refused) and second.index == 1 and "not-an-adf-document" in second.failure.reason
crashed = convert.answer("to-markdown", "not json", convert.container_run)
assert isinstance(crashed, Failure) and crashed.reason.startswith("the adf-codec shim failed with exit 1: ")
# A Jira description as Jira stores it: every block carries a localId, and an empty paragraph spaces two blocks.
JIRA_DESCRIPTION = {
"type": "doc", "version": 1, "content": [
{"type": "heading", "attrs": {"level": 2, "localId": "h-1"}, "content": [{"type": "text", "text": "Goal"}]},
{"type": "paragraph", "attrs": {"localId": "p-1"}, "content": [{"type": "text", "text": "Keep "}, {"type": "text", "text": "this", "marks": [{"type": "strong"}]}]},
{"type": "paragraph", "attrs": {"localId": "p-2"}},
{"type": "bulletList", "attrs": {"localId": "l-1"}, "content": [
{"type": "listItem", "attrs": {"localId": "i-1"}, "content": [{"type": "paragraph", "attrs": {"localId": "p-3"}, "content": [{"type": "text", "text": "one"}]}]},
]},
],
}
@pytest.mark.skipif(shutil.which("docker") is None or subprocess.run(["docker", "info"], capture_output=True, timeout=20).returncode != 0, reason="needs a docker daemon")
def test_the_lossless_pair_gives_back_a_jira_description_identical_and_the_portable_pair_does_not():
markdown = convert.adf_to_markdown_lossless(JIRA_DESCRIPTION)
assert isinstance(markdown, str)
assert convert.markdown_to_adf_lossless(markdown) == JIRA_DESCRIPTION
assert convert.adfs_to_markdown_lossless([JIRA_DESCRIPTION, JIRA_DESCRIPTION]) == [markdown, markdown]
portable = convert.markdown_to_adf(convert.adf_to_markdown(JIRA_DESCRIPTION))
assert portable != JIRA_DESCRIPTION
second = convert.adfs_to_markdown_lossless([JIRA_DESCRIPTION, {"type": "nope"}])
assert isinstance(second, convert.Refused) and second.index == 1
+8 -1
View File
@@ -6,12 +6,13 @@
`Bar = 0.1`
`Next ID = 390`
`Next ID = 392`
## Items
| ID | Release | Blocked by | Exempt | Item | L | E | I | R | S | Moves | Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 391 | 0.12.0 | | | **Read and write a Jira epic issue's description and a Confluence record page through the lossless pair, so saving an unedited document leaves it unchanged.** | 3 | 2 | 3 | 6 | 5 | 1.6 by 6, 1.1 by 3 | 1.9 |
| 319 | 0.12.0 | | | **Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.** | 3 | 2 | 3 | 6 | 4 | 1.1 by 2, 1.4 by 2 | 1.2 |
| 309 | 0.12.0 | | | **Run a scoring run's second pass on one head with two seats in place of four, once `larv-review agreement` shows the seats agree with each other more than a run agrees with its rerun.** | 2 | 2 | 2 | 4 | 2 | 1.4 by 3 | 1.2 |
| 299 | 0.12.0 | | | **Replay the archived judge verdicts through code rules and a classifier trained locally on the archive, and file an item for each verdict kind one of them decides with no missed interrupt.** | 2 | 2 | 2 | 4 | 3 | 1.4 by 3 | 1.0 |
@@ -74,6 +75,12 @@
## Details
### 391. Read and write a Jira epic issue's description and a Confluence record page through the lossless pair, so saving an unedited document leaves it unchanged.
`plan_store/jira_epic.py` reads an issue's description and writes it back on an edit, and `record_store/confluence_page.py` reads a page and saves it back; both use the portable pair, which drops each block's `localId` and the empty paragraphs Jira uses for spacing. `record_store/jira_description.py` only reads, to show a person: it stays on the portable pair.
Open before building: the lossless markdown is less readable than the portable one (upstream larvit/adf-codec#185), and agents and the judge read the plan's details from it. Decide how a detail stays readable, for example by reading lossless and showing a person a portable view. Test that an unedited issue and page save back identical. The item ships alone because its R is above 5.
### 319. Start a fresh session with a brief of the chunk when a rung step changes the worker's model, in place of resuming the old session.
On SWE-bench Verified with Haiku 4.5 and Opus 4.7, handing the strong model the weak model's transcript solved 69.2% at $1.61 a task; starting the strong model fresh solved 79.2% at $0.90 (arXiv 2608.24358, The Handoff Tax). A model switch also rereads the whole context uncached (Claude Code prompt caching docs). A worker that steps from Sonnet 5.5 to Opus 5.5 by `--resume` is the raw handoff; an effort-only step within one model keeps the cache and isn't. The escalated subagent, which starts fresh with the trigger's evidence, already follows the pattern that won.