3
0
mirror of https://github.com/larvit/adf-codec.git synced 2026-11-19 14:45:44 +01:00

Gather an anonymized corpus of real-world ADF documents with each one's source and licence

This commit is contained in:
2026-11-04 21:50:40 +01:00
committed by Lillem4n
parent dbaa6823aa
commit 59954f93b3
168 changed files with 25247 additions and 45 deletions
+28
View File
@@ -1182,3 +1182,31 @@ serves goal 3: a developer choosing a converter sees every one, and what it gets
A converter that starts and converts anything stays in its table, however poorly. Its row says where
it fails ordinary input, such as a list or a quote. A converter that doesn't start, or needs a page
rather than Node, goes in the second table. A wrapper or fork of a listed package is left out.
## The real-world corpus holds anonymized documents from sites whose licence it records
2026-11-04, on the maintainer's answers of 2026-11-03. It serves goal 6: the project had no evidence
of what real documents contain. Valid while no public source with a clearer licence serves ADF.
Each document keeps its exact format: every node, mark, attribute key and the shape of each value.
Its text, names, ids and URLs are replaced with fejkdata's draws, and a document that fails the
shape or leak check stays out. `formatKeys` in `corpus-gathering/anonymize.mjs` lists the attributes
kept because they name a format or a layout. The fetched originals are never committed.
A site joins only with a published licence. All six are hosted by the Linux Foundation:
- The ONAP wiki's own pages disagree. A 2017 page says CC-BY-SA-4.0 and a 2019 page says CC-BY-4.0,
so its documents record CC-BY-SA-4.0, which satisfies both.
- Every other document rests on the foundation's terms of use, which put wikis and other
non-code submissions under CC-BY-3.0. They don't name issue trackers or `*.atlassian.net`, so
their reach to these sites' Jira documents and wikis is an inference. A site's own stated licence
replaces it in `corpus-gathering/sites.json`.
The tool stops at 20 result pages of 100 per source, one request a second as the sites' robots.txt
asks, and 150 documents of at most 64 KiB. Those numbers keep a run under two hours and the corpus
near 1 MB. Selection takes new node and mark types first, then the most new features per byte.
22,769 fetched documents hold no `bodiedSyncBlock`, `decisionItem`, `decisionList`,
`extensionFrame`, `fragment`, `multiBodiedExtension` or `syncBlock`, and neither did the sites'
decision-titled pages. `corpus.test.ts` pins that list, so a regenerated corpus holding one of them
has to shrink it.