Set the conventions the interoperability experiments run and record under

2026-09-05 17:53:33 +02:00
parent 8a565cf557
commit 7d855cfa5d
1 changed files with 75 additions and 0 deletions
@@ -0,0 +1,75 @@
+# interop-tests
+
+How the experiments in [PLAN.md](PLAN.md) are run and recorded. Read PLAN.md first, and the root
+[AGENTS.md](../AGENTS.md) for the conventions all code here follows.
+
+## Layout
+
+| | |
+| --- | --- |
+| `compose.<peer>.yaml` | Overlay on the root `compose.yaml`: the peer's services, a `capture` sidecar in the peer's network namespace, and `node` given `depends_on` the peer |
+| `<peer>.test.ts` | `node:test` file driving our library against the peer, in the style of `test/`. Reads `PEER_HOST`/`PEER_PORT` from env, defaulting to the compose service name and 2775 |
+| `peers/<peer>/` | Dockerfile and config for a peer without a published image, or one that needs config files |
+| `captures/` | pcapng and tshark JSON from a run. Gitignored |
+| `findings/<NN>-<peer>.md` | What a phase found. Committed after every phase |
+| `run.py` | Brings a peer up, runs its tests, stops the capture, tears down, analyses the capture |
+
+## Running
+
+```bash
+./interop-tests/run.py <peer>            # one peer, all its tests
+./interop-tests/run.py <peer> --keep     # leave the peer up for a manual look
+```
+
+`run.py` is the one spelling; it wraps
+
+```bash
+docker compose -f compose.yaml -f interop-tests/compose.<peer>.yaml run --rm --use-aliases node node --test interop-tests/<peer>.test.ts
+docker compose -f compose.yaml -f interop-tests/compose.<peer>.yaml stop capture
+docker compose -f compose.yaml -f interop-tests/compose.<peer>.yaml down -v
+```
+
+and then decodes `captures/<peer>.pcapng` with tshark (`-d tcp.port==<port>,smpp -Y smpp -T json`),
+printing the command histogram and the counts of `_ws.malformed` and error-severity `_ws.expert`.
+Both counts must be zero for a phase to pass.
+
+## Rules for an experiment
+
+1. Peers run in Docker with full patch-version pins. Nothing is installed on the host. Node runs
+   only through the `node` service.
+2. `src/` and `test/` are read-only during an experiment. A defect is recorded with a reproducer,
+   never fixed here — a fix is a separate change with a regression test in `test/`.
+3. No git command that changes state: no add, commit, push, checkout, stash, reset. The
+   orchestrator commits after each phase.
+4. A peer that will not come up is time-boxed: after about an hour of trying, record `blocked` with
+   everything tried, and stop.
+5. `down -v` at the end of every run. Locally built images are kept, and their tag goes in the
+   findings.
+6. Scratch files live outside the repo, in the directory the orchestrator names.
+7. A finding says what happened, what the spec or the peer's docs say, and how to reproduce it.
+   Wording is neutral: a mismatch is a mismatch until a reader decides whose it is.
+
+## Findings file
+
+```markdown
+# <NN> <peer>
+
+Date, images and tags, the commit of this repo, host Docker version.
+
+## Setup
+Commands that worked, and what did not, so the next run starts where this one ended.
+
+## Scenarios
+| Id (from PLAN.md) | Result (pass / fail / blocked / not run) | Evidence (test name, log line, tshark frame) |
+
+## Defects in @larvit/smpp
+One subsection each: what happened, what the spec or the peer's docs say, reproducer (PDU hex or
+test), severity.
+
+## Peer quirks
+Behaviour of the peer worth knowing that is not our defect.
+
+## Open questions
+```
+
+Then set the phase's row in PLAN.md's Status table to `done` or `blocked`, linking the file.