102 lines
4.9 KiB
Markdown
102 lines
4.9 KiB
Markdown
# interop-tests
|
|
|
|
How the experiments in [README.md](README.md) are run and recorded. Read README.md first, and the
|
|
root [AGENTS.md](../AGENTS.md) for the conventions all code here follows.
|
|
|
|
## Layout
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| `compose.<peer>.yaml` | Overlay on the root `compose.yaml`: the peer's services, a `capture` sidecar in the peer's network namespace, and `node` given `depends_on` the peer |
|
|
| `<peer>.test.ts` | `node:test` file driving our library against the peer, in the style of `test/`. Reads `PEER_HOST`/`PEER_PORT` from env, defaulting to the compose service name and 2775 |
|
|
| `peers/<peer>/` | Dockerfile and config for a peer without a published image, or one that needs config files |
|
|
| `captures/` | pcapng and tshark JSON from a run. Gitignored |
|
|
| `findings/<NN>-<peer>.md` | What a phase found. Committed after every phase |
|
|
| `run.py` | Brings a peer up, runs its tests, stops the capture, tears down, analyses the capture |
|
|
|
|
## Running
|
|
|
|
```bash
|
|
./interop-tests/run.py <peer> # one peer, all its tests
|
|
./interop-tests/run.py <peer> --keep # leave the peer up for a manual look
|
|
```
|
|
|
|
`run.py` is the one spelling; it wraps
|
|
|
|
```bash
|
|
docker compose -f compose.yaml -f interop-tests/compose.<peer>.yaml run --rm --use-aliases node node --test interop-tests/<peer>.test.ts
|
|
docker compose -f compose.yaml -f interop-tests/compose.<peer>.yaml stop capture
|
|
docker compose -f compose.yaml -f interop-tests/compose.<peer>.yaml down -v
|
|
```
|
|
|
|
and then decodes `captures/<peer>.pcapng` with tshark (`-d tcp.port==<port>,smpp -Y smpp -T json`),
|
|
printing the command histogram and the counts of `_ws.malformed` and error-severity `_ws.expert`.
|
|
Both counts must be zero for a phase to pass, and an empty capture, or one carrying no bind and its
|
|
response, fails too.
|
|
|
|
A peer whose scenarios send malformed PDUs on purpose — jsmpp does, to prove they are refused — has
|
|
no way to say so, so its run exits non-zero every time and its findings file carries the count that
|
|
is expected. Give the runner an expected count per peer, and a deviation from it becomes the signal
|
|
that a bare threshold cannot be: today a third malformed frame appearing beside jsmpp's two
|
|
deliberate ones looks exactly like the two.
|
|
|
|
## Rules for an experiment
|
|
|
|
1. Peers run in Docker with full patch-version pins. Nothing is installed on the host. Node runs
|
|
only through the `node` service. Every peer service caps its logs (`logging: json-file`,
|
|
`max-size`/`max-file`) and healthchecks something the peer does not log a stack trace for —
|
|
SMPPSim once filled the whole host disk in twenty minutes from a TCP-probe healthcheck.
|
|
2. `src/` and `test/` are read-only during an experiment. A defect is recorded with a reproducer,
|
|
never fixed here — a fix is a separate change with a regression test in `test/`.
|
|
3. No git command that changes state: no add, commit, push, checkout, stash, reset. The
|
|
orchestrator commits after each phase.
|
|
4. A peer that will not come up is time-boxed: after about an hour of trying, record `blocked` with
|
|
everything tried, and stop.
|
|
5. `down -v` at the end of every run. Locally built images are kept, and their tag goes in the
|
|
findings.
|
|
`run.py` runs in the foreground with a long timeout; an agent that backgrounds it is never woken
|
|
when it ends.
|
|
6. Scratch files live outside the repo, in the directory the orchestrator names.
|
|
7. A finding says what happened, what the spec or the peer's docs say, and how to reproduce it.
|
|
Wording is neutral: a mismatch is a mismatch until a reader decides whose it is.
|
|
|
|
## Findings file
|
|
|
|
```markdown
|
|
# <NN> <peer>
|
|
|
|
Date, images and tags, the commit of this repo, host Docker version.
|
|
|
|
## Setup
|
|
Commands that worked, and what did not, so the next run starts where this one ended.
|
|
|
|
## Scenarios
|
|
| Id (from README.md) | Result (pass / fail / blocked / not run) | Evidence (test name, log line, tshark frame) |
|
|
|
|
## Defects in @larvit/smpp
|
|
One subsection each: what happened, what the spec or the peer's docs say, reproducer (PDU hex or
|
|
test), severity.
|
|
|
|
## Peer quirks
|
|
Behaviour of the peer worth knowing that is not our defect.
|
|
|
|
## Open questions
|
|
```
|
|
|
|
Then add the file to README.md's findings table.
|
|
|
|
## Fixing what a phase found
|
|
|
|
Every defect a phase records is fixed before the next phase runs — a fix can change behaviour in
|
|
ways the next experiment must see. One fix per defect class, as its own change:
|
|
|
|
1. A worktree on a branch off `origin/main` (never `origin/v0.4.0`, the 0.4.0 code), named
|
|
for the defect.
|
|
2. Regression tests in `test/` first, naming the behaviour with the reproducer from the findings;
|
|
then the implementation; then the decision record in the root `AGENTS.md` where the fix settles
|
|
a question of the wire or the session's life.
|
|
3. `/larv-review` on the branch, with the pull request based on `main`. When it marks the PR
|
|
ready, fast-forward it.
|
|
4. Back in the experiments worktree: fast-forward `main`, rerun the experiment that found the
|
|
defect, delete the workaround its test carried, and note the fix in the findings file.
|