lilleman
b9167073d4
Mirror / push (push) Has been cancelled
Test / test (18) (pull_request) Successful in 31s
Test / test (22) (pull_request) Successful in 31s
Test / test (26) (pull_request) Successful in 32s
Test / lint (pull_request) Successful in 23s
Test / test (20) (pull_request) Successful in 31s
Test / test (24) (pull_request) Successful in 31s
165 lines
12 KiB
Markdown
165 lines
12 KiB
Markdown
# 07 load
|
|
|
|
Date: 2026-09-06. Repo commit: `ab93833`. Host: AMD Ryzen 9 5950X, 8 vCPUs allotted, 31GB RAM,
|
|
Alpine 6.18.38-0-virt kernel, Docker 29.6.2. Images: `interop-smppload:2.5.3-49fb653` (smppload
|
|
cloned at commit `49fb653`, tag 2.5.3, built with `erlang:27.3.4.17-alpine` + a fresh `rebar3`
|
|
3.27.0 release replacing the commit's own pre-OTP-27 vendored one), `interop-dumbclient:de0334b`
|
|
(vponomarev/libsmpp cloned at commit `de0334b`, built with `golang:1.26.8-alpine3.23` on
|
|
`alpine:3.23.5`), `nicolaka/netshoot:v0.16` (capture sidecar and tshark), `node:24.18.0-bookworm-slim`
|
|
(test runner, from the root `compose.yaml`).
|
|
|
|
## Setup
|
|
|
|
### smppload: builds, binds, then puts a corrupted PDU on the wire - blocked
|
|
|
|
Issue #8 (rebar3/BEAM load errors) is real but resolved in minutes: the commit's own vendored
|
|
`./rebar3` escript predates OTP 27 and fails to load under it
|
|
(`please re-compile this module with an Erlang/OTP 27 compiler`). Replacing it with a fresh rebar3
|
|
3.27.0 release before `make escriptize` fixes the build outright - no further patching needed, and
|
|
`git://` dependency URLs in `rebar.config` resolved fine once rewritten to `https://` (a one-line
|
|
`git config --global url.insteadOf`).
|
|
|
|
The built escript is not usable against any SMSC, though: every `bind_transceiver` it sends is two
|
|
octets short of what its own `command_length` declares. Captured with a raw tshark sidecar against
|
|
`ukarim/smscsim:0.2.0` (independent of both this library and smppload's own logging):
|
|
|
|
```
|
|
002a000000090000000000000001736d7070636c69656e74310070617373776f7264000050010100
|
|
```
|
|
|
|
40 octets on the wire, but `command_length` (the first 4 octets, if the PDU were whole) would need
|
|
to read `0000002a` (42) for a `system_id` "smppclient1" / password "password" bind - the actual
|
|
first two octets of that field are simply missing, so the wire instead starts `002a0000`
|
|
(2,752,512) with `command_id` and everything after shifted two octets early. tshark's own SMPP
|
|
dissector does not recognise the stream as SMPP at all (`-Y smpp` matches zero frames, though the
|
|
raw capture has the SYN/ACK/PSH/FIN sequence and the 40-octet data frame) - a second, independent
|
|
confirmation this is not merely a framing quirk our own codec is stricter about.
|
|
|
|
Traced as far as `oserl`'s `smpp_pdu_syntax:pack/2` (the `trx_deadlock_fix_1` branch smppload's
|
|
`rebar.config` pins), which builds the header as plain 32-bit bit-syntax
|
|
(`<<Len:32, CmdId:32, 0:32, SeqNum:32>>`) - correct on inspection, so the corruption happens
|
|
somewhere between that call and the socket write, not chased further given the time-box. Reproduced
|
|
Re-examined 2026-09-20 to see whether it could be unblocked for the throughput comparison in
|
|
`benchmarks/`. Four things are now established, and one earlier suspicion is ruled out:
|
|
|
|
- **It is one write, not a split one.** A raw listener that accumulates every chunk rather than
|
|
reading the first receives `40 octets across 1 chunks`, byte-identical to the 2026-09-06 capture,
|
|
with `command_length` reading 2,752,512. So the two leading zero octets are absent from the socket
|
|
write itself; nothing about our framing or the capture is involved.
|
|
- **The compiled `pack/2` is correct**, checked in the built tree rather than the repository:
|
|
`Len = size(BodyBin) + 16` written as `<<Len:32, CmdId:32, 0:32, SeqNum:32>>`, returned as the
|
|
iolist `[Header, BodyBin]`. For this bind that is 16 + 26 = 42.
|
|
- **The remaining suspect is `smpp_session.erl:158`**, which writes with `erlang:port_command/2`
|
|
rather than `gen_tcp:send/2` — an undocumented fast path in oserl code that predates OTP 27.
|
|
- **That suspect is untested.** Two attempts to swap it were both invalidated by rebar3 dep caching:
|
|
editing a fetched dependency's source does not rebuild its beam, and the `_checkouts/` route
|
|
re-verifies every dependency, which needs network and git in the build container. Whoever picks
|
|
this up should patch before the first compile, or force the dep to rebuild, and confirm the beam
|
|
actually changed before believing a result.
|
|
|
|
Enough for an upstream report — a reproducer needing no SMSC, the exact octets, and a named
|
|
suspect — but not enough for a patch, since the one-line candidate has never actually run.
|
|
|
|
Recorded as **blocked**; `smppload.test.ts`
|
|
keeps a live reproducer asserting what our server does when it receives it (refuses the stream as
|
|
unframeable - see Scenarios) rather than removing the peer. `smpp-dumb-client` covers S9, and
|
|
substitutes for S6 and (partially) S8 - see below.
|
|
|
|
### smpp-dumb-client: builds and interoperates cleanly
|
|
|
|
No build friction. Two binaries from the same pinned source: `smpp-dumb-client` (unmodified) and
|
|
`smpp-dumb-client-noping` (its two `enquireSender()` call sites in `smpp.go` commented out at build
|
|
time), the second built because every load tool built for this phase sends `enquire_link` on its
|
|
own otherwise (`smpp-dumb-client` every 10s, unconditionally, not configurable) and smppload - the
|
|
one peer that genuinely never does - is blocked, leaving S6 with no peer at all otherwise.
|
|
|
|
One integration snag, not a build one: `smpp.remote` in `config.yml` is fed straight into
|
|
`net.ParseIP` (`hdr.go`) with no DNS resolution at all, so the compose service name cannot appear
|
|
there directly. Fixed in the entrypoint: every `conf/*.yml` carries a `NODE_HOST` placeholder,
|
|
resolved with `getent hosts` and substituted into a writable copy before the real binary starts.
|
|
|
|
The four scenarios share one network namespace, owned by `dumbclient-netns`, a container that
|
|
never exits - they are pure outbound clients with nothing of their own listening, so the only shared
|
|
cost is a source IP, and one capture sidecar sees all four conversations with `node:2775` the same
|
|
way `compose.kannel.yaml`'s does for its four bearerbox variants.
|
|
|
|
Runs 1 and 2 had `dumbclient-w2000` own the namespace. It exits once it has sent its 20,000, which
|
|
took every other client's network with it: the soak's responses stopped arriving, and its log filled
|
|
with `Expired TX packet` lines (`libsmpp`'s 7000ms `TX_MAX_TIMEOUT_MS`). Those runs read that as the
|
|
peer's own window bookkeeping stalling; run 3 (2026-09-26), with the namespace owned by a container
|
|
that outlives them all, reached 173,820 soak messages in 300s where run 2 reached 22,440. The soak
|
|
stays bounded by wall-clock (5 minutes), asserting every arrival answered, nothing duplicated, and
|
|
the memory shape.
|
|
|
|
One test-harness bug found and fixed between the two runs below, not a library defect: the S6 test's
|
|
first version attached its `session.on('close', ...)` listener lazily inside the test body, after
|
|
already waiting on the S9 assertions (which can run for the better part of a minute) - by the time
|
|
the S6 test ran, the idle session had already closed, and an `EventEmitter` never replays a past
|
|
event to a listener added after it fired. Fixed by attaching every session's `close` listener at
|
|
`session`-creation time, recording it in the same per-scenario stats every other assertion reads.
|
|
|
|
Three runs of `./interop-tests/run.py dumbclient`. Run 1 (the original 300,000-count soak) surfaced
|
|
the S6 harness bug above; run 2 fixed it; run 3, with the namespace owner above and the held-message
|
|
throttle in `src/`, is the one Scenarios reports. The capture figures below are run 2's.
|
|
`smppload.test.ts` passed on every run it was given (three, across the investigation above); its one scenario needs no repeat - a second run reproduces the identical corrupted PDU,
|
|
adding nothing.
|
|
|
|
```
|
|
dumbclient run 2: frames 111300, bind_transceiver 4/4, enquire_link 12 (enquire_link_resp 9 - the
|
|
capture stops moments after the test does, catching some requests before their
|
|
response), submit_sm 62441, submit_sm_resp 48830, malformed 0, expert errors 0
|
|
smppload: frames 0 (tshark's own SMPP dissector does not recognise the corrupted stream at all)
|
|
```
|
|
|
|
`submit_sm_resp` reads lower than `submit_sm` in the capture for the same reason
|
|
`enquire_link_resp` does - the sidecar is stopped right after the test file's own `after()` hook
|
|
finishes, which is itself moments after the last response goes out, so a handful of writes land
|
|
after the capture stops seeing them. Not a lost response: `submit_sm` (62441) matches the sum of
|
|
every session's own `arrived` exactly, and every session's own `answered` matches its `arrived` too
|
|
(see Scenarios) - both counted independently, in the server process, of anything on the wire.
|
|
|
|
## Throughput and memory
|
|
|
|
Run 3. `dumb-w500` ran to its full 20,000 in ~44s against a handler serialised to answer roughly
|
|
one message every 2ms (`SLOW_HANDLER_DELAY_MS`), `peakOutstanding` exactly 500. `dumb-w2000`, run
|
|
concurrently, held exactly 1000 and was throttled for the rest.
|
|
|
|
The soak (fast, immediate-response handler; window 100) reached 173,820 `submit_sm` over its fixed
|
|
300s, about 580/s, `peakOutstanding` 15. Sampled every 5s across the whole run (69 samples over
|
|
340s, all four scenarios combined, the harness's own per-message bookkeeping included): rss
|
|
first=165MiB, min=165MiB, max=298MiB, last=298MiB, heapUsed at the last sample 81MiB.
|
|
|
|
## Scenarios (PLAN.md)
|
|
|
|
| Id | Result | Evidence |
|
|
| --- | --- | --- |
|
|
| S6 (idleTimeout, no peer ever pings) | pass | `dumbclient.test.ts` "S6 - idle peer..." - dropped at idleTimeout, `linkTimers - closing an idle peer` logged, no response past the one owed |
|
|
| S8 (throughput, long messages, receipts) | blocked (smppload) / partial substitute | smppload's own scenario is blocked - see Setup. The soak below gives a genuine submit_sm/s figure without long messages or receipts, which `smpp-dumb-client` does not support (`research/esme-clients-and-validators.md` section B) |
|
|
| S9 (bounded window) | pass | Run 3: `dumbclient.test.ts` "S9 - bounded window..." - window 500 20,000/20,000 answered in ~44s, `peakOutstanding` exactly 500; window 2000 5,639 answered and 14,361 throttled, in arrival order, no duplicate ids |
|
|
| Backpressure at the server | pass | Run 3: window 2000 holds exactly 1000 (`maxHeldMessages`, session-options.ts) and the rest is answered `ESME_RTHROTTLED`; smpp-dumb-client counts a throttled message as sent and never resends it; window 500 is never throttled |
|
|
| Long soak | pass | Run 3: `dumbclient.test.ts` "Long soak" - 173,820 arrived, 173,820 answered, 0 duplicates, 0 unanswered errors, `close()` drains with no error; rss 165MiB first, 298MiB max and last, heapUsed 81MiB last |
|
|
| smppload bind corruption (not in PLAN.md - found this phase) | blocked | `smppload.test.ts` - our server refuses the unreadable stream instead of hanging |
|
|
|
|
## Defects in @larvit/smpp
|
|
|
|
None found. `smppload.test.ts`'s own scenario is smppload's defect, not ours: our server's reaction
|
|
(refusing the stream as unframeable, per the decision in the root `AGENTS.md`, "A stream this
|
|
library cannot frame...") is the documented behaviour working exactly as designed against a peer
|
|
that never gets as far as a readable PDU.
|
|
|
|
## Peer quirks
|
|
|
|
- **smppload's `bind_transceiver` is corrupted on the wire** - see Setup. Not chased past `oserl`'s
|
|
`pack/2` (which is correct on inspection) given the time-box.
|
|
- **`smpp-dumb-client` treats `ESME_RTHROTTLED` as final** - a throttled message counts as sent
|
|
and is never resubmitted. Its `enquire_link` interval (10s once bound as an ESME) is hardcoded
|
|
(`smpp.go`, `enquireSender(10)`), not exposed through `config.yml` at all - the no-ping binary
|
|
built for S6 patches the call site out rather than configuring it.
|
|
- **`smpp.remote` takes a literal IP, never a hostname** (`net.ParseIP`, no DNS resolution) - see
|
|
Setup.
|
|
|
|
## Open questions
|
|
|
|
- Whether smppload's bind corruption is in `oserl`'s `gen_esme_session`/`smpp_session` send path
|
|
(not reached, given the time-box) or something specific to this build's dependency versions.
|