Run Jasmin in both directions: message_payload, data_sm and a multipart deadlock confirmed against a production gateway
This commit is contained in:
@@ -0,0 +1,212 @@
|
||||
# 03 jasmin
|
||||
|
||||
Date: 2026-09-06. Repo commit: `db02f00`. Host Docker: 29.6.2. Images: `jookies/jasmin:0.11.0`
|
||||
(both directions), `redis:8.8.2-alpine`, `rabbitmq:3.13.7-management-alpine` (fallback - see
|
||||
below), `python:3.13.7-slim-bookworm` (jcli bootstrap), `nicolaka/netshoot:v0.16` (capture),
|
||||
`node:24.18.0-bookworm-slim` (test runner, from the root `compose.yaml`).
|
||||
|
||||
## Setup
|
||||
|
||||
Two full Jasmin instances (`jasmin`, `jasmin-datasm`), each with its own `redis`/`rabbitmq` pair -
|
||||
kept separate rather than shared, since two unrelated Jasmin instances sharing a broker is the
|
||||
kind of proximity-only coupling the project avoids. `jasmin-datasm` mounts a custom `jasmin.cfg`
|
||||
(`[dlr-thrower] dlr_pdu = data_sm`) for C9/target 4. Both run a `jcli` bootstrap
|
||||
(`interop-tests/peers/jasmin/bootstrap.py`, a plain socket client - no telnet negotiation reply is
|
||||
needed, the server proceeds regardless) that creates a group, two users (`esme1` for most
|
||||
scenarios, `esme2` with a throttled `smpps_throughput` quota for C12), an `smppccm` connector
|
||||
("upstream"/"upstreamds") pointing at our own `node` container, and `mtrouter`/`morouter`
|
||||
`DefaultRoute`s wiring MT to that connector and MO back to `smpps(esme1)`. `interop-tests/jasmin.test.ts`
|
||||
runs one persistent `server()` on `node:2777` as the fake real-world SMSC both connectors bind out
|
||||
to, plus a tiny HTTP listener for Jasmin's DLR-thrower webhook (S7).
|
||||
|
||||
**RabbitMQ 4.3.5 does not work with Jasmin 0.11.0**: its `txamqp` client declares
|
||||
transient/non-exclusive queues, a feature RabbitMQ 4.x refuses by default ("transient_nonexcl_queues
|
||||
... not permitted anymore"), so `RouterPB`/`DLRThrower` fail to start. Fell back to the 3.13 series;
|
||||
`rabbitmq:3.13.7-management-alpine` starts clean.
|
||||
|
||||
Snags fixed while building the harness, roughly in the order found:
|
||||
- **jcli commands after `smppccm -a` need loud failure detection, not keyword-sniffing.** A failed
|
||||
`ok` ("Failed adding connector, check log for details") doesn't contain any of "error"/"unknown"/
|
||||
"invalid" - it left the bootstrap's own session stuck at the `>` sub-prompt, and every later
|
||||
command was misread as a key inside it. Fixed by also checking the last reply lands back on the
|
||||
top-level `jcli :` prompt.
|
||||
- **The connector's password has an 8-character wire maximum.** SMPP's `password` is a C-octet-string
|
||||
with an 8-char + NUL limit; Jasmin's own `smpp.pdu` encoder enforces it strictly when it builds the
|
||||
connector's own bind PDU (our library's encoder is permissive and doesn't). A 10-char password
|
||||
made every single connector bind throw mid-encode (`ValueError: COctetString is longer than
|
||||
allowed maximum size (9)`), logged only in the connector's own per-CID log file
|
||||
(`/var/log/jasmin/default-<cid>.log`), not in the main Jasmin process log.
|
||||
- **`session.userData` is unset when the `session` event fires** (it fires on raw connect, before
|
||||
`authenticate()` runs) - populating a map off it there silently never worked. Fixed like
|
||||
`kannel.test.ts`'s `bindPdus`: read the bind PDU's own `system_id` from `incomingPduObj` instead,
|
||||
and only read `userData` later, from `sms`, once authenticate() has long since run.
|
||||
- **Jasmin's own MT dispatch to one connector is strictly serialized** (see the defect below) -
|
||||
moving `C13`/`S7`'s HTTP-send test ahead of `C3+C7`'s multi-segment sends in file order (both need
|
||||
the same connector's queue to be unstuck) turned three tests that always failed into two that
|
||||
always pass.
|
||||
- The `jasmin`/`jasmin-datasm` healthcheck probes the HTTP API port (1401) with a bare TCP connect,
|
||||
no bytes written - Jasmin's SMPP codec, like SMPPSim's, is not something to probe with an empty
|
||||
PDU. `rabbitmq` uses `rabbitmq-diagnostics -q ping`.
|
||||
- Bootstrap and the whole `docker compose up` sequence are slow and variable on this host - jcli's
|
||||
own connector/router commands (which round-trip through AMQP/redis, unlike the plain in-memory
|
||||
group/user commands) sometimes took most of a minute each under contention; the harness itself
|
||||
budgets for it (`run.py` runs past its own foreground tool timeout and is watched to completion),
|
||||
not something to read as a Jasmin defect.
|
||||
|
||||
Three runs of `./interop-tests/run.py jasmin`, all after the fixes above: all exit non-zero (the
|
||||
4 failing tests below), all `malformed: 0`, `expert errors: 0`. Wire commands, from the last run:
|
||||
|
||||
```
|
||||
bind_transceiver: 14, bind_transmitter: 1, bind_receiver: 6 (+ their _resp)
|
||||
submit_sm: 45, submit_sm_resp: 43
|
||||
deliver_sm: 15, deliver_sm_resp: 1
|
||||
enquire_link: 7, enquire_link_resp: 7
|
||||
unbind: 1, unbind_resp: 1
|
||||
```
|
||||
|
||||
## Scenarios
|
||||
|
||||
| Id | Result | Evidence |
|
||||
| --- | --- | --- |
|
||||
| C1 | pass | `binds transceiver, sees Jasmin's own enquire_link, unbinds clean`; `binds transmitter`; `binds receiver` |
|
||||
| C3+C7 (single-segment) | pass | `single-segment GSM with extension chars`: UUID id, `receipted_message_id`/`message_state` TLVs present, `DELIVERED` |
|
||||
| C3+C7 (multi-segment) | **fail (peer/library interaction, see Defects)** | `2-segment`/`3-segment`/`10-segment GSM`, `2-segment UCS2`: every attempt across 3 runs times out waiting for a receipt |
|
||||
| C8 target 3 (SAR) | pass | `SAR-segmented deliver_sm from the fake upstream`: reassembles into one whole `sms` |
|
||||
| C8 target 3 (UDH) | pass | `UDH-segmented deliver_sm from the fake upstream`: reassembles into one whole `sms` |
|
||||
| C8 target 2 (`message_payload`) | pass (defect confirmed) | `a deliver_sm carrying message_payload instead of short_message`: Jasmin accepts and relays it; our `sms` arrives with an empty message - see Defects |
|
||||
| C9 target 4 (`data_sm` DLR) | pass (defect confirmed) | `a receipt thrown as data_sm is not read as a dlr`: the raw `data_sm` PDU is seen, no `dlr` event ever fires - see Defects |
|
||||
| C11 | pass | `wrong password: one attempt, no retry` (`ESME_RINVPASWD`); `a rebind refused after a live link drops: backs off, never floods` (2-8 attempts, ≥150ms apart) |
|
||||
| C12 | pass | `flooding submits past the quota`: `ESME_RTHROTTLED` from `esme2`'s `smpps_throughput 0.1` quota on some of 15 parallel sends; session stays bound; a later send succeeds once the next slot opens |
|
||||
| C13 | pass | `every send is answered, none lost, order preserved`: 10 distinct ids, arrival order matches send order |
|
||||
| S7 | pass | `a message pushed through /send arrives as submit_sm`: HTTP `/send` → Jasmin → connector → our `server()`; our `sendResp()`+`sendDlr('DELIVERED')` fires Jasmin's own DLR webhook, both the SMSC-ack (`ESME_ROK`) and terminal (`DELIVRD`) callbacks at `dlr-level=3`; long GSM and UCS-2 MO pushes from our server reassemble whole at Jasmin's connector |
|
||||
|
||||
## Defects in @larvit/smpp
|
||||
|
||||
### `message_payload` is never read (target 2) - confirmed against a real peer
|
||||
|
||||
**What happened.** Jasmin's connector accepts a `deliver_sm` with `sm_length` 0 and the text in a
|
||||
`message_payload` TLV from our fake upstream SMSC without complaint, and relays it through
|
||||
`morouter` to our real client bound to Jasmin's `smpps`. The `sms` event fires with the right
|
||||
envelope (`from`/`to`) but `message` is the empty string - the actual text, which only ever existed
|
||||
in `message_payload`, is lost. Matches the target exactly: `incoming-requests.ts` reads
|
||||
`short_message` only.
|
||||
|
||||
**Reproducer.** `interop-tests/jasmin.test.ts`, `C8 (target 2)`: build a `deliver_sm` with
|
||||
`short_message: Buffer.alloc(0)` and `tlvs: { message_payload: { tagValue: Buffer.from(text) } }`,
|
||||
send it over the connector's session, and inspect the `sms` event at a client bound to Jasmin's
|
||||
`smpps` - `sms.message === ''`.
|
||||
|
||||
**Severity.** Medium: silent data loss, not a wire error - the message is fully present on the wire
|
||||
and Jasmin forwards it faithfully; only this library's read of it is incomplete.
|
||||
|
||||
### `sar_*`/UDH segmentation from an upstream SMSC reassembles fine (target 3, MO direction) - not reproduced as a defect here
|
||||
|
||||
C8's SAR and UDH MO pushes both reassembled into one whole `sms` at our real client. This does not
|
||||
confirm target 3 is fixed in general (our own reassembly still keys on UDH only - a SAR-tagged
|
||||
`deliver_sm` from Jasmin's connector would still arrive as an unrelated fragment if Jasmin ever
|
||||
sent SAR MO unprompted), it confirms only that pushing SAR/UDH-tagged `deliver_sm`s ourselves,
|
||||
directly over the connector session, reassembles correctly on the way out through `smpps` - Jasmin
|
||||
does not re-segment or otherwise disturb an already-short single PDU in transit.
|
||||
|
||||
### `data_sm` is refused (target 4) - confirmed against a real peer
|
||||
|
||||
**What happened.** With `jasmin-datasm`'s `[dlr-thrower] dlr_pdu = data_sm`, a receipt requested via
|
||||
`sendSms({ dlr: true, ... })` throws as a `data_sm` PDU on the client's bind, exactly as documented.
|
||||
Our client's `incomingPduObj` sees it arrive; no `dlr` event ever fires. The receipt is lost -
|
||||
`data_sm` falls to the generic unhandled-command path (`ESME_RINVCMDID`), matching the target.
|
||||
|
||||
**Reproducer.** `interop-tests/jasmin.test.ts`, `C9`: bind to `jasmin-datasm`, `sendSms({ dlr: true,
|
||||
... })`, assert an incoming `data_sm` PDU arrives and no `dlr` event ever fires.
|
||||
|
||||
**Severity.** Medium: a real, documented Jasmin configuration (`dlr_pdu = data_sm`) that a receipt
|
||||
depends on silently drops delivery reports, with no error surfaced to the application either.
|
||||
|
||||
### A multi-part MT send deadlocks against Jasmin's serialized per-connector relay - a library/peer interaction, not a wire defect
|
||||
|
||||
**What happened.** A 2-, 3-, 10-segment GSM or UCS-2 message sent through Jasmin's `smpps` (our real
|
||||
client → Jasmin → `mtrouter` → the `upstream` connector → our fake upstream) never gets a receipt,
|
||||
in every one of 3 runs, regardless of segment count or encoding - only the always-unsegmented
|
||||
single-segment case succeeds. Jasmin's own `messages.log` shows why: `SubmitSmPDU[...] request
|
||||
timed out through [cid:upstream], message requeued` after its 120s response timeout, and nothing
|
||||
for the later segments at all (they stay queued behind the stuck one). The mechanism, confirmed by
|
||||
dumping every `submit_sm` this library's `server()` received as Jasmin's fake upstream: segment 1
|
||||
arrives intact (`esm_class: 64`, a correct `05 00 03 <ref> <total> <seq>` UDH, matching exactly what
|
||||
`splitMessage()` itself would produce - Jasmin relays the UDH byte-for-byte) - but segment 2 never
|
||||
arrives. `src/incoming-requests.ts`'s `onMessage()` answers nothing for an incomplete concatenation
|
||||
group (`if (whole) this.emitSms(whole);` - a partial group falls through silently); `sms.sendResp()`
|
||||
only exists once the whole group is in hand, so it answers all segments of a group atomically, in
|
||||
one call, after the last one arrives. Jasmin, on its side, dispatches to a given connector one
|
||||
`submit_sm` at a time, waiting for that one's response before sending the next queued message for
|
||||
the same connector (confirmed indirectly: reordering the test file so `C13`'s 10 *independent*
|
||||
single-segment sends and `S7`'s HTTP-send run *before* any multi-segment send turned both from
|
||||
always-failing into always-passing at sub-second speed, while running them *after* a stuck
|
||||
multi-segment send left them queued behind it for the same reason). Two designs that are each
|
||||
individually reasonable - Jasmin never advances its connector queue past an unanswered request;
|
||||
this library never answers part of a concatenated message - deadlock when composed: Jasmin will not
|
||||
send segment 2 until segment 1 is acked, and this library will not ack segment 1 until segment 2
|
||||
arrives.
|
||||
|
||||
Ruled out before landing on this: RabbitMQ/host CPU contention (real, but the failure is
|
||||
deterministic across three runs regardless of load - not a timing flake); the connector's own
|
||||
`submit_throughput` default of 1 msg/s (set to `0`, no change); the fake upstream's `idleTimeout`
|
||||
dropping the connector mid-flight (widened to 300s, no change - the connector stayed bound
|
||||
throughout, confirmed in its own log).
|
||||
|
||||
**Reproducer.** `interop-tests/jasmin.test.ts`, `C3+C7`, any multi-segment case, run after
|
||||
`waitForUpstreamSession('main')`: `session.sendSms({ dlr: true, message: 'g'.repeat(200), from,
|
||||
to })` against Jasmin's `smpps`, with a `server()` bound as the fake upstream that only calls
|
||||
`sms.sendResp()`/`sendDlr()` once its own `'sms'` event fires. No receipt arrives within Jasmin's
|
||||
own 120s response timeout; `messages.log` on the peer shows the requeue.
|
||||
|
||||
**Severity.** Medium, narrow: needs a real relaying gateway whose own dispatch is serialized
|
||||
per-outbound-link, which is exactly Jasmin's `smppc` connector shape - a directly-connected SMSC
|
||||
(every other peer in this suite) never exhibits it, since it always has a response ready for every
|
||||
segment as they arrive rather than being a relay itself. Worth a decision record either way (answer
|
||||
each segment as it's held, not only once the group completes; or document that a `server()` sitting
|
||||
behind a serializing relay needs its own segment-level ack), but not fixed here per the phase rules.
|
||||
|
||||
## Peer quirks
|
||||
|
||||
- **Jasmin's own `enquireLinkTimerSecs` (30, `[smpp-server]` default) is an idle timer, not a strict
|
||||
period.** Our client's own default 20s keepalive counts as activity and resets it, so Jasmin's own
|
||||
probe never gets a chance to fire on a link that's never quiet for 30s. `C1` disables our own
|
||||
keepalive (`enquireLinkInterval: 0`, widening `idleTimeout` to compensate) to observe it.
|
||||
- **Message ids are UUIDs, self-consistent end to end when the upstream hands out one id per PDU.**
|
||||
Every id the real client's `submit_sm_resp` carried matched exactly what the fake upstream's
|
||||
`sendResp()` assigned; the later receipt named the same id. Since the fake upstream here is this
|
||||
library's own `server()`, a multi-segment message's ids came back in this library's own
|
||||
`<base>-<n>` shape - an artifact of the test rig (our own server reassembling before answering),
|
||||
not evidence that Jasmin itself produces that convention; a real upstream SMSC would very likely
|
||||
hand back unrelated ids per segment, as the research notes expected.
|
||||
- **`smpps_throughput`, not the connector's `submit_throughput`, gates an ESME's own submission
|
||||
rate.** The plan named `submit_throughput` "on the connector" for C12; that setting throttles the
|
||||
connector's own *outbound* rate to its upstream. The knob that actually throttles submissions
|
||||
*into* Jasmin's `smpps` from a bound ESME is a **user**-level quota
|
||||
(`user -u <uid> mt_messaging_cred quota smpps_throughput <n>`), which does answer `ESME_RTHROTTLED`
|
||||
reliably once exceeded.
|
||||
- **`smppccm`'s `cid` must be 3-25 chars, `morouter`'s `smpps(<system_id>)` target is validated by
|
||||
the same regex** (`[A-Za-z0-9_-]{3,25}`) - confirmed from `jasmin/protocols/cli/morouterm.py`.
|
||||
- **`[dlr-thrower] dlr_pdu`** is a config-file setting (`deliver_sm` default, `data_sm` the other
|
||||
option), not a per-connector or per-user key - the only way to test both is two Jasmin instances.
|
||||
- **HTTP DLR callbacks carry fixed query args** (`id`, `level`, `message_status`, `connector`, plus
|
||||
`id_smsc`/`sub`/`dlvrd`/`subdate`/`donedate`/`err`/`text` at `dlr-level` 2/3) appended by
|
||||
`DLRThrower` itself to whatever bare URL `dlr-url` names - unlike Kannel's `%d`/`%F`
|
||||
printf-style placeholders, nothing is written into the URL by the caller. `dlr-level=1` fires once,
|
||||
immediately, with `message_status=ESME_ROK` (an SMSC-ack, not a terminal state); `dlr-level=3`
|
||||
additionally fires once more, later, with the real terminal status (`DELIVRD` here).
|
||||
- **jcli is a plain-text protocol dressed as Telnet** - it sends real `IAC`/option-negotiation bytes
|
||||
and a couple of ANSI escapes in its banner, but never waits for or requires a reply to them; a raw
|
||||
socket client that ignores negotiation entirely and just reads/writes lines works throughout.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Whether Jasmin's own MT dispatch is serialized *per connector* specifically, or *globally* across
|
||||
every connector on the instance - only one connector was configured, so the two are
|
||||
indistinguishable here.
|
||||
- Whether the deadlock above is specific to a message requesting a receipt (`registered_delivery`
|
||||
set) or would also occur for a plain multi-segment send with no `dlr` - not isolated separately,
|
||||
since every `C3+C7` case here requests one.
|
||||
- Whether Jasmin, given an *upstream* connector that itself defaults to SAR (rather than our
|
||||
library's own UDH), would relay an MT message using SAR instead of preserving our UDH bytes - the
|
||||
120s-timeout deadlock always intervened before a second segment could be observed on the wire in
|
||||
either direction.
|
||||
Reference in New Issue
Block a user