Measure throughput, make it goal 6, and gate a release on it #14

Merged
lilleman merged 2 commits from throughput-benchmark into main 2026-09-20 23:39:55 +02:00
8 changed files with 305 additions and 21 deletions
+15 -4
View File
@@ -660,23 +660,29 @@ one wins. They do not override the hard rules below.
about a message the application sent reaches it as a report rather than as an inbound message, and
says whether it is final, so nothing has to read the PDU to tell those apart. An option retunes a
default or opts out of it; an option does not switch on the thing the caller obviously wanted.
6. **Configurable and extendable, never at the defaults' expense.** Where an application needs other
6. **Fast enough that this library is never the bottleneck.** Throughput over a few sessions rather
than many idle ones, which is why this exists on Node at all. On four cores or more, one bound
session sustains at least 5,000 `submit_sm`/s at a send window of 1, 20,000 at the default window
of 10, and 30,000 at 50 or above, and asking for delivery receipts costs nothing measurable.
Memory is bounded per session, never per process. [benchmarks/](benchmarks/README.md) is how those
floors are checked; every release re-measures them and asks what it would take to go faster.
7. **Configurable and extendable, never at the defaults' expense.** Where an application needs other
than the default and cannot build it from what is exported — a rate limit counted per PDU, an
alphabet, a receipt format — it gets an option or a hook rather than a fork. A call that passes no
options stays exactly as easy and as safe, and a hook is a seam the library calls, never a way into
its internals.
7. **A small, stable public surface over reshapeable internals.** Only what `src/index.ts` exports is
8. **A small, stable public surface over reshapeable internals.** Only what `src/index.ts` exports is
published. A new option has to beat "the application can do this itself", and has to keep a
promise this library can verify. The low-level surface is a passthrough: policy binds what the
library composes, never what the caller wrote.
8. **State wider than one session goes through one store.** A pool of sessions, a limit shared
9. **State wider than one session goes through one store.** A pool of sessions, a limit shared
between processes, and what has to survive a restart — receipts still awaited, a message half
reassembled — are held through a store interface and never beside it. Without a store the
application supplies, that state is in memory and ends with the process, and the defaults need
none. The interface carries the library's own versioned records, never an internal shape handed
to the application to persist. Coordinating processes any other way is declined without a fresh
argument each time.
9. **It builds, tests and runs the same everywhere.** Container-only toolchain, no runtime
10. **It builds, tests and runs the same everywhere.** Container-only toolchain, no runtime
dependencies, the Node 18 floor verified in CI rather than asserted, every README example executed
by the suite.
@@ -693,6 +699,11 @@ Who depends on this library, and what they may rely on.
they do in practice outranks what the specification says they should do.
- **The SMSC operator on the far end**, who never sees this API but carries what it does to their
link. A peer they have to complain about is a defect however well the library reads.
- **The developer building the SMPP edge of something else.** What binds to `server()` in practice is
an aggregator's customer-facing edge, a bridge putting SMPP in front of a modern transport, or a
test double standing in for an SMSC. This is not a store-and-forward SMSC and will not become one:
spooling, scheduling, retry policy and billing belong to whatever this is the edge of, and state
shared between instances goes through the store in goal 9.
- **Pre-1.0, so the minor is the breaking unit** and a patch never breaks. What a 0.4.0 consumer has
to change is in [MIGRATION.md](https://gitea.larvit.se/larvit/smpp-js/src/branch/main/MIGRATION.md).
+73
View File
@@ -0,0 +1,73 @@
# Benchmarks
What this library sustains, and against what. Run them before setting or changing a scale goal.
```bash
docker compose run --rm node node benchmarks/run.ts # this library, both ends
COUNT=100000 docker compose run --rm node node benchmarks/run.ts # longer, steadier
```
`run.ts` spawns `smsc-sink.ts` — a server that answers every `submit_sm` `ESME_ROK` and stores
nothing — and drives it with `submit-load.ts` at four window sizes. Point `submit-load.ts` at any
SMSC to compare:
```bash
docker compose -f compose.yaml -f interop-tests/compose.jasmin.yaml run --rm node \
node benchmarks/submit-load.ts --host=jasmin --port=2775 --username=esme1 --password=esme1pw \
--count=20000 --concurrency=50
```
`GATE=1` fails the run when a window falls under goal 6's floor, and refuses to judge at all on
fewer than four cores.
**A short run measures the JIT, not the library.** At 2,000 messages per point the same build
reported 16k/s where 100,000 messages reported 40k/s. Give each point several seconds.
**Cores matter, though the library is single-threaded.** The run is two Node processes, and past
them V8 marks and compiles on threads of its own while the kernel carries loopback TCP. Window 200,
100,000 messages:
| Cores | msgs/s |
| --- | --- |
| 1 | 20,476 |
| 2 | 29,603 |
| 4 | 32,869 |
| 8 | 40,046 |
A window of 1 is unmoved by any of it (7,280–8,659 throughout) because it waits on the round trip
rather than the CPU. The windowed figures are for the pair: one process alone reaches about half.
## Results, 2026-09-20
Single host, 8 cores, Node 24.18.0 in the project container, loopback. Client and SMSC are separate
processes competing for the same CPUs, so these are a floor for split hosts.
`maxOutstanding` is the send window, and it is the setting that matters most:
| Window | msgs/s | with `dlr: true` |
| --- | --- | --- |
| 1 | 7,385 | 7,289 |
| 10 (default) | 25,358 | 24,282 |
| 50 | 37,125 | 37,502 |
| 200 | 40,046 | 39,730 |
Requesting delivery receipts costs nothing measurable at any window. A window of 1 — one request in
flight at a time — costs 5x, which is the round trip rather than the codec.
Against real peers, same driver, window 50, 20,000 messages:
| SMSC | msgs/s | failed |
| --- | --- | --- |
| this library's sink | 37,125 | 0 |
| Jasmin 0.10 | 2,207 | 0 |
| SMPPSim 3.0.0 | — | 19,000 of 20,000 |
Jasmin routes and persists where the sink does neither, so the gap is not an efficiency ratio
between two comparable things — what it establishes is that this library is not the bottleneck
against a production SMSC, by more than an order of magnitude. SMPPSim's store fills at roughly a
thousand messages and it then refuses the rest, so it cannot be loaded; that is a property of the
simulator, not a result.
`smppload`, the one purpose-built SMPP load generator among the peers, is blocked by a bind defect
of its own ([findings/07-load.md](../interop-tests/findings/07-load.md)), so no third-party load
tool drives these numbers.
+95
View File
@@ -0,0 +1,95 @@
import { availableParallelism } from 'node:os';
import { spawn } from 'node:child_process';
import { once } from 'node:events';
import { createInterface } from 'node:readline';
/** Goal 6's floors, per send window. Set GATE=1 to fail the run instead of only reporting. */
const floors: Record<number, number> = { 1: 5000, 10: 20_000, 50: 30_000, 200: 30_000 };
/** Below this, client and sink contend for one core and the windowed floors are unreachable. */
const gateCores = 4;
/**
* Node rather than Python, unlike the repo's other standalone scripts: it spawns the two processes
* it measures, and they must run on the same runtime this library is measured under.
*/
const concurrencies = [1, 10, 50, 200];
const count = Number(process.env.COUNT ?? 5000);
function node(script: string, args: string[] = []) {
return spawn(process.execPath, [`${import.meta.dirname}/${script}`, ...args], {
stdio: ['ignore', 'pipe', 'inherit'],
});
}
async function firstLine(stream: NodeJS.ReadableStream): Promise<string> {
for await (const line of createInterface({ input: stream })) {
return line;
}
return '';
}
const sink = node('smsc-sink.ts');
const listening: unknown = JSON.parse(await firstLine(sink.stdout));
if (typeof listening !== 'object' || listening === null || !('port' in listening)) {
throw new Error('the sink did not report a port');
}
const port = String(listening.port);
const rows: Record<string, unknown>[] = [];
for (const dlr of [false, true]) {
for (const concurrency of concurrencies) {
const load = node('submit-load.ts', [
`--port=${port}`,
`--count=${String(count)}`,
`--concurrency=${String(concurrency)}`,
...(dlr ? ['--dlr'] : []),
]);
const reported: unknown = JSON.parse(await firstLine(load.stdout));
await once(load, 'exit');
if (typeof reported === 'object' && reported !== null) rows.push({ ...reported });
}
}
sink.kill('SIGTERM');
await once(sink, 'exit');
process.stdout.write(`\n${'dlr'.padEnd(6)}${'window'.padEnd(9)}${'msgs/s'.padEnd(10)}seconds\n`);
for (const row of rows) {
const dlr = String(row.dlr).padEnd(6);
const window = String(row.concurrency).padEnd(9);
const rate = String(row.perSecond).padEnd(10);
process.stdout.write(`${dlr}${window}${rate}${String(row.seconds)}\n`);
}
const cores = availableParallelism();
process.stdout.write(`\n${String(cores)} cores\n`);
if (process.env.GATE !== '1') process.exit(0);
if (cores < gateCores) {
process.stdout.write(`refusing to gate on ${String(cores)} cores; goal 6 states four or more\n`);
process.exit(1);
}
const short = rows.filter(row => {
const floor = floors[Number(row.concurrency)];
return floor !== undefined && Number(row.perSecond) < floor;
});
for (const row of short) {
const floor = String(floors[Number(row.concurrency)]);
process.stdout.write(`below goal 6: window ${String(row.concurrency)} ran ${String(row.perSecond)}/s, floor is ${floor}\n`);
}
process.exit(short.length === 0 ? 0 : 1);
+33
View File
@@ -0,0 +1,33 @@
import { server } from '../src/server.ts';
/**
* Answers every submit_sm ESME_ROK and does nothing else, so a measurement against it reads this
* library's own ceiling rather than an SMSC's storage. Prints the bound port on stdout, then waits.
*/
const port = Number(process.env.PORT ?? 0);
const { err, server: smpp } = await server({ port });
if (err) {
process.stderr.write(`sink failed to listen: ${err.message}\n`);
process.exit(1);
}
let answered = 0;
smpp.on('session', session => {
session.on('sms', async sms => {
answered++;
await sms.sendResp();
});
});
smpp.on('serverError', reason => {
process.stderr.write(`sink serverError: ${reason.message}\n`);
});
process.stdout.write(`${JSON.stringify({ port: smpp.port })}\n`);
process.on('SIGTERM', () => {
process.stderr.write(`sink answered ${String(answered)}\n`);
void smpp.close({ signal: AbortSignal.abort() }).then(() => process.exit(0));
});
+72
View File
@@ -0,0 +1,72 @@
import { client } from '../src/client.ts';
/**
* Pushes `count` single-segment messages and reports what the wire carried per second. Keeps
* `concurrency` sends in flight so the send window, not the caller, is what bounds the rate.
*/
function arg(name: string, fallback: string): string {
const found = process.argv.find(one => one.startsWith(`--${name}=`));
return found === undefined ? fallback : found.slice(name.length + 3);
}
const host = arg('host', '127.0.0.1');
const port = Number(arg('port', '2775'));
const count = Number(arg('count', '2000'));
const concurrency = Number(arg('concurrency', '20'));
const message = arg('message', 'benchmark');
const dlr = process.argv.includes('--dlr');
const username = arg('username', 'user');
const password = arg('password', 'pass');
const { err, session } = await client({
host,
maxOutstanding: concurrency,
password,
port,
responseTimeout: 60_000,
username,
});
if (err) {
process.stdout.write(`${JSON.stringify({ error: err.message })}\n`);
process.exit(1);
}
// Narrowing from the guard above does not reach into worker(), which runs after it.
const bound = session;
let issued = 0;
let failed = 0;
let unanswered = 0;
async function worker(): Promise<void> {
while (issued < count) {
issued++;
const sent = await bound.sendSms({ dlr, from: 'BENCH', message, to: '46709771337' });
if (sent.err) failed++;
unanswered += sent.unanswered;
}
}
const started = process.hrtime.bigint();
await Promise.all(Array.from({ length: concurrency }, () => worker()));
const seconds = Number(process.hrtime.bigint() - started) / 1e9;
process.stdout.write(`${JSON.stringify({
concurrency,
count,
dlr,
failed,
perSecond: Math.round(count / seconds),
seconds: Number(seconds.toFixed(3)),
unanswered,
})}\n`);
await bound.close({ signal: AbortSignal.abort() });
process.exit(0);
+10 -10
View File
@@ -36,7 +36,7 @@ rule and an index of the titles below.
- **`PduRefusedError` is exported, and `sessionError` names it in the event's type.** Maintainer's
call, 2026-09-05, from a product review: one event carries both a PDU the peer malformed and the
session's own failure, and `instanceof` is the only way to separate them that hard rule 4 allows —
without the class as a value an application is left string-matching `err.message`. Goal 7 is paid by
without the class as a value an application is left string-matching `err.message`. Goal 8 is paid by
exporting the discriminant and the struct it carries and nothing else: `PduHeader` is named because
an application that logs or forwards a header wants a name for it, `PduRefusalReason` is not
because `reason` is compared against string literals, and an accessor
@@ -63,7 +63,7 @@ rule and an index of the titles below.
runtime: `Object.hasOwn(encodings, x)` is the only test the published surface offered and it
narrows nothing, so `isEncodingName()` is exported beside `isCommandName()` and `isErrorName()`,
which serve their own tables that way. Rejected: a `Result` signature on all three, which costs
every typed consumer a narrow forever — goal 7, and the tag is the last cheap chance to spend it —
every typed consumer a narrow forever — goal 8, and the tag is the last cheap chance to spend it —
to guard a state the compiler refuses. Where the domain really is open the check is already there:
`sendSms()` takes its options as `unknown` and refuses `encoding` by name, which is what a caller
without types gets. `smppTime.encode()` is where that reasoning lands the other way and is recorded
@@ -168,13 +168,13 @@ rule and an index of the titles below.
the suite ran took the 0x40 this library sends on a concatenated segment, but Route Mobile and
Kaleyra both document `esm_class` 0x43 for one, and a caller facing either had to hand-build every
segment through `send()` — giving up the split, the per-segment ids, the send window and the
receipt merge, which is what goal 7 means by beating "the application can do this itself". The four
receipt merge, which is what goal 8 means by beating "the application can do this itself". The four
modes of SMPP 3.4 5.2.12 are a `MESSAGING_MODE` constant group and the option takes one of their
names, so 0x43 is a composition this library makes rather than a value a caller states, and the UDH
indicator a segment carrying a header needs cannot be cleared by anything the option can express.
It takes three of those four: 2.10.3 carries transaction mode on `data_sm` alone, and none goes out
of here, so `FORWARD` stays in the group that mirrors the spec table and `sendSms()` refuses it by
that reason rather than as an unknown name — a mode this library cannot deliver is a promise goal 7
that reason rather than as an unknown name — a mode this library cannot deliver is a promise goal 8
will not let it make. `DATAGRAM` with `dlr: true` is refused on the same footing: 2.10.2 defines the
report away, so arming `DlrMerger` for one is goal 2's wrong answer, where the mode alone and a
report under any other mode both go out untouched. Those three names left `ESM_CLASS`, where they
@@ -231,7 +231,7 @@ rule and an index of the titles below.
`encodingByDataCoding()` reads the alphabet off that same test rather than repeating the group
masks beside it. It is exported for the reason `concatOf()` is — an application that needs a class
other than 0 would otherwise rewrite the read this fixed. Rejected: a `messageClass` field on the
`sms` event, which pays goal 7 for three classes nothing here acts on, where the boolean the
`sms` event, which pays goal 8 for three classes nothing here acts on, where the boolean the
application already had covers the one it does. Compressed text is out of scope and stays out —
nothing here implements 3GPP TS 23.042, so a compressed body reaches the application as whatever
its declared alphabet makes of it — but bit 5 does not move the class bits, so 0x30 is read as
@@ -413,14 +413,14 @@ rule and an index of the titles below.
returned `42 44 46` — `"BDF"` — reported as built, while a string `message_payload` was cut to its
low octets whatever `data_coding` said. Goal 2 owns it, as it owns the `sendSms()` guard above.
The line falls at the string: a `Buffer` is octets the caller already chose and goes out as given
under any `data_coding`, which is what keeps goal 7's escape hatch open — the raw UDH, 8-bit binary
under any `data_coding`, which is what keeps goal 8's escape hatch open — the raw UDH, 8-bit binary
and deliberately malformed bodies `interop-tests/` builds are all still buildable — and a string
with no `data_coding` is untouched, detection carrying every character it was picked for. The
guard is `unencodable()` again rather than a second reading, and `unencodableText()` is the
character, its code point and its index said once for both refusals — unexported where
`unencodable()` is published, since wording `{ char, index }` into a sentence rewrites no read a
caller would get wrong, where asking the codec is, and publishing it would freeze this library's
error prose as API for an application whose own refusal should read like itself. Goal 7, from the
error prose as API for an application whose own refusal should read like itself. Goal 8, from the
architecture review of [#99](https://github.com/larvit/larvitsmpp/pull/99), 2026-09-09. It is
reached through
`encodeBody()` in `message.ts`, which is where the `data_coding`-to-text pair already lives:
@@ -595,7 +595,7 @@ rule and an index of the titles below.
dropped with the link — is traffic the peer will not send again, so each one reaches `sessionError`
as well as the log. Rejected there: an exported `MessageLostError` carrying the group, on the
`PduRefusedError` pattern — no `sms` ever fired for that group, so there is nothing in it the
application could act on, and goal 7 does not buy a second exported class to make a count
application could act on, and goal 8 does not buy a second exported class to make a count
distinguishable. Accepted: a completing segment whose own answer the socket would not carry still
reaches the application, because the message is whole and correct and the failed answer is on
`sessionError` — a peer that re-sends after the drop is the smaller risk than dropping a message
@@ -607,7 +607,7 @@ rule and an index of the titles below.
of the multipart change: `server()` filled the session's only `onRequest` slot, so the escape hatch
the error above names was reachable only by hand-wiring a `Session` over a raw socket, giving up
bind acceptance, `authenticate`, the session set and the drain `close()` runs over it — which is
what goal 7 means by beating "the application can do this itself". What the library verifies is the
what goal 8 means by beating "the application can do this itself". What the library verifies is the
ordering rather than the hook's honesty about answering: the hook is consulted only for a non-bind
request on a session already bound, so no bind — a second one on a live session included — and
nothing a peer sends before one can be intercepted however the hook is written. One
@@ -788,5 +788,5 @@ rule and an index of the titles below.
call, 2026-09-14. Nothing verifies either platform, so the code avoids what is known to differ there:
shelling out, a path joined by hand, a signal Windows does not deliver, a Unix socket or a file mode.
That binds what `dist/` runs; the container tooling, `interop-tests/` and the `package.json` scripts
run on Linux by goal 9. Rejected: macOS and Windows runners, on GitHub's mirror or as Gitea
run on Linux by goal 10. Rejected: macOS and Windows runners, on GitHub's mirror or as Gitea
host-mode runners on a Windows VM and a Mac.
+6 -6
View File
@@ -236,7 +236,7 @@ and is also what the panel ranked hardest — two methods, one answer.
reassembly dispatch, `onRequest` ordering and bind-direction refusal have no unit test because
the class cannot be built without a live socket. Carry the eight members as `IncomingDeps`,
exactly as `sendPastDrain` is carried now. No public surface changes. **Do this before the
store (goal 8), or the back-edge is baked into the store's published interface.**
store (goal 9), or the back-edge is baked into the store's published interface.**
- [ ] **Route `sms.ts` through its handlers, all of it.** `createSms()` already injects
`handlers.send`, and then reaches `sms.session.sendReturn()`, `sms.session.bindAllows()` and
@@ -340,7 +340,7 @@ and is also what the panel ranked hardest — two methods, one answer.
### Doc claims this review falsified
- [ ] **Make "every README example is executed by the suite" true, or stop claiming it.** Goal 9 and
- [ ] **Make "every README example is executed by the suite" true, or stop claiming it.** Goal 10 and
the Done table both promise it; `test/readme.test.ts` transcribes the examples by hand and has
drifted — 15 fenced `javascript` blocks in the README against 10 tests, and the test named "the
documented sending options" passes none of the five options the README's example passes. Read
@@ -459,7 +459,7 @@ and is also what the panel ranked hardest — two methods, one answer.
From comparing 0.5.0 with `smpp`, `@semyonf/smpp`, `@leissner/node-red-smpp`, `node-smpp-next`,
`smpp-js-sdk`, `smppjs`, cloudhopper-smpp, jsmpp, go-smpp, Kannel, Jasmin and php-smpp, 2026-09-14.
Each lands under goal 6: an option or a hook, with the call that passes none unchanged.
Each lands under goal 7: an option or a hook, with the call that passes none unchanged.
### Sending
@@ -621,7 +621,7 @@ Each lands under goal 6: an option or a hook, with the call that passes none unc
- [ ] **Pooling, and state that survives a restart, through an optional store.** Maintainer's call,
2026-09-14. It replaces two declines — merge state surviving a restart, and a pool of sessions —
and goal 8 was rewritten for it. Big: design before code.
and goal 9 was rewritten for it. Big: design before code.
- **What it holds.** Receipts still awaited and the groups `DlrMerger` collects. Segments of a
message already answered but not yet whole, which the peer will not send again (goal 2). The
concatenation reference, so a restart does not reuse one. For a pool, the ids every session
@@ -634,10 +634,10 @@ Each lands under goal 6: an option or a hook, with the call that passes none unc
in-memory store is enough; across processes the application supplies one.
- **The interface.** Narrow, with keys and records of the library's own making, versioned, with
expiry: a record from an older version is read or refused, never misread, and `DlrMerger`'s
group shape is never published (goal 7). What processes share needs an atomic operation —
group shape is never published (goal 8). What processes share needs an atomic operation —
compare-and-set or increment — since get-then-set races.
- **Adapters live elsewhere.** Redis, Postgres or SQLite stores are packages of their own; this
one ships the interface and the in-memory store, and no runtime dependency (goal 9).
one ships the interface and the in-memory store, and no runtime dependency (goal 10).
- **When the store fails.** Open: a send whose awaited receipt cannot be recorded is refused, or
sent and reported as undetermined (goal 2); a pool whose store is down stops, or falls back to
memory. Either way, an application that supplied no store never waits on one.
+1 -1
View File
@@ -24,5 +24,5 @@
"types": ["node"],
"verbatimModuleSyntax": true
},
"include": ["eslint.config.ts", "interop-tests", "src", "test"]
"include": ["benchmarks", "eslint.config.ts", "interop-tests", "src", "test"]
}