§9 JWT signing-key rotation runbook (todo §9); turned the README's 3-line rotation note into an operational runbook and closed the tooling gap that made its documented steps unrunnable — the old "prepend a key / drop it later" meant hand-editing a JSON file holding a private signing key. Tests-first: new pure rotateJwks(current,{prune}) in gen-jwks.ts — --prepend puts a fresh ES256 key first (Kratos signs with keys[0], the old keys still verify in-flight JWTs) and keeps the rest in order; --prune keeps only the newest (drop superseded post-TTL). CLI reads the existing set from a path arg and writes the new set to stdout (header documents the temp-file redirect so the shell's > can't truncate the input). gen-jwks.test.ts covers prepend (length+1, fresh kid first, old set preserved) + prune (→ 1 key). Runbook documents the two-sided install (Kratos signer env/mount + web JWKS_URL; file:// hot-reloads, base64:// immutable), why it's zero-downtime (sign-with-first + verify-by-kid), the scheduled path (prepend → restart kratos → verify new kid → wait ~12min = 10m TTL + skew → prune; rollback before prune) and the emergency path (replace with a single key → every leaked-key token fails signature → forced re-login; the §9 denylist is moot since the signature is already invalid). Verified the CLI live against the committed dev JWKS (bare→1, --prepend→2 with old kid second, --prune→1); jwks.json untouched. Docs/CLI-only, covered by units (per the §9 precedent, no new browser E2E). README Status + Layout updated. typecheck + 335 units green (333 → 335).

README.md

@@ -54,7 +54,8 @@ only where the platform leaves a gap (see [AGENTS.md](AGENTS.md)).
 > SSO, the session→JWT hot path, the users/groups/roles admin screens) and **Hydra's login / consent
 > / logout handlers** — all driven end-to-end by the Playwright suites, plus **production & ops
 > hardening** (the prod compose profile, response security headers, **structured logging + OTLP
-> observability**). What's left is mainly a **JWT key-rotation runbook** — tracked in `todo.md` (§9).
+> observability**, the **[JWT key-rotation runbook](#jwt-signing-key--rotation)**). Remaining
+> polish is tracked in `todo.md` (§9–§10).
 
 ## The MVP — "clone, one command, hack on a plugin"
 
@@ -204,18 +205,71 @@ same way.
 The session tokenizer (§3) signs each session→JWT with an **ES256** key at
 `ory/kratos/tokenizer/jwks.json`. The committed one is a **dev throwaway** (like the
 cookie/cipher secrets in `kratos.yml`) — a clean clone works; **never run it in
-production**. (Re)generate with the bundled generator:
+production**. Mint a fresh key with the bundled generator:
 
 ```bash
 docker compose run --rm -T --no-deps web node src/gen-jwks.ts > ory/kratos/tokenizer/jwks.json
 ```
 
-**Production:** mount a real key over that path, or set
+**Install in production.** Two endpoints must read the *same* key material:
-`SESSION_WHOAMI_TOKENIZER_TEMPLATES_PLAINPAGES_JWKS_URL=base64://<the JWKS JSON, base64>`.
 
-**Rotation (zero downtime):** Kratos signs with the **first** key in the set; the app
+- **Kratos (signer)** — mount the file over `…/tokenizer/jwks.json`, or set
-selects the verify key by `kid` (§4). So prepend a freshly generated key, keep the old
+  `SESSION_WHOAMI_TOKENIZER_TEMPLATES_PLAINPAGES_JWKS_URL=base64://<the JWKS JSON, base64>`.
-one for ~one token TTL (10m) so in-flight JWTs still verify, then drop it.
+- **web (verifier)** — `JWKS_URL` (default `file://…/tokenizer/jwks.json`). A `file://`
+  set is re-read live (5-min TTL, plus an immediate reload on an unknown `kid`); a
+  `base64://` set is immutable and rotates only on a web redeploy. **For rotation, use
+  `file://` on the web side** so it picks up new keys without a restart.
+
+**Why rotation is zero-downtime.** Kratos signs with the **first** key in the set and
+stamps its `kid` in each JWT header; web selects the verify key by that `kid` (§4). So a
+set can hold the new key *and* the old one at once — tokens minted before and after the
+swap both verify.
+
+#### Scheduled rotation
+
+The token TTL is **10 min** (`kratos.yml` → `whoami.tokenizer.…ttl`); the wait window
+below is one TTL + clock skew, round up to **~12 min**. Run from the repo root (paths are
+container-relative; with the dev bind-mount they edit the real file).
+
+1. **Prepend a fresh key** (new key first, old key kept) — write via a temp file so the
+   shell's `>` can't truncate the input before it's read:
+   ```bash
+   docker compose run --rm -T --no-deps web sh -c \
+     'node src/gen-jwks.ts --prepend ory/kratos/tokenizer/jwks.json' > /tmp/jwks.json \
+     && mv /tmp/jwks.json ory/kratos/tokenizer/jwks.json
+   ```
+2. **Restart Kratos** so it signs with the new first key: `docker compose restart kratos`.
+   (web needs no restart — it hot-reloads the file. The hot path verifies JWTs locally, so
+   a brief Kratos blip only touches login/re-mint.)
+3. **Verify** new logins mint the new `kid` — decode the `plainpages_session` cookie's JWT
+   header, or watch web's logs for a `jwks reload on kid miss` debug line as old clients
+   present the new key.
+4. **Wait ~12 min**, then **prune** the superseded key:
+   ```bash
+   docker compose run --rm -T --no-deps web sh -c \
+     'node src/gen-jwks.ts --prune ory/kratos/tokenizer/jwks.json' > /tmp/jwks.json \
+     && mv /tmp/jwks.json ory/kratos/tokenizer/jwks.json
+   ```
+   No Kratos restart needed — it already signs with that key; this only drops a now-unused
+   verify key.
+
+**Rollback** (before the prune): the old key is still in the set, so revert step 1's file
+and `restart kratos` — in-flight tokens never broke.
+
+#### Emergency rotation (key compromise)
+
+Skip the overlap — you want every token signed with the leaked key to die now. **Replace**
+the set with a single fresh key (no `--prepend`):
+
+```bash
+docker compose run --rm -T --no-deps web node src/gen-jwks.ts > ory/kratos/tokenizer/jwks.json
+docker compose restart kratos
+```
+
+Every existing JWT now fails signature verification → its bearer falls back to anonymous
+and must re-authenticate (the §4 re-mint only covers *expired* tokens, not bad signatures,
+so a forged/leaked-key token can't be silently refreshed). The instant-revoke denylist
+(§9) is unnecessary here — the signature itself is already invalid.
 
 ## Type check & tests
 
@@ -669,7 +723,7 @@ src/oauth-login.ts   resolveLoginChallenge(): authenticate a Hydra login challen
 src/oauth-consent.ts resolveConsentChallenge()/acceptConsent()/rejectConsent(): auto-accept first-party, else show the consent screen → grant scopes (§6)
 src/flow-view.ts     buildFlowView(): Kratos self-service Flow → themed view model (fields, hidden csrf, buttons, tone-mapped messages) for views/auth.ejs (§4)
 src/login.ts         completeLogin()/remintSession(): login completion + TTL re-mint — roles from Keto → metadata_public projection → tokenize → session JWT cookie (§4)
-src/gen-jwks.ts      generateJwks() + CLI: mint the ES256 session-tokenizer signing JWKS (§3); see JWT signing key & rotation
+src/gen-jwks.ts      generateJwks()/rotateJwks() + CLI (mint · --prepend · --prune): the ES256 session-tokenizer signing JWKS (§3); see JWT signing key & rotation
 src/bootstrap.ts     One-command bootstrap (§3): idempotent first-boot seed — JWKS-if-absent, demo admin in Kratos, admin role in Keto
 src/cookie.ts        Cookie parse + secure Set-Cookie build (session/CSRF cookies, §4)
 src/csrf.ts          CSRF for our own POST forms (§4): signed double-submit token — issue/verify, cookie, request gate

src/gen-jwks.test.ts

@@ -5,7 +5,7 @@ import { test } from "node:test";
 import assert from "node:assert/strict";
 import { readFileSync } from "node:fs";
 import { createPrivateKey, sign, type JsonWebKey } from "node:crypto";
-import { generateJwks } from "./gen-jwks.ts";
+import { generateJwks, rotateJwks } from "./gen-jwks.ts";
 import { verifyJws } from "./jwt.ts";
 
 const b64url = (s: string) => Buffer.from(s).toString("base64url");
@@ -30,6 +30,21 @@ test("the committed dev JWKS is a valid ES256 signing key importable by node:cry
   assert.doesNotThrow(() => createPrivateKey({ key: k, format: "jwk" }), "Kratos can load it to sign");
 });
 
+test("rotateJwks prepends a fresh signing key, keeping the old ones for in-flight verification", () => {
+  const old = generateJwks(); // a one-key set, as Kratos signs with the first
+  const rotated = rotateJwks(old);
+  assert.equal(rotated.keys.length, old.keys.length + 1);
+  assert.notEqual(rotated.keys[0]!.kid, old.keys[0]!.kid, "the new key is first (Kratos signs with it) with a fresh kid");
+  assert.deepEqual(rotated.keys.slice(1), old.keys, "old keys are preserved in order so unexpired JWTs still verify");
+  assert.equal(rotated.keys[0]!.alg, "ES256");
+});
+
+test("rotateJwks --prune keeps only the newest (first) key, dropping superseded ones", () => {
+  const twoKeys = rotateJwks(generateJwks()); // prepend → 2 keys
+  const pruned = rotateJwks(twoKeys, { prune: true });
+  assert.deepEqual(pruned.keys, [twoKeys.keys[0]], "only the active signing key remains");
+});
+
 test("a JWS signed with a generated key verifies via our own verifier (§4 reads what Kratos signs)", () => {
   const key = generateJwks().keys[0]!;
   const head = b64url(JSON.stringify({ alg: "ES256", kid: key.kid }));

src/gen-jwks.ts

@@ -1,10 +1,15 @@
 import { generateKeyPairSync, randomUUID } from "node:crypto";
+import { readFileSync } from "node:fs";
 import { fileURLToPath } from "node:url";
 
 // ES256 signing JWKS for the Kratos session tokenizer (§3) — Ory-recommended and the
 // verifier's preferred alg (src/jwt.ts). Rotation runbook: README, JWT signing key.
-// (Re)generate the committed dev key (prod supplies its own):
+// CLI (prod supplies its own key; the committed one is a dev throwaway):
-//   docker compose run --rm -T --no-deps web node src/gen-jwks.ts > ory/kratos/tokenizer/jwks.json
+//   gen-jwks.ts                       → a fresh one-key set (mint/replace; emergency rotation)
+//   gen-jwks.ts --prepend <jwks.json> → new key first + the old ones (zero-downtime rotation)
+//   gen-jwks.ts --prune   <jwks.json> → keep only the newest key (drop superseded, post-TTL)
+// All write to stdout; redirect into the JWKS file (use a temp file for --prepend/--prune so
+// the shell's `>` can't truncate the input before it's read).
 
 export interface SigningJwk {
   kid: string;
@@ -26,5 +31,22 @@ export function generateJwks(): JwkSet {
   return { keys: [{ kid: randomUUID(), alg: "ES256", crv, d, kty, use: "sig", x, y }] };
 }
 
-// CLI: print a fresh set to stdout (redirect into the jwks.json above).
+// Rotate a JWKS: prepend a fresh key (Kratos signs with the first; the old keys still verify
-if (process.argv[1] === fileURLToPath(import.meta.url)) process.stdout.write(`${JSON.stringify(generateJwks(), null, 2)}\n`);
+// in-flight JWTs) — or, with `prune`, keep only the newest key (drop superseded ones once the
+// old token TTL has elapsed). Pure list math; the active signing key is always keys[0].
+export function rotateJwks(current: JwkSet, opts: { prune?: boolean } = {}): JwkSet {
+  return opts.prune ? { keys: current.keys.slice(0, 1) } : { keys: [generateJwks().keys[0]!, ...current.keys] };
+}
+
+// CLI: print the resulting set to stdout (see the header for the redirect caveat).
+if (process.argv[1] === fileURLToPath(import.meta.url)) {
+  const args = process.argv.slice(2);
+  const rotate = args.includes("--prepend") || args.includes("--prune");
+  let set: JwkSet;
+  if (rotate) {
+    const path = args.find((a) => !a.startsWith("--"));
+    if (!path) throw new Error("usage: gen-jwks.ts [--prepend|--prune] <existing-jwks.json>");
+    set = rotateJwks(JSON.parse(readFileSync(path, "utf8")) as JwkSet, { prune: args.includes("--prune") });
+  } else set = generateJwks();
+  process.stdout.write(`${JSON.stringify(set, null, 2)}\n`);
+}