diff --git a/AGENTS.md b/AGENTS.md index 7ed9948..f754cf9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,11 +36,13 @@ branch, create a PR and merge it when the CI/CD turns green. ## Project priorities (do not erode) 1. **Simplicity** — prefer the solution that is easiest to understand, smallest, and most readable. -2. **Few dependencies** — runtime deps stay minimal (today `ejs`, `lucide-static`, `@larvit/log`). - Prefer the Node standard library; justify any new dependency; do not add frameworks. The app is - **stateless — no database**. Auth/identity/OAuth are **Ory sidecar services** reached over their - REST APIs with built-in `fetch` — no SDK. New capabilities ship as **plugin folders** under - `plugins/` that fetch their data from upstream services, not as core code. +2. **Few dependencies** — runtime deps stay minimal (today `ejs`, `lucide-static`, `@larvit/log`, + `postgres`). Prefer the Node standard library; justify any new dependency; do not add frameworks. + The **host is stateless — it owns no schema and stores nothing of its own**; a plugin may own a + Postgres database, which the host provisions but never reads or writes inside. Auth/identity/OAuth are + **Ory sidecar services** reached over their REST APIs with built-in `fetch` — no SDK. New + capabilities ship as **plugin folders** under `plugins/` that get their data from an upstream + service or their own database, not as core code. 3. **Strict TypeScript** — `tsconfig.json` is strict (incl. `noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`, `verbatimModuleSyntax`). Keep it that way. Prefer exact types; limit nullable and multi-option types. @@ -79,7 +81,41 @@ Revisit only if the stated reason stops holding. it into `/node_modules`, above every plugin scope. Never let a copy reach a plugin's own `node_modules`: two instances of the barrel break `instanceof` across the boundary, which `plugin-api.test.ts` guards by asserting both paths reach one module. - - **`config/` is still a plain dir — no `package.json` of its own**, or `#menu-config` resolves + - **Plugin storage hands over credentials, not a client** (README → Plugin storage). The host takes + `postgres` to run the provisioning DDL, and `storage-provisioning.ts` is the only module importing + it — `storage.ts` beside it stays pure so `web` never loads a driver (`src/postgres.test.ts` guards + both halves, because one value imported from the wrong module breaks it invisibly). It is never + re-exported through the barrel, so no driver shape enters the contract. Three properties hold the design together, so + don't trade one away in isolation: passwords are `HMAC-SHA256(PLUGIN_DB_SECRET, id)` rather than + stored, which is what keeps the host stateless — whoever holds that secret holds every plugin + database, so it ranks with the DB password itself; the provisioning DSN reaches `bootstrap` only + (`src/compose.test.ts` guards the split); and provisioning never drops anything, so uninstalling a + plugin cannot destroy data — boot logs the orphans instead. Because the host's copy sits in the + ambient `/node_modules`, a plugin can `import "postgres"` without declaring it — incidental, not a + packaging promise, and a plugin must still depend on its own driver. +- **The trust boundary is the `web` process, not the plugin.** Per-plugin databases and roles bound + *accidents*, not hostile plugins: `PLUGIN_DB_SECRET` is in `web`'s environment during `onBoot`, and + a plugin already holds `ctx.system`'s Ory admin clients — so cross-plugin DB isolation is + containment, and README says so rather than implying a sandbox. Consistent with priority #7 + (crash-isolation is a non-goal). `server.ts` still deletes the secret from `process.env` right + after `loadConfig`, which is before discovery imports any plugin module — the ordering is the whole + point, so move it earlier if anything, **never later**. **Valid while plugins are + operator-installed code, not third-party uploads.** +- **`ory/postgres/init/init.sql` is the only home for the Ory databases' ACL** — don't re-assert the + `REVOKE CONNECT` from `bootstrap`. `REVOKE` only *warns* when the caller doesn't own the database, + so under the least-privilege provisioning account the README recommends it would report success + while changing nothing, and it hard-fails whenever `PLUGIN_DB_ADMIN_URL` names a server with no + `kratos`. A volume created before that file gained the revokes keeps the default grant; + `docker compose down -v` is the dev remedy. **Valid while pre-release, with no deployed volumes.** +- **`bootstrap.ts` stays under `src/auth/`** even though it now provisions plugin databases as well + as seeding Ory. It is the one-shot service's entrypoint, not an auth module; moving it to + `src/bootstrap.ts` would edit `compose.yml`, five e2e compose files and `src/compose.test.ts` for a + rename. Reconsider when a third seeding concern lands. +- **`BootContext.storage` keeps all six credential fields, and there is no `onShutdown` hook.** While + `HOST_API_VERSION` is frozen both are free to revisit; after the freeze, adding is compatible and + removing is not, so the shape errs small elsewhere. Pools handed to a plugin are reaped on process + exit — revisit if a plugin ever needs an orderly drain. **Valid while the freeze holds.** +- **`config/` is still a plain dir — no `package.json` of its own**, or `#menu-config` resolves against that instead and boot fails loud. An operator's menu override has no use for dependencies; if that changes, it needs the same package treatment. - **A plugin `package.json` without `"type": "module"` is refused, not warned.** Allowing it costs a diff --git a/README-dockerhub.md b/README-dockerhub.md index 3daac1a..cad8ec2 100644 --- a/README-dockerhub.md +++ b/README-dockerhub.md @@ -2,7 +2,8 @@ A self-hostable foundation for server-rendered web apps — public or gated pages from a zero-JS design system, with a config-driven menu and auth/permissions (Ory) baked in. -Every domain feature is a drop-in plugin folder; the app is stateless, no build step. +Every domain feature is a drop-in plugin folder, with a Postgres database of its own if it wants +one; the host itself is stateless, and there is no build step. **Source, docs & issues: ** ([GitHub mirror](https://github.com/larvit/plainpages)) diff --git a/README.md b/README.md index 076486f..e104215 100644 --- a/README.md +++ b/README.md @@ -89,6 +89,7 @@ From here, render real pages against the app shell and fetch upstream data — s - [hooks](#hooks) - [where they live & mounting](#where-plugins-live-and-how-to-mount-them) - [dependencies](#plugin-dependencies) + - [storage](#plugin-storage) - [local dev & test](#local-dev--test-story) - [The menu system](#the-menu-system) - [Building blocks](#building-blocks) @@ -106,7 +107,7 @@ From here, render real pages against the app shell and fetch upstream data — s - [security model](#security-model) - [Email](#email) - [Architecture](#architecture) - - [Stateless](#stateless) + - [Stateless core](#stateless-core) - [Testing](#testing) - [end-to-end](#end-to-end-playwright) - [the full gate](#the-full-gate-one-command) @@ -380,6 +381,7 @@ folder-derived `id` to produce the loaded `Plugin`. | `permissions` | no | Permissions this plugin gates on. See [Nav & permission gates](#nav--permission-gates). | | `routes` | no | See [Routes & handlers](#routes--handlers). | | `hooks` | no | See [Hooks](#hooks). | +| `storage` | no | `true` ⇒ the host provisions a Postgres database and login role for this plugin and hands the credentials to `onBoot`. See [Plugin storage](#plugin-storage). | A plugin may be routes-only, nav-only, or hooks-only — every collection field is optional. @@ -432,8 +434,8 @@ export async function listThings(ctx: RequestContext) { `requireSession(ctx)`, `can(ctx, permission)` (coarse JWT-claim check, zero I/O), and `check(keto, ctx, {namespace, object, relation})` (a live Keto check; anonymous ⇒ denied). Throw `new GuardError(403, …)` after a failed `can`/`check` to render the 403 page. -- The handler **fetches its own data** from upstream; plugins hold no state (see - [Stateless](#stateless)). +- The handler **fetches its own data** — from upstream, or from the plugin's own + [storage](#plugin-storage); the host holds none of it (see [Stateless core](#stateless-core)). - Default status: `200` for `view`/`html`/`json`, `303` for `redirect`. #### Escaping & the trust boundary @@ -632,10 +634,13 @@ Optional, for reacting to system actions. A plugin's `hooks` may implement: | Hook | When | May | | --- | --- | --- | -| `onBoot()` | after discovery, before the server listens | warm caches, validate upstream config | +| `onBoot(host)` | after discovery, before the server listens | warm caches, validate upstream config, open a [storage](#plugin-storage) connection | | `onRequest(ctx)` | before route matching | inspect, or **short-circuit** by returning a `RouteResult` | | `onResponse(ctx, result)` | after the handler | observe/log; cannot change the response | +`onBoot`'s `host` is a `BootContext`, carrying `storage` for a plugin that declared it. A hook +written without a parameter stays valid. + Hooks run in **discovery order** (plugins sorted by id). `onRequest` fires on every request that reaches routing (static assets bypass it); the **first** hook to return a `RouteResult` short-circuits — later hooks and the route handler are skipped, and that result renders against its own plugin's @@ -653,7 +658,7 @@ getting its folder there. bind-mounts the whole tree (`compose.override.yml`: `.:/app`), so a restart picks it up. **2. A plugin kept in its own repo, or added to a prebuilt image.** Bind-mount the plugin -folder onto `/app/plugins/` with a small compose override. Plugins are stateless, so +folder onto `/app/plugins/` with a small compose override. A plugin folder is code, not data — mount it read-only: ```yaml @@ -724,6 +729,88 @@ barrel's types on disk: typecheck it mounted under the host tree, or vendor a ty `node_modules`** and point tsconfig `paths` at it — a stub inside is the shadowing copy discovery refuses, and it would travel with the folder you mount. +### Plugin storage + +A plugin that needs to keep data sets `storage: true`. The host then provisions a Postgres +**database and login role of its own** — both named `plugin_` — and hands the credentials to +`onBoot`: + +```ts +import postgres from "postgres"; // your dependency, not the host's +import { definePlugin } from "@plainpages/plugin-api"; + +let sql: ReturnType; + +export default definePlugin({ + apiVersion: "1.0.0", + storage: true, + hooks: { + onBoot: async (boot) => { + if (!boot.storage) throw new Error("things: storage was not provisioned"); + sql = postgres(boot.storage.url); + // Every web instance runs onBoot, and concurrent CREATE TABLE IF NOT EXISTS is an error in + // Postgres — the lock is released when the transaction ends. + await sql.begin(async (tx) => { + await tx`SELECT pg_advisory_xact_lock(hashtext('things:schema'))`; + await tx`CREATE TABLE IF NOT EXISTS things (id uuid PRIMARY KEY, name text NOT NULL)`; + }); + }, + }, +}); +``` + +`boot.storage` is a `StorageCredentials` — `database`, `host`, `password`, `port`, `user`, and `url`, +the same values pre-assembled as a DSN, which most clients take directly. It is typed optional, so +the guard above is expected of every storage plugin rather than a sign something is wrong. + +**Credentials, not a client.** The host has no opinion on how you reach Postgres: depend on +`postgres`, `pg`, a query builder or an ORM ([Plugin dependencies](#plugin-dependencies)). No driver +is part of the contract, so upgrading yours is yours alone to time. The flip side is that pool sizing +is yours too — keep yours under `PLUGIN_DB_CONNECTION_LIMIT` (default 10), the per-role ceiling the +host sets so one plugin cannot exhaust the Postgres this stack shares with Ory. + +**The schema is yours, migrations included.** The host creates the database empty, never reads or +writes inside it, and ships no migration machinery — evolving your tables compatibly (expand, then +contract, so a rolled-back version still runs) is yours to own. Create your tables in `onBoot`: it +runs before the server listens, so a failure aborts boot instead of surfacing later as a broken page. + +What the host does guarantee: + +- **One database and one role per plugin**, with `CONNECT` revoked from `PUBLIC`. This bounds + *accidents* — a wrong database name, a mistyped DSN, a stray query — and it is not a security + boundary: plugins share the `web` process, so a plugin that goes looking can reach another's + credentials. Install plugins you trust ([Security model](#security-model)). +- **Provisioning is idempotent and runs every boot**, so a plugin dropped in later is picked up by + the next `docker compose up -d` — the same rule as permission seeding. Each boot re-applies the + role's password, connection limit, and `NOCREATEDB`/`NOCREATEROLE`. +- **Your data is never dropped.** Removing a plugin folder leaves its database untouched; deleting it + is a deliberate act by an operator. Each boot logs any `plugin_*` database no installed plugin + claims, so what you left behind stays findable — read that list before dropping anything, since a + second Plainpages stack sharing this server will have its databases named there too. + +**Passwords are derived, never stored** — each is `HMAC-SHA256(PLUGIN_DB_SECRET, )`, so +`bootstrap` and `web` compute the same value independently and nothing has to be written down. +Rotate every plugin's password by changing `PLUGIN_DB_SECRET` and running `docker compose up -d`, +which re-applies each role's password and leaves the data alone — restart every `web` instance as +part of it, since one still holding the old secret can open no new connections. Treat the secret as +you would a database password: whoever holds it holds every plugin database. Under +`REQUIRE_SECURE_SECRETS` a missing, empty or throwaway secret is refused — in `bootstrap` before it +creates any role, so no database is ever given a password derivable from a constant in this repo. + +**Only `bootstrap` holds provisioning credentials.** It alone gets `PLUGIN_DB_ADMIN_URL`, an account +with `CREATEDB` and `CREATEROLE` (superuser works but is more than it needs; the dev stack simply +reuses Ory's). Keep using the same account: Postgres gives a `CREATEROLE` account admin rights only +over the roles it created itself, so if you swap it for a fresh one, grant that one `ADMIN OPTION` on +the existing `plugin_*` roles first, or the next boot cannot re-apply their passwords. `web` gets `PLUGIN_DB_URL`, which names the server and must carry no credentials — +supply one with a username or password and boot fails, rather than leaving a privileged password in +the process that runs plugin code. Set both, plus `PLUGIN_DB_SECRET` +([Configuration](#configuration)); the dev stack sets them for you. + +Storage stays off until `PLUGIN_DB_URL` is set, and a plugin declaring it while that is unset +**aborts boot** naming itself — rather than serving pages without its data. One naming limit: a +storage plugin's folder may be at most **56 characters**, so `plugin_` fits Postgres' 63-byte +identifier. + ### Local dev & test story A plugin is a normal folder of TypeScript, tested the same way the core is — everything in Docker. @@ -936,7 +1023,7 @@ The app is **environment-agnostic**: no `NODE_ENV`, every behaviour its own expl | `PORT` | `3000` | web listen port | | `CACHE_TEMPLATES` | `false` | cache compiled EJS templates (`true` in prod) | | `SECURE_COOKIES` | `false` | mark our session/CSRF cookies `Secure` (`true` in prod https; off in dev http) | -| `REQUIRE_SECURE_SECRETS` | `false` | when `true`, `CSRF_SECRET` must be supplied and differ from the dev throwaway | +| `REQUIRE_SECURE_SECRETS` | `false` | when `true`, `CSRF_SECRET` — and `PLUGIN_DB_SECRET` once storage is configured — must be supplied and differ from the dev throwaway | | `LOG_LEVEL` | `info` | min severity logged: `error`/`warn`/`info`/`verbose`/`debug`/`silly`/`none` | | `LOG_FORMAT` | `text` | log line format: `text` (human-readable, dev) or `json` (structured, prod) | | `SERVICE_NAME` | `plainpages` | OTLP `service.name` on every log + span — brand it as your own deployment | @@ -952,6 +1039,10 @@ The app is **environment-agnostic**: no `NODE_ENV`, every behaviour its own expl | `REVOCATION_DENYLIST` | `false` | when `true`, enable the optional [instant permission/session revoke denylist](#instant-revoke-the-optional-denylist) | | `REVOCATION_TTL_SEC` | `900` | how long a revoke entry lives; keep ≥ tokenizer TTL (10m) + clock skew | | `CSRF_SECRET` | dev throwaway | signs our double-submit CSRF token; enforced by `REQUIRE_SECURE_SECRETS` | +| `PLUGIN_DB_URL` | _unset_ (dev: `postgres://postgres:5432`) | credential-free Postgres base URL for [plugin storage](#plugin-storage); unset ⇒ storage off, and a plugin declaring it aborts boot | +| `PLUGIN_DB_ADMIN_URL` | _unset_ (dev: the bundled superuser) | the DSN that provisions each plugin's database and role — read by the one-shot `bootstrap` service **only**, never by `web` | +| `PLUGIN_DB_SECRET` | dev throwaway | derives each plugin's database password; `REQUIRE_SECURE_SECRETS` enforces it in `web` once `PLUGIN_DB_URL` is set, and in `bootstrap` whenever a plugin declares storage | +| `PLUGIN_DB_CONNECTION_LIMIT` | `10` | per-role Postgres connection ceiling, so one plugin's pools cannot exhaust the server Ory shares; read by `bootstrap` when provisioning | ### Canonical host (one public URL) @@ -1082,8 +1173,8 @@ records that subject as revoked-now; the hot path then rejects every token for i the revoke and forces a re-mint — which re-reads permissions from Keto, or clears a dead session. A fresh re-login passes, so a downgrade lands immediately without locking the account. -It is an in-memory, auto-evicting map — no database, so it stays inside the stateless model — and -the check is pure CPU, keeping Keto off the hot path. Entries self-evict after `REVOCATION_TTL_SEC` +It is an in-memory, auto-evicting map — host-owned state would break the [stateless +core](#stateless-core) — and the check is pure CPU, keeping Keto off the hot path. Entries self-evict after `REVOCATION_TTL_SEC` (default 900s ≥ the 10m token TTL + skew). Two bounds: it is instant only on the **single instance** that handled the revoke (elsewhere the guarantee falls back to the token TTL — back it with a shared store for hard multi-instance revoke), and a **group** membership change is transitive across many @@ -1198,18 +1289,21 @@ of it over their **REST APIs using Node's built-in `fetch`** — no SDK dependen In **dev** the host-facing Ory ports are published — Kratos public `4433` and Hydra public `4444`; prod keeps them internal. -Runtime deps stay tiny and pinned: **`ejs`**, **`lucide-static`**, and **`@larvit/log`**. Auth, -sessions, SSO and OAuth2 add *services*, not npm packages. +Runtime deps stay tiny and pinned: **`ejs`**, **`lucide-static`**, **`@larvit/log`**, and +**`postgres`** — the last one has no sub-dependencies of its own and is used in a single module, to +provision [plugin storage](#plugin-storage) at boot. Auth, sessions, SSO and OAuth2 add *services*, +not npm packages. -### Stateless +### Stateless core -Plainpages holds **no state of its own**. The only database in the stack is **Postgres**, used by -Ory; the `web` app never connects to it. +The host holds **no state of its own**: it owns no schema and keeps nothing between requests. The +stack's **Postgres** backs Ory, and gives every plugin that asks for one a database of its own +([Plugin storage](#plugin-storage)) — which the host provisions but never reads or writes. -A plugin reads and writes state by **calling an upstream service** from its route handler — a REST -API, an ERP, a plant historian, the customer's own backend — and renders the response with the -building blocks. That keeps `web` trivially scalable and crash-safe: any instance can serve any -request, because the session lives in Kratos and the data lives upstream. +So a plugin gets its data one of two ways: by **calling an upstream service** from its route handler +— a REST API, an ERP, a plant historian, the customer's own backend — or from **its own database**. +Either keeps `web` trivially scalable and crash-safe: any instance can serve any request, because the +session lives in Kratos and the data lives outside the process. ## Testing @@ -1371,6 +1465,11 @@ the one-shot bootstrap) — and mounts no source. Secrets come from the environm running insecure. Before going live, supply the production secrets and any SSO credentials — the **only** manual prep ([What you must supply](#what-you-must-supply-the-only-manual-prep)). +**Back up the `pgdata` volume.** Once a plugin declares [storage](#plugin-storage), Postgres holds +business data that exists nowhere else, alongside Ory's identities — the stack stops being +reproducible from the image and config alone. Snapshot the volume, or `pg_dump` each database on a +schedule, and rehearse the restore. + Every response carries security headers (`src/http/security-headers.ts`): a strict `Content-Security-Policy` (the core is zero-JS — `script-src 'self'`, no inline scripts), `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY` + `frame-ancestors 'none'`, @@ -1521,7 +1620,8 @@ src/ The app — strict tsc, no build step. *.test.ts sit beside i18n/ catalog (parity rules) · locale (resolution) · translate · load · runtime · english · view-locals · locales/ (the core en-US + sv-SE catalogs) plugin-host/ plugin.ts (the contract) · plugin-api.ts (the `@plainpages/plugin-api` barrel) · system.ts - (ctx.system) · discovery · router · hooks · view-resolver + (ctx.system) · discovery · router · hooks · view-resolver · storage (the rules) · + storage-provisioning (the DDL; bootstrap-only, holds the driver) ui/ chrome (the one global menu) · shell-context · dashboard · nav (composeNav) · menu-config (`#menu-config`) · icons (lucide sprite builder) · list-query · paginate diff --git a/ci.sh b/ci.sh index e9782d1..6bbbc33 100755 --- a/ci.sh +++ b/ci.sh @@ -60,6 +60,33 @@ echo "$units" | grep -E '^. (tests|pass|fail) ' || true count=$(echo "$units" | grep -oE 'tests [0-9]+' | grep -oE '[0-9]+' | head -1 || true) [ "${count:-0}" -ge 50 ] || { echo "only ${count:-0} unit tests ran — test glob broken?"; exit 1; } +# Plugin storage against a real Postgres. The step above runs --no-deps, so this suite's integration +# test skips there — and it is the only thing proving the DDL actually grants what it claims, rather +# than that the SQL text is the text we wrote. `node --test` counts a skip, so the floor won't catch it. +step "Plugin storage (real Postgres)" +# Own project name, like every E2E suite below: the default project is the DEV stack, so a bare +# `down -v` here would delete the operator's pgdata — Ory identities and every plugin database. +# --wait, because initdb on a cold volume outlasts the suite's connect timeout. +storage_rc=0 +storage_proj=plainpages-storage +storage_files=(-p "$storage_proj" -f compose.yml) # no override merge, like the e2e suites below +storage_dsn="postgres://${POSTGRES_USER:-ory}:${POSTGRES_PASSWORD:-ory}@postgres:5432/ory" +storage_out="" +docker compose "${storage_files[@]}" up -d --wait postgres >/dev/null || storage_rc=$? +# `if`, not `&&`: a false `&&` returns non-zero, which under `set -e` would exit before teardown. +if [ "$storage_rc" -eq 0 ]; then + # --build like the e2e suites: this stack mounts no source, so without it the step would test + # whatever `web` image that project last baked. + storage_out=$(docker compose "${storage_files[@]}" run --build --rm --no-deps \ + -e "PLUGIN_DB_ADMIN_URL=$storage_dsn" \ + web node --test src/plugin-host/storage.test.ts 2>&1) || storage_rc=$? +fi +docker compose "${storage_files[@]}" down -v >/dev/null 2>&1 || true # also covers a failed `up` +echo "$storage_out" | grep -E '^. (tests|pass|fail|skipped) ' || true +[ "$storage_rc" -eq 0 ] || { echo "$storage_out"; echo "plugin storage integration tests failed (exit $storage_rc)"; exit "$storage_rc"; } +# A skip here exits 0 and proves nothing — the same trap the unit floor above guards against. +echo "$storage_out" | grep -qE '^. skipped 0$' || { echo "storage integration test skipped — PLUGIN_DB_ADMIN_URL not wired through"; exit 1; } + # Run one E2E suite against its OWN named stack, then always tear it down (even on failure). The # per-suite project name keeps a flaky teardown from leaking containers/volumes into the next suite. # --user: the runner writes screenshots + the report into the checkout, so they must belong to diff --git a/compose.override.yml b/compose.override.yml index f4d0176..2b9de80 100644 --- a/compose.override.yml +++ b/compose.override.yml @@ -1,5 +1,9 @@ # Development overrides, merged automatically by `docker compose up`. # Mounts the source for live editing and restarts on change via `node --watch`. + +# web connects with it and bootstrap provisions against it, so the two must agree — one home. +x-plugin-db-url: &plugin-db-url postgres://postgres:5432 + services: web: command: node --watch src/server.ts @@ -13,6 +17,9 @@ services: CACHE_TEMPLATES: "false" LOG_FORMAT: "text" # human-readable logs in dev (base sets json for prod log pipelines) LOG_LEVEL: "debug" # verbose by default while developing (base defaults to info) + # Point plugin storage at the bundled Postgres, so a dropped-in plugin declaring `storage` + # works with no further config; the secret falls back to the dev throwaway (config.ts). + PLUGIN_DB_URL: *plugin-db-url REQUIRE_SECURE_SECRETS: "false" SECURE_COOKIES: "false" # dev serves http — Secure cookies wouldn't be sent SCHEDULING_UPSTREAM: "http://shifts-upstream:4000" # backs the reference plugin once you copy it into plugins/ @@ -29,6 +36,11 @@ services: # It belongs here and not in the base file, where it would desynchronise prod and collide with the # e2e stacks, which bind individual plugins *inside* /app/plugins. bootstrap: + # Provisions the plugin databases web connects to above, as the dev superuser. + environment: + PLUGIN_DB_ADMIN_URL: postgres://${POSTGRES_USER:-ory}:${POSTGRES_PASSWORD:-ory}@postgres:5432/ory + PLUGIN_DB_URL: *plugin-db-url + REQUIRE_SECURE_SECRETS: "false" # dev derives from the throwaway, as web does volumes: - .:/app diff --git a/compose.yml b/compose.yml index 959745f..6e252bc 100644 --- a/compose.yml +++ b/compose.yml @@ -17,10 +17,16 @@ services: CACHE_TEMPLATES: "true" CSRF_SECRET: ${CSRF_SECRET:-dev-insecure-csrf-secret} LOG_FORMAT: "json" # structured logs for prod pipelines; set OTLP_ENDPOINT to also export to a collector + # Per-plugin Postgres storage. Explicit toggle: unset ⇒ off, and a plugin declaring `storage` + # refuses to boot rather than run without its data. The URL carries no credentials — each + # plugin's own password is derived from the secret (README → Plugin storage). + PLUGIN_DB_SECRET: ${PLUGIN_DB_SECRET:-} + PLUGIN_DB_URL: ${PLUGIN_DB_URL:-} REQUIRE_SECURE_SECRETS: "true" SECURE_COOKIES: "true" # prod serves https — mark session/CSRF cookies Secure # Wait for the services the app talks to (kratos + keto + hydra for the OAuth2 login/ - # consent handler) + the one-shot bootstrap (admin + JWKS seed). + # consent handler) + the one-shot bootstrap (admin + JWKS seed). Postgres too: a plugin that + # declares `storage` opens its connection in onBoot, before the server listens. depends_on: bootstrap: condition: service_completed_successfully @@ -30,14 +36,17 @@ services: condition: service_healthy hydra: condition: service_healthy + postgres: + condition: service_healthy # verifier reads the same tokenizer JWKS Kratos signs with (config.ts JWKS_URL). # Read-only — bootstrap is the only writer. volumes: - ./ory/kratos/tokenizer:/etc/config/kratos/tokenizer:ro restart: unless-stopped - # Ory's storage only (Kratos/Keto/Hydra) — the web app never connects here. - # init/init.sql creates one database per service. Dev defaults below; supply + # The stack's storage: one database per Ory service (init/init.sql), plus one per plugin that + # declares `storage` — bootstrap creates those at boot, since only it holds superuser credentials. + # A plugin connects as its own role from inside web. Dev defaults below; supply # POSTGRES_USER/PASSWORD via env in production. postgres: image: postgres:18.6-alpine3.23 @@ -127,6 +136,8 @@ services: condition: service_healthy keto: condition: service_healthy + postgres: + condition: service_healthy environment: ADMIN_EMAIL: ${ADMIN_EMAIL:-admin@plainpages.local} ADMIN_PASSWORD: ${ADMIN_PASSWORD:-admin} @@ -137,6 +148,13 @@ services: JWKS_FILE: /etc/config/kratos/tokenizer/jwks.json KETO_WRITE_URL: http://keto:4467 KRATOS_ADMIN_URL: http://kratos:4434 + # The superuser DSN that creates each plugin's database and role lives ONLY here — never in + # web, so plugin code cannot read it out of its own environment. Unset ⇒ a plugin declaring + # `storage` fails the seed loudly. The secret must match web's; both derive the same passwords. + PLUGIN_DB_ADMIN_URL: ${PLUGIN_DB_ADMIN_URL:-} + PLUGIN_DB_SECRET: ${PLUGIN_DB_SECRET:-} + PLUGIN_DB_URL: ${PLUGIN_DB_URL:-} # only to refuse a mismatch: what bootstrap creates, web connects to + REQUIRE_SECURE_SECRETS: "true" # refuse the throwaway secret here too, before any role is created volumes: - ./ory/kratos/tokenizer:/etc/config/kratos/tokenizer command: node src/auth/bootstrap.ts diff --git a/examples/plugins/scheduling/shifts.test.ts b/examples/plugins/scheduling/shifts.test.ts index 50441fc..e641b66 100644 --- a/examples/plugins/scheduling/shifts.test.ts +++ b/examples/plugins/scheduling/shifts.test.ts @@ -51,7 +51,7 @@ test("the manifest's onBoot hook validates SCHEDULING_UPSTREAM (the binding, not try { const manifest = (await import("./plugin.ts")).default; assert.equal(typeof manifest.hooks?.onBoot, "function"); - assert.throws(() => manifest.hooks!.onBoot!(), /SCHEDULING_UPSTREAM/); // bad upstream → boot fails loud + assert.throws(() => manifest.hooks!.onBoot!({}), /SCHEDULING_UPSTREAM/); // bad upstream → boot fails loud } finally { if (prev === undefined) delete process.env["SCHEDULING_UPSTREAM"]; else process.env["SCHEDULING_UPSTREAM"] = prev; diff --git a/ory/postgres/init/init.sql b/ory/postgres/init/init.sql index e058258..dad615f 100644 --- a/ory/postgres/init/init.sql +++ b/ory/postgres/init/init.sql @@ -1,6 +1,14 @@ -- Runs once on first boot (docker-entrypoint-initdb.d), as the POSTGRES_USER. -- One database per Ory service: each owns its schema and runs its own migrations, --- so they never collide. The web app never connects here (stateless — see README). +-- so they never collide. A plugin's database does not belong here: bootstrap provisions those on +-- every boot, so one dropped in later is picked up too (README → Plugin storage). CREATE DATABASE kratos; CREATE DATABASE keto; CREATE DATABASE hydra; + +-- Postgres grants CONNECT to PUBLIC by default, so every plugin role could otherwise open the auth +-- plane's databases and read pg_catalog; table data stays protected either way. Ory connects as the +-- POSTGRES_USER, which owns these and keeps its access. +REVOKE CONNECT ON DATABASE kratos FROM PUBLIC; +REVOKE CONNECT ON DATABASE keto FROM PUBLIC; +REVOKE CONNECT ON DATABASE hydra FROM PUBLIC; diff --git a/package-lock.json b/package-lock.json index fcaeb22..fe90016 100644 --- a/package-lock.json +++ b/package-lock.json @@ -10,7 +10,8 @@ "dependencies": { "@larvit/log": "2.3.0", "ejs": "6.0.1", - "lucide-static": "1.32.0" + "lucide-static": "1.33.0", + "postgres": "3.4.9" }, "devDependencies": { "@types/ejs": "3.1.5", @@ -400,11 +401,24 @@ } }, "node_modules/lucide-static": { - "version": "1.32.0", - "resolved": "https://registry.npmjs.org/lucide-static/-/lucide-static-1.32.0.tgz", - "integrity": "sha512-lLIg/D57X3QDqAhk0Hoz7Yc36tU9XYgWLXWL21x4TOIulwWyiC9wlHDswCHMbm5O3fFBLsBUWJuDs9rEpm70rg==", + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lucide-static/-/lucide-static-1.33.0.tgz", + "integrity": "sha512-jNGgvTNcLUfVRX4N9PH9pVVTJzoph/BmYmgU838bYBQodkUJL4nAThkuymFz1x3OUYMhJxPndC7rdg1sxOPYKg==", "license": "ISC" }, + "node_modules/postgres": { + "version": "3.4.9", + "resolved": "https://registry.npmjs.org/postgres/-/postgres-3.4.9.tgz", + "integrity": "sha512-GD3qdB0x1z9xgFI6cdRD6xu2Sp2WCOEoe3mtnyB5Ee0XrrL5Pe+e4CCnJrRMnL1zYtRDZmQQVbvOttLnKDLnaw==", + "license": "Unlicense", + "engines": { + "node": ">=12" + }, + "funding": { + "type": "individual", + "url": "https://github.com/sponsors/porsager" + } + }, "node_modules/typescript": { "version": "7.0.2", "resolved": "https://registry.npmjs.org/typescript/-/typescript-7.0.2.tgz", diff --git a/package.json b/package.json index d7a05aa..8227ccd 100644 --- a/package.json +++ b/package.json @@ -19,7 +19,8 @@ "dependencies": { "@larvit/log": "2.3.0", "ejs": "6.0.1", - "lucide-static": "1.32.0" + "lucide-static": "1.33.0", + "postgres": "3.4.9" }, "devDependencies": { "@types/ejs": "3.1.5", diff --git a/src/auth/bootstrap.test.ts b/src/auth/bootstrap.test.ts index 194c207..f57655b 100644 --- a/src/auth/bootstrap.test.ts +++ b/src/auth/bootstrap.test.ts @@ -5,7 +5,10 @@ import { test } from "node:test"; import assert from "node:assert/strict"; import { randomUUID } from "node:crypto"; -import { ensureJwks, firstRunBanner, identityPayload, permissionTuple, seedAdmin, seedPermissions } from "./bootstrap.ts"; +import { ensureJwks, firstRunBanner, identityPayload, permissionTuple, provisionPluginStorage, seedAdmin, seedPermissions, serverMismatch } from "./bootstrap.ts"; +import { createLogger } from "../logger.ts"; +import type { Plugin } from "../plugin-host/plugin.ts"; +import type { ProvisionOptions, ProvisionResult } from "../plugin-host/storage-provisioning.ts"; const json = (status: number, body?: unknown) => new Response(body === undefined ? null : JSON.stringify(body), { @@ -151,3 +154,63 @@ test("ensureJwks generates a key only when the file is absent", () => { assert.equal(ensureJwks(path, { exists: () => true, write }), false); assert.equal(writes.length, 1); // present → nothing written }); + +// --- Plugin storage provisioning ----------------------------------------------------- +// The provisioner is injected, so the branch decisions are testable without a Postgres. + +const SILENT = createLogger({ level: "none" }); +const storagePlugin = (id: string): Plugin => ({ apiVersion: "1.0.0", id, storage: true }); +const EMPTY: ProvisionResult = { orphans: [], provisioned: [] }; + +function recordingProvisioner(result: ProvisionResult = EMPTY) { + const calls: ProvisionOptions[] = []; + return { calls, provision: async (options: ProvisionOptions) => { calls.push(options); return result; } }; +} + +test("provisioning is skipped entirely when nothing declares storage and none is configured", async () => { + const { calls, provision } = recordingProvisioner(); + await provisionPluginStorage({}, [{ apiVersion: "1.0.0", id: "plain" }], SILENT, provision); + assert.deepEqual(calls, []); // no connection attempted, so an unconfigured stack still boots +}); + +// Uninstalling the last storage plugin is exactly when a left-behind database needs naming. +test("provisioning still runs with nothing to provision, so orphans are reported", async () => { + const { calls, provision } = recordingProvisioner({ orphans: ["plugin_gone"], provisioned: [] }); + await provisionPluginStorage({ PLUGIN_DB_ADMIN_URL: "postgres://ory:ory@db:5432/ory" }, [], SILENT, provision); + assert.equal(calls.length, 1); + assert.deepEqual(calls[0]?.pluginIds, []); +}); + +test("a plugin declaring storage without a provisioning DSN fails loud, naming the plugin", async () => { + const { calls, provision } = recordingProvisioner(); + await assert.rejects( + provisionPluginStorage({}, [storagePlugin("things")], SILENT, provision), + /PLUGIN_DB_ADMIN_URL.*things/s, + ); + assert.deepEqual(calls, []); +}); + +test("the connection limit and derived secret reach the provisioner", async () => { + const { calls, provision } = recordingProvisioner(); + const env = { PLUGIN_DB_ADMIN_URL: "postgres://ory:ory@db:5432/ory", PLUGIN_DB_CONNECTION_LIMIT: "25", PLUGIN_DB_SECRET: "real" }; + await provisionPluginStorage(env, [storagePlugin("things")], SILENT, provision); + assert.equal(calls[0]?.connectionLimit, 25); + assert.equal(calls[0]?.secret, "real"); + assert.deepEqual(calls[0]?.pluginIds, ["things"]); +}); + +// bootstrap creates the role on one server; web tells the plugin to connect to another. Left +// unsaid it surfaces inside a plugin as "password authentication failed", naming neither. Warned +// rather than refused: web reaching a pooler bootstrap cannot provision through is legitimate. +test("a storage URL mismatch is reported, and provisioning still runs", async () => { + const { calls, provision } = recordingProvisioner(); + const env = { PLUGIN_DB_ADMIN_URL: "postgres://ory:ory@db-a:5432/ory", PLUGIN_DB_URL: "postgres://db-b:5432" }; + await provisionPluginStorage(env, [storagePlugin("things")], SILENT, provision); + assert.equal(calls.length, 1); +}); + +test("the same server spelled with an implicit port still agrees", () => { + assert.equal(serverMismatch("postgres://ory:ory@db:5432/ory", "postgres://db"), null); // 5432 is the default + assert.equal(serverMismatch("postgres://ory:ory@db:5432/ory", undefined), null); // web's own boot error to raise + assert.equal(serverMismatch("postgres://ory:ory@db:5432/ory", "postgres://db:6543"), "db:5432 vs db:6543"); +}); diff --git a/src/auth/bootstrap.ts b/src/auth/bootstrap.ts index c7088bd..137ef59 100644 --- a/src/auth/bootstrap.ts +++ b/src/auth/bootstrap.ts @@ -8,10 +8,15 @@ // Then prints a first-run banner; fails loud on any unexpected upstream error. import { existsSync, writeFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; +import { resolvePluginDbConnectionLimit, resolvePluginDbSecret } from "../config.ts"; import { discoverPlugins } from "../plugin-host/discovery.ts"; -import { declaredPermissions, isValidPermissionName } from "../plugin-host/plugin.ts"; +import { declaredPermissions, isValidPermissionName, type Plugin } from "../plugin-host/plugin.ts"; +import { provisionStorage } from "../plugin-host/storage-provisioning.ts"; +import { storagePluginIds } from "../plugin-host/storage.ts"; import { generateJwks, type JwkSet } from "./gen-jwks.ts"; -import { createLogger, runWithLog, tracedFetch } from "../logger.ts"; +import { createLogger, runWithLog, tracedFetch, type Log } from "../logger.ts"; + +type Env = Record; // --- Pure payload builders (the Kratos/Keto request contracts) ----------------------- @@ -141,7 +146,7 @@ export function firstRunBanner(opts: { appUrl: string; email: string; password: // --- CLI (the bootstrap container entrypoint) ---------------------------------------- async function main() { - const env = process.env; + const env = { ...process.env }; // snapshot: the storage credentials leave process.env before discovery // Structured like the web app so prod logs stay uniform; honour LOG_FORMAT/SERVICE_NAME. const log = createLogger({ format: env["LOG_FORMAT"] === "json" ? "json" : "text", @@ -150,29 +155,84 @@ async function main() { // runWithLog makes `log` ambient so seedAdmin's tracedFetch traces the Kratos/Keto seed calls. await runWithLog(log, async () => { if (ensureJwks(env["JWKS_FILE"] ?? "/etc/config/kratos/tokenizer/jwks.json")) log.info("generated a JWKS signing key"); - - // Seed every discovered plugin's declared permission names (plus any ADMIN_PERMISSIONS), so the - // shipped example — and any dropped-in plugin — works for the demo admin without a host edit. - const declared = declaredPermissions(await discoverPlugins()).map((decl) => decl.name); - const { ignored, permissions } = seedPermissions(env["ADMIN_PERMISSIONS"], declared); - if (ignored.length > 0) { - log.warn("ignoring ADMIN_PERMISSIONS entries that are not :", { ignored: ignored.join(", ") }); - } - const email = env["ADMIN_EMAIL"] ?? "admin@plainpages.local"; - const password = env["ADMIN_PASSWORD"] ?? "admin"; - const result = await seedAdmin({ - email, - fetchImpl: tracedFetch, - ketoWriteUrl: env["KETO_WRITE_URL"] ?? "http://keto:4467", - kratosAdminUrl: env["KRATOS_ADMIN_URL"] ?? "http://kratos:4434", - password, - permissions, - }); - log.info("admin seeded", { created: result.created, id: result.id, permissions: result.permissions.join(", ") }); - // The banner is human-facing UX (the first-run "you're ready" block), not a log event — print raw. - console.log(firstRunBanner({ appUrl: env["APP_URL"] ?? "http://localhost:3000", email, password })); + // Discovery imports every plugin module — and its dependencies — into *this* process, which holds + // the credential that may CREATE DATABASE/ROLE. Same move as server.ts, on the stronger secret. + delete process.env["PLUGIN_DB_ADMIN_URL"]; + delete process.env["PLUGIN_DB_SECRET"]; + const plugins = await discoverPlugins(); + await provisionPluginStorage(env, plugins, log); + await seedAdminAndPermissions(env, plugins, log); }); await log.end(); // flush any pending OTLP spans/logs before the one-shot exits } +// A database and login role for each plugin that asked for one. It happens here because bootstrap +// holds the stack's only provisioning credentials — web derives the same password and connects as +// the plugin's own role. +export async function provisionPluginStorage(env: Env, plugins: Plugin[], log: Log, provision = provisionStorage): Promise { + const ids = storagePluginIds(plugins); + const adminUrl = env["PLUGIN_DB_ADMIN_URL"]; + // Still connect with nothing to provision, as long as storage is configured: uninstalling the + // last storage plugin is exactly when an orphaned database needs naming. + if (ids.length === 0 && !adminUrl) return; + if (!adminUrl) throw new Error(`bootstrap: PLUGIN_DB_ADMIN_URL must be set — these plugins declare storage: ${ids.join(", ")}`); + // Provisioned here, connected to from web: a different server means the role is created in one + // place and looked for in another, surfacing inside a plugin as "password authentication failed". + // Warned, not refused — web reaching a pooler that cannot run CREATE DATABASE is a legitimate split. + const mismatch = serverMismatch(adminUrl, env["PLUGIN_DB_URL"]); + if (mismatch) log.warn("PLUGIN_DB_ADMIN_URL and PLUGIN_DB_URL name different servers", { servers: mismatch }); + const result = await provision({ + adminUrl, + connectionLimit: resolvePluginDbConnectionLimit(env), + pluginIds: ids, + secret: resolvePluginDbSecret(env), + }); + if (result.provisioned.length > 0) log.info("plugin storage provisioned", { databases: result.provisioned.join(", ") }); + // Never dropped, so an uninstalled plugin's data outlives it — say so, or nobody can find it. + if (result.orphans.length > 0) { + log.warn("plugin databases no installed plugin claims", { databases: result.orphans.join(", ") }); + } +} + +// Describes the disagreement, or null when they agree (or when web's URL is unset — that is web's +// own boot error to raise, naming the plugin that wanted storage). +export function serverMismatch(adminUrl: string, webUrl: string | undefined): string | null { + if (!webUrl) return null; + const [admin, web] = [safeHostPort(adminUrl), safeHostPort(webUrl)]; + if (admin === null || web === null || admin === web) return null; // a malformed URL fails in config.ts + return `${admin} vs ${web}`; +} + +function safeHostPort(url: string): string | null { + try { + const parsed = new URL(url); + return `${parsed.hostname}:${parsed.port || "5432"}`; + } catch { + return null; + } +} + +// Seed every discovered plugin's declared permission names (plus any ADMIN_PERMISSIONS), so the +// shipped example — and any dropped-in plugin — works for the demo admin without a host edit. +async function seedAdminAndPermissions(env: Env, plugins: Plugin[], log: Log): Promise { + const declared = declaredPermissions(plugins).map((decl) => decl.name); + const { ignored, permissions } = seedPermissions(env["ADMIN_PERMISSIONS"], declared); + if (ignored.length > 0) { + log.warn("ignoring ADMIN_PERMISSIONS entries that are not :", { ignored: ignored.join(", ") }); + } + const email = env["ADMIN_EMAIL"] ?? "admin@plainpages.local"; + const password = env["ADMIN_PASSWORD"] ?? "admin"; + const result = await seedAdmin({ + email, + fetchImpl: tracedFetch, + ketoWriteUrl: env["KETO_WRITE_URL"] ?? "http://keto:4467", + kratosAdminUrl: env["KRATOS_ADMIN_URL"] ?? "http://kratos:4434", + password, + permissions, + }); + log.info("admin seeded", { created: result.created, id: result.id, permissions: result.permissions.join(", ") }); + // The banner is human-facing UX (the first-run "you're ready" block), not a log event — print raw. + console.log(firstRunBanner({ appUrl: env["APP_URL"] ?? "http://localhost:3000", email, password })); +} + if (process.argv[1] === fileURLToPath(import.meta.url)) await main(); diff --git a/src/compose.test.ts b/src/compose.test.ts index 1a355c1..fc930ed 100644 --- a/src/compose.test.ts +++ b/src/compose.test.ts @@ -44,10 +44,11 @@ test("long-running Ory services declare readiness healthchecks", () => { `${svc} probes :${port}/health/ready`); }); -test("web waits for kratos, keto and hydra to be healthy before starting", () => { +test("web waits for kratos, keto, hydra and postgres to be healthy before starting", () => { assert.match(webBlock, /depends_on:/, "web declares dependencies"); - // hydra: the OAuth2 login/consent handler talks to its admin API. - for (const svc of ["kratos", "keto", "hydra"]) + // hydra: the OAuth2 login/consent handler talks to its admin API. postgres: a plugin declaring + // `storage` opens its connection in onBoot, before the server listens. + for (const svc of ["kratos", "keto", "hydra", "postgres"]) assert.match(webBlock, new RegExp(`${svc}:\\s*\\n\\s*condition:\\s*service_healthy`), `web waits for ${svc} healthy`); }); @@ -78,6 +79,21 @@ test("prod base supplies the app secret via env and mounts no source; dev overri assert.match(compose, /POSTGRES_PASSWORD:\s*\$\{POSTGRES_PASSWORD\b/, "postgres password via env"); }); +test("the provisioning superuser DSN reaches bootstrap only, never web", () => { + // web runs plugin code, which can read its own environment — so the credentials that may CREATE + // DATABASE/ROLE must never be there. web gets the credential-free base URL and derives each + // plugin's own password from the shared secret instead. + const boot = compose.slice(compose.indexOf("\n bootstrap:")); + const overrideWeb = override.slice(override.indexOf("\n web:"), override.indexOf("\n bootstrap:")); + assert.match(boot, /PLUGIN_DB_ADMIN_URL:/, "bootstrap is given the superuser DSN"); + // Reordering the override's services would empty this slice, and every doesNotMatch below would + // then pass against "". + assert.ok(overrideWeb.includes("PLUGIN_DB_URL"), "sliced the dev override's web block"); + for (const [name, block] of [["base", webBlock], ["dev override", overrideWeb]] as const) + assert.doesNotMatch(block, /PLUGIN_DB_ADMIN_URL/, `${name} web never sees it`); + assert.match(webBlock, /PLUGIN_DB_URL:\s*\$\{PLUGIN_DB_URL/, "base wires web's base URL from env"); +}); + test("a one-shot bootstrap seeds the stack before web starts", () => { // MVP bar: `bootstrap` runs after kratos+keto are healthy, seeds the admin + // JWKS, then exits; web waits for it to complete. Live seeding is boot-verified. diff --git a/src/config.test.ts b/src/config.test.ts index 62ec8ce..62a00d5 100644 --- a/src/config.test.ts +++ b/src/config.test.ts @@ -1,6 +1,6 @@ import assert from "node:assert/strict"; import { test } from "node:test"; -import { loadConfig } from "./config.ts"; +import { loadConfig, resolvePluginDbConnectionLimit, resolvePluginDbSecret } from "./config.ts"; // Explicit secure-secret enforcement (no environment sniffing): secrets are the only // thing a hardened deploy must supply. @@ -9,6 +9,43 @@ const secureEnv = { REQUIRE_SECURE_SECRETS: "true", }; +// web reads the secret through loadConfig and bootstrap through resolvePluginDbSecret; the two +// deriving different passwords is invisible until a plugin's connection is refused at boot. Compose +// passes an unset variable through as "", which is the case that actually drifted. +test("web and bootstrap resolve the same plugin storage secret", () => { + for (const env of [{}, { PLUGIN_DB_SECRET: "" }, { PLUGIN_DB_SECRET: "a-real-secret" }]) { + assert.equal(loadConfig(env).pluginDbSecret, resolvePluginDbSecret(env), `for ${JSON.stringify(env)}`); + } + assert.match(resolvePluginDbSecret({ PLUGIN_DB_SECRET: "" }), /dev-insecure/); // empty is unset, not a secret +}); + +// bootstrap writes these passwords into Postgres, so it must refuse the publicly-known throwaway +// before creating a role with one — not leave web to notice afterwards. +test("bootstrap refuses a missing, empty or throwaway plugin storage secret when hardened", () => { + const hardened = { REQUIRE_SECURE_SECRETS: "true" }; + for (const secret of [undefined, "", "dev-insecure-plugin-db-secret"]) { + const env = secret === undefined ? hardened : { ...hardened, PLUGIN_DB_SECRET: secret }; + assert.throws(() => resolvePluginDbSecret(env), /PLUGIN_DB_SECRET/, `for ${JSON.stringify(secret)}`); + } + assert.equal(resolvePluginDbSecret({ ...hardened, PLUGIN_DB_SECRET: "a-real-secret" }), "a-real-secret"); +}); + +// buildCredentials overwrites the userinfo, so a pasted admin DSN would *work* — and leave a +// privileged password in the process that runs plugin code. Refusing it is the whole guard. +test("PLUGIN_DB_URL carrying credentials is refused, not silently overwritten", () => { + assert.throws(() => loadConfig({ PLUGIN_DB_URL: "postgres://root:hunter2@db:5432/ory" }), /no username or password/); + assert.throws(() => loadConfig({ PLUGIN_DB_URL: "postgres://root@db:5432" }), /no username or password/); + assert.equal(loadConfig({ PLUGIN_DB_URL: "postgres://db:5432" }).pluginDbUrl, "postgres://db:5432"); + assert.equal(loadConfig({}).pluginDbUrl, undefined); // unset ⇒ storage off +}); + +test("the per-role connection ceiling defaults to 10 and rejects nonsense", () => { + assert.equal(resolvePluginDbConnectionLimit({}), 10); + assert.equal(resolvePluginDbConnectionLimit({ PLUGIN_DB_CONNECTION_LIMIT: "25" }), 25); + assert.throws(() => resolvePluginDbConnectionLimit({ PLUGIN_DB_CONNECTION_LIMIT: "0" }), /positive integer/); + assert.throws(() => resolvePluginDbConnectionLimit({ PLUGIN_DB_CONNECTION_LIMIT: "ten" }), /positive integer/); +}); + test("loads dev defaults when the environment is empty", () => { const c = loadConfig({}); assert.equal(c.port, 3000); diff --git a/src/config.ts b/src/config.ts index 137fff7..c46f622 100644 --- a/src/config.ts +++ b/src/config.ts @@ -6,6 +6,35 @@ export const LOG_LEVELS = ["error", "warn", "info", "verbose", "debug", "silly", "none"] as const; export type LogLevel = (typeof LOG_LEVELS)[number]; +const DEV_PLUGIN_DB_SECRET = "dev-insecure-plugin-db-secret"; + +// The one resolution both processes use — they must agree exactly, or web connects with a password +// the role was never given. Compose passes an unset variable through as "", so empty means unset. +// `enforce` says whether storage is actually in play: web once PLUGIN_DB_URL is configured, +// bootstrap once a plugin declares storage. Enforced, the throwaway is refused — bootstrap is what +// writes these passwords into Postgres, so it must refuse *before* creating a role with one. +export function resolvePluginDbSecret(env: Env, enforce?: boolean): string { + return readSecret(env, "PLUGIN_DB_SECRET", DEV_PLUGIN_DB_SECRET, enforce ?? readBool(env, "REQUIRE_SECURE_SECRETS", false)); +} + +// Only bootstrap provisions, so only bootstrap reads this; env still gets read in one place. +export function resolvePluginDbConnectionLimit(env: Env): number { + return readPosInt(env, "PLUGIN_DB_CONNECTION_LIMIT", 10); +} + +// PLUGIN_DB_URL is web's, and web must never hold credentials that outrank a plugin's own role. +// Pasting the admin DSN here would otherwise work — buildCredentials overwrites the userinfo — and +// leave a superuser password in the environment plugin code can read. +function readCredentiallessUrl(env: Env, key: string): string | undefined { + const value = readOptionalUrl(env, key); + if (value === undefined) return undefined; + const url = new URL(value); + if (url.username || url.password) { + throw new Error(`config: ${key} must carry no username or password — each plugin connects as its own role`); + } + return value; +} + export interface Config { appUrl: string | undefined; // canonical public URL; set ⇒ off-host visitors are redirected here. Unset ⇒ no redirect (explicit toggle) cacheTemplates: boolean; @@ -24,6 +53,8 @@ export interface Config { oryTimeoutSec: number; // per-call timeout for outbound Kratos/Keto/Hydra fetches (bounds a hung Ory) otlpEndpoint: string | undefined; // OTLP/HTTP collector base URI; unset ⇒ console-only (no export) otlpProtocol: "http/json" | "http/protobuf"; // OTLP wire format (protobuf for json-averse collectors) + pluginDbSecret: string; // derives each plugin's database password (src/plugin-host/storage.ts) + pluginDbUrl: string | undefined; // credential-free Postgres base URL; unset ⇒ plugin storage is off port: number; revocationDenylist: boolean; // enable the optional instant permission/session revoke denylist revocationTtlSec: number; // how long a revoke entry lives; keep ≥ tokenizer TTL + clock skew @@ -150,6 +181,12 @@ export function loadConfig(env: Env = process.env): Config { oryTimeoutSec: readPosInt(env, "ORY_TIMEOUT_SEC", 5), otlpEndpoint: readOptionalUrl(env, "OTLP_ENDPOINT"), otlpProtocol: readEnum(env, "OTLP_PROTOCOL", ["http/json", "http/protobuf"] as const, "http/json"), + // Per-plugin storage. PLUGIN_DB_URL carries the server and its connection parameters but no + // credentials: the superuser DSN that provisions stays in bootstrap, so a plugin cannot read it + // out of web's environment. Unset ⇒ storage is off and a plugin declaring it fails loud at boot, + // which is also why the secret is only enforced once a URL is configured. + pluginDbSecret: resolvePluginDbSecret(env, requireSecure && Boolean(env["PLUGIN_DB_URL"])), + pluginDbUrl: readCredentiallessUrl(env, "PLUGIN_DB_URL"), port: readPort(env), // Optional instant-revoke, off by default. When on, an admin deactivate/delete or permission // change revokes the subject's live tokens at once; the entry lives ttl seconds (≥ the 10m diff --git a/src/plugin-host/discovery.test.ts b/src/plugin-host/discovery.test.ts index 8bdca5a..91966db 100644 --- a/src/plugin-host/discovery.test.ts +++ b/src/plugin-host/discovery.test.ts @@ -27,13 +27,19 @@ test("a missing plugins/ dir means zero plugins, not an error (clean clone)", as }); test("discovers each folder's manifest, sorted, id derived from the folder name", async (t) => { - const dir = scaffold(t, { "beta/plugin.ts": full("beta"), "alpha/plugin.ts": full("alpha") }); + const dir = scaffold(t, { + "beta/plugin.ts": full("beta"), + "alpha/plugin.ts": full("alpha"), + "gamma/plugin.ts": `export default { apiVersion: "1.0.0", storage: true };`, + }); const plugins = await discoverPlugins({ dir }); - assert.deepEqual(plugins.map((p) => p.id), ["alpha", "beta"]); // deterministic order + assert.deepEqual(plugins.map((p) => p.id), ["alpha", "beta", "gamma"]); // deterministic order assert.equal(plugins[0]?.apiVersion, "1.0.0"); assert.equal(plugins[0]?.nav?.[0]?.label, "alpha"); assert.equal(typeof plugins[0]?.routes?.[0]?.handler, "function"); // handlers survive import + assert.equal(plugins[0]?.storage, undefined); // storage is opt-in, never assumed + assert.equal(plugins[2]?.storage, true); }); // Every per-plugin problem and every error-level conflict aborts boot with a message naming it. @@ -48,6 +54,9 @@ const badCases: Array<{ name: string; files: Record; match: RegE { name: "non-array routes", files: { "weird/plugin.ts": `export default { apiVersion: "1.0.0", routes: "nope" };` }, match: /weird.*routes.*array/s }, { name: "non-function home", files: { "weirdhome/plugin.ts": `export default { apiVersion: "1.0.0", home: "nope" };` }, match: /weirdhome.*home.*function/s }, { name: "non-function dashboard", files: { "weirddash/plugin.ts": `export default { apiVersion: "1.0.0", dashboard: "nope" };` }, match: /weirddash.*dashboard.*function/s }, + { name: "non-boolean storage", files: { "weirdstore/plugin.ts": `export default { apiVersion: "1.0.0", storage: "postgres://db" };` }, match: /weirdstore.*storage.*boolean/s }, + // The folder name becomes a Postgres identifier, which truncates past 63 bytes. + { name: "a storage plugin whose folder name overflows a Postgres identifier", files: { [`${"a".repeat(57)}/plugin.ts`]: `export default { apiVersion: "1.0.0", storage: true };` }, match: /storage.*56 characters/s }, { name: "reserved dashboard id shadows the gated dashboard", files: { "dashboard/plugin.ts": full("dashboard") }, match: /dashboard.*reserved/s }, { name: "duplicate nav id across plugins", files: { "a/plugin.ts": full("a").replace("a:root", "dup"), "b/plugin.ts": full("b").replace("b:root", "dup") }, match: /nav id "dup"/ }, { name: "a route marked public AND permission is contradictory", files: { "contra/plugin.ts": `export default { apiVersion: "1.0.0", routes: [{ method: "GET", path: "/", public: true, permission: "x:read", handler: () => ({ html: "x" }) }] };` }, match: /contra.*public.*permission/s }, diff --git a/src/plugin-host/discovery.ts b/src/plugin-host/discovery.ts index fd36dd7..d4e19c9 100644 --- a/src/plugin-host/discovery.ts +++ b/src/plugin-host/discovery.ts @@ -8,6 +8,7 @@ import { existsSync, readdirSync, readFileSync } from "node:fs"; import { dirname, join } from "node:path"; import { fileURLToPath, pathToFileURL } from "node:url"; import { checkApiVersion, findConflicts, isValidPermissionName, isValidPluginId, RESERVED_PLUGIN_IDS, type Plugin, type PluginManifest } from "./plugin.ts"; +import { isValidStoragePluginId, MAX_STORAGE_PLUGIN_ID_LENGTH } from "./storage.ts"; const rootDir = join(dirname(fileURLToPath(import.meta.url)), "..", ".."); @@ -66,6 +67,13 @@ export async function discoverPlugins(options: DiscoverOptions = {}): Promise { const calls: string[] = []; + const scoped: string[] = []; // each hook is handed a context built for its own plugin + const bootContextFor = (built: Plugin) => { scoped.push(built.id); return {}; }; await runBootHooks([ plugin("a", { onBoot: () => void calls.push("a") }), plugin("b", {}), // no onBoot → skipped plugin("c", { onBoot: async () => void calls.push("c") }), - ]); + ], bootContextFor); assert.deepEqual(calls, ["a", "c"]); + assert.deepEqual(scoped, ["a", "c"]); // and built only for the plugins that have one - await assert.rejects(runBootHooks([plugin("x", { onBoot: () => { throw new Error("boom"); } })]), /boom/); + await assert.rejects(runBootHooks([plugin("x", { onBoot: () => { throw new Error("boom"); } })], () => ({})), /boom/); }); test("runRequestHooks short-circuits on the first RouteResult (with its plugin); later hooks skipped", async () => { diff --git a/src/plugin-host/hooks.ts b/src/plugin-host/hooks.ts index 04fd1c6..0c6cd77 100644 --- a/src/plugin-host/hooks.ts +++ b/src/plugin-host/hooks.ts @@ -4,11 +4,15 @@ // entirely when no plugin declares the hook, so the no-hooks hot path stays free. import type { RequestContext } from "../http/context.ts"; -import type { Plugin, RouteResult } from "./plugin.ts"; +import type { BootContext, Plugin, RouteResult } from "./plugin.ts"; -// After discovery, before the server listens. A throw aborts boot. -export async function runBootHooks(plugins: Plugin[]): Promise { - for (const plugin of plugins) await plugin.hooks?.onBoot?.(); +// After discovery, before the server listens. A throw aborts boot. Each hook gets a context built +// for its own plugin, so one plugin is never handed another's storage credentials. +export async function runBootHooks(plugins: Plugin[], bootContextFor: (plugin: Plugin) => BootContext): Promise { + for (const plugin of plugins) { + const onBoot = plugin.hooks?.onBoot; + if (onBoot) await onBoot(bootContextFor(plugin)); + } } // Before route matching. The first hook to return a RouteResult short-circuits the request — its diff --git a/src/plugin-host/plugin-api.ts b/src/plugin-host/plugin-api.ts index b2a7bd7..1777480 100644 --- a/src/plugin-host/plugin-api.ts +++ b/src/plugin-host/plugin-api.ts @@ -5,7 +5,10 @@ // a plugin should import from here, never reach into deeper modules. See README.md → Building plugins. export { definePlugin, isValidPermissionName } from "./plugin.ts"; -export type { HttpMethod, Plugin, PluginHooks, PluginManifest, PermissionDecl, Route, RouteHandler, RouteResult } from "./plugin.ts"; +export type { BootContext, HttpMethod, Plugin, PluginHooks, PluginManifest, PermissionDecl, Route, RouteHandler, RouteResult } from "./plugin.ts"; +// A plugin's own database, handed to onBoot when the manifest sets `storage`. Credentials, not a +// client — the plugin depends on whichever driver it prefers (README → Plugin storage). +export type { StorageCredentials } from "./storage.ts"; export type { RequestContext, User } from "../http/context.ts"; export type { PageChrome } from "../ui/chrome.ts"; export type { NavNode } from "../ui/nav.ts"; diff --git a/src/plugin-host/plugin.ts b/src/plugin-host/plugin.ts index 9b3d9de..18151af 100644 --- a/src/plugin-host/plugin.ts +++ b/src/plugin-host/plugin.ts @@ -6,6 +6,7 @@ import type { RequestContext } from "../http/context.ts"; import type { NavNode } from "../ui/nav.ts"; +import type { StorageCredentials } from "./storage.ts"; // Bump major on a breaking manifest/handler change, minor on an additive one. export const HOST_API_VERSION = "1.0.0"; @@ -60,9 +61,14 @@ export function declaredPermissions(plugins: Plugin[]): PermissionDecl[] { return [...byName.values()].sort((a, b) => a.name.localeCompare(b.name)); } +// What onBoot receives. A hook declaring no parameter stays valid, so this may grow additively. +export interface BootContext { + storage?: StorageCredentials; // this plugin's own database; present iff the manifest declared `storage` +} + // Optional hooks on system actions. Crash-isolation is a non-goal — a throwing hook fails loud. export interface PluginHooks { - onBoot?: () => Promise | void; // after discovery, before the server listens + onBoot?: (host: BootContext) => Promise | void; // after discovery, before the server listens onRequest?: (ctx: RequestContext) => Promise | RouteResult | void; // may short-circuit onResponse?: (ctx: RequestContext, result: RouteResult | null) => Promise | void; } @@ -80,6 +86,9 @@ export interface PluginManifest { nav?: NavNode[]; // fragment merged into the menu (composeNav); node `icon` is a Lucide sprite id (src/ui/icons.ts), node ids must be globally unique permissions?: PermissionDecl[]; routes?: Route[]; + // Ask for a Postgres database of this plugin's own; its credentials arrive on onBoot's BootContext. + // The host provisions and locks it down but owns no schema inside it, and never drops it. + storage?: boolean; } // A discovered plugin: the manifest plus the `id` the host read from the folder name. Mounted diff --git a/src/plugin-host/storage-provisioning.ts b/src/plugin-host/storage-provisioning.ts new file mode 100644 index 0000000..c58d422 --- /dev/null +++ b/src/plugin-host/storage-provisioning.ts @@ -0,0 +1,44 @@ +// The connecting half of plugin storage: runs the DDL storage.ts plans. Imported by bootstrap +// alone — the only process holding superuser credentials, which is why the driver stops here. + +import postgres from "postgres"; +import { derivePassword, orphanNames, provisionSql, storageName } from "./storage.ts"; + +export interface ProvisionOptions { + adminUrl: string; // needs CREATEDB + CREATEROLE, not superuser + connectionLimit: number; + pluginIds: string[]; + secret: string; +} + +export interface ProvisionResult { + orphans: string[]; // a plugin_ database no installed plugin claims; reported, never dropped + provisioned: string[]; +} + +export async function provisionStorage(options: ProvisionOptions): Promise { + // Notices are left to surface: a REVOKE the account cannot perform only *warns*, and silencing + // that would mean reporting a locked-down database that is still open to PUBLIC. + const sql = postgres(options.adminUrl, { connect_timeout: 10, max: 1 }); + try { + const provisioned: string[] = []; + for (const pluginId of options.pluginIds) { + const name = storageName(pluginId); + const [role] = await sql`SELECT 1 FROM pg_roles WHERE rolname = ${name}`; + const [database] = await sql`SELECT 1 FROM pg_database WHERE datname = ${name}`; + const plan = provisionSql({ + connectionLimit: options.connectionLimit, + databaseExists: database !== undefined, + name, + password: derivePassword(options.secret, pluginId), + roleExists: role !== undefined, + }); + for (const statement of plan) await sql.unsafe(statement); // provisionSql quotes what it interpolates + provisioned.push(name); + } + const existing = await sql<{ datname: string }[]>`SELECT datname FROM pg_database`; + return { orphans: orphanNames(existing.map((row) => row.datname), provisioned), provisioned }; + } finally { + await sql.end({ timeout: 5 }); // a wedged connection would otherwise hang the boot web waits on + } +} diff --git a/src/plugin-host/storage.test.ts b/src/plugin-host/storage.test.ts new file mode 100644 index 0000000..3bc49c1 --- /dev/null +++ b/src/plugin-host/storage.test.ts @@ -0,0 +1,220 @@ +// Guards the per-plugin storage rules: the shared database/role name, the derived password, the DSN +// a plugin receives and the provisioning statements. The integration test runs only when a superuser +// DSN is supplied, so the unit suite needs no Postgres. +import { test } from "node:test"; +import assert from "node:assert/strict"; +import postgres from "postgres"; +import { provisionStorage } from "./storage-provisioning.ts"; +import { + buildCredentials, + derivePassword, + isValidStoragePluginId, + MAX_STORAGE_PLUGIN_ID_LENGTH, + orphanNames, + provisionSql, + quoteIdentifier, + quoteLiteral, + storageName, + storagePluginIds, +} from "./storage.ts"; + +const SECRET = "a-test-secret"; + +test("the database and the role share one plugin_-prefixed name", () => { + assert.equal(storageName("things"), "plugin_things"); + assert.equal(storageName("my-plugin"), "plugin_my-plugin"); +}); + +test("a storage plugin's id must leave the identifier under Postgres' 63 bytes", () => { + assert.equal(MAX_STORAGE_PLUGIN_ID_LENGTH, 56); // 63 - "plugin_" + assert.ok(isValidStoragePluginId("a".repeat(MAX_STORAGE_PLUGIN_ID_LENGTH))); + assert.ok(!isValidStoragePluginId("a".repeat(MAX_STORAGE_PLUGIN_ID_LENGTH + 1))); +}); + +test("the password is derived, so the same one is reachable without storing it", () => { + const derived = derivePassword(SECRET, "things"); + assert.equal(derived, derivePassword(SECRET, "things")); + assert.notEqual(derived, derivePassword(SECRET, "other")); + assert.notEqual(derived, derivePassword("a-rotated-secret", "things")); + assert.match(derived, /^[A-Za-z0-9_-]{43}$/); // base64url of 32 bytes — needs no escaping in a DSN +}); + +test("credentials name the plugin's own database, user and password", () => { + const credentials = buildCredentials("postgres://postgres:5432", "things", SECRET); + assert.deepEqual(credentials, { + database: "plugin_things", + host: "postgres", + password: derivePassword(SECRET, "things"), + port: 5432, + url: `postgres://plugin_things:${derivePassword(SECRET, "things")}@postgres:5432/plugin_things`, + user: "plugin_things", + }); +}); + +test("the base URL's connection parameters survive into the DSN", () => { + const credentials = buildCredentials("postgres://db.example?sslmode=require", "things", SECRET); + assert.equal(credentials.port, 5432); // absent ⇒ Postgres' default, never NaN + assert.equal(credentials.host, "db.example"); + assert.match(credentials.url, /@db\.example\/plugin_things\?sslmode=require$/); +}); + +test("quoting doubles an embedded quote", () => { + assert.equal(quoteIdentifier('we"ird'), '"we""ird"'); + assert.equal(quoteLiteral("we'ird"), "'we''ird'"); +}); + +const ATTRIBUTES = "LOGIN NOCREATEDB NOCREATEROLE CONNECTION LIMIT 10"; + +test("only the plugins that asked for storage are provisioned", () => { + assert.deepEqual( + storagePluginIds([{ apiVersion: "1.0.0", id: "a", storage: true }, { apiVersion: "1.0.0", id: "b" }, { apiVersion: "1.0.0", id: "c", storage: true }]), + ["a", "c"], + ); +}); + +test("an orphan is a plugin_ database no installed plugin claims", () => { + const existing = ["plugin_gone", "plugin_here", "kratos", "ory"]; + assert.deepEqual(orphanNames(existing, ["plugin_here"]), ["plugin_gone"]); // Ory's are not ours to report + assert.deepEqual(orphanNames(existing, ["plugin_here", "plugin_gone"]), []); +}); + +test("provisioning creates the role and the database when neither exists", () => { + const plan = { connectionLimit: 10, databaseExists: false, name: "plugin_things", password: "pw", roleExists: false }; + assert.deepEqual(provisionSql(plan), [ + `CREATE ROLE "plugin_things" ${ATTRIBUTES} PASSWORD 'pw'`, + `GRANT "plugin_things" TO CURRENT_USER`, // else a CREATEROLE (non-superuser) account cannot own it + `CREATE DATABASE "plugin_things" OWNER "plugin_things"`, + `REVOKE ALL ON DATABASE "plugin_things" FROM PUBLIC`, + `GRANT ALL PRIVILEGES ON DATABASE "plugin_things" TO "plugin_things"`, + ]); +}); + +// Re-asserting the attributes, not just the password, is what makes "idempotent" mean the role +// cannot drift — a CREATEDB granted by hand out of band is taken back on the next boot. +test("re-provisioning re-asserts every attribute and creates nothing twice", () => { + const plan = { connectionLimit: 10, databaseExists: true, name: "plugin_things", password: "rotated", roleExists: true }; + assert.deepEqual(provisionSql(plan), [ + `ALTER ROLE "plugin_things" WITH ${ATTRIBUTES} PASSWORD 'rotated'`, + `REVOKE ALL ON DATABASE "plugin_things" FROM PUBLIC`, + `GRANT ALL PRIVILEGES ON DATABASE "plugin_things" TO "plugin_things"`, + ]); +}); + +// The limit is interpolated unquoted, and Postgres reads a negative one as "unlimited". +test("a connection limit that is not a positive integer is refused, not interpolated", () => { + const plan = { databaseExists: false, name: "plugin_things", password: "pw", roleExists: false }; + for (const connectionLimit of [1.5, 0, -1, Number.NaN]) { + assert.throws(() => provisionSql({ ...plan, connectionLimit }), /positive integer/, `for ${connectionLimit}`); + } +}); + +// --- Integration: the statements above, against a real Postgres ----------------------- +// Opt-in via PLUGIN_DB_ADMIN_URL (a superuser DSN); the unit gate runs no Postgres. What the unit +// tests cannot prove lives here: the owner may create tables, and a peer role is locked out. + +const ADMIN_URL = process.env["PLUGIN_DB_ADMIN_URL"] ?? ""; +const integration = ADMIN_URL ? {} : { skip: "set PLUGIN_DB_ADMIN_URL to a superuser DSN to run" }; + +function baseUrlOf(adminUrl: string): string { + const url = new URL(adminUrl); + url.username = ""; + url.password = ""; + url.pathname = ""; + return url.href; +} + +async function queryAs(url: string, statement: string): Promise { + const sql = postgres(url, { connect_timeout: 10, max: 1, onnotice: () => {} }); + try { + return await sql.unsafe(statement); + } finally { + await sql.end(); + } +} + +// Drops what a previous run may have left behind: `finally` does not survive a SIGKILL or a +// cancelled CI job, and the leftovers would otherwise fail every later run on the same server. +async function dropStorage(admin: postgres.Sql, ids: string[]): Promise { + for (const id of ids) { + const name = quoteIdentifier(storageName(id)); + await admin.unsafe(`DROP DATABASE IF EXISTS ${name} WITH (FORCE)`); + await admin.unsafe(`DROP ROLE IF EXISTS ${name}`); + } +} + +test("provisions a database its plugin can use and a peer plugin cannot reach", integration, async () => { + const ids = ["storage-itest-a", "storage-itest-b"]; + const base = baseUrlOf(ADMIN_URL); + const admin = postgres(ADMIN_URL, { connect_timeout: 10, max: 1, onnotice: () => {} }); + try { + await dropStorage(admin, ids); + await provisionStorage({ adminUrl: ADMIN_URL, connectionLimit: 10, pluginIds: ids, secret: SECRET }); + + const owner = buildCredentials(base, "storage-itest-a", SECRET); + await queryAs(owner.url, "CREATE TABLE IF NOT EXISTS notes (body text)"); + await queryAs(owner.url, "INSERT INTO notes (body) VALUES ('persisted')"); + const rows = (await queryAs(owner.url, "SELECT body FROM notes")) as { body: string }[]; + assert.deepEqual(rows.map((row) => row.body), ["persisted"]); + + // A peer holds valid credentials for its OWN database and still cannot reach this one. + const peer = new URL(buildCredentials(base, "storage-itest-b", SECRET).url); + peer.pathname = `/${storageName("storage-itest-a")}`; + await assert.rejects(queryAs(peer.href, "SELECT 1"), /permission denied|not permitted/i); + + // Re-running is idempotent, and a rotated secret lands on the existing role. + const rerun = await provisionStorage({ adminUrl: ADMIN_URL, connectionLimit: 10, pluginIds: ids, secret: "a-rotated-secret" }); + // Scoped to this test's own ids: another plugin's database on the same server is not this + // test's business, and asserting otherwise would make the suite order-dependent. + for (const id of ids) assert.ok(!rerun.orphans.includes(storageName(id)), `${id} is still installed`); + const rotated = buildCredentials(base, "storage-itest-a", "a-rotated-secret"); + const kept = (await queryAs(rotated.url, "SELECT body FROM notes")) as { body: string }[]; + assert.deepEqual(kept.map((row) => row.body), ["persisted"]); // rotating the secret keeps the data + await assert.rejects(queryAs(owner.url, "SELECT 1"), /password authentication failed/i); + + // Uninstalling drops nothing, so what is left behind must be named — including when the LAST + // storage plugin goes and there is nothing left to provision. + const uninstalled = await provisionStorage({ adminUrl: ADMIN_URL, connectionLimit: 10, pluginIds: [], secret: "a-rotated-secret" }); + for (const id of ids) assert.ok(uninstalled.orphans.includes(storageName(id)), `${id}'s database is reported`); + } finally { + try { + await dropStorage(admin, ids); + } finally { + await admin.end({ timeout: 5 }); // its own finally, or a failed DROP leaks the connection + } + } +}); + +// README tells an operator CREATEDB + CREATEROLE is enough and superuser is more than it needs. +// That is a promise about their production credentials, so prove it rather than assert it. +test("provisions through a CREATEDB + CREATEROLE account, without superuser", integration, async () => { + const pluginId = "storage-itest-lowpriv"; + const provisioner = "storage-itest-provisioner"; + const admin = postgres(ADMIN_URL, { connect_timeout: 10, max: 1, onnotice: () => {} }); + try { + // The fresh provisioner below holds no ADMIN option on a role an earlier run left behind, so a + // leftover would fail the ALTER branch rather than the code being wrong. + await dropStorage(admin, [pluginId]); + await admin.unsafe(`DROP ROLE IF EXISTS ${quoteIdentifier(provisioner)}`); + await admin.unsafe(`CREATE ROLE ${quoteIdentifier(provisioner)} LOGIN CREATEDB CREATEROLE PASSWORD 'itest-provisioner'`); + const asProvisioner = new URL(ADMIN_URL); + asProvisioner.username = provisioner; + asProvisioner.password = "itest-provisioner"; + const provision = () => provisionStorage({ adminUrl: asProvisioner.href, connectionLimit: 10, pluginIds: [pluginId], secret: SECRET }); + await provision(); + // Twice: the second run takes the ALTER branch, where naming a superuser-only attribute would + // fail — i.e. every redeploy after the one that worked. + await provision(); + + const owner = buildCredentials(baseUrlOf(ADMIN_URL), pluginId, SECRET); + await queryAs(owner.url, "CREATE TABLE IF NOT EXISTS notes (body text)"); + const rows = (await queryAs(owner.url, "SELECT 1 AS ok")) as { ok: number }[]; + assert.deepEqual(rows.map((row) => row.ok), [1]); // the plugin owns and can use what it was given + } finally { + try { + await dropStorage(admin, [pluginId]); + await admin.unsafe(`DROP ROLE IF EXISTS ${quoteIdentifier(provisioner)}`); + } finally { + await admin.end({ timeout: 5 }); + } + } +}); diff --git a/src/plugin-host/storage.ts b/src/plugin-host/storage.ts new file mode 100644 index 0000000..3516a68 --- /dev/null +++ b/src/plugin-host/storage.ts @@ -0,0 +1,90 @@ +// Per-plugin Postgres storage: the naming, credential and DDL rules (README → Plugin storage). +// Pure — the connecting half lives in storage-provisioning.ts, so `web` never loads a driver. + +import { createHmac } from "node:crypto"; +import type { Plugin } from "./plugin.ts"; + +// Database and role share one name, so reconnecting needs nothing looked up. The prefix also keeps +// a plugin id from ever naming an Ory database. +export const NAME_PREFIX = "plugin_"; + +// Postgres truncates an identifier at 63 bytes, which would silently collide two long ids. +export const MAX_STORAGE_PLUGIN_ID_LENGTH = 63 - NAME_PREFIX.length; + +export interface StorageCredentials { + database: string; + host: string; + password: string; + port: number; + url: string; + user: string; +} + +export function storageName(pluginId: string): string { + return `${NAME_PREFIX}${pluginId}`; +} + +export function isValidStoragePluginId(pluginId: string): boolean { + return Buffer.byteLength(pluginId) <= MAX_STORAGE_PLUGIN_ID_LENGTH; // Postgres counts bytes, not characters +} + +export function storagePluginIds(plugins: Plugin[]): string[] { + return plugins.filter((plugin) => plugin.storage).map((plugin) => plugin.id); +} + +// Derived, never stored — which is what keeps the host free of state it would have to persist. +// Whoever holds the secret holds every plugin's database. +export function derivePassword(secret: string, pluginId: string): string { + return createHmac("sha256", secret).update(pluginId).digest("base64url"); +} + +// `baseUrl` names the server and its connection parameters, and carries no credentials of its own. +export function buildCredentials(baseUrl: string, pluginId: string, secret: string): StorageCredentials { + const name = storageName(pluginId); + const password = derivePassword(secret, pluginId); + const url = new URL(baseUrl); + url.username = name; + url.password = password; + url.pathname = `/${name}`; + return { database: name, host: url.hostname, password, port: Number(url.port) || 5432, url: url.href, user: name }; +} + +// CREATE ROLE/DATABASE bind no parameters, so the name and password are quoted into the statement. +export function quoteIdentifier(name: string): string { + return `"${name.replaceAll('"', '""')}"`; +} + +export function quoteLiteral(value: string): string { + return `'${value.replaceAll("'", "''")}'`; +} + +export function orphanNames(existing: string[], provisioned: string[]): string[] { + return existing.filter((name) => name.startsWith(NAME_PREFIX) && !provisioned.includes(name)).sort(); +} + +export interface ProvisionPlan { + connectionLimit: number; + databaseExists: boolean; + name: string; + password: string; + roleExists: boolean; +} + +export function provisionSql(plan: ProvisionPlan): string[] { + // Interpolated unquoted, and Postgres reads a negative limit as "unlimited" — the opposite of the point. + if (!Number.isSafeInteger(plan.connectionLimit) || plan.connectionLimit < 1) { + throw new Error(`storage: connectionLimit must be a positive integer, got ${plan.connectionLimit}`); + } + const identifier = quoteIdentifier(plan.name); + // No NOSUPERUSER: naming SUPERUSER in an ALTER is superuser-only, and CREATE defaults to it anyway. + const attributes = `LOGIN NOCREATEDB NOCREATEROLE CONNECTION LIMIT ${plan.connectionLimit} PASSWORD ${quoteLiteral(plan.password)}`; + return [ + plan.roleExists ? `ALTER ROLE ${identifier} WITH ${attributes}` : `CREATE ROLE ${identifier} ${attributes}`, + // CREATE DATABASE ... OWNER needs SET ROLE on the owner, and PG16+ gives a CREATEROLE account + // ADMIN but *not* SET on the roles it creates — so it grants itself membership first. A + // superuser could skip this; issuing it anyway is what keeps a least-privilege account working. + ...(plan.databaseExists ? [] : [`GRANT ${identifier} TO CURRENT_USER`, `CREATE DATABASE ${identifier} OWNER ${identifier}`]), + `REVOKE ALL ON DATABASE ${identifier} FROM PUBLIC`, + `GRANT ALL PRIVILEGES ON DATABASE ${identifier} TO ${identifier}`, + ]; +} diff --git a/src/postgres.test.ts b/src/postgres.test.ts index 3ec70b3..e636e92 100644 --- a/src/postgres.test.ts +++ b/src/postgres.test.ts @@ -3,14 +3,46 @@ // verified by booting postgres in CI/e2e; this catches edits. import { test } from "node:test"; import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; +import { readdirSync, readFileSync } from "node:fs"; const read = (p: string) => readFileSync(new URL(`../${p}`, import.meta.url), "utf8"); const ORY_DATABASES = ["hydra", "keto", "kratos"]; // one DB per Ory service -test("init SQL gives each Ory service its own database", () => { +function sourceFiles(dir = "src"): string[] { + const out: string[] = []; + for (const entry of readdirSync(new URL(`../${dir}/`, import.meta.url), { withFileTypes: true })) { + if (entry.isDirectory()) out.push(...sourceFiles(`${dir}/${entry.name}`)); + else if (entry.name.endsWith(".ts")) out.push(`${dir}/${entry.name}`); + } + return out.sort(); +} + +test("init SQL gives each Ory service its own database, and leaves plugin databases to bootstrap", () => { const sql = read("ory/postgres/init/init.sql"); for (const db of ORY_DATABASES) { assert.match(sql, new RegExp(`CREATE DATABASE ${db}\\b`, "i"), `creates ${db}`); } + // This file runs once, on an empty data dir — a plugin database added here would never appear for + // a plugin dropped in later. bootstrap provisions them on every boot instead. + assert.doesNotMatch(sql, /plugin_/i, "no plugin database is seeded here"); + // PUBLIC keeps CONNECT unless it is revoked, which would put every plugin role on the auth plane. + for (const db of ORY_DATABASES) { + assert.match(sql, new RegExp(`REVOKE CONNECT ON DATABASE ${db} FROM PUBLIC`, "i"), `${db} is closed to PUBLIC`); + } +}); + +// AGENTS.md records that the driver runs the provisioning DDL in bootstrap and nothing else. A +// single value imported from the wrong module puts it in web's graph without changing behaviour, +// so nothing but this would notice. +test("the Postgres driver reaches bootstrap only, never web's import graph", () => { + const files = sourceFiles(); + assert.ok(files.length > 40, "walks the source tree"); + assert.deepEqual( + files.filter((f) => /^import .*"postgres"/m.test(read(f))), // an import line, not a mention of one + ["src/plugin-host/storage-provisioning.ts", "src/plugin-host/storage.test.ts"], + ); + assert.deepEqual( + files.filter((f) => !f.endsWith(".test.ts") && /from "[^"]*storage-provisioning\.ts"/.test(read(f))), + ["src/auth/bootstrap.ts"], + ); }); diff --git a/src/server.ts b/src/server.ts index d8d38f6..b489926 100644 --- a/src/server.ts +++ b/src/server.ts @@ -13,8 +13,13 @@ import { createKratosAdmin } from "./auth/kratos-admin.ts"; import { createKratosPublic } from "./auth/kratos-public.ts"; import { createLogger, tracedFetch } from "./logger.ts"; import { loadMenuConfig } from "./ui/menu-config.ts"; +import { buildCredentials, storagePluginIds, type StorageCredentials } from "./plugin-host/storage.ts"; const config = loadConfig(); // validates the env (incl. enforced secrets) — fails loud at boot +// The storage secret is in `config` now, so drop it from the environment before ANY plugin code +// runs: a plugin module's top level evaluates during discovery, long before onBoot. Defence in +// depth, not a boundary (AGENTS.md) — and only ever move this line earlier, never later. +delete process.env["PLUGIN_DB_SECRET"]; // App-level logger: structured, OTLP-capable when OTLP_ENDPOINT is set. The hot path clones it // per request for access logging + a trace span (src/http/app.ts); console-only otherwise. const log = createLogger({ format: config.logFormat, level: config.logLevel, otlpEndpoint: config.otlpEndpoint, otlpProtocol: config.otlpProtocol, serviceName: config.serviceName }); @@ -44,7 +49,28 @@ log.info("plugins discovered", { count: plugins.length, ids: plugins.map((p) => const i18n = createI18n(await loadI18n({ logger: log, pluginIds: plugins.map((p) => p.id) })); log.info("locales loaded", { locales: i18n.available.join(", ") }); -await runBootHooks(plugins); // plugin onBoot — after discovery, before listen; a throw aborts boot +// A plugin's database credentials are derived, never stored — so the only thing that can be missing +// is the server itself. Refuse at boot rather than at that plugin's first query, hours later. +const pluginDbUrl = config.pluginDbUrl; +const declaresStorage = storagePluginIds(plugins); +if (declaresStorage.length > 0 && pluginDbUrl === undefined) { + throw new Error(`config: PLUGIN_DB_URL must be set — these plugins declare storage: ${declaresStorage.join(", ")}`); +} + +const storageCredentials = new Map(); +if (pluginDbUrl !== undefined) { + for (const id of declaresStorage) storageCredentials.set(id, buildCredentials(pluginDbUrl, id, config.pluginDbSecret)); +} +// onBoot is the only way credentials are handed over, so without one the database is provisioned +// and unreachable. A warning, not a refusal — the plugin still works, it just cannot store anything. +const unreachable = plugins.filter((plugin) => plugin.storage && !plugin.hooks?.onBoot).map((plugin) => plugin.id); +if (unreachable.length > 0) log.warn("plugins declare storage but have no onBoot to receive it", { plugins: unreachable.join(", ") }); + +// plugin onBoot — after discovery, before listen; a throw aborts boot. +await runBootHooks(plugins, (plugin) => { + const storage = storageCredentials.get(plugin.id); + return storage ? { storage } : {}; +}); const server = createApp({ // Canonical-host redirect target (off-host GET/HEAD visitors are sent here). Opt-in: omitted unless diff --git a/todo.md b/todo.md index e624651..2723c0a 100644 --- a/todo.md +++ b/todo.md @@ -36,6 +36,7 @@ Prioritized. Overall verdict: architecture is sound; these are refinements. ## Finnished work +- [x] Give a plugin persistent storage: `storage: true` provisions a Postgres database + login role named `plugin_`, credentials arrive on `onBoot`, passwords are derived from `PLUGIN_DB_SECRET` rather than stored. - [x] Refuse a stray `package.json`/`node_modules` in `config/` by name, as plugin folders already are. - [x] Let Renovate reach the example plugins' manifests (`ignorePaths` overrides `config:recommended`). - [x] The seeded admin is granted each permission once — `seedPermissions` dedupes and the grant PUT is idempotent.