Give a plugin a Postgres database of its own #78
@@ -36,11 +36,13 @@ branch, create a PR and merge it when the CI/CD turns green.
|
||||
## Project priorities (do not erode)
|
||||
|
||||
1. **Simplicity** — prefer the solution that is easiest to understand, smallest, and most readable.
|
||||
2. **Few dependencies** — runtime deps stay minimal (today `ejs`, `lucide-static`, `@larvit/log`).
|
||||
Prefer the Node standard library; justify any new dependency; do not add frameworks. The app is
|
||||
**stateless — no database**. Auth/identity/OAuth are **Ory sidecar services** reached over their
|
||||
REST APIs with built-in `fetch` — no SDK. New capabilities ship as **plugin folders** under
|
||||
`plugins/` that fetch their data from upstream services, not as core code.
|
||||
2. **Few dependencies** — runtime deps stay minimal (today `ejs`, `lucide-static`, `@larvit/log`,
|
||||
`postgres`). Prefer the Node standard library; justify any new dependency; do not add frameworks.
|
||||
The **host is stateless — it owns no schema and stores nothing of its own**; a plugin may own a
|
||||
Postgres database, which the host provisions and never reads or writes. Auth/identity/OAuth are
|
||||
**Ory sidecar services** reached over their REST APIs with built-in `fetch` — no SDK. New
|
||||
capabilities ship as **plugin folders** under `plugins/` that get their data from an upstream
|
||||
service or their own database, not as core code.
|
||||
3. **Strict TypeScript** — `tsconfig.json` is strict (incl. `noUncheckedIndexedAccess`,
|
||||
`exactOptionalPropertyTypes`, `verbatimModuleSyntax`). Keep it that way. Prefer exact types;
|
||||
limit nullable and multi-option types.
|
||||
@@ -79,6 +81,16 @@ Revisit only if the stated reason stops holding.
|
||||
it into `/node_modules`, above every plugin scope. Never let a copy reach a plugin's own
|
||||
`node_modules`: two instances of the barrel break `instanceof` across the boundary, which
|
||||
`plugin-api.test.ts` guards by asserting both paths reach one module.
|
||||
- **Plugin storage hands over credentials, not a client** (README → Plugin storage). The host takes
|
||||
`postgres` to run the provisioning DDL in `bootstrap` and nothing else: it is never re-exported
|
||||
through the barrel, so no driver shape enters the contract and a plugin depends on whichever client
|
||||
it likes. Three properties hold the design together, so don't trade one away in isolation:
|
||||
passwords are `HMAC-SHA256(PLUGIN_DB_SECRET, id)` rather than stored, which is what keeps the host
|
||||
stateless; the superuser DSN reaches `bootstrap` only, because plugin code runs inside `web` and
|
||||
can read that process' environment (`src/compose.test.ts` guards the split); and provisioning
|
||||
never drops anything, so uninstalling a plugin cannot destroy data. Because the host's copy sits in
|
||||
the ambient `/node_modules`, a plugin can `import "postgres"` without declaring it — that is
|
||||
incidental, not a packaging promise, and a plugin must still depend on its own driver.
|
||||
- **`config/` is still a plain dir — no `package.json` of its own**, or `#menu-config` resolves
|
||||
against that instead and boot fails loud. An operator's menu override has no use for
|
||||
dependencies; if that changes, it needs the same package treatment.
|
||||
|
||||
+2
-1
@@ -2,7 +2,8 @@
|
||||
|
||||
A self-hostable foundation for server-rendered web apps — public or gated pages from a
|
||||
zero-JS design system, with a config-driven menu and auth/permissions (Ory) baked in.
|
||||
Every domain feature is a drop-in plugin folder; the app is stateless, no build step.
|
||||
Every domain feature is a drop-in plugin folder, with a Postgres database of its own if it wants
|
||||
one; the host itself is stateless, and there is no build step.
|
||||
|
||||
**Source, docs & issues: <https://gitea.larvit.se/larvit/plainpages>**
|
||||
([GitHub mirror](https://github.com/larvit/plainpages))
|
||||
|
||||
@@ -89,6 +89,7 @@ From here, render real pages against the app shell and fetch upstream data — s
|
||||
- [hooks](#hooks)
|
||||
- [where they live & mounting](#where-plugins-live-and-how-to-mount-them)
|
||||
- [dependencies](#plugin-dependencies)
|
||||
- [storage](#plugin-storage)
|
||||
- [local dev & test](#local-dev--test-story)
|
||||
- [The menu system](#the-menu-system)
|
||||
- [Building blocks](#building-blocks)
|
||||
@@ -106,7 +107,7 @@ From here, render real pages against the app shell and fetch upstream data — s
|
||||
- [security model](#security-model)
|
||||
- [Email](#email)
|
||||
- [Architecture](#architecture)
|
||||
- [Stateless](#stateless)
|
||||
- [Stateless core](#stateless-core)
|
||||
- [Testing](#testing)
|
||||
- [end-to-end](#end-to-end-playwright)
|
||||
- [the full gate](#the-full-gate-one-command)
|
||||
@@ -380,6 +381,7 @@ folder-derived `id` to produce the loaded `Plugin`.
|
||||
| `permissions` | no | Permissions this plugin gates on. See [Nav & permission gates](#nav--permission-gates). |
|
||||
| `routes` | no | See [Routes & handlers](#routes--handlers). |
|
||||
| `hooks` | no | See [Hooks](#hooks). |
|
||||
| `storage` | no | `true` ⇒ the host provisions a Postgres database and login role for this plugin and hands the credentials to `onBoot`. See [Plugin storage](#plugin-storage). |
|
||||
|
||||
A plugin may be routes-only, nav-only, or hooks-only — every collection field is optional.
|
||||
|
||||
@@ -432,8 +434,8 @@ export async function listThings(ctx: RequestContext) {
|
||||
`requireSession(ctx)`, `can(ctx, permission)` (coarse JWT-claim check, zero I/O), and
|
||||
`check(keto, ctx, {namespace, object, relation})` (a live Keto check; anonymous ⇒ denied). Throw
|
||||
`new GuardError(403, …)` after a failed `can`/`check` to render the 403 page.
|
||||
- The handler **fetches its own data** from upstream; plugins hold no state (see
|
||||
[Stateless](#stateless)).
|
||||
- The handler **fetches its own data** — from upstream, or from the plugin's own
|
||||
[storage](#plugin-storage); the host holds none of it (see [Stateless core](#stateless-core)).
|
||||
- Default status: `200` for `view`/`html`/`json`, `303` for `redirect`.
|
||||
|
||||
#### Escaping & the trust boundary
|
||||
@@ -632,10 +634,13 @@ Optional, for reacting to system actions. A plugin's `hooks` may implement:
|
||||
|
||||
| Hook | When | May |
|
||||
| --- | --- | --- |
|
||||
| `onBoot()` | after discovery, before the server listens | warm caches, validate upstream config |
|
||||
| `onBoot(host)` | after discovery, before the server listens | warm caches, validate upstream config, open a [storage](#plugin-storage) connection |
|
||||
| `onRequest(ctx)` | before route matching | inspect, or **short-circuit** by returning a `RouteResult` |
|
||||
| `onResponse(ctx, result)` | after the handler | observe/log; cannot change the response |
|
||||
|
||||
`onBoot`'s `host` is a `BootContext`, carrying `storage` for a plugin that declared it. A hook
|
||||
written without a parameter stays valid.
|
||||
|
||||
Hooks run in **discovery order** (plugins sorted by id). `onRequest` fires on every request that
|
||||
reaches routing (static assets bypass it); the **first** hook to return a `RouteResult` short-circuits
|
||||
— later hooks and the route handler are skipped, and that result renders against its own plugin's
|
||||
@@ -653,7 +658,7 @@ getting its folder there.
|
||||
bind-mounts the whole tree (`compose.override.yml`: `.:/app`), so a restart picks it up.
|
||||
|
||||
**2. A plugin kept in its own repo, or added to a prebuilt image.** Bind-mount the plugin
|
||||
folder onto `/app/plugins/<id>` with a small compose override. Plugins are stateless, so
|
||||
folder onto `/app/plugins/<id>` with a small compose override. A plugin folder is code, not data —
|
||||
mount it read-only:
|
||||
|
||||
```yaml
|
||||
@@ -724,6 +729,67 @@ barrel's types on disk: typecheck it mounted under the host tree, or vendor a ty
|
||||
`node_modules`** and point tsconfig `paths` at it — a stub inside is the shadowing copy discovery
|
||||
refuses, and it would travel with the folder you mount.
|
||||
|
||||
### Plugin storage
|
||||
|
||||
A plugin that needs to keep data sets `storage: true`. The host then provisions a Postgres
|
||||
**database and login role of its own** — both named `plugin_<id>` — and hands the credentials to
|
||||
`onBoot`:
|
||||
|
||||
```ts
|
||||
import postgres from "postgres"; // your dependency, not the host's
|
||||
import { definePlugin, type StorageCredentials } from "@plainpages/plugin-api";
|
||||
|
||||
let sql: ReturnType<typeof postgres>;
|
||||
|
||||
export default definePlugin({
|
||||
apiVersion: "1.0.0",
|
||||
storage: true,
|
||||
hooks: {
|
||||
onBoot: async (host) => {
|
||||
if (!host.storage) throw new Error("things: storage was not provisioned");
|
||||
sql = postgres(host.storage.url);
|
||||
await sql`CREATE TABLE IF NOT EXISTS things (id uuid PRIMARY KEY, name text NOT NULL)`;
|
||||
},
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
`host.storage` is a `StorageCredentials` — `database`, `host`, `password`, `port`, `user`, and `url`,
|
||||
the same values pre-assembled as a DSN, which most clients take directly.
|
||||
|
||||
**Credentials, not a client.** The host has no opinion on how you reach Postgres: depend on
|
||||
`postgres`, `pg`, a query builder or an ORM ([Plugin dependencies](#plugin-dependencies)). No driver
|
||||
is part of the contract, so upgrading yours is yours alone to time.
|
||||
|
||||
**The schema is yours.** The host creates the database empty and never reads or writes inside it.
|
||||
Create your tables in `onBoot`: it runs before the server listens, so a failure aborts boot instead
|
||||
of surfacing later as a broken page.
|
||||
|
||||
What the host does guarantee:
|
||||
|
||||
- **One database and one role per plugin.** `CONNECT` is revoked from `PUBLIC`, so another installed
|
||||
plugin's role cannot reach your database.
|
||||
- **Provisioning is idempotent and runs every boot**, so a plugin dropped in later is picked up by
|
||||
the next `docker compose up -d` — the same rule as permission seeding.
|
||||
- **Your data is never dropped.** Removing a plugin folder leaves its database untouched; deleting it
|
||||
is a deliberate act by an operator.
|
||||
|
||||
**Passwords are derived, never stored** — each is `HMAC-SHA256(PLUGIN_DB_SECRET, <plugin id>)`, so
|
||||
`bootstrap` and `web` compute the same value independently and nothing has to be written down.
|
||||
Rotate every plugin's password by changing `PLUGIN_DB_SECRET` and running `docker compose up -d`,
|
||||
which re-applies each role's password and leaves the data alone.
|
||||
|
||||
**Only `bootstrap` holds superuser credentials.** It alone gets `PLUGIN_DB_ADMIN_URL`, the DSN that
|
||||
may `CREATE DATABASE`/`CREATE ROLE`. `web` gets `PLUGIN_DB_URL`, which names the server and carries
|
||||
no credentials — so plugin code, which runs inside `web`, cannot read a superuser password out of its
|
||||
own environment. Set both, plus `PLUGIN_DB_SECRET` ([Configuration](#configuration)); the dev stack
|
||||
sets them for you.
|
||||
|
||||
Storage stays off until `PLUGIN_DB_URL` is set, and a plugin declaring it while that is unset
|
||||
**aborts boot** naming itself — rather than serving pages without its data. One naming limit: a
|
||||
storage plugin's folder may be at most **56 characters**, so `plugin_<id>` fits Postgres' 63-byte
|
||||
identifier.
|
||||
|
||||
### Local dev & test story
|
||||
|
||||
A plugin is a normal folder of TypeScript, tested the same way the core is — everything in Docker.
|
||||
@@ -936,7 +1002,7 @@ The app is **environment-agnostic**: no `NODE_ENV`, every behaviour its own expl
|
||||
| `PORT` | `3000` | web listen port |
|
||||
| `CACHE_TEMPLATES` | `false` | cache compiled EJS templates (`true` in prod) |
|
||||
| `SECURE_COOKIES` | `false` | mark our session/CSRF cookies `Secure` (`true` in prod https; off in dev http) |
|
||||
| `REQUIRE_SECURE_SECRETS` | `false` | when `true`, `CSRF_SECRET` must be supplied and differ from the dev throwaway |
|
||||
| `REQUIRE_SECURE_SECRETS` | `false` | when `true`, `CSRF_SECRET` — and `PLUGIN_DB_SECRET` once storage is configured — must be supplied and differ from the dev throwaway |
|
||||
| `LOG_LEVEL` | `info` | min severity logged: `error`/`warn`/`info`/`verbose`/`debug`/`silly`/`none` |
|
||||
| `LOG_FORMAT` | `text` | log line format: `text` (human-readable, dev) or `json` (structured, prod) |
|
||||
| `SERVICE_NAME` | `plainpages` | OTLP `service.name` on every log + span — brand it as your own deployment |
|
||||
@@ -952,6 +1018,9 @@ The app is **environment-agnostic**: no `NODE_ENV`, every behaviour its own expl
|
||||
| `REVOCATION_DENYLIST` | `false` | when `true`, enable the optional [instant permission/session revoke denylist](#instant-revoke-the-optional-denylist) |
|
||||
| `REVOCATION_TTL_SEC` | `900` | how long a revoke entry lives; keep ≥ tokenizer TTL (10m) + clock skew |
|
||||
| `CSRF_SECRET` | dev throwaway | signs our double-submit CSRF token; enforced by `REQUIRE_SECURE_SECRETS` |
|
||||
| `PLUGIN_DB_URL` | _unset_ (dev: `postgres://postgres:5432`) | credential-free Postgres base URL for [plugin storage](#plugin-storage); unset ⇒ storage off, and a plugin declaring it aborts boot |
|
||||
| `PLUGIN_DB_ADMIN_URL` | _unset_ (dev: the bundled superuser) | the DSN that provisions each plugin's database and role — read by the one-shot `bootstrap` service **only**, never by `web` |
|
||||
| `PLUGIN_DB_SECRET` | dev throwaway | derives each plugin's database password; enforced by `REQUIRE_SECURE_SECRETS` once `PLUGIN_DB_URL` is set |
|
||||
|
||||
### Canonical host (one public URL)
|
||||
|
||||
@@ -1082,8 +1151,8 @@ records that subject as revoked-now; the hot path then rejects every token for i
|
||||
the revoke and forces a re-mint — which re-reads permissions from Keto, or clears a dead session. A
|
||||
fresh re-login passes, so a downgrade lands immediately without locking the account.
|
||||
|
||||
It is an in-memory, auto-evicting map — no database, so it stays inside the stateless model — and
|
||||
the check is pure CPU, keeping Keto off the hot path. Entries self-evict after `REVOCATION_TTL_SEC`
|
||||
It is an in-memory, auto-evicting map — host-owned state would break the [stateless
|
||||
core](#stateless-core) — and the check is pure CPU, keeping Keto off the hot path. Entries self-evict after `REVOCATION_TTL_SEC`
|
||||
(default 900s ≥ the 10m token TTL + skew). Two bounds: it is instant only on the **single instance**
|
||||
that handled the revoke (elsewhere the guarantee falls back to the token TTL — back it with a shared
|
||||
store for hard multi-instance revoke), and a **group** membership change is transitive across many
|
||||
@@ -1198,18 +1267,21 @@ of it over their **REST APIs using Node's built-in `fetch`** — no SDK dependen
|
||||
In **dev** the host-facing Ory ports are published — Kratos public `4433` and Hydra public `4444`;
|
||||
prod keeps them internal.
|
||||
|
||||
Runtime deps stay tiny and pinned: **`ejs`**, **`lucide-static`**, and **`@larvit/log`**. Auth,
|
||||
sessions, SSO and OAuth2 add *services*, not npm packages.
|
||||
Runtime deps stay tiny and pinned: **`ejs`**, **`lucide-static`**, **`@larvit/log`**, and
|
||||
**`postgres`** — the last one has no sub-dependencies of its own and is used in a single module, to
|
||||
provision [plugin storage](#plugin-storage) at boot. Auth, sessions, SSO and OAuth2 add *services*,
|
||||
not npm packages.
|
||||
|
||||
### Stateless
|
||||
### Stateless core
|
||||
|
||||
Plainpages holds **no state of its own**. The only database in the stack is **Postgres**, used by
|
||||
Ory; the `web` app never connects to it.
|
||||
The host holds **no state of its own**: it owns no schema and keeps nothing between requests. The
|
||||
stack's **Postgres** backs Ory, and gives every plugin that asks for one a database of its own
|
||||
([Plugin storage](#plugin-storage)) — which the host provisions but never reads or writes.
|
||||
|
||||
A plugin reads and writes state by **calling an upstream service** from its route handler — a REST
|
||||
API, an ERP, a plant historian, the customer's own backend — and renders the response with the
|
||||
building blocks. That keeps `web` trivially scalable and crash-safe: any instance can serve any
|
||||
request, because the session lives in Kratos and the data lives upstream.
|
||||
So a plugin gets its data one of two ways: by **calling an upstream service** from its route handler
|
||||
— a REST API, an ERP, a plant historian, the customer's own backend — or from **its own database**.
|
||||
Either keeps `web` trivially scalable and crash-safe: any instance can serve any request, because the
|
||||
session lives in Kratos and the data lives outside the process.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
@@ -13,6 +13,9 @@ services:
|
||||
CACHE_TEMPLATES: "false"
|
||||
LOG_FORMAT: "text" # human-readable logs in dev (base sets json for prod log pipelines)
|
||||
LOG_LEVEL: "debug" # verbose by default while developing (base defaults to info)
|
||||
# Point plugin storage at the bundled Postgres, so a dropped-in plugin declaring `storage`
|
||||
# works with no further config; the secret falls back to the dev throwaway (config.ts).
|
||||
PLUGIN_DB_URL: postgres://postgres:5432
|
||||
REQUIRE_SECURE_SECRETS: "false"
|
||||
SECURE_COOKIES: "false" # dev serves http — Secure cookies wouldn't be sent
|
||||
SCHEDULING_UPSTREAM: "http://shifts-upstream:4000" # backs the reference plugin once you copy it into plugins/
|
||||
@@ -29,6 +32,9 @@ services:
|
||||
# It belongs here and not in the base file, where it would desynchronise prod and collide with the
|
||||
# e2e stacks, which bind individual plugins *inside* /app/plugins.
|
||||
bootstrap:
|
||||
# Provisions the plugin databases web connects to above, as the dev superuser.
|
||||
environment:
|
||||
PLUGIN_DB_ADMIN_URL: postgres://${POSTGRES_USER:-ory}:${POSTGRES_PASSWORD:-ory}@postgres:5432/ory
|
||||
volumes:
|
||||
- .:/app
|
||||
|
||||
|
||||
+19
-3
@@ -17,10 +17,16 @@ services:
|
||||
CACHE_TEMPLATES: "true"
|
||||
CSRF_SECRET: ${CSRF_SECRET:-dev-insecure-csrf-secret}
|
||||
LOG_FORMAT: "json" # structured logs for prod pipelines; set OTLP_ENDPOINT to also export to a collector
|
||||
# Per-plugin Postgres storage. Explicit toggle: unset ⇒ off, and a plugin declaring `storage`
|
||||
# refuses to boot rather than run without its data. The URL carries no credentials — each
|
||||
# plugin's own password is derived from the secret (README → Plugin storage).
|
||||
PLUGIN_DB_SECRET: ${PLUGIN_DB_SECRET:-}
|
||||
PLUGIN_DB_URL: ${PLUGIN_DB_URL:-}
|
||||
REQUIRE_SECURE_SECRETS: "true"
|
||||
SECURE_COOKIES: "true" # prod serves https — mark session/CSRF cookies Secure
|
||||
# Wait for the services the app talks to (kratos + keto + hydra for the OAuth2 login/
|
||||
# consent handler) + the one-shot bootstrap (admin + JWKS seed).
|
||||
# consent handler) + the one-shot bootstrap (admin + JWKS seed). Postgres too: a plugin that
|
||||
# declares `storage` opens its connection in onBoot, before the server listens.
|
||||
depends_on:
|
||||
bootstrap:
|
||||
condition: service_completed_successfully
|
||||
@@ -30,14 +36,17 @@ services:
|
||||
condition: service_healthy
|
||||
hydra:
|
||||
condition: service_healthy
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
# verifier reads the same tokenizer JWKS Kratos signs with (config.ts JWKS_URL).
|
||||
# Read-only — bootstrap is the only writer.
|
||||
volumes:
|
||||
- ./ory/kratos/tokenizer:/etc/config/kratos/tokenizer:ro
|
||||
restart: unless-stopped
|
||||
|
||||
# Ory's storage only (Kratos/Keto/Hydra) — the web app never connects here.
|
||||
# init/init.sql creates one database per service. Dev defaults below; supply
|
||||
# The stack's storage: one database per Ory service (init/init.sql), plus one per plugin that
|
||||
# declares `storage` — bootstrap creates those at boot, since only it holds superuser credentials.
|
||||
# A plugin connects as its own role from inside web. Dev defaults below; supply
|
||||
# POSTGRES_USER/PASSWORD via env in production.
|
||||
postgres:
|
||||
image: postgres:18.6-alpine3.23
|
||||
@@ -127,6 +136,8 @@ services:
|
||||
condition: service_healthy
|
||||
keto:
|
||||
condition: service_healthy
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
ADMIN_EMAIL: ${ADMIN_EMAIL:-admin@plainpages.local}
|
||||
ADMIN_PASSWORD: ${ADMIN_PASSWORD:-admin}
|
||||
@@ -137,6 +148,11 @@ services:
|
||||
JWKS_FILE: /etc/config/kratos/tokenizer/jwks.json
|
||||
KETO_WRITE_URL: http://keto:4467
|
||||
KRATOS_ADMIN_URL: http://kratos:4434
|
||||
# The superuser DSN that creates each plugin's database and role lives ONLY here — never in
|
||||
# web, so plugin code cannot read it out of its own environment. Unset ⇒ a plugin declaring
|
||||
# `storage` fails the seed loudly. The secret must match web's; both derive the same passwords.
|
||||
PLUGIN_DB_ADMIN_URL: ${PLUGIN_DB_ADMIN_URL:-}
|
||||
PLUGIN_DB_SECRET: ${PLUGIN_DB_SECRET:-}
|
||||
volumes:
|
||||
- ./ory/kratos/tokenizer:/etc/config/kratos/tokenizer
|
||||
command: node src/auth/bootstrap.ts
|
||||
|
||||
@@ -51,7 +51,7 @@ test("the manifest's onBoot hook validates SCHEDULING_UPSTREAM (the binding, not
|
||||
try {
|
||||
const manifest = (await import("./plugin.ts")).default;
|
||||
assert.equal(typeof manifest.hooks?.onBoot, "function");
|
||||
assert.throws(() => manifest.hooks!.onBoot!(), /SCHEDULING_UPSTREAM/); // bad upstream → boot fails loud
|
||||
assert.throws(() => manifest.hooks!.onBoot!({}), /SCHEDULING_UPSTREAM/); // bad upstream → boot fails loud
|
||||
} finally {
|
||||
if (prev === undefined) delete process.env["SCHEDULING_UPSTREAM"];
|
||||
else process.env["SCHEDULING_UPSTREAM"] = prev;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
-- Runs once on first boot (docker-entrypoint-initdb.d), as the POSTGRES_USER.
|
||||
-- One database per Ory service: each owns its schema and runs its own migrations,
|
||||
-- so they never collide. The web app never connects here (stateless — see README).
|
||||
-- so they never collide. A plugin's database does not belong here: bootstrap provisions those on
|
||||
-- every boot, so one dropped in later is picked up too (README → Plugin storage).
|
||||
CREATE DATABASE kratos;
|
||||
CREATE DATABASE keto;
|
||||
CREATE DATABASE hydra;
|
||||
|
||||
Generated
+15
-1
@@ -10,7 +10,8 @@
|
||||
"dependencies": {
|
||||
"@larvit/log": "2.3.0",
|
||||
"ejs": "6.0.1",
|
||||
"lucide-static": "1.31.0"
|
||||
"lucide-static": "1.31.0",
|
||||
"postgres": "3.4.9"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/ejs": "3.1.5",
|
||||
@@ -405,6 +406,19 @@
|
||||
"integrity": "sha512-XFX9NO+gcLsOkXISmeQZYGJa2siGWZc/lgT/BK1b83h9RdOiz3cj1eQP5nWhELiB1KhW5ryk0nmMopOgm/CuSA==",
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/postgres": {
|
||||
"version": "3.4.9",
|
||||
"resolved": "https://registry.npmjs.org/postgres/-/postgres-3.4.9.tgz",
|
||||
"integrity": "sha512-GD3qdB0x1z9xgFI6cdRD6xu2Sp2WCOEoe3mtnyB5Ee0XrrL5Pe+e4CCnJrRMnL1zYtRDZmQQVbvOttLnKDLnaw==",
|
||||
"license": "Unlicense",
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
"funding": {
|
||||
"type": "individual",
|
||||
"url": "https://github.com/sponsors/porsager"
|
||||
}
|
||||
},
|
||||
"node_modules/typescript": {
|
||||
"version": "7.0.2",
|
||||
"resolved": "https://registry.npmjs.org/typescript/-/typescript-7.0.2.tgz",
|
||||
|
||||
+2
-1
@@ -19,7 +19,8 @@
|
||||
"dependencies": {
|
||||
"@larvit/log": "2.3.0",
|
||||
"ejs": "6.0.1",
|
||||
"lucide-static": "1.31.0"
|
||||
"lucide-static": "1.31.0",
|
||||
"postgres": "3.4.9"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/ejs": "3.1.5",
|
||||
|
||||
+16
-1
@@ -8,8 +8,10 @@
|
||||
// Then prints a first-run banner; fails loud on any unexpected upstream error.
|
||||
import { existsSync, writeFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { resolvePluginDbSecret } from "../config.ts";
|
||||
import { discoverPlugins } from "../plugin-host/discovery.ts";
|
||||
import { declaredPermissions, isValidPermissionName } from "../plugin-host/plugin.ts";
|
||||
import { provisionStorage } from "../plugin-host/storage.ts";
|
||||
import { generateJwks, type JwkSet } from "./gen-jwks.ts";
|
||||
import { createLogger, runWithLog, tracedFetch } from "../logger.ts";
|
||||
|
||||
@@ -151,9 +153,22 @@ async function main() {
|
||||
await runWithLog(log, async () => {
|
||||
if (ensureJwks(env["JWKS_FILE"] ?? "/etc/config/kratos/tokenizer/jwks.json")) log.info("generated a JWKS signing key");
|
||||
|
||||
const plugins = await discoverPlugins();
|
||||
|
||||
// A database and login role for each plugin that asked for one. It happens here because
|
||||
// bootstrap holds the stack's only superuser credentials — web derives the same password from
|
||||
// PLUGIN_DB_SECRET and connects as the plugin's own role, never as a superuser.
|
||||
const storagePlugins = plugins.filter((plugin) => plugin.storage).map((plugin) => plugin.id);
|
||||
if (storagePlugins.length > 0) {
|
||||
const adminUrl = env["PLUGIN_DB_ADMIN_URL"];
|
||||
if (!adminUrl) throw new Error(`bootstrap: PLUGIN_DB_ADMIN_URL must be set — these plugins declare storage: ${storagePlugins.join(", ")}`);
|
||||
const provisioned = await provisionStorage({ adminUrl, pluginIds: storagePlugins, secret: resolvePluginDbSecret(env) });
|
||||
log.info("plugin storage provisioned", { databases: provisioned.join(", ") });
|
||||
}
|
||||
|
||||
// Seed every discovered plugin's declared permission names (plus any ADMIN_PERMISSIONS), so the
|
||||
// shipped example — and any dropped-in plugin — works for the demo admin without a host edit.
|
||||
const declared = declaredPermissions(await discoverPlugins()).map((decl) => decl.name);
|
||||
const declared = declaredPermissions(plugins).map((decl) => decl.name);
|
||||
const { ignored, permissions } = seedPermissions(env["ADMIN_PERMISSIONS"], declared);
|
||||
if (ignored.length > 0) {
|
||||
log.warn("ignoring ADMIN_PERMISSIONS entries that are not <resource>:<action>", { ignored: ignored.join(", ") });
|
||||
|
||||
+16
-3
@@ -44,10 +44,11 @@ test("long-running Ory services declare readiness healthchecks", () => {
|
||||
`${svc} probes :${port}/health/ready`);
|
||||
});
|
||||
|
||||
test("web waits for kratos, keto and hydra to be healthy before starting", () => {
|
||||
test("web waits for kratos, keto, hydra and postgres to be healthy before starting", () => {
|
||||
assert.match(webBlock, /depends_on:/, "web declares dependencies");
|
||||
// hydra: the OAuth2 login/consent handler talks to its admin API.
|
||||
for (const svc of ["kratos", "keto", "hydra"])
|
||||
// hydra: the OAuth2 login/consent handler talks to its admin API. postgres: a plugin declaring
|
||||
// `storage` opens its connection in onBoot, before the server listens.
|
||||
for (const svc of ["kratos", "keto", "hydra", "postgres"])
|
||||
assert.match(webBlock, new RegExp(`${svc}:\\s*\\n\\s*condition:\\s*service_healthy`),
|
||||
`web waits for ${svc} healthy`);
|
||||
});
|
||||
@@ -78,6 +79,18 @@ test("prod base supplies the app secret via env and mounts no source; dev overri
|
||||
assert.match(compose, /POSTGRES_PASSWORD:\s*\$\{POSTGRES_PASSWORD\b/, "postgres password via env");
|
||||
});
|
||||
|
||||
test("the provisioning superuser DSN reaches bootstrap only, never web", () => {
|
||||
// web runs plugin code, which can read its own environment — so the credentials that may CREATE
|
||||
// DATABASE/ROLE must never be there. web gets the credential-free base URL and derives each
|
||||
// plugin's own password from the shared secret instead.
|
||||
const boot = compose.slice(compose.indexOf("\n bootstrap:"));
|
||||
const overrideWeb = override.slice(override.indexOf("\n web:"), override.indexOf("\n bootstrap:"));
|
||||
assert.match(boot, /PLUGIN_DB_ADMIN_URL:/, "bootstrap is given the superuser DSN");
|
||||
for (const [name, block] of [["base", webBlock], ["dev override", overrideWeb]] as const)
|
||||
assert.doesNotMatch(block, /PLUGIN_DB_ADMIN_URL/, `${name} web never sees it`);
|
||||
assert.match(webBlock, /PLUGIN_DB_URL:\s*\$\{PLUGIN_DB_URL/, "base wires web's base URL from env");
|
||||
});
|
||||
|
||||
test("a one-shot bootstrap seeds the stack before web starts", () => {
|
||||
// MVP bar: `bootstrap` runs after kratos+keto are healthy, seeds the admin +
|
||||
// JWKS, then exits; web waits for it to complete. Live seeding is boot-verified.
|
||||
|
||||
+11
-1
@@ -1,6 +1,6 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { loadConfig } from "./config.ts";
|
||||
import { loadConfig, resolvePluginDbSecret } from "./config.ts";
|
||||
|
||||
// Explicit secure-secret enforcement (no environment sniffing): secrets are the only
|
||||
// thing a hardened deploy must supply.
|
||||
@@ -9,6 +9,16 @@ const secureEnv = {
|
||||
REQUIRE_SECURE_SECRETS: "true",
|
||||
};
|
||||
|
||||
// web reads the secret through loadConfig and bootstrap through resolvePluginDbSecret; the two
|
||||
// deriving different passwords is invisible until a plugin's connection is refused at boot. Compose
|
||||
// passes an unset variable through as "", which is the case that actually drifted.
|
||||
test("web and bootstrap resolve the same plugin storage secret", () => {
|
||||
for (const env of [{}, { PLUGIN_DB_SECRET: "" }, { PLUGIN_DB_SECRET: "a-real-secret" }]) {
|
||||
assert.equal(loadConfig(env).pluginDbSecret, resolvePluginDbSecret(env), `for ${JSON.stringify(env)}`);
|
||||
}
|
||||
assert.match(resolvePluginDbSecret({ PLUGIN_DB_SECRET: "" }), /dev-insecure/); // empty is unset, not a secret
|
||||
});
|
||||
|
||||
test("loads dev defaults when the environment is empty", () => {
|
||||
const c = loadConfig({});
|
||||
assert.equal(c.port, 3000);
|
||||
|
||||
@@ -6,6 +6,15 @@
|
||||
export const LOG_LEVELS = ["error", "warn", "info", "verbose", "debug", "silly", "none"] as const;
|
||||
export type LogLevel = (typeof LOG_LEVELS)[number];
|
||||
|
||||
const DEV_PLUGIN_DB_SECRET = "dev-insecure-plugin-db-secret";
|
||||
|
||||
// bootstrap resolves the plugin-storage secret through this, web through loadConfig — and the two
|
||||
// must agree exactly or web connects with a password the role was never given. Compose passes an
|
||||
// unset variable through as "", so empty means throwaway here as it does in readSecret.
|
||||
export function resolvePluginDbSecret(env: Env): string {
|
||||
return env["PLUGIN_DB_SECRET"] || DEV_PLUGIN_DB_SECRET;
|
||||
}
|
||||
|
||||
export interface Config {
|
||||
appUrl: string | undefined; // canonical public URL; set ⇒ off-host visitors are redirected here. Unset ⇒ no redirect (explicit toggle)
|
||||
cacheTemplates: boolean;
|
||||
@@ -24,6 +33,8 @@ export interface Config {
|
||||
oryTimeoutSec: number; // per-call timeout for outbound Kratos/Keto/Hydra fetches (bounds a hung Ory)
|
||||
otlpEndpoint: string | undefined; // OTLP/HTTP collector base URI; unset ⇒ console-only (no export)
|
||||
otlpProtocol: "http/json" | "http/protobuf"; // OTLP wire format (protobuf for json-averse collectors)
|
||||
pluginDbSecret: string; // derives each plugin's database password (src/plugin-host/storage.ts)
|
||||
pluginDbUrl: string | undefined; // credential-free Postgres base URL; unset ⇒ plugin storage is off
|
||||
port: number;
|
||||
revocationDenylist: boolean; // enable the optional instant permission/session revoke denylist
|
||||
revocationTtlSec: number; // how long a revoke entry lives; keep ≥ tokenizer TTL + clock skew
|
||||
@@ -150,6 +161,12 @@ export function loadConfig(env: Env = process.env): Config {
|
||||
oryTimeoutSec: readPosInt(env, "ORY_TIMEOUT_SEC", 5),
|
||||
otlpEndpoint: readOptionalUrl(env, "OTLP_ENDPOINT"),
|
||||
otlpProtocol: readEnum(env, "OTLP_PROTOCOL", ["http/json", "http/protobuf"] as const, "http/json"),
|
||||
// Per-plugin storage. PLUGIN_DB_URL carries the server and its connection parameters but no
|
||||
// credentials: the superuser DSN that provisions stays in bootstrap, so a plugin cannot read it
|
||||
// out of web's environment. Unset ⇒ storage is off and a plugin declaring it fails loud at boot,
|
||||
// which is also why the secret is only enforced once a URL is configured.
|
||||
pluginDbSecret: readSecret(env, "PLUGIN_DB_SECRET", DEV_PLUGIN_DB_SECRET, requireSecure && Boolean(env["PLUGIN_DB_URL"])),
|
||||
pluginDbUrl: readOptionalUrl(env, "PLUGIN_DB_URL"),
|
||||
port: readPort(env),
|
||||
// Optional instant-revoke, off by default. When on, an admin deactivate/delete or permission
|
||||
// change revokes the subject's live tokens at once; the entry lives ttl seconds (≥ the 10m
|
||||
|
||||
@@ -27,13 +27,19 @@ test("a missing plugins/ dir means zero plugins, not an error (clean clone)", as
|
||||
});
|
||||
|
||||
test("discovers each folder's manifest, sorted, id derived from the folder name", async (t) => {
|
||||
const dir = scaffold(t, { "beta/plugin.ts": full("beta"), "alpha/plugin.ts": full("alpha") });
|
||||
const dir = scaffold(t, {
|
||||
"beta/plugin.ts": full("beta"),
|
||||
"alpha/plugin.ts": full("alpha"),
|
||||
"gamma/plugin.ts": `export default { apiVersion: "1.0.0", storage: true };`,
|
||||
});
|
||||
const plugins = await discoverPlugins({ dir });
|
||||
|
||||
assert.deepEqual(plugins.map((p) => p.id), ["alpha", "beta"]); // deterministic order
|
||||
assert.deepEqual(plugins.map((p) => p.id), ["alpha", "beta", "gamma"]); // deterministic order
|
||||
assert.equal(plugins[0]?.apiVersion, "1.0.0");
|
||||
assert.equal(plugins[0]?.nav?.[0]?.label, "alpha");
|
||||
assert.equal(typeof plugins[0]?.routes?.[0]?.handler, "function"); // handlers survive import
|
||||
assert.equal(plugins[0]?.storage, undefined); // storage is opt-in, never assumed
|
||||
assert.equal(plugins[2]?.storage, true);
|
||||
});
|
||||
|
||||
// Every per-plugin problem and every error-level conflict aborts boot with a message naming it.
|
||||
@@ -48,6 +54,9 @@ const badCases: Array<{ name: string; files: Record<string, string>; match: RegE
|
||||
{ name: "non-array routes", files: { "weird/plugin.ts": `export default { apiVersion: "1.0.0", routes: "nope" };` }, match: /weird.*routes.*array/s },
|
||||
{ name: "non-function home", files: { "weirdhome/plugin.ts": `export default { apiVersion: "1.0.0", home: "nope" };` }, match: /weirdhome.*home.*function/s },
|
||||
{ name: "non-function dashboard", files: { "weirddash/plugin.ts": `export default { apiVersion: "1.0.0", dashboard: "nope" };` }, match: /weirddash.*dashboard.*function/s },
|
||||
{ name: "non-boolean storage", files: { "weirdstore/plugin.ts": `export default { apiVersion: "1.0.0", storage: "postgres://db" };` }, match: /weirdstore.*storage.*boolean/s },
|
||||
// The folder name becomes a Postgres identifier, which truncates past 63 bytes.
|
||||
{ name: "a storage plugin whose folder name overflows a Postgres identifier", files: { [`${"a".repeat(57)}/plugin.ts`]: `export default { apiVersion: "1.0.0", storage: true };` }, match: /storage.*56 characters/s },
|
||||
{ name: "reserved dashboard id shadows the gated dashboard", files: { "dashboard/plugin.ts": full("dashboard") }, match: /dashboard.*reserved/s },
|
||||
{ name: "duplicate nav id across plugins", files: { "a/plugin.ts": full("a").replace("a:root", "dup"), "b/plugin.ts": full("b").replace("b:root", "dup") }, match: /nav id "dup"/ },
|
||||
{ name: "a route marked public AND permission is contradictory", files: { "contra/plugin.ts": `export default { apiVersion: "1.0.0", routes: [{ method: "GET", path: "/", public: true, permission: "x:read", handler: () => ({ html: "x" }) }] };` }, match: /contra.*public.*permission/s },
|
||||
|
||||
@@ -8,6 +8,7 @@ import { existsSync, readdirSync, readFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { checkApiVersion, findConflicts, isValidPermissionName, isValidPluginId, RESERVED_PLUGIN_IDS, type Plugin, type PluginManifest } from "./plugin.ts";
|
||||
import { isValidStoragePluginId, MAX_STORAGE_PLUGIN_ID } from "./storage.ts";
|
||||
|
||||
const rootDir = join(dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
|
||||
@@ -66,6 +67,13 @@ export async function discoverPlugins(options: DiscoverOptions = {}): Promise<Pl
|
||||
const shape = shapeError(manifest);
|
||||
if (shape) { fail(shape); continue; }
|
||||
|
||||
// The folder name becomes a Postgres identifier, which truncates past 63 bytes — two long ids
|
||||
// would then share one database. Only checked for a plugin that asked for storage.
|
||||
if (manifest.storage === true && !isValidStoragePluginId(id)) {
|
||||
fail(`declares storage, so its folder name must be at most ${MAX_STORAGE_PLUGIN_ID} characters`);
|
||||
continue;
|
||||
}
|
||||
|
||||
plugins.push({ ...manifest, id }); // identity is the folder, not the manifest
|
||||
}
|
||||
|
||||
@@ -131,6 +139,8 @@ function shapeError(manifest: PluginManifest): string | null {
|
||||
for (const slot of ["home", "dashboard"] as const) {
|
||||
if (manifest[slot] !== undefined && typeof manifest[slot] !== "function") return `"${slot}" must be a function (a route handler)`;
|
||||
}
|
||||
// A truthy non-boolean (a DSN, say) must not quietly read as "provision me one".
|
||||
if (manifest.storage !== undefined && typeof manifest.storage !== "boolean") return `"storage" must be a boolean`;
|
||||
// `public` and `permission` are contradictory on the same route/nav node — "open to all" vs
|
||||
// "needs this permission". Refuse rather than silently pick one, so the author's intent is unambiguous.
|
||||
for (const route of Array.isArray(manifest.routes) ? manifest.routes : []) {
|
||||
|
||||
@@ -12,14 +12,17 @@ function plugin(id: string, hooks: PluginHooks): Plugin {
|
||||
|
||||
test("runBootHooks runs each onBoot in order, skips plugins without one, and a throw aborts", async () => {
|
||||
const calls: string[] = [];
|
||||
const scoped: string[] = []; // each hook is handed a context built for its own plugin
|
||||
const bootContextFor = (built: Plugin) => { scoped.push(built.id); return {}; };
|
||||
await runBootHooks([
|
||||
plugin("a", { onBoot: () => void calls.push("a") }),
|
||||
plugin("b", {}), // no onBoot → skipped
|
||||
plugin("c", { onBoot: async () => void calls.push("c") }),
|
||||
]);
|
||||
], bootContextFor);
|
||||
assert.deepEqual(calls, ["a", "c"]);
|
||||
assert.deepEqual(scoped, ["a", "c"]); // and built only for the plugins that have one
|
||||
|
||||
await assert.rejects(runBootHooks([plugin("x", { onBoot: () => { throw new Error("boom"); } })]), /boom/);
|
||||
await assert.rejects(runBootHooks([plugin("x", { onBoot: () => { throw new Error("boom"); } })], () => ({})), /boom/);
|
||||
});
|
||||
|
||||
test("runRequestHooks short-circuits on the first RouteResult (with its plugin); later hooks skipped", async () => {
|
||||
|
||||
@@ -4,11 +4,15 @@
|
||||
// entirely when no plugin declares the hook, so the no-hooks hot path stays free.
|
||||
|
||||
import type { RequestContext } from "../http/context.ts";
|
||||
import type { Plugin, RouteResult } from "./plugin.ts";
|
||||
import type { BootContext, Plugin, RouteResult } from "./plugin.ts";
|
||||
|
||||
// After discovery, before the server listens. A throw aborts boot.
|
||||
export async function runBootHooks(plugins: Plugin[]): Promise<void> {
|
||||
for (const plugin of plugins) await plugin.hooks?.onBoot?.();
|
||||
// After discovery, before the server listens. A throw aborts boot. Each hook gets a context built
|
||||
// for its own plugin, so one plugin is never handed another's storage credentials.
|
||||
export async function runBootHooks(plugins: Plugin[], bootContextFor: (plugin: Plugin) => BootContext): Promise<void> {
|
||||
for (const plugin of plugins) {
|
||||
const onBoot = plugin.hooks?.onBoot;
|
||||
if (onBoot) await onBoot(bootContextFor(plugin));
|
||||
}
|
||||
}
|
||||
|
||||
// Before route matching. The first hook to return a RouteResult short-circuits the request — its
|
||||
|
||||
@@ -5,7 +5,10 @@
|
||||
// a plugin should import from here, never reach into deeper modules. See README.md → Building plugins.
|
||||
|
||||
export { definePlugin, isValidPermissionName } from "./plugin.ts";
|
||||
export type { HttpMethod, Plugin, PluginHooks, PluginManifest, PermissionDecl, Route, RouteHandler, RouteResult } from "./plugin.ts";
|
||||
export type { BootContext, HttpMethod, Plugin, PluginHooks, PluginManifest, PermissionDecl, Route, RouteHandler, RouteResult } from "./plugin.ts";
|
||||
// A plugin's own database, handed to onBoot when the manifest sets `storage`. Credentials, not a
|
||||
// client — the plugin depends on whichever driver it prefers (README → Plugin storage).
|
||||
export type { StorageCredentials } from "./storage.ts";
|
||||
export type { RequestContext, User } from "../http/context.ts";
|
||||
export type { PageChrome } from "../ui/chrome.ts";
|
||||
export type { NavNode } from "../ui/nav.ts";
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
|
||||
import type { RequestContext } from "../http/context.ts";
|
||||
import type { NavNode } from "../ui/nav.ts";
|
||||
import type { StorageCredentials } from "./storage.ts";
|
||||
|
||||
// Bump major on a breaking manifest/handler change, minor on an additive one.
|
||||
export const HOST_API_VERSION = "1.0.0";
|
||||
@@ -60,9 +61,14 @@ export function declaredPermissions(plugins: Plugin[]): PermissionDecl[] {
|
||||
return [...byName.values()].sort((a, b) => a.name.localeCompare(b.name));
|
||||
}
|
||||
|
||||
// What onBoot receives. A hook declaring no parameter stays valid, so this may grow additively.
|
||||
export interface BootContext {
|
||||
storage?: StorageCredentials; // this plugin's own database; present iff the manifest declared `storage`
|
||||
}
|
||||
|
||||
// Optional hooks on system actions. Crash-isolation is a non-goal — a throwing hook fails loud.
|
||||
export interface PluginHooks {
|
||||
onBoot?: () => Promise<void> | void; // after discovery, before the server listens
|
||||
onBoot?: (host: BootContext) => Promise<void> | void; // after discovery, before the server listens
|
||||
onRequest?: (ctx: RequestContext) => Promise<RouteResult | void> | RouteResult | void; // may short-circuit
|
||||
onResponse?: (ctx: RequestContext, result: RouteResult | null) => Promise<void> | void;
|
||||
}
|
||||
@@ -80,6 +86,9 @@ export interface PluginManifest {
|
||||
nav?: NavNode[]; // fragment merged into the menu (composeNav); node `icon` is a Lucide sprite id (src/ui/icons.ts), node ids must be globally unique
|
||||
permissions?: PermissionDecl[];
|
||||
routes?: Route[];
|
||||
// Ask for a Postgres database of this plugin's own; its credentials arrive on onBoot's BootContext.
|
||||
// The host provisions and locks it down but owns no schema inside it, and never drops it.
|
||||
storage?: boolean;
|
||||
}
|
||||
|
||||
// A discovered plugin: the manifest plus the `id` the host read from the folder name. Mounted
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
// Guards the per-plugin storage rules: the shared database/role name, the derived password, the DSN
|
||||
// a plugin receives and the provisioning statements. The integration test runs only when a superuser
|
||||
// DSN is supplied, so the unit suite needs no Postgres.
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import postgres from "postgres";
|
||||
import {
|
||||
buildCredentials,
|
||||
derivePassword,
|
||||
isValidStoragePluginId,
|
||||
MAX_STORAGE_PLUGIN_ID,
|
||||
provisionSql,
|
||||
provisionStorage,
|
||||
quoteIdentifier,
|
||||
quoteLiteral,
|
||||
storageName,
|
||||
} from "./storage.ts";
|
||||
|
||||
const SECRET = "a-test-secret";
|
||||
|
||||
test("the database and the role share one plugin_-prefixed name", () => {
|
||||
assert.equal(storageName("things"), "plugin_things");
|
||||
assert.equal(storageName("my-plugin"), "plugin_my-plugin");
|
||||
});
|
||||
|
||||
test("a storage plugin's id must leave the identifier under Postgres' 63 bytes", () => {
|
||||
assert.equal(MAX_STORAGE_PLUGIN_ID, 56); // 63 - "plugin_"
|
||||
assert.ok(isValidStoragePluginId("a".repeat(MAX_STORAGE_PLUGIN_ID)));
|
||||
assert.ok(!isValidStoragePluginId("a".repeat(MAX_STORAGE_PLUGIN_ID + 1)));
|
||||
});
|
||||
|
||||
test("the password is derived, so the same one is reachable without storing it", () => {
|
||||
const derived = derivePassword(SECRET, "things");
|
||||
assert.equal(derived, derivePassword(SECRET, "things"));
|
||||
assert.notEqual(derived, derivePassword(SECRET, "other"));
|
||||
assert.notEqual(derived, derivePassword("a-rotated-secret", "things"));
|
||||
assert.match(derived, /^[A-Za-z0-9_-]{43}$/); // base64url of 32 bytes — needs no escaping in a DSN
|
||||
});
|
||||
|
||||
test("credentials name the plugin's own database, user and password", () => {
|
||||
const credentials = buildCredentials("postgres://postgres:5432", "things", SECRET);
|
||||
assert.deepEqual(credentials, {
|
||||
database: "plugin_things",
|
||||
host: "postgres",
|
||||
password: derivePassword(SECRET, "things"),
|
||||
port: 5432,
|
||||
url: `postgres://plugin_things:${derivePassword(SECRET, "things")}@postgres:5432/plugin_things`,
|
||||
user: "plugin_things",
|
||||
});
|
||||
});
|
||||
|
||||
test("the base URL's connection parameters survive into the DSN", () => {
|
||||
const credentials = buildCredentials("postgres://db.example?sslmode=require", "things", SECRET);
|
||||
assert.equal(credentials.port, 5432); // absent ⇒ Postgres' default, never NaN
|
||||
assert.equal(credentials.host, "db.example");
|
||||
assert.match(credentials.url, /@db\.example\/plugin_things\?sslmode=require$/);
|
||||
});
|
||||
|
||||
test("quoting doubles an embedded quote", () => {
|
||||
assert.equal(quoteIdentifier('we"ird'), '"we""ird"');
|
||||
assert.equal(quoteLiteral("we'ird"), "'we''ird'");
|
||||
});
|
||||
|
||||
test("provisioning creates the role and the database when neither exists", () => {
|
||||
assert.deepEqual(provisionSql("plugin_things", "pw", { databaseExists: false, roleExists: false }), [
|
||||
`CREATE ROLE "plugin_things" LOGIN PASSWORD 'pw'`,
|
||||
`CREATE DATABASE "plugin_things" OWNER "plugin_things"`,
|
||||
`REVOKE ALL ON DATABASE "plugin_things" FROM PUBLIC`,
|
||||
`GRANT ALL PRIVILEGES ON DATABASE "plugin_things" TO "plugin_things"`,
|
||||
]);
|
||||
});
|
||||
|
||||
test("re-provisioning re-sets the password and creates nothing twice", () => {
|
||||
assert.deepEqual(provisionSql("plugin_things", "rotated", { databaseExists: true, roleExists: true }), [
|
||||
`ALTER ROLE "plugin_things" WITH LOGIN PASSWORD 'rotated'`,
|
||||
`REVOKE ALL ON DATABASE "plugin_things" FROM PUBLIC`,
|
||||
`GRANT ALL PRIVILEGES ON DATABASE "plugin_things" TO "plugin_things"`,
|
||||
]);
|
||||
});
|
||||
|
||||
// --- Integration: the statements above, against a real Postgres -----------------------
|
||||
// Opt-in via PLUGIN_DB_ADMIN_URL (a superuser DSN); the unit gate runs no Postgres. What the unit
|
||||
// tests cannot prove lives here: the owner may create tables, and a peer role is locked out.
|
||||
|
||||
const ADMIN_URL = process.env["PLUGIN_DB_ADMIN_URL"] ?? "";
|
||||
const integration = ADMIN_URL ? {} : { skip: "set PLUGIN_DB_ADMIN_URL to a superuser DSN to run" };
|
||||
|
||||
function baseUrlOf(adminUrl: string): string {
|
||||
const url = new URL(adminUrl);
|
||||
url.username = "";
|
||||
url.password = "";
|
||||
url.pathname = "";
|
||||
return url.href;
|
||||
}
|
||||
|
||||
async function queryAs(url: string, statement: string): Promise<unknown> {
|
||||
const sql = postgres(url, { connect_timeout: 10, max: 1, onnotice: () => {} });
|
||||
try {
|
||||
return await sql.unsafe(statement);
|
||||
} finally {
|
||||
await sql.end();
|
||||
}
|
||||
}
|
||||
|
||||
test("provisions a database its plugin can use and a peer plugin cannot reach", integration, async () => {
|
||||
const ids = ["storage-itest-a", "storage-itest-b"];
|
||||
const base = baseUrlOf(ADMIN_URL);
|
||||
const admin = postgres(ADMIN_URL, { connect_timeout: 10, max: 1, onnotice: () => {} });
|
||||
try {
|
||||
await provisionStorage({ adminUrl: ADMIN_URL, pluginIds: ids, secret: SECRET });
|
||||
|
||||
const owner = buildCredentials(base, "storage-itest-a", SECRET);
|
||||
await queryAs(owner.url, "CREATE TABLE IF NOT EXISTS notes (body text)");
|
||||
await queryAs(owner.url, "INSERT INTO notes (body) VALUES ('persisted')");
|
||||
const rows = (await queryAs(owner.url, "SELECT body FROM notes")) as { body: string }[];
|
||||
assert.deepEqual(rows.map((row) => row.body), ["persisted"]);
|
||||
|
||||
// A peer holds valid credentials for its OWN database and still cannot reach this one.
|
||||
const peer = new URL(buildCredentials(base, "storage-itest-b", SECRET).url);
|
||||
peer.pathname = `/${storageName("storage-itest-a")}`;
|
||||
await assert.rejects(queryAs(peer.href, "SELECT 1"), /permission denied|not permitted/i);
|
||||
|
||||
// Re-running is idempotent, and a rotated secret lands on the existing role.
|
||||
await provisionStorage({ adminUrl: ADMIN_URL, pluginIds: ids, secret: "a-rotated-secret" });
|
||||
const rotated = buildCredentials(base, "storage-itest-a", "a-rotated-secret");
|
||||
const kept = (await queryAs(rotated.url, "SELECT body FROM notes")) as { body: string }[];
|
||||
assert.deepEqual(kept.map((row) => row.body), ["persisted"]); // rotating the secret keeps the data
|
||||
await assert.rejects(queryAs(owner.url, "SELECT 1"), /password authentication failed/i);
|
||||
} finally {
|
||||
for (const id of ids) {
|
||||
const name = quoteIdentifier(storageName(id));
|
||||
await admin.unsafe(`DROP DATABASE IF EXISTS ${name} WITH (FORCE)`);
|
||||
await admin.unsafe(`DROP ROLE IF EXISTS ${name}`);
|
||||
}
|
||||
await admin.end();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,104 @@
|
||||
// Per-plugin Postgres storage (README → Plugin storage). Only bootstrap provisions, because only it
|
||||
// is given superuser credentials; web derives the same passwords and never sees them.
|
||||
|
||||
import { createHmac } from "node:crypto";
|
||||
import postgres from "postgres";
|
||||
|
||||
// Database and role share one name, so reconnecting needs nothing looked up. The prefix also keeps
|
||||
// a plugin id from ever naming an Ory database.
|
||||
const NAME_PREFIX = "plugin_";
|
||||
|
||||
// Postgres truncates an identifier at 63 bytes, which would silently collide two long ids.
|
||||
export const MAX_STORAGE_PLUGIN_ID = 63 - NAME_PREFIX.length;
|
||||
|
||||
// `url` pre-assembles the other fields as a DSN, which most drivers take directly.
|
||||
export interface StorageCredentials {
|
||||
database: string;
|
||||
host: string;
|
||||
password: string;
|
||||
port: number;
|
||||
url: string;
|
||||
user: string;
|
||||
}
|
||||
|
||||
export function storageName(pluginId: string): string {
|
||||
return `${NAME_PREFIX}${pluginId}`;
|
||||
}
|
||||
|
||||
export function isValidStoragePluginId(pluginId: string): boolean {
|
||||
return pluginId.length <= MAX_STORAGE_PLUGIN_ID;
|
||||
}
|
||||
|
||||
// Derived, never stored — which is what keeps the host free of state it would have to persist.
|
||||
export function derivePassword(secret: string, pluginId: string): string {
|
||||
return createHmac("sha256", secret).update(pluginId).digest("base64url");
|
||||
}
|
||||
|
||||
// `baseUrl` names the server and its connection parameters, and carries no credentials of its own.
|
||||
export function buildCredentials(baseUrl: string, pluginId: string, secret: string): StorageCredentials {
|
||||
const name = storageName(pluginId);
|
||||
const password = derivePassword(secret, pluginId);
|
||||
const url = new URL(baseUrl);
|
||||
url.username = name;
|
||||
url.password = password;
|
||||
url.pathname = `/${name}`;
|
||||
return { database: name, host: url.hostname, password, port: Number(url.port) || 5432, url: url.href, user: name };
|
||||
}
|
||||
|
||||
// CREATE ROLE/DATABASE bind no parameters, so the name and password are quoted into the statement.
|
||||
export function quoteIdentifier(name: string): string {
|
||||
return `"${name.replaceAll('"', '""')}"`;
|
||||
}
|
||||
|
||||
export function quoteLiteral(value: string): string {
|
||||
return `'${value.replaceAll("'", "''")}'`;
|
||||
}
|
||||
|
||||
export interface ProvisionState {
|
||||
databaseExists: boolean;
|
||||
roleExists: boolean;
|
||||
}
|
||||
|
||||
// One plugin's full plan, in order. The password is re-set on every run so rotating the secret needs
|
||||
// no separate step, and PUBLIC loses CONNECT so no other plugin's role can reach this database.
|
||||
export function provisionSql(name: string, password: string, state: ProvisionState): string[] {
|
||||
const identifier = quoteIdentifier(name);
|
||||
const secret = quoteLiteral(password);
|
||||
return [
|
||||
state.roleExists
|
||||
? `ALTER ROLE ${identifier} WITH LOGIN PASSWORD ${secret}`
|
||||
: `CREATE ROLE ${identifier} LOGIN PASSWORD ${secret}`,
|
||||
...(state.databaseExists ? [] : [`CREATE DATABASE ${identifier} OWNER ${identifier}`]),
|
||||
`REVOKE ALL ON DATABASE ${identifier} FROM PUBLIC`,
|
||||
`GRANT ALL PRIVILEGES ON DATABASE ${identifier} TO ${identifier}`,
|
||||
];
|
||||
}
|
||||
|
||||
export interface ProvisionOptions {
|
||||
adminUrl: string; // superuser DSN — the rights to create a database and a role
|
||||
pluginIds: string[];
|
||||
secret: string;
|
||||
}
|
||||
|
||||
// Idempotent, and it drops nothing: an uninstalled plugin keeps its data until an operator removes
|
||||
// it deliberately.
|
||||
export async function provisionStorage(options: ProvisionOptions): Promise<string[]> {
|
||||
const sql = postgres(options.adminUrl, { connect_timeout: 10, max: 1, onnotice: () => {} });
|
||||
try {
|
||||
const names: string[] = [];
|
||||
for (const pluginId of options.pluginIds) {
|
||||
const name = storageName(pluginId);
|
||||
const [role] = await sql`SELECT 1 FROM pg_roles WHERE rolname = ${name}`;
|
||||
const [database] = await sql`SELECT 1 FROM pg_database WHERE datname = ${name}`;
|
||||
const plan = provisionSql(name, derivePassword(options.secret, pluginId), {
|
||||
databaseExists: database !== undefined,
|
||||
roleExists: role !== undefined,
|
||||
});
|
||||
for (const statement of plan) await sql.unsafe(statement); // provisionSql quotes what it interpolates
|
||||
names.push(name);
|
||||
}
|
||||
return names;
|
||||
} finally {
|
||||
await sql.end();
|
||||
}
|
||||
}
|
||||
@@ -8,9 +8,12 @@ import { readFileSync } from "node:fs";
|
||||
const read = (p: string) => readFileSync(new URL(`../${p}`, import.meta.url), "utf8");
|
||||
const ORY_DATABASES = ["hydra", "keto", "kratos"]; // one DB per Ory service
|
||||
|
||||
test("init SQL gives each Ory service its own database", () => {
|
||||
test("init SQL gives each Ory service its own database, and leaves plugin databases to bootstrap", () => {
|
||||
const sql = read("ory/postgres/init/init.sql");
|
||||
for (const db of ORY_DATABASES) {
|
||||
assert.match(sql, new RegExp(`CREATE DATABASE ${db}\\b`, "i"), `creates ${db}`);
|
||||
}
|
||||
// This file runs once, on an empty data dir — a plugin database added here would never appear for
|
||||
// a plugin dropped in later. bootstrap provisions them on every boot instead.
|
||||
assert.doesNotMatch(sql, /plugin_/i, "no plugin database is seeded here");
|
||||
});
|
||||
|
||||
+13
-1
@@ -13,6 +13,7 @@ import { createKratosAdmin } from "./auth/kratos-admin.ts";
|
||||
import { createKratosPublic } from "./auth/kratos-public.ts";
|
||||
import { createLogger, tracedFetch } from "./logger.ts";
|
||||
import { loadMenuConfig } from "./ui/menu-config.ts";
|
||||
import { buildCredentials } from "./plugin-host/storage.ts";
|
||||
|
||||
const config = loadConfig(); // validates the env (incl. enforced secrets) — fails loud at boot
|
||||
// App-level logger: structured, OTLP-capable when OTLP_ENDPOINT is set. The hot path clones it
|
||||
@@ -44,7 +45,18 @@ log.info("plugins discovered", { count: plugins.length, ids: plugins.map((p) =>
|
||||
const i18n = createI18n(await loadI18n({ logger: log, pluginIds: plugins.map((p) => p.id) }));
|
||||
log.info("locales loaded", { locales: i18n.available.join(", ") });
|
||||
|
||||
await runBootHooks(plugins); // plugin onBoot — after discovery, before listen; a throw aborts boot
|
||||
// A plugin's database credentials are derived, never stored — so the only thing that can be missing
|
||||
// is the server itself. Refuse at boot rather than at that plugin's first query, hours later.
|
||||
const pluginDbUrl = config.pluginDbUrl;
|
||||
const declaresStorage = plugins.filter((plugin) => plugin.storage).map((plugin) => plugin.id);
|
||||
if (declaresStorage.length > 0 && pluginDbUrl === undefined) {
|
||||
throw new Error(`config: PLUGIN_DB_URL must be set — these plugins declare storage: ${declaresStorage.join(", ")}`);
|
||||
}
|
||||
|
||||
// plugin onBoot — after discovery, before listen; a throw aborts boot.
|
||||
await runBootHooks(plugins, (plugin) =>
|
||||
plugin.storage && pluginDbUrl !== undefined ? { storage: buildCredentials(pluginDbUrl, plugin.id, config.pluginDbSecret) } : {},
|
||||
);
|
||||
|
||||
const server = createApp({
|
||||
// Canonical-host redirect target (off-host GET/HEAD visitors are sent here). Opt-in: omitted unless
|
||||
|
||||
@@ -3,6 +3,9 @@
|
||||
## Unfinnished work
|
||||
|
||||
- [ ] Add a way to configure plugins directly when installing. **Decided: the manifest declares it, not an `.env`** — a declared schema is validatable at boot, so a missing or mistyped setting fails loud and named the way a stray `package.json` now does, and the picker/docs can be generated from the declaration. Open: where the operator *supplies* the values (env var per key, a `config/` file, or both), and whether a secret may be declared at all.
|
||||
- [ ] A provisioned plugin database is invisible and unremovable. Nothing lists what exists, so an operator cannot see that `plugin_<id>` is there, nor that its plugin is gone — and uninstalling never drops it (deliberately, so data survives), leaving orphans with no supported way to clean them. Same shape as the uninstalled-plugin grant gap above. Sketch: a read-only "provisioned, but no installed plugin claims it" list plus a documented drop procedure.
|
||||
- [ ] Decide who owns the connection ceiling for plugin storage. Each storage plugin opens its own pool against one Postgres, whose default `max_connections` is 100, and nothing warns as plugins are added. Either state a per-plugin pool ceiling in README → Plugin storage or record that the plugin owns it.
|
||||
- [ ] E2E that a plugin's data actually survives a restart. `storage.test.ts` proves provisioning against a real Postgres (opt-in via `PLUGIN_DB_ADMIN_URL`), but no Playwright flow writes through a plugin page and reads it back after `docker compose restart web`.
|
||||
- [ ] Rename the plugin "admin" to something less generic, like "auth-admin" or "users-groups-admin".
|
||||
- [ ] Guard the group paths against self-lockout, or accept them explicitly. The self-revoke guard covers only your own *direct* grants on the Users screen; unticking a permission on a group you belong to, removing yourself from that group, or deleting it can all still strip your own effective access with no warning. Recorded in AGENTS.md as a known gap — the robust fix is a "last effective holder" check, which needs a reverse Keto query.
|
||||
- [ ] The permission picker has no concurrency baseline, so two operators editing the same user/group silently discard each other's change. Sketch: post the rendered set as a hidden baseline; if it no longer matches Keto, re-render with "this changed while you had the page open" rather than applying.
|
||||
@@ -36,6 +39,7 @@ Prioritized. Overall verdict: architecture is sound; these are refinements.
|
||||
|
||||
## Finnished work
|
||||
|
||||
- [x] Give a plugin persistent storage: `storage: true` provisions a Postgres database + login role named `plugin_<id>`, credentials arrive on `onBoot`, passwords are derived from `PLUGIN_DB_SECRET` rather than stored.
|
||||
- [x] Refuse a stray `package.json`/`node_modules` in `config/` by name, as plugin folders already are.
|
||||
- [x] Let Renovate reach the example plugins' manifests (`ignorePaths` overrides `config:recommended`).
|
||||
- [x] The seeded admin is granted each permission once — `seedPermissions` dedupes and the grant PUT is idempotent.
|
||||
|
||||
Reference in New Issue
Block a user