Skip to content

Capacity

The governed request path costs the database a fixed amount of work per request. This page gives the measured cost of the current release, the budget the repository enforces on it, and the procedure to measure an installation's ceiling yourself.

Purpose

A Mission Control gateway records every model or tool call as a decision, an admission and a usage attempt before the upstream call, and settles them after it. That bookkeeping is what a capacity plan has to fit: the number of statements and write transactions per request, whether any of them takes the database-wide write lock, and the requests per second one or several gateway processes sustain against one PostgreSQL database.

Prerequisites

  • The repository checked out with its platform dependencies installed (pnpm install in platform/; see Install).
  • The throwaway test PostgreSQL the platform tests use, started from platform/ with pnpm db:test:up.
  • Node.js 24 and pnpm 11.22.0, as platform/package.json declares.

The harness runs in process: it builds the real inference composition (governance, kernels, routes) over a template-cloned test database, with the model provider and the Connector server stubbed to answer instantly or after a fixed delay. It is not a load test of a running installation; it measures the gateway's own cost per request. With --instances N it starts N gateway processes over that one database and splits the closed-loop clients between them, which measures how the cost scales across processes.

Measure

From platform/:

pnpm --filter @mission-control/gateway capacity --profile direct
pnpm --filter @mission-control/gateway capacity --profile direct --instances 2
pnpm --filter @mission-control/gateway capacity --profile direct --instances 2 --restarts 2
pnpm --filter @mission-control/gateway capacity --profile hung
pnpm --filter @mission-control/gateway capacity --profile mcp --no-assert
pnpm --filter @mission-control/gateway capacity --profile managed --no-assert

Options: --concurrency 16 (closed-loop clients, in total), --instances 1 (gateway processes over the one database; the clients are split between them), --restarts 0 (with two or more instances, direct profile: the instances serve over HTTP behind a balancer model that probes /readyz/traffic, and that many are stopped one at a time and replaced during the window; any failed request fails the budget), --seconds 20 (measured window), --warmup 3, --stub-latency-ms 0 (upstream delay), --json <path> (where the report is written; default platform/apps/gateway/capacity-results/<timestamp>-<profile>.json, ignored by git) and --no-assert (report without the budget). The hung profile fixes its own clients and instance and ignores --concurrency, --instances and --restarts.

Profiles: direct is a Chat Completions request with a virtual key; mcp is an MCP tools/call with a virtual key; managed is a Chat Completions request signed by an agent runtime's execution transport for a dispatched turn; hung is the resilience gate described below.

Expected result, for the direct profile:

profile direct, instances 1, concurrency 16, 20 s, stub 0 ms, release <version>
requests 1730  requests/s 86.5  latency ms p50 142.6 p95 451.8 p99 577.2 max 669.6
per request: statements 57.9  write transactions 2.0 (serialized 0.0)  read snapshots 1.0  savepoints 3.0  db 45.7 ms  lock held 0.0 ms  queue wait 0.0 ms  lock wait 0.0 ms
budget: passed (at most 2 write transactions, 0 of them serialized, and 60 statements per request, at least 40 requests/s)

The report's perRequest block is the same figure per request: statements, writeTransactions (outermost write transactions), serializedTransactions (those that took the database-wide write lock; none on the governed path), readTransactions (read snapshots, which take no lock), savepoints, dbMs, lockHeldMs, queueWaitMs (the wait for this process's turn at the lock) and lockWaitMs (the wait for the lock itself). With several instances the report also carries instances and perInstanceRequestsPerSecond, and the per-request figures are summed over the instances.

Budget

The direct profile fails the run, and the weekly workflow's capacity job, when a request costs more than 2 write transactions, when any of them takes the database-wide write lock, when it runs more than 60 statements, or when 16 concurrent clients sustain fewer than 40 requests per second in total against an instant stub, and, during a rolling run, when any request fails. The MCP and managed profiles are reported without a budget; the hung profile has its own gate, described next.

The hung-provider gate

The hung profile proves the program's resilience gate: one provider that never answers leaves the other traffic at its baseline latency and is refused quickly. The instance runs with an in-flight maximum of 12 (GATEWAY_MAX_IN_FLIGHT), so each Team's share is 6. The seeded tenant has two Teams: the healthy Team's 6 clients call support throughout; the other Team's 8 clients call support for a five-second baseline, then hung, a second virtual model whose only deployment never answers and carries firstTokenMs 2000. A client the other Team has past its share is refused with retry-after: 1, which it honors; each hung request holds a slot until its first-token timeout and ends 503 provider_unavailable; after three timeouts the instance skips the deployment and, since it is the only deployment of its model, still tries it once per request while extending the skip (Model gateway).

From platform/:

pnpm --filter @mission-control/gateway capacity --profile hung

Expected result, measured on 2026-09-26 on the machine of the tables below:

profile hung, instances 1, restarts 0, concurrency 14, 20 s, stub 0 ms, release <version>
requests 2129  requests/s 106.5  latency ms p50 53.7 p95 73.9 p99 84.1 max 373.8
per request: statements 57.4  write transactions 2.0 (serialized 0.0)  read snapshots 1.0  savepoints 3.0  db 35.2 ms  lock held 0.0 ms  queue wait 0.0 ms  lock wait 0.0 ms
hung provider: baseline p50 197.8 ms (14 clients on support, 0 refused by the Team share); healthy p50 53.7 ms (6 clients, 0 refused, 0 failed) while 8 clients call hung: 100 hung requests, 60 timed out, 40 refused, 0 unexpected, slowest 2372 ms, 1 skip(s) opened; in-flight maximum 12, firstTokenMs 2000
budget: passed (healthy p50 within 20% + 20 ms of the baseline, no healthy request refused or failed, every hung request refused or timed out within 2 × firstTokenMs)

The top-level figures are the healthy Team's during the measured window; the report's hung block carries the baseline, the healthy side and the hung side (timedOut, refused, succeeded, other, maxMs, skipsOpened); the per-request figures average every answer of the window, refusals included. The budget fails the run when the healthy p50 exceeds the baseline p50 by more than 20% plus 20 ms, when a baseline or healthy request fails, when a healthy request is refused, when no hung request ended, when a hung request succeeded or ended other than refused or timed out, or when the slowest hung request took longer than twice firstTokenMs. Without the first-token timeout the eight hung clients would hold their slots for good and no hung request would end; without the Team share they would hold eight of the twelve slots and the healthy Team would be refused.

Measured cost of this release

Measured on 2026-09-25 with the harness above: 16 clients in total, 20 seconds, instant stubs, one PostgreSQL 16 in Docker on a developer machine. The "before" columns are the same measurement of the release before the governed request path was rebuilt.

Profile Statements before → now Write transactions before → now (serialized now) Requests/s Latency p50 / p95 ms
Direct Chat Completions 353 → 58 15 → 2 (0) 87 143 / 452
MCP tools/call 365 → 67 16 → 2 (0) 85 171 / 312
Managed chat hop 984 → 111 15 → 3 (1) 49 334 / 350

Several gateway processes over the same database, direct profile, the 16 clients split between them:

Instances Requests/s (per instance) Latency p50 / p95 ms Database ms per request
1 87 143 / 452 46
2 234 (117 + 117) 67 / 83 67
4 222 (56 each) 69 / 84 70
2, each restarted once during the window 216 (107 + 108) 71 / 101 67

What the numbers mean:

  • Every governed request writes exactly two transactions: one before the upstream call (the decision, its admissions and its attempt) and one after it (the settlement, the releases and the terminal state). Neither takes the database-wide write lock: they lock the rows they change, so requests run in parallel with one another and with management writes. Authorization reads in one snapshot. A tool that needs a person's approval writes one more, small, transaction so the approval can reference its decision.
  • Requests that share a finite budget cap wait for one another on the cap's policy rows for the duration of one dispatch transaction, about ten milliseconds, so two requests admitted at once never exceed the cap; an Organization without finite caps, like the harness tenant, runs its requests fully in parallel.
  • The managed hop's third transaction is the runtime's turn-authority read, which validates the execution's funding under the database-wide lock. It is the one serialized transaction left on a request path, and the hop's latency at concurrency 16 is the queue for that lock. See the gateway's Health and logging for the leases a hop renews.
  • A rolling restart of two instances, each stopped with SIGTERM and started again once during the window, costs about 5% of the two-instance rate and no failed request: the balancer model probes /readyz/traffic every 200 ms, the stopping instance drains for a second while the probe says so, and its requests in flight finish before its listeners close (Run several gateway instances).
  • One gateway process is bound by its own event loop before the database is: two processes deliver more than twice one process's rate, and four processes sharing the same 16 clients hold about the same total as two while the database's time per request stays near 70 ms. On this machine one PostgreSQL carries well over 200 direct requests per second from two or more gateway instances; an installation's own ceiling comes from running the harness with --instances against its database.

Verification

  • pnpm --filter @mission-control/gateway capacity --profile direct prints budget: passed.
  • pnpm --filter @mission-control/gateway capacity --profile hung prints budget: passed.
  • The weekly workflow's capacity job runs the direct profile alone, with two instances and with a rolling restart, then the hung, MCP and managed profiles, and uploads every report as the capacity-results artifact.

Failure recovery

Symptom Recovery
capacity: N requests failed A request answered other than 200. Run the gateway test suite (pnpm --filter @mission-control/gateway test) to find the failing path; the harness has no retries by design.
budget: FAILED with more write transactions A change added a write transaction to the request path. Compare the perRequest block with the table above and look for a new transaction in the kernels or the services they call.
budget: FAILED with a serialized write transaction A change put a transaction(), which takes the database-wide write lock, on the request path. The governed path runs concurrentTransaction() under row locks; find the new site in the kernels, the dispatch service or a service they call.
budget: FAILED with fewer requests per second on a loaded machine The budget assumes an otherwise idle machine and an instant stub. Rerun alone, or with --seconds 40.
budget: FAILED with failed requests during a rolling restart A stopping instance refused or dropped a request: its drain no longer serves through the window, or its listeners closed before the requests in flight finished. Run the gateway's shutdown tests (pnpm --filter @mission-control/gateway exec vitest run src/instance/graceful-shutdown.test.ts).
budget: FAILED with a healthy request refused during the hung run A hung request held a slot past its first-token timeout, or the Team share no longer holds the other Team to half the instance. Run pnpm --filter @mission-control/gateway exec vitest run src/instance/in-flight.test.ts src/invocation/model-invocation-kernel.test.ts.
budget: FAILED with the slowest hung request past 4000 ms The first-token deadline did not abort the provider call; run the kernel's deadline tests (src/invocation/model-invocation-kernel.test.ts).
Connection refused The test PostgreSQL is not running: pnpm db:test:up from platform/.