Skip to content

Run several gateway instances

Several gateway processes serve one installation from behind the platform's load balancer or ingress, over one PostgreSQL database, and restart one at a time without a failed request. This page gives the contract the balancer must follow, the rolling restart procedure and how to verify it.

Purpose

One gateway process is bound by its own event loop before the database is (Capacity): two instances serve more than twice one instance's requests per second. Several instances also make a deploy or a configuration change a rolling restart instead of a service window, and a crashed instance's in-flight direct requests are recovered by the others within about a minute (Health and logging).

Prerequisites

  • A load balancer or ingress that health-checks each instance and stops routing to one that fails the check: a cloud load balancer, a reverse proxy with active checks, or Kubernetes readiness probes. Route inference traffic to every healthy instance's public port; no session affinity is needed, because MCP ingress is stateless and every other piece of request state is a row.
  • One PostgreSQL database and one DATABASE_URL for every instance, the Console and the tools (Configure the services). Allow max_connections for one pool of at most 10 connections per instance.
  • Identical configuration files (global and organization YAML) and secrets on every instance. An instance loads them at startup and ignores later edits, so a change is applied by the rolling restart below.
  • The control listener (CONTROL_GATEWAY_HOST, CONTROL_GATEWAY_PORT) bound on an address the balancer's health check and the other services reach, never a public one. The public inference listener answers 404 to every probe.
  • One Console, one Agent runtime and one Collector. Their heartbeat destinations (CONTROL_GATEWAY_URL, MISSION_CONTROL_CONTROL_URL) may name any gateway instance or an internal balancer over the control listeners: heartbeats are shared through the database, so every instance's /readyz sees them.
  • Operator access to the hosts or the orchestrator. No Console role is involved.

Procedure

  1. Start the instances with the same environment and files. Each registers itself in gateway_instance and logs Gateway instance registered with its id; one of them, the holder of the loop lease, runs the background loops and reports mission_control_loop_leader 1, the others log Another gateway instance runs the loops; this one retries every 30 s.
  2. Point the balancer's health check at GET /readyz/traffic on the control port. Choose the check interval and the unhealthy threshold so that their product stays under GATEWAY_DRAIN_SECONDS (default 10 seconds): for example an interval of 3 seconds with a threshold of 2. The probe answers 200 only when the instance has bound its listeners, its database answers and it is not stopping; it does not depend on the Console, the runtime or the Collector. It also answers 503 saturated while the instance holds its in-flight maximum (GATEWAY_MAX_IN_FLIGHT, default 256) and until it is back under 80% of it, so the balancer sends new requests to the other instances; a request that reaches a full instance meanwhile is refused 503 instance_saturated with retry-after: 1 (Model gateway).
  3. To apply a configuration change or a new release, roll the instances:
    1. For a new release, run the migration once, from the new release's platform/ with DATABASE_URL set: pnpm --filter @mission-control/database db:migrate. Instances of the previous release keep running against the migrated database, because migrations stay additive within a release (Upgrade and rollback).
    2. For each instance in turn, stop it (SIGTERM, or the supervisor's stop). Both readiness probes answer 503 draining at once while the instance keeps serving for the drain window, so the balancer stops routing to it; the instance then closes its listeners, finishes the requests and streams in flight (bounded by GATEWAY_SHUTDOWN_TIMEOUT_SECONDS, default 60 seconds), stops its loops, records its stop and exits.
    3. Start the instance again with the reviewed files or the new release, and wait until its GET /readyz/traffic answers 200 before stopping the next one.

Expected result

Every instance's /readyz/traffic answers {"status":"ready"}; exactly one instance reports mission_control_loop_leader 1; during a roll the balancer removes the stopping instance within the drain window, its requests in flight complete, and no request fails or is refused.

Verification

  • On each instance: curl --fail http://<control-host>:4001/readyz/traffic prints {"status":"ready"}.
  • In Prometheus: sum(mission_control_loop_leader) is 1, and mission_control_build_info shows the same version on every instance after a release roll.
  • From a checkout with the test PostgreSQL running (pnpm db:test:up in platform/), the harness performs the roll against a balancer model and asserts that no request failed:
pnpm --filter @mission-control/gateway capacity --profile direct --instances 2 --restarts 2

Expected result: rolling restart: 2 instance restart(s), failed requests 0 and budget: passed. Measured on 2026-09-26 on a developer machine: 216 requests per second across two instances with two restarts, 0 failed requests, p95 101 ms, against 228 requests per second without restarts (Capacity).

Failure recovery

Symptom Recovery
Requests fail with connection errors while an instance stops The balancer removed the instance only after it closed its listeners: its check interval times its unhealthy threshold exceeds GATEWAY_DRAIN_SECONDS. Shorten the check or raise the drain window on every instance.
A new instance refuses to start with STORAGE_MIGRATION_REQUIRED Run the release's db:migrate first (step 3.1). The instances of the previous release keep running meanwhile.
/readyz/traffic answers 503 not_ready The instance cannot reach the database; see Database startup and storage errors.
/readyz/traffic answers 503 saturated on every instance The fleet is full: add an instance, or raise GATEWAY_MAX_IN_FLIGHT when the instances have CPU and database headroom (Capacity). MissionControlInstanceSaturated fires after ten minutes of refusals.
/readyz answers 503 on every instance while /readyz/traffic answers 200 A component (Console, Agent runtime or Collector) is not heartbeating, or heartbeats to an address no instance serves. The inference plane keeps serving; fix the component's CONTROL_GATEWAY_URL or MISSION_CONTROL_CONTROL_URL.
No instance reports mission_control_loop_leader 1 for more than 30 seconds The holder lost its database connection and stopped its loops; another instance takes the lease on its next attempt. Check the holder's logs for Loop leadership will retry and its database connectivity.
A stop takes longer than expected The drain window (10 seconds by default) plus the requests in flight, up to the shutdown timeout (60 seconds); a stream that never ends is cut at the timeout.