Health and Logging¶
The gateway process binds public inference and private control separately. Only private control exposes process liveness, compatibility readiness and the traffic probe a load balancer uses. The Console answers health on a private listener when CONSOLE_INTERNAL_PORT is set; without it, monitor its sign-in page through operator-managed checks.
Private control probes¶
/healthz always returns HTTP 200 with:
/readyz requires completed startup, including a reachable PostgreSQL database with every bundled migration applied, plus Agent runtime, Collector, and Console heartbeats. It returns HTTP 200 with {"status":"ready"} only when all components report mission-control.trace-native/v3, or HTTP 503 with a bounded dependency status. Its body also reports the gateway's release version and the version each component last reported, unknown for one that reports none or whose heartbeat is stale:
{
"status": "ready",
"contractVersion": "mission-control.trace-native/v3",
"dependencies": {
"agentRuntime": "mission-control.trace-native/v3",
"collector": "mission-control.trace-native/v3",
"console": "mission-control.trace-native/v3"
},
"version": "0.1.0",
"componentVersions": {
"agentRuntime": "0.1.0",
"collector": "unknown",
"console": "0.1.0"
}
}
A build without a stamped release, such as a local checkout, reports 0.0.0-dev. The Collector reports no version yet.
/readyz/traffic is the load balancer's probe:
It returns HTTP 200 with {"status":"ready"} when the instance has bound its listeners, its database answers and it is not stopping, and HTTP 503 with {"status":"starting"}, {"status":"not_ready"} (the database is unreachable), {"status":"saturated"} (the instance holds its in-flight maximum, GATEWAY_MAX_IN_FLIGHT, and keeps answering so until it holds fewer than 80% of it) or {"status":"draining"} otherwise. It does not read the component heartbeats, so a Console, Agent runtime or Collector outage never removes the inference plane from the balancer. During a stop, /readyz also answers HTTP 503 with "status": "draining". Point the balancer's health check at this path on the control port, with its interval times its unhealthy threshold under GATEWAY_DRAIN_SECONDS (Run several gateway instances).
Inference must return 404 for every probe. A safe inference liveness check expects authentication failure without a key:
Expected status: 401. Strict YAML and required-secret validation complete before listeners bind. Readiness makes no upstream model/MCP calls and does not wait for OAuth consent or discovery. Use manual model Test connection and MCP Refresh connection for diagnostics.
Console private probe¶
Set CONSOLE_INTERNAL_PORT (and CONSOLE_INTERNAL_HOST when the probe is not local) and restart the Console. From the Console host:
Expected result: HTTP 200 with {"status":"ok","version":"0.1.0"} (the port here is an example). The listener reads no session and no database, so it keeps answering while PostgreSQL is down; it proves the process is serving, not that sign-in works. Every other path answers 404. Keep the port off the public ingress. An invalid port fails Console startup with CONSOLE_INTERNAL_PORT must be a port number.
Both private listeners also serve Prometheus metrics at /metrics; see Metrics and alerts.
Agent runtime probe¶
The private agent runtime service exposes a liveness route on its bind address:
It returns HTTP 200 with {"status":"ok"} when the process is serving. It does not prove gateway reachability or matching runtime inbound/heartbeat secrets; a failed chat run reports runtime_unavailable for those cases. The runtime logs no request bodies, headers, transcripts, or keys.
Probe use¶
- Use liveness to decide whether the gateway process should be restarted.
- Use readiness to keep the compatible component set out of service while storage or a required heartbeat is unavailable.
- Do not restart repeatedly for an upstream provider or MCP failure when gateway readiness remains healthy.
- Alert separately on manual model diagnostics and Connector connection states, which the gateway's unified refresh keeps current (
MCP_DISCOVERY_INTERVAL_SECONDS, default 300 seconds;0disables scheduled refresh); a scheduled failure logs onewarnline with the organization, the server id and the safe error category, never the credential or the upstream response, and appears in Connector diagnostics. Readiness never depends on a Connector.
Database startup and storage errors¶
The gateway, the Console and connector:rotate-credentials connect to DATABASE_URL at startup and check that every migration their release bundles is recorded. A missing migration stops the process before it serves anything:
Run that command from platform/ of the release you are starting, with the same DATABASE_URL, then start the process again. Nothing migrates at startup, so a restart alone never clears this error. Bootstrap applies missing migrations itself; config:check does not check migrations.
A DATABASE_URL that the Postgres driver cannot parse, for example one with a port out of range, is refused before anything connects. The message leaves the URL out because it holds the password:
storage_invalid_url: the database URL could not be parsed; it is not shown because it may contain a password
Correct the URL in the service environment (shared.env on the existing VPS), percent-encoding reserved characters in the user name and password, then start the process again. The deployment helper's check-database-url phase refuses such a URL before it stops anything.
db:migrate first checks how the database orders text, and so do bootstrap and the deployment helper's migrate phase, which run the same migrator. The check refuses a database whose collation, character type or locale provider is not C, C and libc, before any migration is applied:
storage_collation_refused: the database uses the collation C.UTF-8 and character type C.UTF-8 (locale provider libc); Mission Control needs C for both, with the libc provider, so text sorts bytewise. As the superuser, drop it and create it with: create database mission_control owner mission_control template template0 lc_collate 'C' lc_ctype 'C'; then run db:migrate again
The message names the database's actual settings and prints the statement that recreates it. C.UTF-8 is refused on purpose: its character type makes lower(), which uniqueness checks and ordering use, fold non-ASCII letters, which SQLite never did. An empty database can simply be dropped and created again with the printed statement by the Postgres superuser. A database that already holds data must keep it: back it up, then restore the backup with Restore into a new database, which recreates the database with the right settings.
Each process opens a pool of at most 10 connections, waits at most 30 seconds for a free one, and names itself in pg_stat_activity through application_name: for example mission-control-gateway, mission-control-console, mission-control-migrate, mission-control-bootstrap, mission-control-config-check or mission-control-import. Use that column to see which process holds a connection or a lock:
select application_name, state, wait_event_type, now() - xact_start as transaction_age
from pg_stat_activity where datname = current_database() order by xact_start;
Every transaction that may write takes one database-wide write lock, first in an in-process queue and then as a Postgres advisory lock, so write transactions run one at a time, as they did on SQLite. These storage errors can appear in the gateway and Console logs:
storage_transaction_interleaved: the write lock was not acquired within 30000 ms …: a write transaction waited 30 seconds for the write lock. Either a transaction body awaited a statement that lost its transaction context, which is a defect to report with the log line, or the write queue is overloaded. The message states how many transactions were queued ahead. Checkpg_stat_activityfor a long-running transaction.storage_write_lock_timeout: the write lock could not be acquired in time: another process held the database-wide write lock past the same timeout. Look for a long transaction from anotherapplication_name; Troubleshooting shows how to find the holder and end it.storage_transaction_interleavedwith a message about nested transactions: a code defect in transaction nesting. Report it with the log line.storage_empty_list: an empty list cannot be bound as a parameter list on Postgres: a code defect in a query; the request fails without writing. Report it with the log line.storage_write_inside_read_transaction: a code defect that opened a write inside a read transaction. Report it.storage_transaction_aborted: a statement failed inside the transaction and its error was caught; nothing was committed: a code defect. A statement failed, and the code caught its error and carried on. On Postgres a failed statement aborts the whole transaction, so nothing the request wrote was kept, and the request fails rather than report a success. Report it with the log line. For code authors: catch a statement error around a nestedtransaction(), which is a savepoint that rolls back alone, never inside the body of the outermost transaction.
The storage layer logs no statements, and errors from statements issued through its facade carry the database message without the statement's parameters. The database's error detail can still repeat key or row values, for example Key (…)=(…) already exists or Failing row contains (…) for unique, foreign-key and check violations, so a log line that prints a whole error object includes them. A failed query-builder statement, such as one issued during sign-in, can include its parameters in the error message; keep process logs access-controlled.
Gateway logs¶
The gateway uses structured Pino logs. Configure verbosity with LOG_LEVEL; default is info. The logger redacts these field paths:
Application code also sanitizes provider/MCP failures before storing or relaying them. Keep logs access-controlled because metadata such as server IDs, tool names, correlations, timings, and error categories can still reveal operational context.
The resilience path logs Upstream skipped (kind, id, forMs) when an instance stops trying a model deployment or Connector server first, Upstream recovered when it answers again, and guardrail_classifier_unavailable (stage, outcome) when the guardrail classifier could not answer; the capacity refusals instance_saturated and team_share_exceeded count in mission_control_admission_refusals_total (Metrics and alerts).
The console writes framework and mail-transport output to its process streams. Development console mail contains live magic/invitation URLs and must never feed a shared production log system.
Next.js development request/server-function logging is disabled because request URLs can contain one-time authentication material. Keep application and ingress access logs configured to remove query strings and authorization headers. Do not restore verbose request logging to diagnose telemetry.
Telemetry Collector health and logs¶
The production Collector health listener is fixed to loopback 127.0.0.1:13133:
Server available means Collector components started. It does not prove producer authentication, writer/storage success or query visibility. Monitor memory limits, bounded batch rejection, stock exporter retries, signed writer availability and ClickHouse latency/failures separately.
Collector logs contain safe categories and generated correlations only. Never enable a debug exporter or log bodies, protobuf, headers, AuthData, attributes, exception text or row values. The current Collector has no persisted queue. Captured content resides only in its bounded processing path and permitted Team payload storage, never diagnostic logs.
The Console exports no process metrics. The public Mission Control Collector remains Trace-only and does not accept metrics.
Gateway instances and the loop leader¶
Each gateway process registers itself in the gateway_instance table at startup (its id, release and hostname), moves its last_seen_at every ten seconds with one statement outside any transaction, and records stopped_at at shutdown once its listeners drained. The log line Gateway instance registered carries the id.
The Agent runtime's, the Collector's and the Console's heartbeats are rows as well (component_heartbeat): a heartbeat sent to any instance counts for every instance's /readyz, and a second identity of one component within the heartbeat TTL is an identity conflict, as before. Accepted internal request nonces are rows (internal_request_nonce), so a signed internal request replayed against another instance is refused there too.
Every admission a request takes records the instance that dispatched it. Recovery treats a lease as abandoned when it has expired or when its instance stopped or has not been seen for 30 seconds (three heartbeats): a crashed gateway's in-flight direct requests are finished as indeterminate with gateway_interrupted_recovery and their admissions released on the next recovery tick, within about a minute, instead of after the ten-minute lease. Managed turns keep the ten-minute lease. An instance that stops heartbeating for 30 seconds while it still runs is recovered the same way; a finish that arrives afterwards finds the recovered outcome and is refused as a settlement conflict. Admissions written before this release carry no instance and are recovered by their lease alone.
One instance runs the background loops: governance recovery, the connector and experiment schedulers, bundle-staging cleanup and observation maintenance. At startup every instance tries the loop lease, a session-level advisory lock on a dedicated database connection. The holder starts the loops and reports mission_control_loop_leader 1; every other instance logs Another gateway instance runs the loops; this one retries every 30 s and takes the loops over within 30 seconds of the holder stopping or losing its connection. A holder that loses its database connection stops its loops at once. Rows in gateway_instance accumulate one per process start; nothing removes them.
Graceful restart¶
The gateway handles SIGINT and SIGTERM in order. Its readiness probes answer draining while both listeners keep serving for GATEWAY_DRAIN_SECONDS (default 10 seconds), so a load balancer stops routing to it. It then closes the listeners: idle keep-alive connections end at once, requests and streams in flight finish, bounded by GATEWAY_SHUTDOWN_TIMEOUT_SECONDS (default 60 seconds), after which the remaining connections are destroyed. It then stops the background loops and releases the loop lease when it holds them, drains owned OAuth token exchanges, records its stop in gateway_instance, flushes tracing and closes its database pool. A second signal is ignored. OAuth exchanges retain their bounded transport deadlines even when the original caller has cancelled; this allows a rotated refresh token to be saved before shutdown. A single instance without a balancer may set GATEWAY_DRAIN_SECONDS=0, since nothing observes the window; the test suites do. Verify both private probes after restart.
The Console handles SIGTERM and SIGINT the same way for its database: a transaction already open finishes on its connection, while any new statement during shutdown fails at once with storage_shutting_down: the Console is shutting down instead of opening a new connection. Requests still draining at that point report a temporary storage failure.
Canonical budget definitions, reservations, physical-attempt claims/settlements, Requests and management audit are durable in the PostgreSQL database. Active I/O renews its existing lease without another request or spend admission. Unknown termination retains its debit/capacity until confirmation or actual lease recovery; startup recovery does not turn unknown work into free usage. The gateway uses that database's transactional admission; the supported deployment is one Console and one or several gateway instances per database (Configure the services, Run several gateway instances). No distributed guarantee across independent databases is claimed.
Current observability boundary¶
Observe → Requests reads canonical decision evidence; Usage reads ledger amounts. They remain available independently of observed span storage. Traces and Sessions read Team spans under actual scope. Metadata, captured content and export are separately permissioned. The Collector synchronously forwards through the gateway writer; acknowledged export is not a private Chat completion or an additional financial fact.
Agent runtime, Collector, and Console send a private compatibility heartbeat at startup and every 30 seconds. Private /readyz is 503 for a missing, mismatched, identity-conflicting, or older-than-120-second heartbeat. The query service runs inside Console and has no separate heartbeat.
There is no public metrics/logs/profiles ingestion, arbitrary Collector pipeline, live tailing, attachments, alerts, incident workspace, public observation API, arbitrary workflow recovery, or distributed health aggregation. Use supervision, split-listener probes, safe structured logs, writer/ClickHouse health, Observability, model/Connector health, governed decisions, and Audit. Scores and score comments belong to Evaluate.
See Telemetry operations, Troubleshooting, Backup and restore, Upgrade and rollback, and Security behavior.