Skip to content

Metrics and Alerts

The gateway and the Console expose Prometheus metrics on their private listeners. The repository ships alert rules, with unit tests, and a Grafana dashboard under deploy/monitoring/. This page covers scraping them, what each metric and alert means, and what to do when an alert fires.

Prerequisites

  • Prometheus 2.40 or later (the rules are tested with Prometheus 3.14) with network access to the private listeners, and optionally Grafana 10 or later.
  • The Console's private listener enabled: set CONSOLE_INTERNAL_PORT (and CONSOLE_INTERNAL_HOST when Prometheus is not on the same host) and restart the Console. See Health and logging.

Metrics carry no request content, key material, customer subjects, Organization or Team identifiers, model names or raw paths. Keep both listeners off public networks all the same: they reveal traffic volume and timing.

Scrape the services

Add both jobs to Prometheus's scrape_configs. The job names matter: the alert rules select on them. Hosts and ports are examples:

scrape_configs:
  - job_name: mission-control-gateway
    metrics_path: /metrics
    static_configs:
      - targets: ["gateway.internal.example.test:4001"]
  - job_name: mission-control-console
    metrics_path: /metrics
    static_configs:
      - targets: ["console.internal.example.test:3101"]
rule_files:
  - /etc/prometheus/mission-control/alerts.yaml

Copy deploy/monitoring/alerts.yaml to the rule_files path and reload Prometheus.

Expected result: on the Prometheus Targets page both jobs are UP. From the gateway host:

curl --fail --silent http://127.0.0.1:4001/metrics | grep mission_control_build_info

prints mission_control_build_info{version="0.1.0"} 1. The inference listener answers /metrics with 404, and so does the Console's public port.

Metrics

Every series from the Console also carries process="console". The process metrics (mission_control_process_*, mission_control_nodejs_*) are the standard Node.js set.

Metric Type Labels Meaning
mission_control_build_info gauge version Always 1; the running release version.
mission_control_http_requests_total counter listener, operation, status_class Gateway requests. operation is one of chat_completions, responses, embeddings, rerank, models, deployment_invoke, mcp, oauth, management, telemetry, internal, probe, other.
mission_control_http_request_duration_seconds histogram listener, operation Gateway time to response headers. Streamed responses count until the stream starts.
mission_control_provider_attempts_total counter outcome_class Provider attempts: succeeded, cancelled, rejected_request, account_rejected, rate_limited, unavailable, response_invalid.
mission_control_admission_refusals_total counter reason Governance admission refusals by public error code, including the capacity refusals instance_saturated and team_share_exceeded.
mission_control_inflight_requests gauge none Governed requests this instance holds between the end of authorization and the end of the request.
mission_control_inflight_limit gauge none The instance's in-flight maximum (GATEWAY_MAX_IN_FLIGHT, default 256). At the maximum, new requests are refused 503 instance_saturated and /readyz/traffic answers saturated.
mission_control_upstream_skips_total counter kind Skips opened on a model deployment or a connector server after three consecutive unavailable outcomes or a rate limit (Model gateway).
mission_control_upstream_skipped gauge kind Deployments or Connector servers this instance skips right now.
mission_control_trace_spans_dropped_total counter none Gateway request spans the trace exporter dropped because its queue of 16 384 spans was full.
mission_control_guardrail_classifier_unavailable_total counter stage, outcome Guardrail evaluations whose classifier could not answer: degraded (a detect-only input or output stage passed without it) or refused (a stage with a blocking rule failed closed with guardrail_unavailable).
mission_control_governance_stage_duration_seconds histogram stage Time a governed request spent in each stage: prepare (authorization reads), dispatch (the pre-dispatch write transaction, including its lock wait), upstream (from dispatch to the provider or Connector response) and finish (the terminal write transaction).
mission_control_storage_write_wait_seconds histogram stage Time a serialized write transaction waited: queue (this process's turn) then lock (the database-wide write lock). Governed requests take neither.
mission_control_storage_write_hold_seconds histogram none Time an outermost write transaction held the write lock.
mission_control_storage_write_timeouts_total counter stage Writes refused because the queue or the lock timed out (storage_write_lock_timeout).
mission_control_storage_write_transactions_total counter mode Outermost write transactions committed: serialized (behind the database-wide write lock: management, evaluation, chat and background writes) or concurrent (row locks only: the governed request path).
mission_control_loop_leader gauge none 1 on the gateway instance that holds the loop lease and runs the background loops (recovery, the connector and experiment schedulers, cleanup), 0 on every other instance.
mission_control_db_pool_connections gauge state Database pool connections: total, idle, waiting.
mission_control_scheduler_last_run_timestamp_seconds gauge scheduler When the experiments or connectors scheduler last completed a pass. Lag is time() minus this value.
mission_control_span_writes_total counter outcome Producer spans the gateway stored, and span writes it refused.

The storage wait, hold and timeout, pool and build metrics come from both processes; the others, including the write-transaction counter and the loop leader gauge, from the gateway only.

Alerts

Each alert carries severity (critical or warning) and a runbook annotation pointing at its section here.

MissionControlGatewayDown

The gateway's control listener could not be scraped for 2 minutes. Check that the gateway process is running and that /readyz answers; a gateway that exits at startup logs why (for example STORAGE_MIGRATION_REQUIRED or a configuration error). See Health and logging.

MissionControlHighErrorRate

More than 5% of inference requests answered 5xx for 10 minutes. Look at mission_control_provider_attempts_total by outcome_class: unavailable or rate_limited point at a provider; storage timeouts point at the write lock. Canonical request detail under Observe → Requests names each failure's error code.

MissionControlSlowRequests

Chat completion p95 time to response headers stayed above 30 seconds for 15 minutes. Compare provider latency (provider outcome and Trace timings) with mission_control_governance_stage_duration_seconds: the dispatch and finish stages are the request's two write transactions, which wait on row locks only, so a slow upstream stage is the provider and slow write stages are the database.

MissionControlWriteLockTimeouts

A write was refused with storage_write_lock_timeout. The write lock serializes every write transaction, and a timeout means one writer held it for about 10 seconds. Find long transactions in PostgreSQL (pg_stat_activity where wait_event is advisory) and check mission_control_storage_write_hold_seconds. Governed requests take no part in this lock: they run under row locks. A hold of seconds is a migration, a restore, a long management write or a background loop, which runs on the instance that holds the loop lease (mission_control_loop_leader). Callers see the refusal as a retryable error.

MissionControlWriteLockContention

Writers waited more than a second at p95 for the write lock for 10 minutes. The lock's users are management, evaluation, chat and background writes, which run one at a time across processes; governed requests do not take it, so their latency is affected only through the database's load. mission_control_storage_write_transactions_total{mode="serialized"} by process shows who writes behind the lock, and mission_control_storage_write_hold_seconds for how long. Reduce write-heavy background work, or look for long transactions as for MissionControlWriteLockTimeouts. When the dispatch and finish stages of mission_control_governance_stage_duration_seconds grow at the same time, the database itself is the ceiling (Capacity).

MissionControlPoolSaturated

Callers waited for a database connection for 5 minutes. Each process keeps at most 10 connections. Check for slow queries and for PostgreSQL connection limits; a request that cannot get a connection within 30 seconds fails.

MissionControlSchedulerStalled

The experiment scheduler has not completed a pass for 5 minutes, so queued experiments do not advance. Check the gateway's logs for Experiment pass failed; the scheduler recovers on the next pass after the cause clears. The alert only fires where experiments are enabled.

MissionControlSpanWritesRefused

The gateway refused span writes for 10 minutes: the Collector cannot store spans, so Traces stop updating. Requests and the usage ledger are unaffected. Check ClickHouse availability and the gateway's ClickHouse settings; see Operate telemetry.

MissionControlProviderErrors

More than 20% of provider attempts failed on the provider's side (account_rejected, rate_limited, unavailable, response_invalid) for 10 minutes. rejected_request is excluded: it reflects the caller's request. Check the provider's status and the credential; model Test connection confirms reachability.

MissionControlSpansDropped

The gateway's trace exporter dropped spans for 10 minutes: the Collector is slower than the gateway's span rate, or down, and the exporter's queue (16 384 spans) stays full. Requests are unaffected by design; Traces miss the dropped spans. Check the Collector's health and its ClickHouse write path (Operate telemetry), then the gateway's MISSION_CONTROL_OTLP_ENDPOINT.

MissionControlInstanceSaturated

A gateway instance refused requests for capacity for 10 minutes: it held its in-flight maximum (instance_saturated) or one Team held half of it (team_share_exceeded). Compare mission_control_inflight_requests with mission_control_inflight_limit and the p95 of mission_control_http_request_duration_seconds: slow providers hold slots, so check mission_control_upstream_skipped and the provider attempts first. Add an instance behind the balancer (Run several gateway instances), or raise GATEWAY_MAX_IN_FLIGHT when the instance has CPU and database headroom (Capacity). A single Team past its share is that Team's burst; its budgets carry concurrency caps.

Load the dashboard

In Grafana, Dashboards → New → Import, upload deploy/monitoring/dashboard.json and pick the Prometheus data source. Expected result: the Mission Control platform dashboard shows request rate, errors and latency by operation, provider outcomes, admission refusals, write-lock wait and hold, write transactions by mode, governance stage p95, the loop leader, pool connections, scheduler lag, span writes, in-flight requests against the limit, dropped trace spans, skipped upstream targets, guardrail classifier outages and the running version.

Verification

From the repository root, the alert rules and their tests pass with promtool:

docker run --rm -v "$PWD/deploy/monitoring:/rules:ro" -w /rules --entrypoint promtool \
  prom/prometheus:v3.14.0 test rules alerts.test.yaml

Expected result: SUCCESS. CI runs the same check on every pull request.

Failure recovery

Symptom Recovery
The gateway target is down but the gateway serves inference Prometheus cannot reach port 4001; the control listener binds CONTROL_GATEWAY_HOST, loopback by default.
The Console target answers 404 CONSOLE_INTERNAL_PORT is unset on the Console, or Prometheus scrapes the public port.
No process="console" series The Console's private listener is not enabled; see the prerequisites.
Alerts never fire in a test The scrape job names differ from mission-control-gateway and mission-control-console.