Skip to content

Troubleshooting

Use safe error codes, correlation IDs, private probes, and authorization-aware Observability. Never paste a raw key, signature, prompt, tool payload, provider body, or foreign identifier into diagnostics.

Symptom Inspect Recovery
Private /readyz is 503 dependencies.agentRuntime, collector, console; component versions/instance IDs; last heartbeat Start matching mission-control.trace-native/v3 components, correct INTERNAL_SERVICE_SECRET/control URL, and wait for a fresh heartbeat.
/readyz appears on inference or /v1/models appears on control Reverse-proxy/listener mapping Route inference only to port 4000 and private control only to 4001.
Key returns 401 Bearer syntax, pepper, key expiry/revocation, owner/key/Team status and exact membership Install a current key or correct coordinated pepper; do not issue another key to bypass access/limits.
Model returns unsupported_feature Strict Chat Completions/Responses fields Remove hosted/stateful/provider-only fields; use store: false.
Model returns provider_rejected_request Canonical Request outcome and, when captured and separately permitted, diagnostic detail under observability.content.read Correct the request or the deployment's vendor profile; the provider, not Mission Control, refused it. Retrying unchanged cannot succeed.
Model returns provider_account_rejected Provider credential, workspace and billing state for every deployment of the virtual model Correct the environment credential, restart the gateway, then validate the files and restart gateway and Console.
Model returns provider_rate_limited with Retry-After Provider rate limits across the deployments Wait the stated seconds or add a fallback deployment.
Model/MCP returns operation_denied catalog, Team allowance and virtual-key policy Correct the actual allowance/key policy or owner state.
MCP returns tool_ineligible Enabled server, matching target-keyed discovery, declared schema hash Refresh connection and reconcile the declaration; never call upstream directly with its credential.
MCP returns HTTP 400/-32020 MCP-Protocol-Version, Mcp-Method, Mcp-Name, Mcp-Param-*, and matching body fields Correct missing/malformed/disagreeing headers; do not retry unchanged bytes.
MCP returns HTTP 400/-32022 or a Connector test cannot discover Client/server protocol support Upgrade and pin the peer to 2026-07-28; there is no initialize or legacy fallback.
Browser MCP preflight fails Exact MCP_ALLOWED_ORIGINS, requested POST method and bounded request-header set, gateway restart Correct the exact origin/header request and restart after allowlist changes; never add *.
MCP request ends with 499/no body Client/proxy closed the per-request response; Connector evidence state Reconcile authoritative evidence before retrying a side effect; an unconfirmed dispatched cancellation remains indeterminate.
Inference returns invalid_trace_context W3C traceparent shape and managed execution Trace agreement Send a valid non-zero Trace/Span or omit it and use returned context.
429 limit code Model string code or MCP application code -31029; safe retry detail Honor timing or change the binding limit. Keys do not split counters.
Agent invoke says unavailable/not published/not authorized deployment status/channels, audience/API owner grants, entry key, origin and runtime Correct the relevant state; no alternate execution path exists.
Chat says delivery unknown Authoritative thread GET through Console, stored message IDs Reload/reconcile before another send; never duplicate silently.
OTLP unauthenticated/denied ordinary key with telemetry.ingest or dedicated managed capability, current owner/Team/policy Correct actual identity/policy; caller attributes cannot establish scope.
OTLP partial success Per-Trace rejection reason, duplicate/ambiguous Session IDs, bounds Retry only safe rejected partitions after correction.
OTLP resource exhausted batch size, memory, writer/control and ClickHouse health Reduce batch/concurrency, restore the writer/backend, and let stock exporters retry.
Observability no access observability.read scope Grant a current containing role; changing URL filters cannot widen access.
No activity / filtered empty Absolute UTC window and filters Widen preset or clear/correct filters.
Observed telemetry partial ClickHouse reader, final tables, writer/Collector Keep governed evidence, restore observed backend, retry same URL.
Provider operations unknown Provider span support Use request evidence/metrics; provider-hosted parsing is deferred.
Gateway, Console or connector:rotate-credentials exits with STORAGE_MIGRATION_REQUIRED Migrations recorded in the database at DATABASE_URL against the release being started Run pnpm --filter @mission-control/database db:migrate from that release's platform/ with the same DATABASE_URL, then start again; nothing migrates at startup.
db:migrate, bootstrap or the deployment's migrate phase exits with storage_collation_refused The collation, character type and locale provider the message names Recreate the database with the libc C collation and character type, as described in Health and logging; restore a database that holds data rather than recreating it empty.
Logs show storage_transaction_interleaved or storage_write_lock_timeout The session holding the write lock (Find the write-lock holder); queue depth in the message Wait for a holder that is still working. End a stuck one: restart the service its application_name names, or run select pg_terminate_backend(<pid>) as the database owner or the Postgres superuser. Report a repeated message with its log line. See Health and logging.
Logs show storage_transaction_aborted The failed request and its log line Nothing the request wrote was kept. Report the log line: a statement failed and the code caught its error inside the transaction.
Gateway, Console or a command-line tool refuses DATABASE_URL with storage_invalid_url The URL's host, port and percent-encoding in shared.env Percent-encode reserved characters in the user name and password, correct the host or port, then start the process again.
Deployment refuses shared.env or DATABASE_URL Retired DATABASE_PATH/APPLICATION_DATABASE_PATH, URL scheme, password encoding or password= parameter, reachability, server and client tool versions Correct shared.env as described in Deploy to an existing VPS; import SQLite data with Migrate from SQLite.
import-sqlite refuses or fails The named table, row and column; target database migrated and empty Correct the cause and rerun; see Migrate from SQLite.
ClickHouse empty-reset migration refuses Any row or unexpected table in the dedicated development ClickHouse storage Stop all components, replace the exact empty development volume, redeploy complete set. Do not copy old rows.
ClickHouse down migration refuses Migration 6 is irreversible Keep final schema; recover by empty reset and complete forward redeploy.

Find the write-lock holder

Every write transaction holds one database-wide advisory lock, key 7266147, until it ends. When writes fail with storage_write_lock_timeout or storage_transaction_interleaved, find the session that holds it. On the database host, as the Postgres superuser:

sudo -u postgres psql -X -d mission_control -c "select a.datname, a.pid, a.application_name, a.state, now() - a.xact_start as held_for from pg_locks l join pg_stat_activity a using (pid) where l.locktype = 'advisory' and l.objid = 7266147 and l.granted;"

Expected result: at most one row per database, naming the holding process (application_name, for example mission-control-gateway), its state and how long its transaction has been open. No row means the lock is free again.

  • active: the transaction is still working. Wait for it; the queued writers proceed once it ends.
  • idle in transaction and a held_for that keeps growing: the transaction is stuck, and nothing ends it but its process. Restart the service its application_name names (for example systemctl --user restart mission-control-gateway.service as the service account), or stop the command-line tool it names. If that is not possible, end the session with select pg_terminate_backend(<pid>), run as the database owner (the role in DATABASE_URL) or the Postgres superuser.

Either way the stuck transaction rolls back and writes nothing. Run the query again: no row means the other writers can take the lock. Report a stuck transaction with the log lines of the process that held it.

Useful checks

curl --fail http://127.0.0.1:4001/healthz
curl --fail http://127.0.0.1:4001/readyz
curl --write-out '%{http_code}\n' --output /dev/null \
  --request POST http://127.0.0.1:4000/runtime/v1/runs

Expected retired-route status: 404.