Skip to content

Legacy Trace Cutover

Historical cutover only

This destructive procedure applies only to the earlier Run-based development schema. For current data, use Upgrade and roll back. Do not reset current data.

The procedure was written for the SQLite releases, whose data lived in two SQLite files. The current release stores the same tables in one PostgreSQL database; move an installation that still runs a SQLite release with Migrate from SQLite, which preserves data.

Purpose

Deploy the Trace-native gateway, telemetry plane, Agent runtime, Collector, and Console as one compatible development release. This cutover intentionally does not preserve data from the earlier Run-based development schema.

There is no compatibility mode, route redirect, dual write, or data-preserving rollback across this boundary.

Prerequisites

  • A reviewed checkout using Node.js 24 and pnpm 11.22.0.
  • Permission to stop every Mission Control component and replace the development SQLite and ClickHouse storage.
  • The final environment described in Configure the services, including separate inference and control listener URLs.
  • No application, evidence, or observation rows that must be retained.
  • A recorded commit revision and configuration source for reproducing the environment.

Procedure

  1. Record the current commit and create integrity-checked diagnostic backups of both SQLite files and the YAML catalog.
  2. Stop the Console, gateway process, Agent runtime, Collector, and external integration processes.
  3. Verify the two configured SQLite paths are the intended development files. Do not use a home directory, repository root, or filesystem root as a reset target.
  4. Replace only those two development SQLite files with empty SQLite files. Keep the diagnostic copy outside the active paths; it is not an input to the new release.
  5. Verify the dedicated development ClickHouse tables, discard their rows, and apply migrations through irreversible migration 8. Migration 6 refuses any non-empty otel_traces, observations, observed_spans, or observed_trajectory_events table; migration 7 creates spans, trace_rollup, and request_minute; migration 8 drops the interim observed tables and is deliberately irreversible.
  6. Convert service configuration to split inference/private-control listeners and the Trace-native runtime, Collector, and Console heartbeat URLs. Install the Collector service override that follows the immutable current release.
  7. From the reviewed checkout, run the non-destructive deployment helper:
scripts/deploy_vps.sh
  1. For the first empty-store start, run the idempotent bootstrap command from the promoted release before signing in.
  2. Start the compatible release with reviewed organization files and required secrets; current startup loads them automatically, with no Apply confirmation.
  3. Let the script require private readiness only after Agent runtime, Collector, and Console heartbeats, and require the inference, retired-route, process-path, and Console probes.

The deployment helper builds the platform, Agent runtime, and pinned Collector; backs up the already-reset SQLite files; applies both SQLite migration chains; promotes one immutable release; and never resets or restores storage itself.

Expected result

  • Control SQLite contains request_decision, limit_usage_event, and telemetry_capture_override; the retired trace_index, trace_telemetry_index, gateway_attempt, provider_invocation, connector_invocation, and trace_attribution tables are absent.
  • Application SQLite contains chat_execution and does not contain chat_run or chat_message.chat_run_id.
  • ClickHouse contains spans, trace_rollup, request_minute, and their two materialized views; otel_traces, observations, observed_spans, and observed_trajectory_events are absent.
  • Private /readyz reports ready only when Agent runtime, Collector, and Console all report mission-control.trace-native/v2 within the heartbeat TTL.
  • /v1/chat/completions, /v1/responses, /v1/models, /mcp, and /v1/agents/{agentId}:invoke exist only on inference.
  • Private configuration, heartbeat, telemetry-control, health, and readiness routes exist only on the control listener.
  • Retired /runtime/v1/*, /apps/v1/*, and Console /runs routes return 404 with no redirect.

Verification

From the repository root, run the release tests that still cover this boundary before starting traffic. The SQLite reset test was retired with the SQLite storage; the ClickHouse reset test and the Console journey remain:

pnpm --dir platform --filter @mission-control/telemetry exec vitest run \
  src/clickhouse/trace-native-reset.integration.test.ts
pnpm --dir platform --filter @mission-control/console exec playwright test \
  e2e/trace-native-cutover.spec.ts

Then verify private readiness from the control host:

curl --fail http://127.0.0.1:4001/readyz

Verify a retired inference route stays absent:

curl --request POST --write-out '%{http_code}\n' --output /dev/null \
  http://127.0.0.1:4000/runtime/v1/runs

Expected status: 404.

Failure recovery

Stop every component. Correct the incompatible binary, configuration, credential, or migration input. Reset the exact empty development SQLite files and dedicated ClickHouse storage again, then redeploy the complete compatible set from step 7.

Do not start an older component against the cutover schema. Do not recreate a retired route, restore old rows into final tables, copy observations between schemas, or attempt to cross ClickHouse migration 6 with a down migration. Its down file is deliberately irreversible and the migrator rejects the attempt before executing any rollback statement.