Skip to content

Operate Telemetry

Purpose

Operate authenticated OTLP ingestion, the Team span repository, capture/retention and isolated backup/restore while keeping financial evidence independent.

Prerequisites

Use compatible gateway, Collector, runtime and Console (mission-control.trace-native/v3), a migrated PostgreSQL database and the complete ClickHouse migration chain through migration 9. Use separate writer, reader, cleanup, migration and backup credentials. Never run tests or a development cutover against a live/reference instance.

Procedure

  1. Apply current ClickHouse migrations on the intended compatible instance. Working directory: platform/.
pnpm --filter @mission-control/telemetry clickhouse:migrate -- up
  1. Configure gateway CLICKHOUSE_URL, CLICKHOUSE_DATABASE, CLICKHOUSE_COLLECTOR_USERNAME/PASSWORD and CLICKHOUSE_ADMIN_USERNAME/PASSWORD; optional CLICKHOUSE_TLS_CA_PATH verifies private TLS. The historical collector-named database account belongs to the gateway writer, not the Collector process. A partial set fails startup.
  2. Configure Collector MISSION_CONTROL_INTERNAL_URL to private control and MISSION_CONTROL_INTERNAL_SERVICE_SECRET to the Console/Collector transport secret. Set OTLP listener TLS and memory bounds. Build from telemetry/collector/ with make build and run its reviewed config.
  3. The current pipeline is OTLP receiver → memory_limiter → mc_writer, with mc_producer authentication. There is no batch processor, persisted queue, direct ClickHouse exporter or credential store in the Collector. It emits a startup heartbeat and then every 30 seconds.
  4. Configure Console read-only ClickHouse settings. Canonical Requests and ledger Usage remain available from the PostgreSQL database if observed span storage is unavailable. Restart affected processes after environmental changes.
  5. Verify a real standalone external export, a managed execution and a gateway request through the actual Collector, in an Organization whose telemetry.standardOtlp.enabled is true. An Organization with it off stores none of these spans; its Requests and ledger Usage are still recorded. Use exact Organization/Team/Trace scope in the Console.

The gateway exports its own request spans to MISSION_CONTROL_OTLP_ENDPOINT through a batch processor whose queue holds 16 384 spans. When the Collector is slow or down, the spans past the queue are dropped rather than delaying requests; mission_control_trace_spans_dropped_total counts them and MissionControlSpansDropped fires after ten minutes of drops (Metrics and alerts). Requests and the usage ledger are unaffected.

Team storage and maintenance

team_span_payloads, team_span_heads and team_span_deletions are the serving contract. Migration metadata and historical tables may exist, but are not a substitute serving backend. Composite source/producer/span identity deduplicates retries. Heads retain minimum expiry; permanent tombstones prevent old batches resurrecting deleted spans, including across partitions.

The gateway writer needs SELECT/INSERT on payloads and heads, and SELECT on deletions. Console reader needs SELECT on all three. The cleanup/admin identity needs SELECT/ALTER DELETE on payloads, SELECT on heads, and SELECT/INSERT on deletions. Backup/restore uses its own privileges and encrypted backup disk. Do not hand these credentials to the Collector or application SDK.

Maintenance runs at startup and every 60 seconds, coalescing physical cleanup. It removes expired/tombstoned payloads but retains permanent heads and tombstones. Reads already hide expired/deleted spans. team.manage authorizes Delete Trace observations for the exact Team/Trace, including standalone traces. The service records durable audit intent before the external operation, rechecks authority and audits completion/failure. The PostgreSQL database and ClickHouse are separate operations, so no distributed atomic erase is claimed. The result distinguishes logical hidden state and eventual physical cleanup; usage facts are untouched.

Backup and isolated restore

The operator-only TeamObservationMaintenance.backup(id) includes all three Team tables on the configured encrypted disk. restoreIsolated(id) restores under distinct mission_control_restore names. This is a whole-three-table maintenance helper, not a public scoped/signed-manifest API. Retain the database backup plus encryption keys separately; compare all three counts, identities, expiry/tombstone behavior and permitted content on isolated restoration. Never overwrite serving tables or omit heads/deletions.

For old-generation telemetry, use its matching binaries and backup procedure; the current helper does not interpret or migrate that format. Follow safe cutover to prepare a fresh dedicated instance/volume when crossing generations.

Isolated integration verification

Use the Collector's external-process tests to verify protocol, producer isolation, retry and backup behavior against disposable local storage. Prerequisites are Docker, the repository's pinned Go toolchain, and unused test ports. Run one suite per project/port pair at a time.

  1. From the repository root, run the build-coupled suite with an explicit test target:
COMPOSE_PROJECT_NAME=mission-control-governance-verify \
CLICKHOUSE_HTTP_PORT=28123 \
CLICKHOUSE_NATIVE_PORT=29000 \
CLICKHOUSE_TEST_URL=http://127.0.0.1:28123 \
CLICKHOUSE_TEST_NATIVE_ENDPOINT=tcp://127.0.0.1:29000 \
make -C telemetry/collector test-integration
  1. Confirm the suite passes. It starts/migrates only the named test ClickHouse and uses synthetic credentials from the reference example configuration. Replace the final target with test-integration-race for the race-enabled suite.
  2. After testing, remove only that disposable project's containers and volumes:
docker compose -p mission-control-governance-verify \
  --env-file telemetry/reference/.env.example \
  -f telemetry/reference/compose.yaml down --volumes --remove-orphans

Without overrides, the Go integration harness uses mission-control-collector-test on ports 28123/29000. Custom project names must contain only lowercase ASCII letters, digits, and hyphens, start with mission-control-, and carry a -test, -tests, or -verify marker, optionally followed by a hyphenated suffix. HTTP/native ports must be distinct, between 1024 and 65535, and cannot be 8123 or 9000. Query/exporter URLs must exactly match their 127.0.0.1 Compose bindings. All Compose operations, including backup-container lookup, use the same validated project.

If validation fails, correct the test variables before retrying. If ports are occupied, choose unused ports and update both port variables and endpoint URLs. The harness never needs a production credential, application restart, or new API key.

Expected result

Current producer authentication and capture policy govern every ingestion. The Collector contains no broad runtime or ClickHouse authority. Financial ledger amounts cannot be created by uploaded span totals.

Verification

Confirm mission-control.trace-native/v3 readiness, current tables and a real queryable Trace. Stop only a disposable test storage service and confirm canonical Requests remain available while OTLP reports retryable failure. Restore it and verify deduplication and original minimum expiry. Delete a synthetic Trace, replay its prior batch and confirm it remains hidden, then verify eventual payload cleanup.

Failure recovery

Correct invalid key/input before retrying. For control/storage outage restore service and allow stock exporter backoff; do not replay inference to recreate telemetry. Missing read-only configuration is explicit unavailable state, never fabricated spans. Preserve failed backup/cutover evidence and restore only to isolated paths. A generation refusal never authorizes an automatic reset.