Skip to content

ADR-0011: Governance Record and Span Store

Status

Accepted — 2026-09-03

Context

ADR-0010 made every inference request Trace-native and replaced the Run lifecycle with caller-owned W3C context, standard OTLP ingestion, and an authorization-aware Console. Its first implementation kept two planes merged: SQLite held a trace index, per-request attempt and invocation rows, and attribution rows; ClickHouse held sanitized observed spans and a content-free AG-UI trajectory; the Console merged governed evidence with observed spans through fingerprints, coverage notices, opaque references, and encrypted cursors.

That shape cost more than it returned. Every trace needed rows in six SQLite tables before a span could be read, the gateway wrote to ClickHouse directly, the Collector carried custom exporters, trace identity was pinned in a table that every request had to touch, and the Console explained provenance instead of showing what an agent did. Industry practice (OpenTelemetry-native stores such as those behind Arize and Bifrost) keeps a request-decision record separate from a span store and reads traces from spans alone.

Decision

  • Two planes. The governance plane is one SQLite table, request_decision: one row per gateway request written before dispatch with the decision it received, updated on completion with route, usage, and outcome, swept to indeterminate after a crash. The observability plane is one ClickHouse span store: spans in the OpenTelemetry Collector column set plus derived scope, kind, outcome, capture, and retention columns, with trace_rollup and request_minute materialized from it.
  • The gateway emits spans, not rows. It records a request span (its span id equals the decision row's span id) and one client span per provider try, exported over OTLP to the Collector. It holds no ClickHouse credentials and writes no trajectory.
  • The Collector stamps scope and applies content policy. mcscope overwrites tenant scope from the authenticated credential; mcpolicy drops content under metadata capture and hard-excludes credentials, authorization, secrets, hidden reasoning, files, and foreign data under full; the contrib ClickHouse exporter writes spans.
  • Trace binding is stateless. The gateway continues the incoming traceparent, honours the reserved root a managed runtime asked for, or mints a root; nothing is persisted. Capture policy is resolved from configuration on every request. The acting managed execution is read from its agent.invoke decision.
  • Plain identifiers. Trace and session URLs carry the W3C trace id and the conversation id. Authorization happens on every read; cursors are keyset cursors bound to their filter and scope.
  • The Console reads each plane for what it is. Overview, Audit › Decisions, model pages, and connector pages read request_decision. Observability offers Traces, Sessions, Requests, and Metrics from the span store, with one tree per trace, a span panel whose Request tab links to the decision, and exception lines for denied requests, calls not routed through Mission Control, content not captured, and spans still arriving.
  • Retired stores are dropped. SQLite migration 29 drops trace_index, trace_telemetry_index, trace_attribution, gateway_attempt, provider_invocation, and connector_invocation; ClickHouse migration 8 drops observed_spans and observed_trajectory_events. Both are development resets without data migration, as ADR-0010 established.
  • One compatible set. Gateway, Console, Agent runtime, Collector, and ClickHouse schema ship together under mission-control.trace-native/v2; the readiness heartbeat rejects a mismatch.

Consequences

  • A request is visible under Audit the moment it is admitted; its trace appears once the Collector exports, which the Console names as "arriving" rather than hiding.
  • Deleting a trace removes its spans and rollup rows and leaves the decision rows; the governance record is the audit trail.
  • Span content now travels through the Collector, so the content policy carries the privacy control that the sanitizer and the SQLite hard exclusions used to share; it has the same fixture coverage.
  • Sessions are scoped to one project and grouped by the conversation id the client supplied; no lifecycle API exists.
  • Duplicate exports can drift the materialized rollups until reads hit FINAL; lists tolerate the drift.
  • SQLite remains the governance store and keeps the gateway single-instance; the decision record is designed to move unchanged.

2026-09-23: storage moved to Postgres

The SQLite statements in this record describe the release it was written for. ADR-0012 moved the governance and application tables, including request_decision, into one PostgreSQL database; the two-plane split and the span store are unchanged.

2026-09-24: live governance record

The live governance record is now inference_decision, read under Observe → Requests; each physical attempt's route, usage and cost are usage-ledger facts. Audit › Decisions and the Metrics view are retired.

Superseded decisions

This decision supersedes the following portions of ADR-0010: "Mission Control records request spans and Trace-scoped evidence" as a trace index with attribution rows; "AG-UI adapters … feed a content-free trajectory" as a stored trajectory; and "authorization-aware Logs, Traces, Sessions, and Metrics over authoritative gateway evidence plus independently observed spans and trajectory". The public inference and private control listeners, caller-owned W3C context, standard OTLP as the only transport, the deferral of provider-hosted tools, and the big-bang development cutover remain accepted as written.

Rejected alternatives

  • Keep the merged model and add a span store beside it. Rejected because every trace would still need six SQLite rows and two provenance vocabularies; the Console would keep explaining coverage instead of showing the trace.
  • Store span content in SQLite. Rejected because content volume and retention belong in a columnar store with TTL, and SQLite must stay small enough for the gateway's synchronous admission path.
  • Keep opaque references and encrypted cursors. Rejected because authorization on read already prevents disclosure; opaque references added secret rotation coupling and shared-URL breakage without adding safety.