Instrument with OpenTelemetry

Purpose

Add a root span to your Python application, propagate it to a governed model request, export completed spans through standard OTLP, and find the combined Trace. No Mission Control adapter or custom telemetry API is required.

Prerequisites

• Python 3.12–3.14 and an active ordinary virtual key stored in your backend environment.
• A generation model allowed by catalog, Team allowance and key model.generate, plus telemetry.ingest for export; see keys and permissions.
• The inference origin and Collector HTTP/protobuf endpoint supplied by your operator, plus a Console account allowed to inspect that Agent's activity.

Procedure

1. In a new application directory, install the SDK and HTTP exporter versions used by the repository's independent OTLP integration:

python3 -m venv .venv
.venv/bin/python -m pip install 'httpx>=0.28,<0.29' \
  'opentelemetry-api>=1.44,<1.45' 'opentelemetry-sdk>=1.44,<1.45' \
  'opentelemetry-exporter-otlp-proto-http>=1.44,<1.45'
export MISSION_CONTROL_GATEWAY_URL=https://gateway.example.test
export MISSION_CONTROL_API_KEY=mck_example_not_a_real_key
export MISSION_CONTROL_VIRTUAL_MODEL=support
export MISSION_CONTROL_OTLP_ENDPOINT=https://collector.example.test/v1/traces

Replace the fictional values. The inference URL is an origin; the Collector URL includes /v1/traces. Use HTTPS outside local development. Restart the process when changing these environment values. No Agent registration or publication is needed. Standalone OTLP may be exported without any preceding model call.

1. Save this complete program as traced_request.py in the same directory:

import json
import os

import httpx
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

key = os.environ["MISSION_CONTROL_API_KEY"]
session_id = "support-conversation-42"
provider = TracerProvider(resource=Resource.create({"service.name": "support-worker"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint=os.environ["MISSION_CONTROL_OTLP_ENDPOINT"],
    headers={"authorization": f"Bearer {key}"},
    timeout=10,
)))
tracer = provider.get_tracer("support-worker")
try:
    with tracer.start_as_current_span("support.request", attributes={
        "gen_ai.operation.name": "invoke_agent",
        "gen_ai.conversation.id": session_id,
    }) as root:
        trace_id = f"{root.get_span_context().trace_id:032x}"
        headers = {
            "authorization": f"Bearer {key}",
            "x-mission-control-session-id": session_id,
        }
        TraceContextTextMapPropagator().inject(headers)
        with httpx.Client(timeout=30) as client:
            response = client.post(
                os.environ["MISSION_CONTROL_GATEWAY_URL"].rstrip("/") + "/v1/chat/completions",
                headers=headers,
                json={"model": os.environ["MISSION_CONTROL_VIRTUAL_MODEL"],
                      "messages": [{"role": "user", "content": "Say hello briefly."}],
                      "max_completion_tokens": 128},
            )
            response.raise_for_status()
            returned = response.headers.get("x-mission-control-trace-id")
            if returned != trace_id:
                raise RuntimeError("Gateway returned a different Trace ID")
            print(response.json()["choices"][0]["message"]["content"])
            print(json.dumps({"traceId": trace_id, "sessionId": session_id}))
finally:
    # End the root span before flushing, including on request failure.
    flushed = provider.force_flush(timeout_millis=15000)
    print(json.dumps({"flushCompleted": flushed}))
    provider.shutdown()

1. Run .venv/bin/python traced_request.py. Expect a model reply and a JSON Trace ID matching the response header. flushCompleted reports processor flushing, not proof of Collector delivery or storage; check exporter diagnostics and the Console.
2. In Observe → Traces, choose the actual Team and the recent time range, then locate that Trace ID. Confirm the application root and gateway request share the Trace, and inspect the request decision. In Sessions, locate support-conversation-42. Use a fresh root Trace per application operation while reusing a Session ID for one conversation.

Session grouping

On governed model requests and MCP tools/call, x-mission-control-session-id supplies an optional client Session ID: nonempty, at most 200 UTF-8 bytes, with no control characters. It groups under the actual ordinary key's server-resolved observation namespace; user keys are supported. It is grouping metadata, never a credential, MCP transport session, or lifecycle handle. Model discovery ignores this header. Managed deployment invocation uses the AG-UI body's threadId for its Session.

On OTLP spans use the gen_ai.conversation.id span attribute, with the same value as the request header. OTLP does not read the inference Session header. Session and Trace identifiers alone grant no access. Keep IDs opaque and free of personal data; use consistent values on related spans and requests. The gateway's request evidence remains authoritative when interpreting observed data.

Expected result

A gateway request is a child of the injected application span. Only the inference call creates a governed request decision; exporting a span cannot authorize a call or fabricate a decision. Metadata policy may replace the caller span name and remove content while retaining trace structure. See capture modes.

Verification

The source checkout's integrations/opentelemetry-python package contains a broader model/MCP/HTTP/gRPC conformance process. From that directory, run uv sync --locked --all-groups and uv run --locked pytest -q. Its tests exercise propagation, exporters, failures, and separate telemetry status. Verify live delivery using the Trace lookup above.

Failure recovery

For 401 check the virtual key; for 403 check key/owner/Team state and model allowance/key policy; use Gateway errors for the exact protocol contract. If inference works but spans are missing, check the Collector URL, protocol, authentication, exporter logs, writer/storage health, and the Console time filter. Flush after spans end. Do not replay a model call just because telemetry export failed. Policy drops/truncation are reported through Collector metrics rather than OTLP partial-success responses.

Related links

• OTLP contract
• Observability concepts
• Operate telemetry
• PydanticAI adapter