Skip to content

Standard OTLP Trace Ingestion

Purpose

Export application spans using standard OTLP HTTP/protobuf (POST /v1/traces) or OTLP gRPC (TraceService/Export). The Collector is Trace-only; no preceding inference or custom lifecycle registration is required.

Prerequisites

Use an active ordinary user/service virtual key with telemetry.ingest, a valid Team/owner and an authorized observation namespace. The key's Organization must set telemetry.standardOtlp.enabled: true in its organization YAML; otherwise every export is refused as permission denied, the managed runtime's included. Configure the standard exporter with that key as its per-request bearer credential. The managed runtime uses a separate short-lived OTLP capability. A Collector-wide API key is never an appropriate replacement.

Protocol and identity

The Collector authenticates the producer, then synchronously forwards a bounded batch through mc_writer to the signed TypeScript gateway writer. The writer reauthenticates current key/credential/member/service/Team/policy and resolves capture/namespace before storage. There is no persisted credential queue and no ClickHouse credential in the Collector.

The gateway exports its own request spans the same way, only for Organizations with standard OTLP enabled. One gateway batch mixes Organizations: the writer resolves each span's Request separately, and a span whose Organization has standard OTLP off is dropped alone while the rest of the batch is stored.

A Trace is Organization + Team + Trace ID. Span identity additionally includes source, authenticated producer and span ID. Sources distinguish gateway, managed execution and external logical key. External secret rotation changes credential-version provenance without creating a new logical producer. Managed producer and credential-version identity are the execution ID. Uploaded service.name, mc.*, customer, Team or payer labels cannot override authenticated scope.

The managed OTLP capability authenticates producer identity; it does not attest body integrity or prevent bearer replay. Storage deduplicates composite identity, preserves minimum expiry and enforces permanent tombstones across retries.

Capture modes

Organization maximum policy is narrowed by Team, applicable key and deployment policy. metadata removes content; full retains bounded permitted content after hard exclusions. Names, errors, events, previews, text search, attributes and URLs are content. Credentials, private transport proofs, hidden reasoning, unrelated memory/files and signing material are excluded even in full capture. Current policy is resolved per ingestion; it does not rewrite historical data.

Retention uses server time and minimum historical expiry; forged start time or replay cannot extend retention. Expired/tombstoned spans are hidden before asynchronous physical cleanup. Financial ledger and private conversation retention are independent.

Bounds and response semantics

HTTP/gRPC ingress is bounded to 4 MiB. The writer accepts at most 1,000 spans and 4 MiB serialized JSON per batch with a five-second internal request deadline. Client exporters should use bounded batches and ordinary backoff.

Invalid credentials, invalid input or permanent scope/policy failure reject the request. Control/storage outages return retryable service failure to stock HTTP/gRPC exporters. A lost response or partial physical write may cause a retry; composite identity and tombstones keep the read result stable. Policy drops/truncation do not masquerade as OTLP partial success proving storage. Exporter flush success alone is not evidence that a span is queryable.

Procedure

  1. Configure an OpenTelemetry exporter or PydanticAI integration with the actual key and Collector endpoint.
  2. Start an application root span and propagate W3C context to model/MCP calls. End spans before flushing.
  3. Use gen_ai.conversation.id for optional Session grouping under the key's server-resolved namespace; direct inference uses x-mission-control-session-id.
  4. Find the exact Team/Trace in Observe → Traces. A standalone external Trace need not have a canonical Request.

Expected result

Gateway, managed and external spans can correlate without uploaded telemetry granting authority. Canonical Requests remain in the PostgreSQL database and usage comes only from physical ledger facts. Team storage uses team_span_payloads, team_span_heads and team_span_deletions together.

Verification

Verify one standalone export and a related model request, then test an invalid key and a foreign Team lookup. A metadata-only viewer must not read captured text. Rotate the key and verify its logical producer remains stable.

Failure recovery

Restore Collector/control/ClickHouse availability and let the exporter retry telemetry. Do not replay an already paid inference request solely because export failed. Correct permanent authority/input errors before retry. Deletion hides spans through permanent tombstones; physical cleanup is eventual and does not clear usage facts.