Skip to content

Sample production traffic

Purpose

Continuously grade a sample of new production Traces with a saved evaluator. Begin with a narrow filter and modest sampling rate, then inspect actual cost and coverage before expanding it.

Prerequisites

Select one Team with an active evaluator and saved version. Creating or editing an auto-evaluation and pausing, resuming or archiving it each need evaluation.manage; listing needs evaluation.read. LLM judgments require captured input/output evidence and Team funding. Metadata-only capture ends those judgments as evidence_uncaptured; assertions with inaccessible required evidence abstain. Capture and content access are separate; see observability.

Procedure

  1. Open a saved evaluator's Auto-evaluation section and choose New auto-evaluation. Enter a name.
  2. Narrow the selection by Agent, Deployment, Key, Outcome, From and To as needed. The time window is UTC. Outcomes are Completed, Failed, Unresolved or Completed or failed. An empty filter means All production Traces.
  3. Set Sampling rate to the percentage of matching Traces to judge, then choose Save auto-evaluation.
  4. Watch Sampled, Enqueued and Scanned to. The last field records how far the sampler scanned finished traffic; it shows — before sampling. Each selected Trace is judged once per auto-evaluation with the evaluator version current when the judgment is queued.
  5. Inspect a judgment on the Trace's Scores panel or Evaluate → Scores, source Judge. Judge execution opens the call and its Trace, Request and cost. The evaluator's Executions table marks Auto-evaluation under Origin; the execution page offers Open judged Trace.
  6. Use Edit to change the filter, Pause to stop sampling while queued judgments still finish, or Resume to start sampling again from now. Archive hides the rule; its executions and the scores they wrote stay, and its name stays taken.

Key selection and later edits

The Key picker searches Team keys by name, owner or pasted ID. Revoked keys are not offered. Any key removes that filter. At 500 or more eligible keys, only the first 500 by name are offered and the form explains the limit.

A saved Agent or Deployment that leaves the picker stays selected as its identifier. A saved revoked Key stays selected as Unavailable. An unrelated edit therefore cannot silently broaden the rule to the whole Team. Changing the sampling rate does not re-judge settled traffic.

The Auto-evaluation section lists each auto-evaluation with its filter, sampling rate, status and counters

The filter and counters describe a subscription to new production Traces. The evaluator on this page supplies the version used for each queued judgment.

Expected result

Matching new production Traces receive judge scores. Values are metadata; reasoning is content. Traffic that finished before the rule existed is never sampled retrospectively.

Judgments share the Team's Judge spend cap; the rule has no separate cap. When funding is exhausted, queued judgments wait as Delayed, while new traffic is skipped. After the window rolls over, sampling resumes with new traffic and does not replay the skipped interval. A paused or archived evaluator also stops sampling.

Verification

Create a narrowly scoped rule, send a new permitted request, and check Sampled/Enqueued and the judgment origin. Open judged Trace and inspect its separate judge call. A sampling rate below 100% need not select every request. Pause the rule and confirm no new matching traffic is sampled while queued judgments can still settle. Live rule edits require no restart or new session.

Failure recovery

The judge execution errors also apply here.

Symptom Meaning and recovery
evidence_unavailable Spans have not arrived. The judgment waits as Delayed and retries for up to 15 minutes after queuing.
evidence_uncaptured Capture was metadata-only or evidence did not arrive within that interval. The judgment fails without another attempt; correct capture for future traffic.
Waiting for the Judge spend window; nothing new is being sampled. Restore permitted funding or wait for its window. Traffic skipped during the pause is not judged later.
Evaluator is paused. Nothing is sampled for it until it is resumed. / An archived evaluator samples nothing. Check the evaluator's status separately from the rule's own status.
Save a version before pointing this judge at production. Save a complete evaluator version first.
Missing New auto-evaluation form Confirm an active saved evaluator and evaluation.manage.
An auto-evaluation with this name exists Choose another name; archived rules retain theirs.
That key is no longer available. Choose another key, or Any key. Review the refreshed list. Choose an eligible key; choose Any key only if the broader filter is intended.