Skip to content

Read an evaluation run summary

Purpose

Use the private management API to read one experiment's generation and grading stages without downloading answer bodies. Every newly started experiment stores an immutable mission-control.experiment-manifest/v1 identity for its exact dataset version, item revisions and repetitions, subject definition, effective tool policy, attached evaluator versions, loaded model routes and profiles, declared tool schema hashes, and the available runtime identity. Credentials are never included. Historical experiments receive a deterministic legacy projection when read; unavailable model, provider, tool, or runtime revision facts stay explicitly unavailable and the read does not rewrite the record.

The existing experiment status remains the generation lifecycle used by existing clients. The summary is additive and derives a separate overallStatus across generation and the selected grading pass (the original pass by default). Console now uses live summary polling for separate generation and grading cards, known spend and coverage, worker health, and bounded row metadata. See Investigate progress and errors for the Console procedure.

Prerequisites

  • Use the private control listener, such as http://127.0.0.1:4001 for a local installation. The route is not mounted on the inference listener.
  • Use an mcm_… management token whose current human owner and token both grant evaluation.read in the experiment's Team.
  • Working directory: any trusted client directory. Set MC_CONTROL_URL, MC_MANAGEMENT_TOKEN, MC_TEAM_ID, and MC_EXPERIMENT_ID to safe values from your installation.

Procedure

  1. Read the stage-aware summary:
curl --fail-with-body \
  --header "Authorization: Bearer $MC_MANAGEMENT_TOKEN" \
  "$MC_CONTROL_URL/management/v1/evaluation/experiments/$MC_EXPERIMENT_ID/summary?teamId=$MC_TEAM_ID"

  1. Read generation as counts of queued, invoking, completed, failed, blocked, and cancelled items. Cases that produced no answer remain visible in these counts.

  2. Read grading for the selected pass; omit gradingPassId to use the original attached evaluator versions. expected is the number of selected completed answers multiplied by the number of selected evaluators. A judgment that has not been enqueued yet still counts as pending; Test on samples and production auto-evaluations do not enter this experiment summary.

  3. Read overallStatus using this model:

Status Meaning
queued No generation item has started.
running Generation has started and queued or invoking items remain.
grading Generation is terminal and at least one selected judgment is pending or executing.
waiting Generation is terminal and every outstanding selected judgment is delayed.
finished All selected work is terminal with no generation failure, blocked item, cancellation, or judge error. A run with zero configured graders can finish here.
finished_with_errors All selected work is terminal, at least one answer completed, and a generation failure, blocked/cancelled item, or judge error occurred.
failed Generation is terminal and produced no completed answer.
cancelled The experiment was explicitly cancelled; this remains true while already-running work drains.
  1. Treat queueReason as diagnosis rather than an ETA. worker_unavailable means the durable evaluation worker heartbeat is absent or stale; generation_queue, grading_queue, and grading_delayed name the current stage. lastWorkerActivityAt is the last persistent worker heartbeat and advances independently of this GET request.

Expected result

The response is a data object containing generation, grading, overallStatus, selectedPassId, commandRevision, generationStatus, errorCode, generationClaimed, archivedAt, manifestHash, queueReason, lastWorkerActivityAt, workerStale, costs, durations, items, failures, revision, and updatedAt. selectedPassId identifies the selected persistent pass. Add gradingPassId to select a regrade; omitted selectors use the original pass. Selection never changes the stored or historical manifest identity. See Regrade saved answers.

errorCode is the existing safe run-level failure code, or null. It is separate from failures: a pre-claim abandonment such as starter_unauthorized, stale_preflight, or preflight_blocked cancels queued items without creating attempt failures or measured durations. The summary preserves that cause even when failures is empty.

commandRevision is the experiment command concurrency revision used by Cancel and Archive, separate from the summary revision. generationStatus preserves the legacy generation status; generationClaimed remains true until the worker releases its run claim. Console waits for that release as well as terminal stage counters before stopping its loop or offering Archive. Commands still check the supplied revision and current execution authority, so a race returns a conflict rather than bypassing concurrency checks. archivedAt is null until archival.

The summary's revision and updatedAt belong to this projection. They advance when a generation item, any persistent grading judgment, or reconciled attempt cost/failure changes the summary, independently of the experiment record's optimistic-concurrency revision. A repeated reconciliation that finds identical usage does not advance them. Test-on-samples executions, auto-evaluations, idle worker heartbeats, and repeated GET requests do not advance them. A newer response can therefore have the same revision and a newer lastWorkerActivityAt or different workerStale. Process fresh liveness fields even at an equal revision. For cancellation, wait until generation queued + invoking and grading pending + executing + delayed are zero before stopping reads.

costs.generation, costs.grading, and costs.combined each contain knownMicros, coverage, and basis. Coverage is complete only when every recorded paid boundary has measured priced usage; partial retains a known subtotal beside missing facts, and unknown returns knownMicros: null. configured_rate means measured provider usage was priced with the configured rate. external_reported is reserved for an external runner's reported cost, and an aggregate spanning both is mixed. Reservation estimates remain estimates and do not enter knownMicros or prove complete coverage. A measured free attempt can therefore be complete with zero, while an old stored zero remains unknown. Mixed-upgrade runs retain historical item totals as partial facts and combine them with new attempts without rewriting that evidence.

Generation items and grading executions carry separate capture-history markers. During upgrade, distinguishable historical positive cost is frozen as a legacy partial fact. A stored current-success total remains a separate provisional fallback, bound to its governance receipt and Trace when that identity is available. A captured known-cost attempt with the same execution and Trace replaces that fallback as soon as the cost fact arrives; it is not added as another charge. Unbound positive fallback remains visible and conservative. An unrelated failed attempt, matching amount, generation marker, or retry count is not overlap evidence, and a genuinely distinct frozen legacy expense remains additive. If earlier paid history cannot be proved, the known new subtotal remains partial rather than becoming complete.

items contains at most 1,000 content-free rows with experimentItemId, generation status, cost (known amount, coverage and basis), latencyMs, traceId, and entryDecisionId. Each row belongs to the authorized Organization, Team and experiment. Generation failures remain shared; grading failures and failed-duration statistics follow the selected pass. Run costs include all passes. failures contains at most 1,000 bounded records: server-associated experimentItemId, stage, stable allowlisted code, safe resourceId when known, retryable, durationMs, traceId, and the actual failed spanId when available. Provider messages, prompts, tool arguments, results, and secrets are never returned. durations.completedCaseP50Ms measures completed generation cases; durations.failedAttemptP50Ms measures known whole-attempt execution time for failed, blocked, or interrupted attempts, including multi-turn work. It uses the lower median over the entire authorized matching population, including shared generation and selected-pass grading attempts; the duration population is not truncated by the 1,000-row failure-list cap. Crash recovery uses a matching durable terminal timestamp when one exists and never counts worker downtime as execution. A missing measured end or component duration remains null; summary freshness still advances at the recovery mutation time.

Verification

Call the summary while an experiment is queued, after one answer completes, after a typed failure, and after its attached judgments finish. Confirm generation counts always add up to the experiment's item count, missing judgments stay pending, a failed paid attempt contributes cost, and the terminal label is finished_with_errors when any generation item failed even if every expected judgment completed. Repeat the GET and confirm it creates no manifest, heartbeat, usage, or revision writes.

Failure recovery

A 403 response means the token or its current human owner lacks evaluation.read in the Team. A 404 response deliberately does not reveal whether the experiment exists in another Team. worker_unavailable means the private gateway evaluation loop is disabled or has stopped updating its heartbeat; check the gateway and EXPERIMENTS_ENABLED before retrying. Use a returned retryable failure code to diagnose the named trace/resource, but retry remains an explicit user action; the scheduler does not automatically repeat an uncertain tool side effect. A missing span ID means only the trace was known and must not be treated as a guessed span. An unavailable provider or runtime revision in the manifest is a recorded limitation, not a cue to substitute a guessed version.

The worker rotates a durable bounded reconciliation cursor, so late usage for newer runs is not starved by old runs and restart resumes the scan. It also closes attempt records left open by a crash only when the exact item or grading execution state identifies the outcome. A stale cancelled invocation is interrupted and its item is cancelled with durable administrative terminal evidence. That marker remains unmeasured across another crash/restart, so its timestamp never becomes execution duration; unrelated or still-active work is not guessed complete.

For current managed runs, provider_revision_unavailable means the manifest pins Mission Control's route, provider model alias and request profile but the provider supplies no immutable build revision. runtime_revision_unavailable means the managed runtime supplied no immutable revision at launch. Historical projections use the corresponding legacy_…_unavailable limitation codes.

Selected measurement coverage

The summary also returns interpretationId and a measurements array per selected evaluator version/pass, including eligible, scored, abstained, notApplicable, error, pending and cancelled counts. Coverage is scored divided by eligible minus notApplicable; no applicable population is null. Generation reliability and all-pass costs remain independent. See Measurements and coverage.