Skip to content

Investigate evaluation progress and errors

Purpose

Follow generation and grading without reloading an experiment, then open the recorded failure evidence for an item. Finished with errors means the work ended with failures; it does not mean every case succeeded. A run with one completed answer and thirteen failures remains visibly unsuccessful.

Prerequisites

  • Sign in and select the experiment's Team. You need evaluation.read in that Team. Access is checked again on every live request.
  • Answer bodies need evaluation.content.read. Trace and span destinations apply their own observability.read and content permissions.
  • Resource IDs are safe metadata. A readable Connector label is resolved only with organization.settings.manage; otherwise the view retains the resource ID or a generic stage name. A resource label grants no inference permission.
  • No new key, session, restart, or configuration activation is needed to use this view after upgrading Console.

Procedure

  1. Open Evaluate → Experiments and open an experiment. Read Generation and Grading separately. Completed generation can still leave the run Grading, or Waiting when every outstanding judge is delayed.
  2. Keep the tab visible. Console reads its session-authorized /{organizationSlug}/evaluate/experiments/{experimentId}/summary?teamId={teamId} endpoint every 2 seconds. It pauses when hidden, refreshes on focus, and retries transport failures after 10 seconds. Responses are private, no-store and contain no answer bodies. Last updated initially shows the recorded summary update time, then the latest successful polling read; worker freshness comes from a separate durable heartbeat, never from polling.
  3. Check the waiting explanation and stale-worker notice. Cancellation stops new work while in-flight generation and grading drain; polling continues until those counters are terminal and the worker releases its generation claim, even when the overall label already says Cancelled. Cancel and Archive use the latest experiment command revision, separately from summary freshness. A concurrent mutation can still return a conflict; refresh before trying again.
  4. Compare Generation cost, Grading cost, and Total known spend with their coverage labels. Unknown is not zero. A known zero marked incomplete can still have missing paid work. Failed attempts contribute to known spend when measured usage is available.
  5. If Run error appears, read that experiment-level cause first. starter_unauthorized means the starter lost execution authority before the worker claimed the run; restore the required permissions or ask an authorized teammate to check readiness and start a new experiment. stale_preflight means checked configuration changed before execution; Check readiness again before starting a new experiment. preflight_blocked means the claim-time readiness checks blocked execution; resolve those checks and check readiness again. These cases can have only cancelled rows, no attempt failures, and no measured component durations. The cause appears both when opening the failed run and when polling observes the failure.
  6. Open a failed row with Open item. Failures reads the newest 1,000 safe failures for this authorized item independently of the run-wide failure cap, and shows the stage, stable code, resource ID, retryability, and measured component timeout when recorded. Open failure span goes directly to the verified trace and selected span (span query parameter); Open failure trace appears when only the trace was recorded. Missing component duration stays unknown, even when a whole-attempt duration exists.
  7. Read judgement errors separately from scores. A failed judgement is not a low score. Inputs, outputs, and scores reflect the page load; reopen an item or refresh the page to fetch those protected results. Automatic progress updates preserve focused controls, expanded rows, filters, and scroll.

Synthetic experiment showing Grading while generation has one completed answer and thirteen failed cases, with separate spend coverage

Notice that grading remains active even though generation has ended, and the known generation spend does not imply complete coverage.

Synthetic experiment showing Finished with errors after one successful generation item, thirteen failed items, and one judgement error

The final label and both stage counters describe the failures rather than presenting a generic completed result.

Synthetic failed experiment item showing search_articles timing out after thirty seconds and an Open failure span link above protected answer panels

The failure's own span link is available directly from the item. The timeout is the recorded component duration, not the experiment or recovery duration.

Expected result

A partially failed run says Finished with errors and reports failed and successful generation counts. An all-failed generation run says Failed and explains that no generation items succeeded. A run abandoned before claim also says Failed, with its recorded Run error and recovery guidance instead of a directive to inspect nonexistent failed invocations. Grading errors remain separate from low numeric or boolean scores. Duplicate row statuses, costs, latencies, and trace/request links use the same bounded metadata projection as the live cards.

Verification

Observe a run while judges are pending and confirm Grading appears without reloading. Confirm failed items remain counted when the final label appears, and that a failure-span link includes the recorded Trace ID and span value. Compare the same run with the private summary API; items and each failure's experimentItemId identify the owning item without exposing inputs, outputs, prompts, arguments, or provider messages.

Failure recovery

If updates are interrupted, the last successful state remains visible with a retry notice. An unavailable session or scope stops automatic reads and asks you to reload to check access; unavailable subjects deliberately return a generic 404. For a stale worker, ask an operator to check the private gateway and EXPERIMENTS_ENABLED. For delayed judges, inspect the linked evaluator and its execution details for availability or spend limits. Repair the resource or configuration and check readiness before starting another experiment; retryability is diagnostic metadata and does not automatically repeat a side effect.

For a persistent regrade, choose Grading pass before investigating. Grading counts, failure cards and failed-attempt duration use that pass; generation failures are shared. Grading cost (all passes) includes earlier paid work. Changing the pass aborts obsolete summary requests. See Regrade saved answers.