Skip to content

Retry selected failures

Purpose

Create a paid child experiment for failed or cancelled cases while retaining the original run's evidence. A retry uses the original dataset revision, exact selected repetition numbers, subject, evaluator versions and tool policy. Advancing the dataset or evaluator's current version does not replace those pins.

Prerequisites

  • Hold evaluation.read, evaluation.execute, and the subject's execution permission: agent.test for a snapshot or deployment.execute for a published version, in the selected Team. Content still requires evaluation.content.read.
  • The original experiment must be unarchived with terminal generation and a released generation claim. Select only failed or cancelled items. Completed, queued, invoking and blocked-on-approval items cannot be retried.
  • The original subject, input bodies, evaluator versions, known model routing and tool schemas must remain available. Current resource policies and execution identities still apply. Started all-tools work is refused when safe replay cannot be established; retryability metadata never authorizes an uncertain side effect.

Procedure

  1. Open Evaluate → Experiments, then the original experiment. Inspect its errors and traces before paying for another attempt.
  2. In Retry selected failures, check the failed or cancelled item/repetition pairs to repeat. Each checkbox reads the case's title, else its external ID, else its dataset item identifier, followed by its repetition; without evaluation.content.read the case fields are unavailable and the identifier stands. Selecting repetition 3 creates one child item at repetition 3; it does not recreate repetitions 1 and 2.
  3. Choose Retry selected items. The gateway checks current authority, original resource availability and configuration. The worker checks them again before claiming the child; funding and runtime admission remain governed on every invocation.
  4. Choose Open retry. Follow its own generation, grading and recorded spend. Use Original experiment to return; Attempt history links the original to its child runs, including archived children.

Expected result

The child has a new experiment ID and fresh item conversations. parentExperimentId names the immediate original run; each item's parentItemId names the original item. Outputs, scores, statuses, counts, costs and attempts on the original are retained without mutation. Child costs belong to the child. A subset retry or paid smoke cannot replace full-dataset publish-gate evidence unless its immutable membership covers the entire historical dataset version and all repetitions of that run. Existing gate metric thresholds are unchanged; this does not introduce a minimum score-coverage requirement.

Retry does not establish an identical upstream model binary. Known routing, tool and subject facts are compared; unknown provider/runtime revisions stay explicitly unknown. A legacy manifest without sufficient model or tool fingerprints cannot establish a same-configuration retry.

No restart, new key or new session is needed to submit an authorized retry. Restoring configuration may require the normal administrator configuration activation; the retry does not perform it.

Verification

Confirm the child contains only the checked pairs, with the original dataset version, evaluator versions and manifest configuration. Confirm the original's failed rows and scores remain present. Open a child item and verify its conversation/Trace belongs to the child. Navigate between the child and original using the history links.

Failure recovery

A lost response can be submitted again with the same selection and request key: it returns the originally committed child. Keys are scoped to Organization, Team, acting membership and operation kind. Reusing a key with a different body returns 409 request_key_conflict. Current read and execution permissions are checked even on replay, but later configuration changes do not prevent retrieving an already-created result.

retry_configuration_changed or retry_configuration_unavailable means the original configuration cannot be verified. Restore the original known configuration, or check readiness and start a new experiment with current settings. retry_subject_unavailable, retry_input_unavailable, retry_evaluator_unavailable, retry_resource_unavailable, retry_resource_denied, and retry_execution_identity_unavailable identify an unavailable historical resource or current policy restriction. Do not substitute new content under the old manifest.

retry_side_effect_uncertain requires inspection and reconciliation of the original action before starting new work. retry_parent_not_terminal requires waiting for generation and its worker claim to finish. retry_item_not_terminal rejects live and approval-blocked rows; retry_item_not_failed_or_cancelled rejects completed rows. retry_selection_invalid, retry_item_not_found, retry_selection_unavailable, or retry_manifest_changed requires refreshing the original selection. A refused submission creates no visible child or operation result. A refusal at claim remains visible on the child as Run error.

Tool-validator retries retain the pinned kind-specific configuration reference/hash and the original grading plan, including enrichment selections. Creation and worker claim recheck current registered schema/classification, source-content authority and effective tool allowance. A missing or changed protected evaluator body or validator registration returns retry_evaluator_unavailable; saving a newer evaluator version does not replace the parent’s pinned version.