Regrade saved answers¶

Purpose¶

Regrade answers creates a persistent grading pass over selected saved experiment outputs. It makes no generation invocation: original inputs, answers, runtime identity, generation states and scores remain intact. You can run the same evaluator version twice and inspect both passes. An evaluator Test on samples is temporary evaluator testing and writes no persistent experiment score; use regrading when the new judgments must become selectable evidence.

Prerequisites¶

Current evaluation.read, evaluation.execute and evaluation.content.read on the experiment's Team. Human permissions are separate from virtual-key inference policy.
An unarchived experiment with completed outputs and retained protected output and dataset evidence. The original generation model or tools need not remain available.
An active evaluator with a saved version and retained prompt or assertion configuration. Each request selects 1–1,000 completed experiment item IDs and 1–10 distinct evaluators, with an explicit version for each.
The gateway evaluation worker enabled, and, for LLM graders, current grader model policy, evaluation identity and Judge spend funding available. No new key or session is required. Policy and permission changes apply on subsequent checks without restart.

Procedure¶

Open Evaluate → Experiments, then the experiment. Grading pass initially shows Original grading. Previous passes remain in the selector.
Under Regrade answers, select the saved outputs by case and repetition. Each checkbox reads the case's title, else its external ID, else its dataset item identifier, followed by its repetition; the case body is never part of that label, and a case whose fields you may not read keeps its identifier. Under Evaluator versions and models, tick each evaluator to grade with: it starts at its newest version, its version list offers every saved version, and the label names that version's grader model or Assertion · no model. An evaluator the list no longer offers — archived or paused while the form stood — stays ticked and is marked Unavailable, so you can see what would be submitted and untick it. Check Selected output count, grader-call count and Cost estimate. For LLM graders the estimate is unknown because token usage varies. Assertions have zero model cost and require no model allowance or Judge spend funding. Selected grader count is shown and actual costs are recorded as work executes; funding is not reserved.
Optionally enter a reason. It is pass metadata and is never appended to the evaluator prompt.
Select Regrade selected answers, then Open grading pass. The worker executes only the selected graders over the pinned saved answers.
Use Grading pass → View pass to inspect previous progress and scores. Compare supports choosing the same experiment as baseline, with separate Candidate grading pass and Baseline grading pass. Export selected comparison retains both selections.

Expected result¶

A pass records its creator, creation time, exact evaluator versions and source output hashes. The original pass is created with the experiment, including retry children; each output binding is filled once when generation saves the answer. Regrade bindings are fully pinned when created. Neither action rewrites the run manifest hash. Historical rows are assigned a deterministic original pass; missing legacy hashes or content remain unavailable, without invented evidence.

Progress, automated score quality and grading failures follow the selected pass. Generation failures are shared. Costs remain all passes and include earlier paid work; child retry costs remain on the child. Manual, feedback and external API scores keep their own provenance and remain independent of automated pass selection. Automated scores keep authorUserId:null; the grading pass and score audit identify the human who requested them.

The Review page and evaluator execution/calibration controls also support pass selection. Human review grades remain shared evidence. Calibration without an explicit pass retains the legacy version-wide latest_agreement contract using original grades. Explicit calibration writes separate evaluator-version/pass agreement; it does not copy or overwrite legacy agreement.

An unchanged publish gate continues to use original grades and legacy calibration. Explicitly selecting a pass in Publish gate requires that pass to cover all completed outputs of a full-dataset candidate run. A partial regrade produces no evidence for that gate. When an agreement bar is configured, the selected evaluator version needs that pass's own calibration. This selection rule does not add a score-coverage threshold to original gates.

Verification¶

Compare the selected pass with Original grading: output hashes and item identities are unchanged, while automated score IDs and pass IDs differ. Export the run and inspect selectedPassId and each automatic score's gradingPassId. Check Grading cost (all passes) and confirm no new generation item or conversation appeared. Re-evaluate an unchanged publish gate after regrading; it continues to use its original evidence selection.

Failure recovery¶

A response lost after creation can be submitted again without changing the form. The stable requestKey returns the original committed pass. A changed body under that key returns 409 request_key_conflict; changing the selection creates a new request key. Current authority is checked even when replaying a committed receipt.

grading_output_unavailable, grading_output_changed or grading_evidence_unavailable means retention or corruption removed executable evidence. Restore the exact retained content if available, or run a new experiment; regrading never reconstructs or regenerates it. Missing protected evaluator configuration returns grading_evaluator_unavailable during creation.

A paused evaluator accepts no new pass and leaves already-queued work waiting. An archived evaluator accepts no new work and cancels its queued executions. Insufficient funding delays queued LLM grades under the existing Judge spend rules. grading_creator_unauthorized means the pass creator lost required current membership or permissions before execution; the original experiment starter's authority is not substituted. Inspect the pass error and create a new authorized pass when ready.

Exports retain pass and evaluator IDs as provenance, including the selected pass's kind, creation time and evaluator versions. Deterministic initial pass IDs contain the experiment ID and permit correlation; exports are not anonymized. Separate experiment/item ID fields, the generation starter, thread and decision IDs remain omitted. Export costs and comparison run costs include all passes; per-item cost fields remain generation-only.

A pass may also be viewed with an explicit saved legacy interpretationId. This changes the selected measurement projection without editing scores. Omitted interpretation selectors preserve raw categories; calibration and gates retain their own explicitly selected evidence. See Measurements and coverage.

See Deterministic assertion graders for the assertion author/test workflow and trusted-tool evidence requirements.

An optional explicit grading plan pins one registered tool-validator enrichment stage and independent generation/enrichment judges. Choose the validator as Enrichment validator and set the dependent evaluators' evidence to Enrichment output; the ticked versions and the plan are one control, so they always match, and View as JSON shows the plan. Waiting for source capture resumes the saved paid result without another business call; terminal enrichment failure or abstention terminates its dependents. See Protected evidence and tool validators.