Skip to content

Understand measurements and coverage

Purpose

Distinguish a valid score from a grader that abstained, an answer outside a metric's scope, and a failed grader invocation. Inspect one evaluator version in one selected grading pass, then explicitly choose whether coverage should block publication.

Prerequisites

  • Team evaluation.read to inspect measurement metadata. Reasoning, raw grader output and answer bodies still require evaluation.content.read.
  • evaluation.manage to save evaluator versions, legacy interpretations, calibration or publish gates. Review grades require evaluation.review and the subject's read permission. These human permissions are separate from virtual-key inference policy.
  • Completed experiment answers and an attached evaluator. No new key, session, restart or configuration activation is needed for these controls. Saving a version affects future judgments; use Regrade answers to apply it to existing answers.

Procedure

  1. Open Evaluate → Evaluators and an evaluator. Save version pins the output schema, Metric direction, optional Pass threshold, and Coverage notice threshold, together with the prompt, model and score definition. New versions use schema v2. The default notice threshold is 0.8 (80%). Categorical definitions use Unordered categories; numeric definitions default to Higher is better. An optional threshold records how a scored value is interpreted; publish gate quality thresholds remain separately configured.
  2. Open the experiment and choose Grading pass. Measurement coverage lists each evaluator version's scored, abstained, not-applicable, error, pending/active/unavailable and cancelled counts. It also states selected completed answers versus completed answers across the run. Execution completion and score coverage are different: a completed grader can abstain.
  3. For historical categorical schema-v1 series, expand Save a legacy interpretation. Name the interpretation, choose the evaluator series and explicitly map labels to Abstained or Not applicable. Save at least one mapping. Select the saved interpretation by name, or Raw categories to return to historical values. The saved policy has an immutable interpretationId; saving it never changes default readers, old scores or unchanged gates. Labels such as -1 and -2 have no automatic meaning.
  4. Select Review/calibrate. Create or open a review queue for this experiment and score definition, grade the cases, then choose that queue, the evaluator Version, Grading pass, and optional Saved interpretation under Calibrate. When a scoped link omits the version, calibration selects the version pinned by that pass or interpretation, even if a newer evaluator version exists. An explicitly mismatched version stays refused. Choose the intended selectors and use Apply calibration scope to refresh compatible queues without saving agreement; this control remains available while Calibrate is disabled and requires no queue. Only Calibrate writes agreement. A requested pass and interpretation remain selected even when no compatible queue exists; Calibrate is disabled until a matching queue is available. Another experiment's queue is never substituted. The displayed compared count includes only compatible scored human/judge pairs. Abstentions, not-applicable judgments and interpreted sentinel labels cannot agree merely because both lack values.
  5. In Compare, choose candidate and baseline passes and, when relevant, their saved interpretations. Export selected comparison retains the selections. Compatible numeric directions determine improvement/regression; incompatible directions or unordered numeric policy show Comparison withheld with a reason while retaining both values. A genuinely missing value still shows No score. Selecting one legacy interpretation changes only its bound evaluator/version; other evaluator results and aggregates remain visible. Candidate and baseline interpretations remain selected when first choosing a baseline or recovering from a refused comparison. An automatic series with no scored judgments stays automatic even when human grades exist; an explicit score-source choice remains authoritative.
  6. To enforce coverage, open the Agent's Evaluate → Publish gate. Select Judge, optionally the exact Grading pass and Saved interpretation, and deliberately set Minimum measurement coverage (optional) to a share from 0 to 1. Keep it blank to leave coverage unenforced. Quality and optional agreement thresholds remain separate. A required gate blocks when its configured coverage minimum is missed.

Expected result

A measurement is one of:

Disposition Stored result
scored A valid numeric, boolean or categorical score under its definition. Existing score-value constraints still apply.
abstained No score value; a stable reasonCode and protected reasoning. It remains an applicable gap.
not_applicable No score value; a stable reasonCode and protected reasoning. It is excluded from applicability but its count stays visible.

Execution errors are outside this measurement union. Cancellation is a separate execution state, not an abstention or failed answer.

Coverage is scored / (eligible − notApplicable). Eligible means completed generation answers selected for this evaluator-version/pass series. For 14 eligible answers, 4 scored, 9 abstained and 1 not applicable, coverage is 4/13, about 31%. Thirteen abstentions with no other outcomes yield 0%. With no eligible or applicable answers, coverage is null, displayed as No applicable measurements. Even a gate minimum of zero cannot turn all-NA results into qualifying evidence.

Pending, executing, delayed, error and cancelled judgments, missing expected executions and unavailable or deleted score evidence remain applicable gaps. Independent manual/API grades do not fill missing automatic judgments. Unselected outputs are not not-applicable; a partial regrade still cannot satisfy a gate requiring the full dataset's completed outputs. Generation reliability and run-wide costs remain separate; costs include all grading passes.

The 80% default is a notice, never automatic gate enforcement. Existing gate records have no coverage minimum until explicitly edited. Omitted pass and interpretation selectors preserve original grades, raw historical categories and legacy global calibration. Explicit raw-pass calibration is stored separately from each interpreted calibration, so neither can overwrite the other's agreement or comparable count.

Verification

  • Check the evaluator's Versions → Measurement policy and the selected pass population. Counts sum to eligible answers; cancelled is counted explicitly.
  • Verify a low-coverage notice says it does not enforce a publish gate. Inspect the saved gate text for Minimum coverage before expecting enforcement.
  • Compare raw and interpreted views. Historical score records and exports' raw score values remain unchanged; derived coverage names the selected interpretation. Experiment exports include the policy mapping and selected measurement counts; comparison exports also include the baseline policy.
  • Check Calibrate shows the intended evaluator version/pass/interpretation and actual compared count. A zero comparable sample cannot be saved as perfect agreement.

Failure recovery

  • For pending or failed judgments, use Run summaries, failure details and Regrade answers. Deleted evidence is not reconstructed.
  • A policy from another Team, pass or evaluator version is rejected. Choose the matching pass and series; no latest-policy lookup occurs. A gate interpretation requires an explicit matching pass and Judge. Saving a gate, including an interpreted gate, requires an existing, unarchived evaluator answering the selected score definition.
  • A legacy interpretation accepts only labels in a schema-v1 categorical definition. Create a new named interpretation to correct a mapping; saved policies are immutable.
  • Malformed v2 output, a value on abstention/NA, or a missing/invalid scored value fails grading as invalid_output. Correct the rubric/schema and save a new version.
  • If no compatible review pairs exist, grade the same experiment subjects under the same score definition before calibrating. The legacy calibration option and explicit pass calibration have separate stored evidence.