Evaluation

Evaluation measures whether an Agent's answers satisfy your expectations. Mission Control connects versioned test cases, recorded execution, automated grading and human review so a quality claim can be traced back to the exact evidence behind it.

From a case to a release decision

Consider a support Agent that must answer an order-status question without inventing an order number:

1. A dataset item records the input and expected behavior. A dataset version fixes the cases used for a comparison.
2. An experiment invokes a saved Agent snapshot or published version against those cases. It records outputs, failures, cost coverage and request evidence.
3. A score definition establishes what a value means: yes/no, a numeric range or categories.
4. A grader produces measurements. It may use an LLM judge, deterministic assertions or a permitted read-only tool validator.
5. A review queue lets people independently grade selected evidence. Calibration compares those reviews with a specific grader version and pass.
6. A publish gate checks recorded evidence for the exact version being promoted against a chosen threshold.

Start with a small dataset and one clear criterion. Add broader coverage once you can explain an individual result.

Keep execution and grading separate

Generation can succeed while grading fails. A grader can abstain because the required evidence is unavailable. A case can be inapplicable to a particular measure. None of those outcomes is equivalent to a quality score of zero.

A grading pass applies pinned graders to saved answers. Regrading can change the judgments without regenerating the answers. A retry executes selected failed cases again and creates linked child evidence; it may incur generation costs and repeat tool activity.

Sample tests help develop a grader. They retain their own results and costs but do not add experiment scores, automated coverage or publish-gate evidence.

Read quality together with coverage

A high average from two scored answers does not establish quality across a hundred cases. Read generation reliability, eligible cases, scored cases, abstentions and missing evidence beside the score. Compare compatible grader versions and passes explicitly.

Human, judge and API scores remain independent. A human correction proposes expected output; applying it creates a new dataset version and leaves the earlier experiment's cases unchanged.

Production evaluation

An auto-evaluation samples matching production Traces for a grader. It processes new traffic rather than replaying everything already stored. Sampling and judgments obey Team funding and capture requirements. Pausing, unavailable evidence and exhausted budgets affect what can be judged.

A publish gate reads existing results. It makes no new model call. Advisory gates report unmet criteria; Required gates block publication when the criteria are not met or evidence is missing.

Evidence and access

Score values are metadata. Prompts, answers, reasoning and case bodies require content permissions. Graders that read captured tools or source evidence also require current observation-content access and retained evidence. Copying a Trace into a dataset creates a separate retained copy; deleting the Trace does not erase that dataset item.

Use the evaluation workflow to choose a task, measurement guide to interpret coverage, and permissions reference for exact authority.