Evaluation workflow¶
Measure Agent quality with versioned cases, experiments, automated graders and human review. Start with one small experiment, then add comparisons and production checks as needed. Evaluation concepts explains the relationships before you begin.
Purpose¶
Turn a quality question into repeatable evidence: what was tested, which Agent version answered, how it was graded, and whether coverage supports the conclusion.
Prerequisites¶
Select one Team in the workspace header. All authorized Teams asks you to select a Team before showing evaluation records or creation forms. All Evaluate tabs share that context; switching Teams resets detail selections, filters and unsaved forms without granting new authority. No new session or restart is needed.
| Work | Permission boundary |
|---|---|
| Read values and metadata | evaluation.read |
| Read cases, outputs, prompts and comments | evaluation.content.read |
| Author cases, definitions and graders | evaluation.manage |
| Run experiments or grading | evaluation.execute, plus the subject's execution permission |
| Record human reviews | evaluation.review |
| Read captured production evidence | observability.read and observability.content.read in the same Team |
Each task guide names its full prerequisites. Executing an experiment, LLM judge or tool validator can consume Team funding. Static readiness, saved definitions and deterministic assertions make no model call.
Procedure¶
Get your first result¶
- Create a small dataset with representative input and expected behavior.
- Define a score and optionally create a grader.
- Run an experiment against a saved snapshot or published version. Check readiness before starting.
- Inspect the output, failures, measurements and cost coverage. Keep the experiment ID or export as your evidence.
Choose the next task¶
| You want to | Guide |
|---|---|
| Add case identity, tags, user context or slices | Author cases and versioned slices |
| Use LLM judgments | Operate LLM judges |
| Check outputs deterministically | Assertion graders |
| Grade tool results or retrieved evidence | Protected evidence and validators |
| Compare models or Agent changes | Create and compare model runs |
| Diagnose incomplete execution | Progress and errors |
| Execute failed cases again | Retry selected failures |
| Apply another grader to existing answers | Regrade saved answers |
| Monitor new production traffic | Sample production traffic |
| Collect human judgment and calibrate | Review and calibrate |
| Block a release without sufficient evidence | Publish gates |
| Run from your build pipeline | Evaluation in CI |
| Import a complete suite | Import an evaluation suite |
Expected result¶
You can identify the dataset version, subject, grading pass and grader version behind a result. Generation success, grading success, quality and coverage remain distinct. Missing evidence or unknown cost is visible rather than counted as success or zero spend.
Verification¶
Open one case and follow its recorded Request and Trace when available. Compare the expected output, actual answer and selected measurement. Check the number of eligible and scored cases before interpreting an average. Keep comparisons on compatible versions and explicitly chosen passes.
Failure recovery¶
| Problem | Recovery |
|---|---|
| No cases or missing content | Dataset curation |
| Readiness blocked | Readiness and operator handoff |
| Generation or grading failed | Investigate progress and errors |
| Judge delayed or malformed output | Judge recovery |
| Production traffic is not sampled | Auto-evaluation recovery |
| Nothing to review or calibrate | Review recovery |
| Publication blocked | Gate recovery |
| Export cannot be imported | Portability formats and recovery |
Before launching an experiment¶
Check readiness, then optionally run a paid smoke on one original case. A smoke is real execution; its partial selection does not establish full-dataset release evidence.
Measurement coverage¶
Read Measurements and coverage to distinguish scored, abstained and not-applicable outcomes. Human reviews remain separate from automated coverage.
Repeatable model comparisons¶
Use comparison groups and templates for pinned dataset selections, grader versions and model arms.
Import a reusable suite¶
Import a complete evaluation suite with a read-only preview, explicit resource mapping and atomic publication of validated resources.