Gate publication on evidence¶
Purpose¶
Require recorded evaluation evidence before publishing an Agent version, or show the same quality check as an advisory warning.
Prerequisites¶
evaluation.manage sets the Agent’s gate; evaluation.read reads it. Publication review requires evaluation.read alongside deployment.publish. First publication or access/account changes also need deployment.access.manage. Have completed evidence for the exact definition being promoted; a Preview or grader sample test does not satisfy the gate.
Procedure¶
- Open the Agent's Evaluate tab and find Publish gate under Experiments. Choose the Score definition, Dataset and Score source: Judge, Reviewer, Chat feedback or External scorer.
- Set the Threshold. Numeric definitions use Threshold · Minimum mean. Categorical and yes/no definitions use Threshold · Label and Threshold · Minimum share between 0 and 1.
- Choose Advisory or Required mode. Optionally choose a Judge and Minimum judge agreement, then Save gate. Agreement stays disabled with Calibrate a judge before requiring agreement. until the judge has a calibrated version. One gate stands per Agent; until configured the section says No publish gate.
- In Build, choose Review & publish for the saved snapshot. The review automatically checks access, accounts, funding and evaluation requirements. Inspect Gate, its reasons and the named experiment. Open the comparison retains any explicitly selected grading pass.
- Resolve an unmet Required gate before publishing. An unmet Advisory gate adds a warning. Once ready, choose Publish agent or Publish update; inspect the published version under Live.
This synthetic example shows the current publication review without a configured gate. When a gate is configured, the same review adds its verdict and links to the experiment comparison.
Expected result¶
A gate reads recorded evidence; it never starts an experiment. It matches the definition hash of the exact snapshot or published version being promoted. By default it reads the newest completed, unarchived eligible experiment on its dataset. An explicit grading pass pins its eligible experiment before recent-run limiting, so newer runs cannot displace that evidence.
Only the selected score source counts. A gate on Judge does not include human reviews. A full-dataset gate excludes incompatible partial runs; a successful smoke is insufficient. See slices and release evidence.
No evidence means the necessary experiment, scores or calibration are missing. Required mode blocks publication for this outcome as well as for an unmet threshold. A Minimum judge agreement applies to every judge version that produced counted scores; an uncalibrated version does not silently pass.
The check makes no new paid calls. A successful publication carries the verdict in its audit entry; a refused publication creates no publish entry.
Verification¶
Review a snapshot with matching completed evidence and confirm the gate names the intended experiment. Review a changed snapshot without that evidence and verify a Required gate blocks publication. Record the gate verdict with the release decision. Gate edits and publication require no service restart or new session.
Failure recovery¶
| Symptom | Recovery |
|---|---|
| No experiment has been executed on this exact version. | Run an experiment on the intended snapshot, then reopen Review & publish. Changing the gate policy is a separate authorized decision. |
| A required gate needs a threshold. | Set the minimum mean, or both label and minimum share. |
| This Team has no calibrated version of that evaluator. | Calibrate the relevant judge version against a review queue before requiring agreement. |
| Publish gate changed. Refresh and try again. | Reload and review the current policy before saving. |
| Missing gate section or editor | Reading needs evaluation.read; editing needs evaluation.manage. Publication requires its separate permissions. |