Check evaluation readiness and run a paid smoke¶

Use static readiness before spending on an experiment, then use a paid smoke to collect execution evidence for one original dataset case. Static readiness makes no model invocation, business tool call, discovery request, credential grant or funding reservation.

Prerequisites¶

You need the Team's evaluation.read and evaluation.execute, readable Agent metadata (agent.read), and agent.test for a snapshot or deployment.execute for a published deployment. Have a saved subject, an active dataset with cases, and any saved active evaluators you intend to attach. Ordinary evaluation users do not need Organization configuration administration or financial-read permission.

Procedure¶

Open Evaluate → Experiments, choose New experiment, and select the subject, dataset, tool policy, repetitions and evaluators.
Select Check readiness. The Console asks the private gateway to inspect the intended run. Review the per-check status, code and severity, and the separate generation case-execution and grading judgment counts.
Review Paid smoke case. It initially selects the first active original case. Select another case if needed, then select Check readiness again. Changing any form setting invalidates the displayed readiness.
To collect execution evidence, select Run paid smoke. The smoke uses the displayed selected-case preflight identity, one repetition and the same subject, tool policy and evaluator versions. Each selected judge runs once on the completed case, even when the full run has several repetitions.
Open the resulting Smoke · … experiment and inspect its item, Trace, recorded costs and judgments. For a full dataset run, select Start experiment after a current readiness check. Its identity is separate from the smoke's one-case identity.

Expected result and verification¶

Required checks combine into blocked if any required check is blocked, otherwise unknown if any required check is unknown, otherwise ready. Advisory checks do not change this verdict. A blocked result disables paid launch. Unknown evidence permits an explicit launch or paid smoke, and the live execution path can still refuse or delay it. Readiness is evidence at checkedAt, not a bypass of admission.

The checks cover subject permissions and settings, original dataset selection, Team model/tool allowances, network evaluator identities, available routes, declared structured-output support, cached tool schema discovery, evidence capture and worker status. An idle worker can be healthy: its heartbeat runs independently of active work. A disabled worker is blocked; a missing or stale heartbeat is unknown. Cached discovery older than five minutes or a latest discovery failure is unknown. Missing or changed schemas with fresh discovery block the selected tool. Tool readiness uses the same effective set as runtime exposure: read-only includes Data and Decision tools, excludes Action and unclassified tools; no tools excludes all; all tools includes even unavailable tools so their refusal remains visible. Excluded tools do not undergo allowance or discovery checks.

Credential configuration, gateway-resolved credentials, connection probes and successful invocation are distinct facts. An LLM judge needs at least one individually eligible JSON-capable route. Assertions need neither a judge route nor an evaluation identity. An incompatible fallback does not block a usable route; credential readiness is assessed only on the candidates that satisfy the actual judge request. Ambient provider credentials remain unverified until execution; preflight does not resolve them. A stored successful model-list probe does not prove tool continuation or JSON judging and does not change routing. The paid smoke provides actual run evidence. Tool validators, enrichment plans, and judge prompts using tool_calls or evidence require full evidence capture and an available observation store; ordinary input/output judging reads the experiment's stored bodies.

Generation counts include every selected case, repetition and arm; judgment counts include each attached evaluator per completed case. plannedInvocationCount adds generation cases, LLM judgments and tool-validator invocations, excluding deterministic assertions; it is not a fixed number of provider/tool requests. Agent model rounds, output tokens and tool usage vary. estimatedMicros is null when cost cannot be estimated from available facts; assertion-only grading has an estimate of zero model cost; validators retain unknown fee estimates rather than appearing free. Assumptions and unknowns travel separately from recorded {knownMicros,coverage,basis} costs. No cost upper bound or guaranteed free run is promised. Funding information is advisory unknown: live funding admission remains authoritative, and a smoke can reach the normal budget refusal or judge-delay path.

Verify that the smoke has one item, retains the original dataset ID/version and selected item revision, and has one planned judgment per attached evaluator. Its immutable manifestHash is an audit identity. The deterministic intentHash excludes creation time, request keys, run IDs and names. Configuration/allowance drift invalidates a supplied preflight at launch and the recorded guard at claim. Existing API callers that start without a preflight retain the previous start contract and normal live admission.

Failure recovery and operator handoff¶

stale_preflight: refresh the form and run Check readiness again. No new experiment is created by a refused launch. A queued run whose recorded guard changed ends with this failure before any item is invoked.
preflight_blocked: review the blocked code. A Team administrator with team.allowance.manage can correct an allowance; evaluation.manage is needed to repair or save an evaluator and its execution identity. No new key/session is needed solely for a live allowance change.
No judge models: the picker distinguishes unavailable generation routes from an empty Team allowance; users without evaluator-management authority cannot edit the selection.
Missing routes, provider settings or discovery/schema drift: hand the safe resource ID, check code and checked time to an operator with organization.settings.manage. The operator edits and validates the configuration file, restarts the gateway and Console to load the same catalog, and refreshes connector discovery through the existing administration flow. Preflight never changes the catalog itself. Run readiness again after the restart/discovery refresh.
Missing evidence: ask the telemetry administrator to review Organization capture settings and any Team/deployment narrowing. Configuration changes require process restart; live capture overrides apply to subsequent invocations. A completed run cannot recover evidence that was never captured.
Worker disabled: the operator enables EXPERIMENTS_ENABLED and restarts the gateway. A stale heartbeat may indicate a stopped worker; inspect gateway logs. A funding delay requires the existing budget owner/operator workflow and another worker pass, not a new credential grant.
request_key_conflict: reuse a smoke requestKey only for the identical body. Generate a new key for a new intended smoke. Identical retries return the original stored result under current authorization, including when the experiment has since progressed.

For assertion authoring, bounds and trusted-tool evidence limitations, see Deterministic assertion graders. Assertions retain ordinary evaluation and pass-creator authorization while bypassing only model admission and funding.

A supplied grading plan pins enrichment and every evidenceFrom selection in the deterministic intent. The Start experiment form always supplies the plan it shows under View as JSON, so a Console start pins each ticked evaluator's version. Validator checks include current tool policy, schema, content authority and the evaluation key’s effective full capture. See Protected evidence and tool validators.