Protected evidence and tool validators¶

Purpose¶

Use a registered Tool validator to check a saved answer against actual sources. Optionally pass its frozen source bundle to independent judges in the same grading pass. Measurement values, coverage, human review and selected-pass gates keep their existing meanings.

Prerequisites¶

Creating or importing a validator/version requires evaluation.manage. Configuration requires evaluation.read and evaluation.content.read to inspect. Running and inspecting source evidence additionally requires observability.read and observability.content.read in the same Team; execution also requires evaluation.execute.
An administrator registers and discovers the MCP tool in the existing connector configuration, pins its sha256: schema hash, and permits it in the Team allowance. Only configured Data and Decision classifications satisfy the shared read-only policy. Action and unknown classifications are refused. This differs from trace read/write classifications.
Saving/importing a validator explicitly provisions or updates the Team's Evaluation execution managed key with the selected validated tools and retained allowed evaluator models/tools. A tool-only identity needs no model. The save is audited. Preflight, claims and invocations never grant or repair that policy; current allowance, identity, tool rules, connector authentication and funding still apply on every invocation.
Enable full capture and the observation store. YAML registration/capture changes require the normal configuration check and process restart. Team/key capture overrides apply to subsequent invocations. Apply the release's migrations to the shared PostgreSQL database before starting the services. No new user session is needed to save a version.

Procedure¶

Open Evaluate → Evaluators → New evaluator, choose Tool validator, select or create the score definition its canonical Measurement must satisfy, then choose Next: criteria.
Under Validator configuration, open Tool and choose the registered tool by name. The list offers only registered tools with a Data or Decision classification that the Team allowance permits, grouped by connector, and search matches a name or a pasted tool ID. Choosing a tool pins its discovered schema hash, shown under the picker; no ID and no hash is typed. Choose Next: review and Create evaluator. Existing graders use Save version for later edits. There is no arbitrary URL field.
Under Arguments, every property of the tool's input schema is a row marked Required or Optional. Map a row From the case with a JSON pointer, or as a Fixed value written as JSON; an optional row may stay Not sent, and a required argument cannot be left unmapped. Saved-answer roots include input, output, expected_output and metadata; captured evidence is available under evidence and complete captured calls under tool_calls when explicitly requested. Missing required evidence refuses execution before a business call. Timeout (milliseconds) and Maximum response size (bytes) accept the bounds below. View as JSON shows the exact document the form submits.
Select completed answers under Experiment and Samples, then Start test. A sample creates no score. For persistent grades, attach the validator to a new run or regrade saved answers.
To enrich independent judges, choose the validator as Enrichment validator on the start or regrade form and set each dependent evaluator's evidence to Enrichment output. Ticking evaluators and planning them is one control, so the chosen versions and the plan cannot disagree, and View as JSON shows the plan. Use {{evidence}} in a downstream LLM template or /evidence/sources in an assertion pointer to inspect the frozen bundle.
Check readiness before a new run. Inspect registration, capture, content authority and funding readiness. Open the execution and its Protected evidence panel to see completeness, hash, call IDs, arguments/results, source IDs and precise Trace/span identity.

{
  "toolResourceId": "tool:corpus/validate-answer",
  "schemaHash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "arguments": {"answer": {"fromPointer": "/output/text"}},
  "timeoutMs": 30000,
  "maxResponseBytes": 262144
}

This is the document View as JSON shows and the management API accepts. The form has no JSON editing mode; an empty pointer is an unmapped argument, so a configuration that maps the whole case with the empty pointer is written through the API.

Configuration is limited to 64 KiB, 100 mapped fields and literal JSON depth 10. Effective mapped arguments are capped at 256 KiB, including repeated pointer values. Values must be finite JSON; functions, undefined fields, sparse arrays and non-JSON objects are refused. JSON pointers read own properties and canonical array indices; /length does not expose array implementation properties. Default timeout is 30 seconds, maximum 60 seconds. The transport/result ceiling is 256 KiB; the registered transport stops reading oversized response streams, including SSE framing. The enclosing private API body remains 64 KiB, and Console import remains 16 MiB.

A saved tool that is no longer registered, read-only or allowed for the Team stays chosen and is marked Unavailable; choose another tool to save a new version. When the tool's schema changed after the configuration was saved, the editor says so: check the arguments, then choose Use the current schema to pin the current hash. A saved argument the schema no longer lists stays visible as Not in the tool's schema until you remove it. A refused save names the reason beside the editor and reads the tool list again.

The MCP result supplies structuredContent, or one JSON text content block, with this strict shape. Hash each source's canonical JSON content using SHA-256 (sorted object keys, preserved array order). Non-finite or non-JSON source content is rejected before hashing. Every usedSourceIds entry must identify a returned source with matching exact content/hash. No returned URL is fetched; URL attribution may remain ordinary source data.

{
  "measurement": {
    "disposition": "scored",
    "value": 1,
    "stringValue": "yes",
    "reasoning": "The returned source supports this check."
  },
  "sources": [],
  "usedSourceIds": []
}

An empty source array is valid only when the validator makes no source reference. To abstain, return disposition:"abstained", a bounded reasonCode and reasoning, with no value/stringValue. not_applicable uses the same no-value shape. Provider lookup failure, invalid output, missing referenced source and schema drift are execution errors, never invented scores. Corpus interpretation and comprehensive citation validation remain the registered tool's responsibility; Mission Control includes no second legal-corpus parser.

{
  "enrichment": {"evaluatorId": "evaluator_sources", "version": 1},
  "judges": [
    {"evaluatorId": "evaluator_support", "version": 2, "evidenceFrom": "enrichment"},
    {"evaluatorId": "evaluator_style", "version": 1, "evidenceFrom": "generation"}
  ]
}

At most one tool-validator enrichment stage precedes independent judges; all stage identities are unique and the total remains at most ten. Cycles, missing enrichment, unknown/cross-Team versions and ambiguous plans are refused. New run intent/preflight hashes include the complete plan. Older stored manifest bytes/hashes remain unchanged and older attachments default to generation evidence. Retry children keep their pinned plan with fresh outputs; regrades create a new pass over saved outputs without generation.

Expected result and retention¶

Protected application content is durable before metadata publishes its reference/hash. A successful validator result waits durably for its governed captured span. Collector delay or worker restart rereads that exact saved result; it does not invoke or charge the tool again or consume its invocation retry budget. Waiting for evidence is visible in pass progress. Dependent judges consume the exact published enrichment hash after it is readable. Failed, cancelled, abstained or unavailable enrichment terminates dependents with enrichment_unavailable.

Authenticated gateway tool argument/result fields now retain up to 256 KiB UTF-8 each through capture and storage, with a 1 MiB total gateway observation-content bound. Other attributes and external/managed producer tool fields keep their 16,384-character bound; model message capture keeps its existing limits. The protected evidence projection remains limited to 512 KiB including repeated source data. These combined bounds are separate from the 256 KiB transport ceiling and do not guarantee that every allowed response fits a complete projection. Sanitized/redacted/truncated values cannot become complete enrichment. Governance result-rule transformations are retained as field-specific transformedFields provenance and make toolResult partial, even when the transformed JSON and source hashes remain valid. Captured arguments describe the actual post-rule arguments sent to the tool. A response can pass the transport limit and still fail with evidence_partial. Frozen bundles are rendered whole for downstream judges; they are not silently clipped and advertised as complete.

New gateway captures carry trusted per-field completeness plus capture-time call/resource/schema identity. The projection’s optional messages entries retain traceId, spanId, captured string input/output, inputComplete/outputComplete flags and metadata. Message fields are frozen in the projection; source-consuming trace assertions and LLM templates read those pinned fields after source verification. Independently incomplete message fields remain unavailable to assertions, while complete fields retain their exact captured bytes. Legacy full-marked or short bodies do not prove completeness. Tool calls correlate by original invocation Trace and call ID, including earlier turns recorded on an item's generation-attempt traces. Replayed tool messages and later conversation activity do not become new calls. Assistant output and dataset metadata never establish tool execution or aliases.

Every evidence read rechecks current evaluation and observability content authority and retained source spans. A result that consumes protected input retains server-published parent projection references and hashes in dependencies; copied arguments and derived output remain subject to those parents and their sources. The earliest inherited expiry applies. Both inspection selectors validate all bound sources before returning judgment output. User-authored projection-shaped data never supplies this authority. Capture-off, expiry, deletion, changed source bytes and missing protected content remain unavailable; a surviving hash never resurrects a shadow copy. Tool-result ingestion waits at most five minutes before missing content becomes unavailable. Projection expiry follows the earliest source expiry. Output-only assertions and templates keep their existing evaluation-content permissions; optional tool capture does not hide their results. Evidence-derived reasoning remains in protected execution content and is inspected with its source evidence, rather than copied into score comments or metadata summaries.

Configured fixed tool fees are recorded as paid tool work; an explicit zero stays known zero, and absent pricing stays unknown. Execution detail and Cost (7 d) use the same governed fee/adjustment aggregation, including failed paid work without a terminal Trace. Mixed known/unknown totals show partial coverage. Run totals include enrichment and every grading pass. Assertions still make no model call. Neither missing pricing nor failed work is presented as free evidence.

Verification¶

Inspect the version's tool/schema hash, then run a synthetic sample. Confirm one governed evaluation Request, the expected Measurement, call-ID correlation and source hashes. A metadata-only viewer must see no arguments, results or source reasoning. Confirm that deleting or expiring a source makes the protected inspection unavailable. Test a collector delay and confirm the recorded tool invocation count and spend remain unchanged while the result waits. Check the selected pass and full-run spend independently.

Failure recovery¶

For registration/schema drift, refresh discovery, review the changed schema and save a new version. Do not rewrite an old version. For revoked allowance or funding, restore the intended policy through its authorized management flow; retries never grant it automatically. Fix missing mappings or invalid response shape before a new sample/pass. A timeout or oversized response is a terminal validator error and may already have incurred a tool fee. For capture/retention/deletion failures, retain the honest error or abstention and produce new authorized evidence; regrading cannot reconstruct deleted content. If an enrichment stage fails, create a new pass after correcting it.