Deterministic assertion graders

Purpose

Use an Assertion evaluator for bounded checks over a saved answer. Assertions produce a yes/no measurement without calling a model. They use the existing experiment, sample-test and regrade queue, version selection, coverage, calibration and gate workflows.

Prerequisites

• Select the correct Organization and Team. Creating an evaluator or version requires evaluation.manage; reading configuration and explanations requires evaluation.content.read alongside evaluation.read. Testing requires evaluation.execute. Regrading additionally requires current read and content authority on the selected saved evidence.
• Create an active Yes or no score definition. Assertion kind and score definition are fixed when the evaluator is created.
• Run the gateway evaluation worker and retain completed experiment output and dataset content. Assertion work requires no judge model, model allowance, evaluation key or Judge spend funding. Generation still requires its own authorization, policy and funding.
• Apply the normal database migrations when deploying this release. Saving a version requires no configuration activation, new session or restart. Existing attached versions stay pinned.

Procedure

1. Open Evaluate → Evaluators → New evaluator. Set Evaluator kind to Assertion, select or create a yes/no definition, then choose Next: criteria.
2. In Assertion configuration, enter the JSON below, choose Next: review, and Create evaluator. Open the evaluator to use Save version for later edits. Each save creates an immutable version with a configuration hash. No model or prompt field is needed.
3. Attach the evaluator to a new experiment, or use Regrade answers on retained outputs. Check readiness counts assertion judgments but adds no judge model invocations or model cost.
4. To inspect behavior without recording scores, choose a completed Experiment and 1–5 Samples, then Start test → Open test results.
5. Open an execution to inspect Reasoning and Assertion output. A failed check records its operator, pointer, expected check and observed value. Configuration and explanation bodies are protected content.

{
  "semantics": "unicode-script-letters/v1",
  "assertion": {
    "op": "all",
    "assertions": [
      {"op": "contains", "path": "/output/text", "text": "Article 39"},
      {"op": "count", "path": "/output/text", "min": 1, "max": 10000}
    ]
  }
}

Supported checks and exact bounds

Configuration is at most 64 KiB of serialized UTF-8 JSON, with at most 100 assertion nodes and depth 10 (the root assertion is depth 1). Composite children cannot be empty. Unknown fields, non-JSON values, invalid pointers and invalid ranges are rejected before saving. Literal JSON nesting is independently bounded to 10 levels. The private management API's 64 KiB request-body limit includes its envelope, so its usable configuration size is slightly smaller.

Pointers use RFC 6901 JSON Pointer syntax: "" selects the evidence root; /output/text selects the answer text; ~1 escapes / and ~0 escapes ~. Arrays use zero-based canonical indices such as /output/items/0; 01 and - do not select an element. Access reads own properties only, with no expressions, code or prototype traversal.

Saved experiment evidence has input, output, expected_output and metadata, retaining their original JSON types. expected_output:null is a real value when no expectation was supplied. Sampled production trace assertions use complete, retained new gateway capture under evaluation and observability content authority. Their input/output retain the captured JSON shape, which can differ from an experiment answer. Legacy captures without trusted completeness, or redacted/truncated/deleted bodies, still abstain. Assertion rules still enqueue and report the coverage gap independently of LLM funding.

Operator	Required fields	Behavior
equals	path, value	Deep JSON equality; object key order is irrelevant and array order matters. Null and empty values remain distinct.
contains	path, text	Case-sensitive substring of a string, with no Unicode normalization. An empty substring matches any string.
regex	path, pattern	RE2 Unicode search over a string. Use anchors for a whole-string check. Backreferences, lookahead and other unsupported RE2 syntax are rejected at authoring/import; patterns never run in JavaScript's backtracking engine. A string longer than 1,310,720 bytes of UTF-8 is searched in overlapping windows; a search those windows cannot decide does not pass. Patterns run in their own RE2 instance, apart from tool rules and guardrails: a pattern too large for its memory is rejected at authoring/import, and a check that still runs the instance out of memory does not pass, while the next check uses a fresh instance. See Regular expressions.
shape	path, required, allowAdditional	Requires a direct object property for each required entry, with type string, number, boolean, array, object or null. False allowAdditional forbids other direct properties. Use separate pointers for nested checks. This is not full JSON Schema.
count	path, min, max	Inclusive array-element or string-code-point count. Bounds are nonnegative safe integers and min <= max. Objects and other types fail.
tool_called	name	Matches a trusted captured runtime tool name, or a canonical name/alias explicitly declared by trusted runtime or registration evidence. Bare names match exactly; there is no global application-specific alias map.
script_ratio	path, script, min, max	script is Arabic or Latin; inclusive finite ratio bounds satisfy 0 <= min <= max <= 1. See the frozen semantics below.
all / any	assertions	Evaluates each child. all fails if any child fails; any passes if any child passes. Otherwise unavailable evidence makes the composite abstain; fully available children use ordinary all/any logic.

A missing pointer or inaccessible required evidence yields abstained, without a score or fabricated value. An existing value of the wrong type fails the check. A shape with a missing required direct property fails. Successful and failed checks produce normalized numeric 1/0 and labels yes/no; protected execution output follows the existing boolean value contract.

tool_called uses complete protected gateway capture, including original earlier turns. It abstains with tool_evidence_unavailable when required trusted capture is unavailable or incomplete. An assistant-authored output.toolCalls or dataset metadata list is never proof that a tool executed. The internal runtime evidence interface distinguishes missing capture from an explicitly captured empty list; a captured empty list fails the check. Literal pointer checks may inspect output fields as ordinary response content.

Frozen script ratio semantics

unicode-script-letters/v1 uses frozen Unicode 17.0 Arabic/Latin Script-and-Letter tables. It normalizes text to NFC, removes case-sensitive [[CITE:...]] markers whose contents contain no ], and counts Unicode code points that are letters in either table. The denominator is Arabic letters + Latin letters, excluding all other scripts, digits, combining marks, punctuation and whitespace. The numerator is the selected script's letter count. If the denominator is zero, the check abstains with no_script_letters. Empty or Unicode-whitespace-only text never passes a language check merely by default.

These examples show the frozen letter-counting behavior:

Input	Assertion Arabic ratio
عé	0.5
Arabic numerals/marks only, or Unicode whitespace only	Abstained
Arabic Extended-A letter ࢠ followed by A	0.5
ع [[CITE:ABC]]	1

Expected result

Each completed assertion has measured execution duration and complete zero model cost. It creates no model Request or Trace. Generation and earlier grader costs remain in all-pass spend; no claim is made that external evidence was free. A sample test writes no score. A persistent scored grade has machine author null, and the pass/audit retain creator attribution. Abstentions remain coverage gaps.

Structured explanations show bounded previews: serialized expected/observed values or pointers longer than 160 characters carry truncated:true. Composites prioritize failed checks, then abstentions, then passes, and retain at most 20 checks and a 16,000-character detail budget, with omittedChecks for additional checks. All assertions still execute; truncation changes only the explanation. Read the protected source/configuration to inspect a full value.

Verification

Confirm the version lists Assertion · no model, the saved configuration and schema v2. Run a sample with matching and nonmatching output; inspect Yes/No, the explanation and zero cost. Test a missing pointer and confirm abstained with no value. Verify that a reader without content permission sees configuration hashes and hidden bodies, never expected literals or observed answer text.

Failure recovery

Fix invalid JSON, pointers, ranges, oversized trees or unsupported regex syntax and save a new version. Existing versions cannot be edited. Update attachments or create a new regrade to use the new version. For sampled trace assertion_evidence_unavailable or tool_evidence_unavailable, review capture, source retention and current observability content authority; adding a claimed tool list to an answer does not resolve it. For missing retained output/configuration, restore exact retained bytes or generate new evidence. A paused evaluator waits; an archived evaluator cancels queued work. Lost regrade-creator authority remains grading_creator_unauthorized even though no model is called.

Related links

• Evaluate
• Measurements and coverage
• Regrade saved answers
• Management API

See Protected evidence and tool validators for capture limits, exact call identity and source inspection. Literal-only saved-output checks keep their original evaluation-content permissions.