Evaluate a change in CI

Purpose

Measure a proposed Agent version from a continuous-integration job and post the comparison back to the pull request. The job holds a scoped mcm_… management token and no browser: it finds the dataset, starts an experiment on the snapshot or published version under test, waits for that experiment to finish, reads the comparison against a baseline, fails the job on a regression, and prints the Console link a reviewer opens. Mission Control notifies nobody — the job owns the comment.

Prerequisites

A management token scoped to one Team, holding evaluation.read, evaluation.manage and evaluation.execute, plus the subject's own right: agent.test for an Agent snapshot or deployment.execute for a published version. Create it under account Access as a scoped expiring token and store the one-time secret as a repository secret; the default lifetime is capped at 30 days, and revoking the person's membership revokes their tokens immediately.

The job sets these itself:

Variable	Meaning
MC_MANAGEMENT_TOKEN	The mcm_… secret, from the repository secret store
MC_CONTROL_URL	The trusted private control origin, for example http://127.0.0.1:4001
MC_CONSOLE_URL	The Console origin, used only to compose the link
MC_ORG_SLUG	The Organization slug in Console addresses
MC_TEAM_ID	The Team every call is scoped to
MC_AGENT_ID, MC_SNAPSHOT_ID	The subject under test
MC_BASELINE_ID	The completed experiment to compare against
MC_DEFINITION_ID, MC_EVALUATOR_ID, MC_EVALUATOR_VERSION	The exact intended primary definition and automatic grader/version

The control listener is private: never expose it, or the token, to an untrusted browser or backend client. The same token cannot invoke a public model. Bodies are strict JSON at most 64 KiB, responses are {"data":…}, and errors are {"error":{"code","message"}}.

Procedure

1. Find the dataset, or create it once

curl --fail-with-body --silent \
  --header "Authorization: Bearer $MC_MANAGEMENT_TOKEN" \
  "$MC_CONTROL_URL/management/v1/evaluation/datasets?teamId=$MC_TEAM_ID" \
  | jq -r '.data[] | select(.name == "Support questions") | .id'

Store the id as MC_DATASET_ID. A Team that has none yet creates one, which needs evaluation.manage:

curl --fail-with-body --silent \
  --header "Authorization: Bearer $MC_MANAGEMENT_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{"teamId":"'"$MC_TEAM_ID"'","name":"Support questions"}' \
  "$MC_CONTROL_URL/management/v1/evaluation/datasets"

A dataset starts empty at version 0. Add items with POST /management/v1/evaluation/datasets/{id}/items, or read a mission-control.dataset/v1 file back with POST /management/v1/evaluation/datasets/{id}/import, which takes {"teamId","expectedRevision","document"}.

2. Start the experiment

EXPERIMENT_ID=$(curl --fail-with-body --silent \
  --header "Authorization: Bearer $MC_MANAGEMENT_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "teamId":"'"$MC_TEAM_ID"'",
    "name":"Pull request '"$PR_NUMBER"'",
    "datasetId":"'"$MC_DATASET_ID"'",
    "subject":{"kind":"agent_snapshot","agentId":"'"$MC_AGENT_ID"'","snapshotId":"'"$MC_SNAPSHOT_ID"'"},
    "toolPolicy":"read_only",
    "evaluatorIds":["'"$MC_EVALUATOR_ID"'"],
    "repetitions":1
  }' \
  "$MC_CONTROL_URL/management/v1/evaluation/experiments" | jq -r '.data.id')

The answer is 201 with the whole experiment record. A published version is the subject instead as {"kind":"agent_version","agentId":"…","deploymentId":"…"}. Attach judges with "evaluatorIds":["evaluator_…"]; each attached evaluator contributes the version that stands when the experiment starts. Set MC_EVALUATOR_VERSION to that intended pinned version; a mismatch is refused by the comparison policy rather than silently selecting another version. An experiment consumes Team funding, and evaluation.execute is the permission that says so.

3. Wait for it to finish

Read the stage-aware summary until generation and grading have drained. The legacy experiment record's status covers generation only; completed is not proof that attached judges have finished.

while :; do
  SUMMARY=$(curl --fail-with-body --silent \
    --header "Authorization: Bearer $MC_MANAGEMENT_TOKEN" \
    "$MC_CONTROL_URL/management/v1/evaluation/experiments/$EXPERIMENT_ID/summary?teamId=$MC_TEAM_ID") || exit 1
  OPEN=$(printf '%s' "$SUMMARY" | jq -er '.data | .generation.queued + .generation.invoking + .grading.pending + .grading.executing + .grading.delayed') || exit 1
  [ "$OPEN" = "0" ] && break
  sleep 10
done
printf '%s' "$SUMMARY" | jq -e '.data.overallStatus == "finished"'

finished_with_errors, failed, and cancelled fail this recipe. Cancellation keeps its label while in-flight work drains, so the loop uses the stage counters. costs pairs known spend with coverage, and items and associated failures carry bounded content-free metadata. No configured graders means zero expected grading work. Existing legacy record endpoints remain compatible. Apply a job timeout for workers or judges that cannot recover; the waiting reason is not an ETA.

4. Read the comparison and decide

curl --fail-with-body --silent \
  --header "Authorization: Bearer $MC_MANAGEMENT_TOKEN" \
  "$MC_CONTROL_URL/management/v1/evaluation/experiments/$EXPERIMENT_ID/compare?teamId=$MC_TEAM_ID&baseline=$MC_BASELINE_ID&primaryDefinitionId=$MC_DEFINITION_ID&sourceKind=judge&evaluatorId=$MC_EVALUATOR_ID&evaluatorVersion=$MC_EVALUATOR_VERSION" \
  | jq -e '.data | .comparability.compatible == true and .summary.pairedScoredCases > 0 and .summary.excludedFromQuality == 0 and .summary.unmatchedRepetitions == 0 and ([.rows[].verdict] | all(. != "regression" and . != "no_score"))'

jq -e exits non-zero when the expression is false, which fails the job. Set MC_DEFINITION_ID, MC_EVALUATOR_ID and MC_EVALUATOR_VERSION to the intended exact series. This illustrative job policy refuses incompatible, missing and unmatched evidence; it is not an automatic model recommendation or a replacement for a configured publish gate. Each aggregate names one score definition and one source — judge and human values are separate series and are never merged — with count, candidateMean, baselineMean, candidateTrueRate, baselineTrueRate and regressions. Narrow to one series with &definitionId=…&sourceKind=judge. The whole experiment, its items, every live score and these aggregates are also available as one file from GET /management/v1/evaluation/experiments/{id}/export?teamId=…&baseline=….

5. Compose the link and post it

The API answers with ids and never with absolute URLs, so the job builds the address from its own Console origin:

echo "$MC_CONSOLE_URL/$MC_ORG_SLUG/evaluate/experiments/$EXPERIMENT_ID/compare?teamId=$MC_TEAM_ID&baseline=$MC_BASELINE_ID"

That is the Console comparison path. Add the same primary metric, grader/version, pass and interpretation selectors to preserve the evidence the job read.

6. A GitHub Actions job

The keys in the example below belong to GitHub's own workflow schema, not to Mission Control.

name: Evaluate the pull request
on: pull_request
jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    env:
      MC_MANAGEMENT_TOKEN: ${{ secrets.MC_MANAGEMENT_TOKEN }}
      MC_CONTROL_URL: ${{ vars.MC_CONTROL_URL }}
      MC_CONSOLE_URL: ${{ vars.MC_CONSOLE_URL }}
      MC_ORG_SLUG: ${{ vars.MC_ORG_SLUG }}
      MC_TEAM_ID: ${{ vars.MC_TEAM_ID }}
      MC_AGENT_ID: ${{ vars.MC_AGENT_ID }}
      MC_SNAPSHOT_ID: ${{ vars.MC_SNAPSHOT_ID }}
      MC_BASELINE_ID: ${{ vars.MC_BASELINE_ID }}
      PR_NUMBER: ${{ github.event.number }}
    steps:
      - uses: actions/checkout@v4
      - name: Start the experiment, wait for it and compare
        id: evaluate
        run: ./.github/scripts/evaluation.sh
      - name: Comment the comparison
        if: always() && steps.evaluate.outputs.link != ''
        env:
          GH_TOKEN: ${{ github.token }}
        run: 'gh pr comment "$PR_NUMBER" --body "Evaluation: ${{ steps.evaluate.outputs.link }}"'

./.github/scripts/evaluation.sh holds steps 1 to 5 and ends by appending link=<the composed URL> to $GITHUB_OUTPUT.

Expected result

The experiment is a record: it names the dataset version, the subject it resolved with its definition hash, the tool policy and the repetitions, and nothing about it changes afterwards. Each item is one invocation funded by the Team, with its own Trace and Request. The comparison pairs immutable dataset item revisions and repetitions within one dataset. Different dataset versions can be explored; new, missing, revised cases and unequal repeats are listed as unmatched. See Compare evaluation evidence for sample limits and denominator definitions.

A job that starts an experiment on the same snapshot as the Team's publish gate produces exactly the evidence that gate reads: the gate joins a proposed release to its experiments by content hash, so a comparison a pull request executed is what the release review shows later.

Verification

Read the same experiment in the Console at $MC_CONSOLE_URL/$MC_ORG_SLUG/evaluate/experiments/$EXPERIMENT_ID?teamId=$MC_TEAM_ID and confirm the counts match the record the job read. Confirm the token is attenuated as intended by asking for something outside its scope — a Team it does not name answers 403 not_authorized, never a listing.

Failure recovery

• 401 authentication_failed — the token is expired, revoked, or the bearer is not an mcm_… secret. Virtual keys are refused at this door.
• 403 not_authorized — the token lacks a permission at the scope it asked for, or the caller lacks the subject's own right: agent.test for a snapshot, deployment.execute for a published version. A token can never hold more than its owner holds today.
• 404 not_found — the id belongs to another Team, or nothing by that id exists. Existence is never revealed before the permission is checked.
• 409 revision_conflict — a dataset import named a revision that has moved; read the dataset again and repeat with the revision it now carries.
• 409 resource_conflict — a score was posted twice with the same reference. One live API score stands per reference and author, so a retried step is safe: treat this as success.
• 400 invalid_request — an unknown body field, an unknown query key, a duplicate query key, or a body above 64 KiB. Import a file larger than that through the Console, which accepts up to 16 MiB.
• An experiment that stays queued — the gateway is not executing experiments, or the Team's funding or its Judge spend cap stopped it. The Console's experiment page states which.
• An experiment that ends failed — the record names an error code, and each failed item names its own. starter_unauthorized means the permissions the token's owner held when the experiment started had lapsed by the time the gateway claimed it.
• A comparison refused — compare needs two completed, unarchived experiments in the same dataset. Select compatible grader/version/interpretation evidence explicitly; unmatched revisions stay excluded and visible.

Related links

• Management API
• API overview
• Evaluate
• Roles and permissions

Repeatable comparison jobs

The private comparison group/template APIs let a job launch 2–5 immutable arms with one scoped printable-ASCII request key. Check readiness first, retain the returned intent/configuration hashes, then retry the identical launch body/key after response loss. Pin the historical dataset version (and optional slice version), complete enrichment/judge plan, and primary definition plus exact grader/version. Group metadata returns each experiment ID and current stage-aware summary; poll until its generation and grading settle, then retain the original pass IDs in Compare URLs.

Group admission is limited to 1,000 generation-plus-model/tool-grader worker attempts, including the current five-attempt grading recovery policy. Assertions add no model/tool call. Model rounds, provider retries and token/fee prices remain separate financial uncertainties. Versioned templates retain explicit references; an edit never changes an already-launched group. A hard-crash group's invocation_indeterminate item requires deliberate recovery, not an automatic paid replay. No production source-suite acceptance is implied by synthetic group verification.

Create a complete grader once

Use private POST /management/v1/evaluation/evaluators/setup with evaluation.manage to save a score definition (new or existing), grader kind/configuration and first immutable version together. The complete setup API example shows an assertion with no model identity or spend. Pin the returned evaluatorId and evaluatorVersion in the experiment/grading plan. Model/tool configurations validate current catalog registration and Team allowances before publication; a failed setup creates no visible partial grader or widened identity policy.

Keep a stable requestKey (1–100 printable ASCII characters) and the exact submitted body through connection loss. Identical retries return the original safe IDs/version; a changed body returns 409 request_key_conflict. Use a new key for an intentionally new setup. Current manage permission is checked even during receipt recovery. Legacy two-step grader creation and single-resource evaluator imports remain supported.

The Console's Test on samples is an explicit paid action for model/tool graders and retains its results/costs, but it creates no experiment scores and cannot satisfy CI/publish gates. It tests the current saved version under the actual requester's live authority. Use a persistent grading pass when CI needs saved judgments and coverage. A sample test alone does not establish calibration.

Reconcile a versioned suite bundle

Use the private suite bundle API to stage a mission-control.evaluation-bundle/v1 file, preview references and resource bindings, then atomically commit the complete suite. Working directory for the example/generator commands is the repository root. Keep the source namespace and object keys stable across source revisions; record the source commit in case metadata instead of creating a new namespace for every commit.

Upload at most 16 MiB in 40 KiB raw chunks encoded as canonical base64, within the unchanged 64 KiB management request limit. Sessions expire after 24 hours and belong to their creating membership. Use current evaluation read/manage/content permissions on each write and preview. Bind only existing permitted models, read-only validator tools and saved same-Team snapshots. Templates need the corresponding agent permissions too.

Persist the upload session ID, original file hash and request key so a restarted job can resend identical chunks. Preview is read-only and returns field-addressed issues plus current revisions and create/adopt/unchanged/update/conflict decisions. A changed or locally edited object requires an explicit resolutions[key]="update"; abort and correct the file/bindings if that is not intended. Dataset updates merge supplied case identities and retain omitted cases. They do not erase historical experiments or versions.

Commit with the exact preview hash and a stable commit request key. Current authorization and references are checked again. A stale preview conflicts; a lost successful response can replay the same safe receipt even after staging bytes expire. A fresh session containing the same unchanged source/bindings reuses the same resource IDs and versions. The operation publishes all accepted objects together or none; it never imports executed experiments or scores. Updating existing graders can affect their active auto-evaluation rules, which is shown in preview.

Use platform/packages/control-plane/testdata/evaluation-bundle.synthetic.json to verify an importer against synthetic data. It includes a definition, assertion, two cases (including context and multiple turns), a smoke slice and a template. Map its two model aliases and snapshot alias to real allowed test resources. When adapting a source suite, preserve original case identities, slice membership and source settings, and report unsupported settings explicitly. Source low/high model-selection directives must not silently become reasoning effort. Importing a suite does not verify equivalence to an external application; external execution adapters are not included in the current release.