Model Management¶
Target architecture — not the current operating contract
This page includes the broader model-management product. For implemented administration and API behavior, see Model administration and the Model Gateway.
Organization, Team, member, service-account, key and customer spend, request, token and concurrency caps are implemented; see Budgets. Reusable input/output guardrail policies with Organization baselines and virtual-model assignments are implemented through reviewed YAML; see Model guardrails. Purpose/data-class routing, policy authoring UI, Team/key guardrail assignments, Team/resource/Run/outcome attribution, soft alerts, custom windows, and invoice reconciliation below remain target architecture.
Purpose¶
Model Management separates stable product capability from provider-specific deployments. Agents and Workflows request a Virtual Model; the Model Gateway resolves that logical identity through an approved routing policy to a healthy, permitted deployment.
Model resources are completely separate from MCP Servers and the Connector Catalog. They use an OpenAI-compatible gateway contract, provider credential isolation, model-specific routing and guardrails, and model economics.
Components¶
| Component | Responsibility |
|---|---|
| Provider Registry | Register hosted, private, sovereign, or local inference providers with owners, egress posture, environments, and customer-secret references |
| Deployment Catalog | Record model identity, provider deployment, region, capability, context limit, pricing metadata, health, and lifecycle |
| Virtual Models | Provide stable logical identities that decouple consumers from provider deployments |
| Routing Policies | Resolve approved deployments using environment, use case, data class, latency, quality, availability, residency, and cost |
| Fallback Manager | Define ordered, pre-approved fallback chains and behavior during capacity or provider failure |
| Access Grants | Control which Agents, Workflows, teams, applications, services, and environments may request a Virtual Model |
| Budget and Quota Manager | Extend the current focused limits with team, resource, Run, and outcome attribution, soft thresholds, additional windows, and reconciliation |
| Guardrail Profiles | Apply typed input and output validation, content filtering, redaction, purpose, and data-handling controls |
| Model Evaluation | Compare deployments and routes using approved datasets, production cohorts, quality, safety, latency, cost, and outcomes |
| Health and Economics | Show availability, latency, errors, tokens, estimated cost, capacity, routing distribution, and affected consumers |
Virtual-model resolution¶
Agent or Workflow
-> Virtual Model
-> current Routing Policy
-> approved Model Deployment
-> Provider
Resolution follows these rules:
- The caller presents a registered workload identity, resource and immutable version, environment, Run, Trace, and requested Virtual Model capability.
- The Control Kernel validates the grant and contextual policy.
- The Model Gateway loads the current approved routing policy for that Virtual Model and scope.
- The route filters deployments by environment, capability, data class, use case, residency, guardrail compatibility, quota, and health.
- The gateway selects an approved deployment, isolates its provider credential, invokes it, validates the response, and emits route, usage, cost, status, and evidence metadata.
- A retry or fallback remains inside the approved route and uses the declared reliability semantics.
The resolved route is recorded in Run evidence. Changing the provider or deployment does not change the Agent, Workflow, AI Application, Virtual Model, policy, or historical identity.
Connections¶
| Connection | Contract |
|---|---|
| Control Kernel | Provider, deployment, Virtual Model, policy, grant, owner, lifecycle, configuration, evidence, evaluation, and incident identity |
| Model Gateway | Exclusive inference route and enforcement point for resolution, credentials, guardrails, budgets, quotas, usage, cost, Trace, and evidence |
| Agent Studio | Agents request Virtual Models and declare capability and quality requirements without embedding provider credentials or endpoints |
| Automations | Workflow Agent tasks inherit the Agent's Virtual Model contract; deterministic steps do not make untyped provider calls |
| Retrieval | Embedding and reranking use approved Virtual Models when those operations require model inference |
| AI Applications | Applications bind registered Agents or Workflows, never a provider endpoint from browser code |
| Operations and Assurance | Correlates health, routing distribution, tokens, cost, errors, evaluations, consumers, incidents, and outcomes |
Non-goals¶
Model Management is not:
- Part of the MCP Connector Catalog.
- A provider-specific client embedded in an Agent or application.
- A promise that fallback may ignore data, quality, residency, cost, or policy constraints.
- An inference runtime that can bypass the Model Gateway.
- A training, fine-tuning, or general machine-learning platform in the initial scope.
- A store for prompt and response content by default.
Current governance mapping¶
The current source release uses Organizations, Teams, real user/service-account virtual keys and separate human management roles. Agent is managed application content; deployments own secretless execution access and publisher funding. Observe uses Requests, Traces, Sessions and Usage with separate metadata/content/export permissions. Historical identity terms in this target proposal are not the operating contract. See Current release.