AI model routing is the runtime decision that assigns each agent step to an eligible model and endpoint. A useful router does not merely send easy prompts to a cheaper model and hard prompts to a larger one. It considers data sensitivity, residency, tool capability, output contract, task risk, measured quality, latency objectives, provider state and fallback constraints. The policy must explain why a route was allowed and what evidence supports the expected outcome.
Agent workflows contain different steps: classify a request, retrieve records, draft text, choose a tool, validate an action and summarize a result. Assigning one model to the whole run can waste capacity or expose data unnecessarily. Routing per step improves control only when context and state remain coherent and evaluations are step-specific. Otherwise the system exchanges a visible model choice for hidden handoff errors.
Inventory steps before ranking models
Describe each step with input classes, output contract, required capabilities, authority, consequence of error and recovery options. A nonbinding summary may tolerate small wording variation and a delayed retry. A tool-selection step that can trigger a refund needs stronger evidence, bounded output and possibly approval. Record whether the step needs images, long context, strict structured output, low latency, regional processing or specific tool semantics. This becomes the route request; raw prompt inspection alone is an unreliable classifier.
Keep step definitions in a registry owned by the workflow team. The registry should state allowed model classes, current evaluation suite, maximum data classification, approval requirements and fallback behavior. Model names are deployment details that can change. Stable policy refers to capabilities and evidence thresholds, then a controlled catalog maps those requirements to concrete endpoints. The OpenAI Agents SDK models guide demonstrates how an agent runtime can work with different model providers; enterprise routing adds qualification and policy around that flexibility.
Filter eligibility before optimizing quality, latency or cost
| Gate | Question | Evidence | Failure behavior |
|---|---|---|---|
| Data handling | May this endpoint process the classified input? | Approved use, retention and training terms | Exclude endpoint |
| Residency | Does processing and telemetry stay in an allowed region? | Endpoint architecture and contract | Route only to approved region |
| Capability | Can the model satisfy modality, context and output needs? | Contract tests and provider documentation | Exclude or redesign step |
| Safety and security | Has this model passed threat and misuse scenarios? | Current evaluation release | Quarantine version |
| Authority | May this model participate in this consequential decision? | Risk-tier policy | Require stronger route or human control |
| Availability | Is endpoint health inside the route threshold? | Recent telemetry and circuit state | Use eligible fallback or queue |
Eligibility is boolean for a specific request. Do not let a low price compensate for prohibited data handling or a missing capability. Apply gates to the whole path, including gateway, provider, region, telemetry exporters and caches. If context contains mixed classifications, route according to the highest material classification or split the step using a reviewed transformation. Redaction can lower exposure only when the runtime proves protected fields were removed and the remaining text cannot trivially re-identify the subject.
Assign task risk tiers with concrete consequences
Risk combines likelihood and impact of a wrong, delayed or exposed result. Define tiers using business consequences, not model complexity. Tier 0 might cover internal formatting with no sensitive data. Tier 1 can include reversible drafting. Tier 2 may influence customer service or employee operations. Tier 3 can propose financial, legal or security actions. A tier determines minimum evaluation evidence, allowed endpoints, logging, reviewer involvement and fallback. Avoid labeling an entire application high risk when only one step creates the consequence.
The NIST Generative AI Profile encourages organizations to map risks and manage them through measured controls. Translate that guidance into route policy: a higher tier requires fresher evaluations, stricter data paths, stronger structured-output validation and narrower fallbacks. Review tiers when the workflow gains a new tool or authority. A drafting step can become consequential when its output starts flowing directly into an execution API.
Rank eligible routes with evaluated utility
Once eligibility is satisfied, rank routes by expected utility. Use step-specific quality, end-to-end latency, marginal cost, availability and operational preference. Normalize measures over a declared evaluation window and keep uncertainty visible. A model with a slightly higher average score but sparse evidence on a critical language should not automatically win. Use a conservative confidence bound or require a minimum sample for high-risk routes. The objective function and weights should be versioned and reviewed by the product owner.
| Step | Primary objective | Important constraint | Useful outcome measure |
|---|---|---|---|
| Intent classification | Fast correct dispatch | Stable label contract | Correct route without rework |
| Evidence synthesis | Grounded completeness | Approved sources and context limit | Supported claims retained |
| Customer draft | Usable response | Tone, privacy and no action authority | Acceptance with minimal edit |
| Tool proposal | Correct arguments and abstention | Strict schema and permission context | Valid safe proposal |
| Action verification | Detect disallowed or inconsistent request | Independent evidence path | Unsafe action blocked |
| Final summary | Accurate concise record | No new unsupported facts | Trace-consistent completion |
Optimize the completed workflow, not isolated token price. A cheaper classifier that sends more cases to an expensive recovery route may increase total cost and latency. Include retries, tool calls, human review and failed outcomes in unit economics. The LLM cost controls guide explains portfolio budgets; routing should provide step-level attribution and a counterfactual estimate from shadow evaluations rather than claiming savings from list prices alone.
Constrain fallbacks as carefully as primary routes
A fallback is a new route decision, not a transparent retry. Reapply data, region, capability and risk gates. Define triggers such as timeout, throttling, provider incident or quality circuit breaker. Keep semantic failures separate from transport failures: sending the same ambiguous task to another model may produce a different answer without fixing missing evidence. For consequential steps, queueing or human review may be safer than switching providers. Limit fallback depth to avoid expensive loops and confusing trace trees.
Preserve idempotency across retries. Assign a workflow action identifier before model selection and prevent multiple tool executions if two routes finish. Streaming complicates failover because partial output may already be visible; decide whether to abandon, label or restart the response. Capture the exact reason, prior route state and whether context was changed. Users should not receive a seamless-looking answer if the fallback reduced capability or changed an important data-handling condition.
Qualify routes with offline and shadow evaluation
Build a scenario set for each step and risk tier. Include ordinary cases, boundaries, missing evidence, adversarial instructions, long context, multilingual inputs, tool errors and abstention. Score output and trajectory where relevant. Promote a model version only when it passes minimum gates and comparative thresholds. The model evaluation engineering notes provide a release-evidence framework; add route eligibility and fallback assertions so evaluation results become executable policy.
Shadow routing records what an alternative would have received and produced without allowing side effects. Remove or protect sensitive data according to endpoint eligibility even in shadow mode; experimentation is not an exemption. Compare outcomes with blinded review where practical, and segment by scenario. Canary a new route for low-risk traffic before higher tiers. Keep model and endpoint versions pinned during analysis so an upstream silent change does not contaminate the comparison.
Trace route decisions and detect drift
The OpenTelemetry GenAI conventions offer common attributes for generative AI operations. Add policy version, step type, risk tier, eligible set hash, selected route, exclusion reasons, fallback chain and evaluation release. Record usage and latency at each call, then join them with business outcome and review. Avoid high-cardinality sensitive values in general attributes. Store governed references when operators need to inspect the underlying case.
Monitor selection distribution, gate failures, fallback rate, quality by route, cost per completed workflow and human correction. A shift in route mix may signal input drift, endpoint instability or a policy deployment. The model observability guide helps connect traces to evaluations and review. Alert when an owner can act: quarantine a model, open a circuit, lower rollout, refresh an evaluation or investigate a new input cohort.
Treat routing policy as production code
Store policy, model catalog and evaluation releases under change control. Require review for edits that expand data access, weaken risk thresholds or add fallbacks. The NIST SSDF AI profile extends secure software practices to AI development; apply it to route configuration, adapter libraries and evaluation pipelines. Validate signed catalog updates, restrict who can approve endpoints and rehearse rollback when a policy deploy unexpectedly excludes all routes.
Provide a deterministic explanation surface for operators: request classification, failed gates, selected objective score and policy version. Do not ask another model to explain the router after the fact. Product teams should be able to simulate a proposed policy against recent traces without sending raw data to newly considered endpoints. A risk-aware router earns trust when its decisions are reproducible, its constraints are visible and its optimization target matches useful work.
Key takeaways
- Route individual workflow steps using declared capabilities and consequences.
- Filter data, residency, capability, security and authority eligibility before optimization.
- Use risk tiers to set evaluation, logging, approval and fallback requirements.
- Rank only eligible routes by step-specific expected utility and uncertainty.
- Reapply every gate to fallbacks and preserve idempotency across attempts.
- Version routing policy and join each decision to evaluations and business outcomes.
Frequently asked questions
Should a router use an LLM to classify tasks? It can assist on low-risk ambiguity, but deterministic workflow metadata and policy should lead. Consequential eligibility decisions should not depend on an unqualified classifier.
Is the cheapest eligible model always preferred? No. Choose by completed-work utility, including quality, retries, review and latency. Price per token is only one input.
How quickly can a new model enter production routing? Only as quickly as capability tests, security review and scenario evidence support. Low-risk canaries can begin earlier than consequential routes, but they still need explicit eligibility.
Conclusion
AI model routing should make risk constraints stronger, not disguise them behind dynamic selection. Inventory steps, declare route requirements, eliminate ineligible endpoints, rank the remaining choices with measured utility and constrain every fallback. Trace policy decisions and feed real outcomes back into evaluation. This turns a model catalog into an accountable runtime system that can use different capabilities and economics without treating sensitive data, consequential authority or workflow quality as negotiable variables.