AI agent evaluation scenarios are executable descriptions of a workflow situation, its initial state, available tools, expected boundaries and acceptable outcomes. They test more than the final answer. An agent can produce plausible text after choosing the wrong tool, leaking data to an ineligible route, skipping approval or leaving a partial business action behind. A scenario library makes those trajectory failures repeatable before release and comparable after models, prompts, tools or policies change.
Traditional question-and-answer sets remain useful for component quality, but agents operate through stateful sequences. The OpenAI Agents SDK tracing guide records model generations, tool calls, handoffs and guardrails, while its results guide exposes final output, new items, agent state and interruptions. A scenario harness can use those surfaces to assert what the workflow did, not only what it said.
Build the library from workflow risk
Map the workflow as states and transitions. For each step, identify authority, required evidence, mutable dependencies, side effects and recovery. Then list failure modes: wrong tool, wrong arguments, stale data, denied permission, tool timeout, conflicting evidence, missing reviewer, duplicated action, handoff loop or misleading completion. Use product incidents, support cases, policy exceptions and threat models as inputs. The library should represent how work fails in the organization, not a generic set of clever prompts.
Prioritize by impact, likelihood, detectability and control dependence. A rare irreversible action deserves a scenario even if it contributes little to average traffic. Include ordinary cases because a suite containing only traps can optimize the agent for defensive behavior and reduce usefulness. The NIST Generative AI Profile frames risk through governance, mapping, measurement and management; scenario coverage is practical measurement evidence tied to that map.
Define a portable scenario contract
| Field | What it specifies | Why it matters | Example assertion |
|---|---|---|---|
| Initial state | Records, permissions, conversation and clock | Makes the run reproducible | Invoice is pending and version 7 |
| Stimulus | User event or upstream trigger | Separates intent from hidden setup | Requester asks to approve above limit |
| Available capabilities | Models, tools, agents and policies | Prevents accidental extra authority | Read tool allowed; payment tool gated |
| Fault schedule | Timeouts, stale reads or malformed responses | Exercises recovery deterministically | Second supplier lookup times out |
| Acceptable trajectories | Required, optional and forbidden transitions | Allows multiple safe solutions | Escalation or refusal both accepted |
| Outcome invariants | Business state and evidence after run | Tests effects, not narration | No payment created; review item exists |
| Observability requirements | Spans, reasons and correlation | Proves operators can investigate | Approval interruption linked to proposal |
Use stable fixture identifiers and a virtual clock. Mock external tools at the contract boundary, but preserve their real schemas, error categories and authorization behavior. Seed random or sampling configuration where possible and run repeated trials where nondeterminism matters. Avoid requiring one exact natural-language response when several answers are acceptable. Assert structured facts, prohibited claims, required disclosures and workflow state, while scoring prose separately with a rubric or calibrated judge.
Balance happy paths, boundaries and failures
| Family | Purpose | Representative case | Release use |
|---|---|---|---|
| Golden path | Protect core user value | Complete a routine request with current evidence | Every change |
| Boundary | Test limits and abstention | Amount exactly at an approval threshold | Every policy or tool change |
| Dependency failure | Verify retry and degradation | Tool timeout after a read but before action | Runtime and provider releases |
| Adversarial | Exercise manipulation and data exposure | Retrieved document instructs agent to bypass policy | Security gate |
| Human control | Test review packet and resume | Reviewer edits one argument then approves | Approval UI and state changes |
| Recovery | Validate partial-failure handling | First side effect succeeds; second fails | Workflow release |
| Drift sentinel | Detect changing production inputs | Recent anonymized edge-case template | Scheduled monitoring |
Cover tool choice, argument construction, sequencing, handoffs, stopping, approval and recovery. For every scenario that expects action, add a nearby scenario that lacks one prerequisite and must not act. For every retry path, add a duplicate-prevention assertion. For every handoff, cap allowed transitions and assert context preservation. The NIST adversarial ML taxonomy supplies terminology for attacker goals and lifecycle stages; use it to structure threat scenarios instead of collecting prompt-injection strings without an attack model.
Use layered oracles for trajectory and outcome
A test oracle decides whether a run passed. Use deterministic checks first: state transition, tool allowlist, schema validity, argument invariants, number of side effects, approval identity, source references and final database version. Add rule-based text checks for mandatory notices or forbidden sensitive values. Human or model-based grading can evaluate usefulness, but should not override a deterministic safety failure. Keep grader prompts, models and calibration sets versioned like production components.
Allow acceptable trajectory sets rather than one brittle script. An agent may safely ask for missing data or escalate immediately. Both can pass if state remains unchanged and the user receives an accurate next step. Score efficiency only after correctness and safety gates: unnecessary tool calls, repeated retrieval and handoff loops matter, but a short unsafe path must never outrank a longer controlled one. The LLM evaluation practical framework offers component-level guidance that can complement these workflow oracles.
Build a deterministic harness around nondeterministic agents
The harness should create isolated state, load fixtures, configure eligible models and tools, inject scheduled faults, run within limits, capture trace and state, execute oracles and destroy the environment. Give each run a unique correlation ID. Tool mocks should support idempotency and realistic pagination. For integration tiers, use sandbox accounts and reversible resources rather than production. Never let an evaluation suite send email, move money or alter customer records merely because the model selected a tool.
Set budgets for steps, model calls, tokens, time and side-effect attempts. A budget breach is a result, not a hung test. Preserve enough trace data to diagnose failures while redacting sensitive fixtures. Re-run failed stochastic scenarios to estimate frequency, but do not erase an important one-off safety failure by averaging. Report pass rate with confidence and failure categories. For low-frequency critical controls, a single violation should block release until triage establishes that the scenario or product expectation was wrong.
Turn the library into release gates
Tag scenarios by workflow, risk, control, language, data class and change sensitivity. A tool-schema change should run affected tool and recovery scenarios; a global model change should run the full critical suite plus sampled broader coverage. Define non-regression gates for safety invariants and bounded tolerances for probabilistic quality. Compare against the deployed baseline, not an obsolete perfect score. Require owners to approve waived failures with expiry and a compensating control.
The model evaluation engineering notes emphasize release evidence rather than a single benchmark number. A scenario report should identify code, prompt, model, tool, policy and fixture versions; scenario counts by risk; changed outcomes; unresolved failures; and the deployment decision. Keep the evidence with the release so an incident review can determine which relevant scenarios existed and whether they passed.
Feed production evidence back without copying sensitive cases
Mine traces, reviewer corrections, escalations, abandoned workflows and incidents for missing scenario patterns. Convert them into synthetic or minimized fixtures that preserve the decisive structure without retaining unnecessary personal data. Record the production issue and scenario that guards it. Sample successful traces too, because drift may appear as increasing tool count or review effort before failures. The model observability guide explains how traces and evaluations form an operating loop.
Monitor library health: critical-risk coverage, stale fixtures, flaky rate, unowned scenarios, median diagnosis time and production defects without a corresponding test. Review scenarios when policies or business processes change; a test can pass while enforcing yesterday's rule. Archive obsolete cases with rationale instead of deleting history. A scenario library is an institutional memory of expected behavior, but unlike free-form lessons learned, it can run against the next release.
Govern scenarios as product assets
Assign a domain owner for expected outcomes, an engineering owner for harness reliability and a risk owner for critical controls. Review changes to high-risk oracles independently from changes to the agent, preventing one commit from weakening the test and behavior together. Protect fixtures, tool credentials and traces. Separate development examples from regulated evidence sets where access and retention differ. Version the scenario schema and provide migration tools so old cases remain executable.
Measure diversity rather than raw test count. Ten variations of the same ordinary request do not replace a missing partial-failure case. Create coverage views over states, transitions, tools, risk controls and failure classes. For retrieval-heavy workflows, link to the RAG evaluation guide for source and answer measures, but keep agent assertions on selection, action and recovery. Component quality and workflow reliability are related, not interchangeable.
Key takeaways
- Model each scenario with initial state, capabilities, faults, acceptable trajectories and outcome invariants.
- Derive coverage from business risk, incidents and threat models.
- Assert tools, state, approval and side effects before judging response style.
- Use isolated harnesses with virtual time, realistic tool contracts and strict run budgets.
- Gate releases by risk-tagged non-regression evidence and explicit waivers.
- Convert production patterns into minimized governed scenarios that remain executable.
Frequently asked questions
How many scenarios are enough? There is no universal count. Coverage should represent material states, transitions, controls, user cohorts and failure classes. Missing one irreversible-action case matters more than adding many ordinary paraphrases.
Can a model grade agent trajectories? It can assist with qualitative dimensions, but deterministic state, authority and side-effect assertions should remain primary. Calibrate any model grader against expert decisions.
Should production traces become test fixtures automatically? No. Minimize, de-identify, review expected behavior and confirm rights before adding them. A production outcome may itself be incorrect and should not define the oracle uncritically.
Conclusion
AI agent evaluation scenarios make workflow behavior testable across models, tools, policies and failures. Start from risk, write portable stateful contracts, use layered oracles, isolate side effects and attach scenario evidence to releases. Keep the library alive through production feedback and domain ownership. Teams then stop asking only whether an agent produced a good answer and can prove whether it gathered the right evidence, used the right authority, recovered from failure and left the business in an acceptable state.