Model evaluation is the engineering discipline of collecting evidence about whether an AI-enabled system performs a defined job acceptably under the conditions in which it will be used. It is not one benchmark, one accuracy score, or a final test before launch. For a support summarizer, a procurement classifier, or a tool-using assistant, the evaluation must represent the workflow, data, people, controls, failure costs, and connected software. A strong score on an unrelated public dataset cannot establish production fitness.
This model evaluation guide complements Edilec's foundational evaluation notes, tool calling implementation guide, and AI workflow approvals plan. Engineering owns the repeatable evidence pipeline, but domain owners must define what a useful and harmful outcome means. Neither group can substitute for the other.
Key takeaways
- Write a workflow contract before selecting metrics: user, task, inputs, permitted action, prohibited outcome, fallback, and baseline.
- Construct evaluation data from real operating slices, including ambiguity, missing evidence, adversarial content, and cases that should abstain.
- Measure the complete system, not only the base model: retrieval, prompts, tools, policy, latency, cost, interface, and human review all affect outcomes.
- Use release gates by critical slice and failure type; an aggregate average must not cancel a severe regression.
- Connect production samples and incidents back to versioned datasets without turning live user data into an uncontrolled test archive.
Define the evaluation contract
Start with a sentence: for this user and context, the system receives these permitted inputs, produces this decision support or action, and must defer when these conditions hold. Record the current human or software baseline, acceptable variation, consequence of false positive and false negative, latency expectation, cost ceiling, affected groups, and accountable owner. A customer reply draft and a payment approval recommendation may use similar models but require radically different evidence and authority.
| Contract element | Claims-routing example | Evaluation consequence |
|---|---|---|
| User and task | Operations analyst assigns a warranty claim to a specialist queue | Judge routing usefulness, not general conversation |
| Permitted evidence | Claim form, product record, current policy, and service history | Test missing, conflicting, stale, and restricted sources |
| Output authority | Recommend queue and reason; do not approve payment | Any payment action is a critical failure |
| Fallback | Uncertain or policy-triggered case enters manual triage | Measure safe deferral and queue burden |
| Service target | Result within the analyst's intake window | Include end-to-end latency and availability |
| Change boundary | Prompt, model, policy, retrieval, taxonomy, or tool update | Each material change triggers an appropriate evaluation |
Translate risks into observable failures. Examples include unsupported facts, wrong classification, omitted material evidence, policy bypass, privacy leakage, unsafe advice, excessive deference, poor subgroup performance, tool misuse, and an explanation that does not match the action. Rank by consequence and detectability. A low-frequency security or financial failure can deserve a zero-tolerance release gate even when ordinary task quality is high. The NIST AI RMF provides Govern, Map, Measure, and Manage functions that help keep these decisions connected.
Build a representative, versioned evaluation dataset
Sample from the actual distribution, then deliberately enrich rare but important slices. Record source, consent or permitted use, time range, language, customer or workflow segment, difficulty, expected behavior, annotation guidance, and sensitive-data handling. Separate development examples from the held-out release set. Prevent near duplicates and templated variants from crossing splits. For changing knowledge, store the reference corpus or snapshot and effective date so a future run can distinguish model change from source change.
| Dataset slice | Why it belongs | Expected behavior |
|---|---|---|
| Routine cases | Represents common value and establishes baseline | Correct, useful completion within budget |
| Boundary cases | Values sit near policy or classification thresholds | Consistent treatment with clear evidence |
| Incomplete cases | Real workflows often lack required information | Ask, abstain, or route rather than invent |
| Conflicting evidence | Sources can disagree in production | Expose conflict and follow authority rules |
| Adversarial content | Inputs or retrieved text may attempt instruction override | Keep policy and prevent restricted action |
| Changed conditions | Taxonomy, source, role, or dependency has moved | Detect incompatibility or request reevaluation |
| Previously observed failures | Regression tests protect hard-earned fixes | Pass without damaging nearby cases |
Annotation needs the same rigor as code review. Write a rubric with examples and escalation rules. Use multiple reviewers for subjective or consequential cases, measure disagreement, and let an adjudicator preserve uncertainty instead of forcing false consensus. Some tasks have several acceptable responses; score required facts, prohibited claims, tone constraints, evidence use, and actionability separately. Keep reviewer identity and rubric version. A label without provenance becomes difficult to challenge when the product changes.
Choose metrics that lead to decisions
Use deterministic metrics where the task supports them: precision and recall for classification, exact or semantic checks for structured extraction, citation entailment for grounded answers, policy-rule pass rates for actions, and latency or cost percentiles for operation. Add rubric-based human review for usefulness and material omissions. Model-based graders can accelerate evaluation, but validate them against expert judgments, inspect disagreement by slice, protect them from injected content, and version the grader like any other component.
| Metric | Decision it supports | Misuse to avoid |
|---|---|---|
| Precision and recall by class | Choose threshold and review capacity for asymmetric errors | Reporting accuracy only on an imbalanced dataset |
| Grounded claim rate | Assess whether factual output is supported by allowed evidence | Treating citation presence as support |
| Safe deferral rate | Confirm the system stops when authority or evidence is missing | Calling every refusal a failure |
| Critical policy violations | Block releases that cross prohibited boundaries | Averaging severe failures into a quality score |
| Human correction time | Estimate operational effort and usability | Counting acceptance clicks without checking outcome |
| Latency and cost per accepted task | Select route and capacity for a useful result | Optimizing cheap incomplete outputs |
Define thresholds before looking at the candidate result. Compare with the current production version and, where useful, the human or non-AI baseline. Require no regression on critical slices and a meaningful improvement on the target outcome. Include confidence intervals or uncertainty when sample sizes are small. The NIST AI Resource Center provides resources for testing, evaluation, verification, and validation; teams still need a local decision rule that states who can accept residual risk.
Test the whole AI system and its controls
Run offline tests at several layers: unit checks for transforms and policy, component checks for retrieval or tools, scenario tests for multi-step behavior, adversarial tests, and end-to-end workflow trials with the user interface and fallback. Include network timeout, malformed tool output, permission denial, stale cache, unavailable source, duplicate request, and partial write. The OWASP LLM application project is a useful security test inventory, while the NCSC secure AI guidelines connect testing to the full development and operational lifecycle.
Do not grade only the final text. Inspect evidence selected, tool calls, policy decisions, arguments, side effects, user confirmation, and resulting system state. For agents, test stopping: unnecessary loops can increase cost and create additional opportunities for error. For retrieval systems, separate retrieval quality from answer synthesis so a source-index defect is not misdiagnosed as a prompt problem. The NIST Generative AI Profile adds generative-AI-specific considerations that can inform this threat and measurement plan.
Make release and production evaluation one loop
- Register the candidate's model, prompt, retrieval corpus, tool schemas, policy, grader, dataset, and application versions.
- Run deterministic checks first, then representative quality, security, performance, and operational suites.
- Review failures by severity and slice; document accepted limitations and controls rather than deleting uncomfortable cases.
- Shadow or canary the candidate on an approved population with no or limited side effects.
- Compare user outcomes, corrections, abstentions, latency, cost, policy denials, incidents, and subgroup behavior with the baseline.
- Promote, restrict, roll back, or collect more evidence through an accountable release decision.
In production, sample according to risk and drift, not convenience alone. Join model version to input category, retrieved source versions, tool activity, review decision, correction, and final outcome while minimizing sensitive retention. Watch changes in input distribution, missing sources, acceptance, override, complaint, policy denial, and cost. A trigger needs an owner and action: investigate, increase review, disable a tool, narrow a population, return to the previous version, or rebuild the evaluation slice.
Engineering review checklist
- The workflow contract states intended use, prohibited actions, owners, affected users, baseline, and fallback.
- Evaluation data has lawful provenance, representative and adverse slices, split hygiene, annotation guidance, and versioning.
- Metrics distinguish critical failures, task quality, safe deferral, operational burden, latency, and cost.
- System tests cover retrieval, tools, policy, permissions, user interface, failures, retries, side effects, and recovery.
- Release thresholds were defined in advance and apply by material slice, with an accountable risk decision.
- Production sampling can connect observed outcomes and incidents to a reproducible test without uncontrolled data collection.
Frequently asked questions
How large should an evaluation set be?
Large enough to estimate the decisions you need by critical slice. Start with dozens of carefully reviewed examples per important behavior, then expand where uncertainty or consequence is high. Thousands of low-quality labels do not compensate for missing failure cases or unclear rubrics.
Can an LLM grade another LLM?
Yes, as one instrument. Validate the grader against expert judgments, use a structured rubric, inspect disagreement and bias by slice, defend against injected content, and route consequential or ambiguous cases to people. Do not treat grader agreement as ground truth.
When should evaluation run?
Run fast checks on every relevant change, broader suites before release, and scheduled or triggered production reviews. Changes to models, prompts, retrieval sources, policy, tools, permissions, user populations, or downstream systems can all require reevaluation.
Conclusion
Useful model evaluation is an operating feedback system. It starts with a bounded job, turns risks into testable behavior, uses representative evidence, blocks unacceptable regressions, and learns from production without losing provenance. Engineering teams gain confidence not by proving that a model is generally good, but by showing that a specific system remains useful, controlled, and recoverable for the work it is allowed to perform.