AI Safety Reviews Before Production Launch should answer a practical question: can this feature fail in a way that the people operating it can see, contain, and explain? A polished demonstration rarely answers that question. It tends to use cooperative prompts, clean records, and a developer standing nearby. A launch review should instead start with intended use, affected people, decision authority, data classes, and the consequence of a wrong or unavailable answer. The NIST AI Risk Management Framework offers useful organizing language, but the work becomes real only when owners can point to product behavior, test evidence, and an escalation path.
Define the launch boundary before testing
Write a short release statement that says who may use the feature, what it may do, what it must not decide, and which records it may access. Include the model and provider assumptions, enabled tools, languages or document types in scope, and known limits. This is not legalistic paperwork. It prevents a helpful internal drafting aid from becoming an unreviewed approval mechanism because someone notices it can populate a form. When the boundary is explicit, testers can exercise the work the feature is actually authorized to perform and reviewers can recognize scope creep before it reaches customers.
| Review area | Question to resolve | Launch evidence |
|---|---|---|
| Intended use | What decision or task is the feature allowed to assist? | Named user, workflow, and prohibited actions. |
| Data boundary | Which records and fields can enter prompts or retrieval? | Permission tests and retention decision. |
| Action authority | Can the feature change a system or only prepare work? | Tool scopes, approval rule, and audit event. |
| Failure path | What happens after uncertainty, abuse, or outage? | Visible message, owner, and recovery exercise. |
Test misuse as carefully as happy paths
Create test cases from the ways real work becomes messy: a document that contains misleading instructions, a user who asks the assistant to bypass a policy, a retrieved record that is stale, a tool response that is incomplete, and a prompt that requests information outside the user’s role. The OWASP Top 10 for LLM applications is a useful starting point for input, data, and tool-related failure modes. Do not turn its headings into a box-ticking exercise. Translate each relevant risk into a specific product scenario and define the expected refusal, warning, escalation, or safe completion.
Collect evidence that a reviewer can inspect
Use a versioned evaluation set with examples drawn from intended use, known edge cases, and past incidents where available. For each case, record the input class, expected behavior, actual output, citations or tool calls, evaluator judgment, and failure category. Define who can change the set and how a material change triggers re-evaluation. The goal is not a single score that implies universal safety. It is a traceable picture of where the feature performs reliably, where it needs human judgment, and where it should not be offered. The NIST Generative AI Profile supports this contextual approach to risk.
| Scenario | Expected behavior | Reviewer check |
|---|---|---|
| Unsupported question | State the limit and offer an approved next step. | No invented authority or fabricated source. |
| Prompt injection in a record | Ignore untrusted instructions and preserve task boundary. | No unexpected tool call or privilege expansion. |
| Sensitive record request | Apply the user’s real permission before retrieval. | Denied content does not appear in context or output. |
| Provider outage | Fail visibly and retain unfinished work when needed. | User sees ownership and recovery information. |
Write a launch decision that survives later scrutiny
The review output should be a decision record, not a meeting note that says “approved.” Record the intended use and prohibited uses, affected groups, system and model versions, connected data and tools, evaluation-set version, unresolved limitations, mitigations, residual risks, operating owner, approval conditions, and next review trigger. NIST’s Generative AI Profile organizes generative-AI risk work across governance, context mapping, measurement, and management. The record should connect each material risk to an implemented behavior or a consciously accepted limitation.
Make approval conditional when evidence has a shelf life. A provider model update, new retrieval source, broader user group, additional language, changed tool permission, or materially different traffic pattern should trigger targeted re-evaluation. Use MITRE ATLAS to develop adversarial scenarios and OWASP’s LLM risk guidance to challenge prompt, retrieval, output, agency, and supply-chain assumptions. Link this review to Edilec’s model monitoring guide so the pre-launch evidence becomes a post-launch control plan.
| Change after approval | Minimum response | Reason |
|---|---|---|
| Model or major prompt version | Run the affected evaluation suites and compare regressions | Behavior can change without interface changes |
| New data source or connector | Recheck permission filtering, provenance, retention, and injection paths | The trust boundary has expanded |
| New tool or write action | Threat-model side effects and rehearse authorization and rollback | Consequences are no longer limited to generated text |
| New audience, language, or jurisdiction | Test representative users and review legal and support duties | Prior evidence may not cover the new context |
| Unexpected harmful event | Contain, preserve evidence, investigate, and re-authorize before resuming | The accepted risk assumptions have been contradicted |
Make approval and recovery operational
A launch decision should name an accountable product owner, technical owner, risk or domain reviewer where warranted, and the person who can disable the feature. Record the approved scope, residual risks, monitoring signals, and the date for reassessment. Rehearse disabling a tool, rolling back a prompt or model configuration, and locating the affected requests. A recovery plan that exists only in a document is too slow when an incident is unfolding. The MITRE ATLAS knowledge base can help teams think through adversarial techniques, while the release plan should stay specific to the product’s actual exposure.
- Assign one owner for the product decision and one for the technical controls.
- Test representative safe, uncertain, malicious, and unavailable conditions.
- Keep an approval record that states residual risk and the next review date.
- Alert on action failures, policy denials, unusual tool patterns, and user corrections.
- Give support staff a way to report harmful output with enough context to investigate.
Treat launch as the start of observation
Make the review record easy to find later. Link the launch scope, test set, residual-risk decision, monitoring dashboard, and incident route from the product release record. When a customer reports a surprising answer months later, the team should be able to see what was approved, which model and tools were in use, and what evidence was considered. That continuity prevents a safety review from becoming a forgotten meeting and makes later changes easier to assess against the original intent.
Keep the user experience honest during rollout. Tell users what the feature can help with, how to verify material outputs, and how to report a problem. Do not imply that a generated draft is an official decision or that an answer is complete when the service has no authorized source. Clear product language reduces risky overreliance and provides better feedback when the feature is uncertain. It also gives support teams a stable explanation while configuration, monitoring thresholds, and operating procedures mature in the early weeks of production use.
Create a short launch dashboard before the first user arrives. It should show volume by approved workflow, source and tool failures, refusals, escalations, reviewer corrections, and the status of any action queue. Add a qualitative route for frontline staff to flag a concerning answer even when they cannot classify the technical cause. During the first weeks, hold a brief review with the product owner and operators to inspect representative cases. This makes it possible to distinguish a harmless formatting issue from a pattern that needs a pause, a revised boundary, or direct notification to affected users.
Safety review also needs to account for dependencies that change outside the release. A provider can update a model, a knowledge source can be revised, a permission group can be reconfigured, and a connected tool can introduce a new response shape. Keep a record of material dependencies and define which changes require a fresh evaluation. Where a change cannot be controlled, monitor for unexpected output, tool calls, or error patterns and retain the option to disable the affected capability. This turns ongoing uncertainty into a managed operating responsibility rather than a reason to abandon useful features.
Initial traffic will show cases that pre-launch testing did not anticipate. Review sampled conversations or work packets under appropriate access controls, track corrections and escalations, and compare actual usage with the approved boundary. Watch for a feature being repurposed into decisions it was not designed to support. Changes to prompts, retrieval, models, tools, permissions, or source records can alter behavior even when the interface looks unchanged. Establish a threshold for pausing rollout and a small, regular review that turns observed failures into test cases and product improvements.
Key takeaways
- A credible review defines the feature’s authorized use before it measures behavior.
- Test unsafe and ambiguous conditions with product-specific scenarios.
- Evidence should connect an input, expected behavior, actual result, and reviewer judgment.
- Approval needs named owners, a disable path, and a rehearsed recovery process.
- Post-launch observation is part of safety, not an optional analytics phase.
Frequently asked questions
How large should a pre-launch evaluation set be?
There is no universally sufficient number. Start with enough examples to cover each intended workflow, relevant user role, data condition, tool action, and known failure mode. Add cases from pilots and incidents over time. A smaller, reviewed set tied to product decisions is more useful than a large collection whose expected outcomes nobody can explain.
Who should approve an AI launch?
The people who own the customer outcome, technical implementation, data or security boundary, and regulated or high-impact domain concerns should participate according to the feature’s risk. Approval is not a ceremonial committee. Each named person should be able to explain the scope, evidence, unresolved risk, and condition that would require a pause.
Conclusion
A useful AI safety review turns vague reassurance into observable product behavior. Define the boundary, test the ways work can go wrong, retain evidence, and make recovery possible without improvisation. That gives a team a responsible launch decision and a much clearer way to learn after release.