Human approval is useful only when it changes the safety of an AI-enabled workflow. Adding an Approve button after a model has already sent a payment, changed a record or disclosed data is ceremony, not control. A sound design decides which actions need review before execution, gives the reviewer enough evidence to make an independent judgment, preserves the review state durably and records the outcome. It also avoids sending every routine event to people, because an overloaded queue encourages rubber-stamping and workarounds.
The design problem is therefore not simply whether a human is in the loop. It is what authority the person has, when the workflow pauses, what they can see, how conflicts are handled and how the organization learns from overrides. NIST's AI RMF calls for defined human-AI roles, documented oversight and measurement under conditions similar to deployment. For organizations subject to the EU AI Act, Article 14 creates specific human-oversight obligations for high-risk systems; legal applicability and implementation should be assessed with qualified counsel.
Place approval gates according to risk
Begin with the proposed action, not the model's confidence score. Assess consequence if wrong, reversibility after execution, exposure of sensitive data, uncertainty in the evidence and whether policy or law reserves the decision for a person. A fluent answer can still support a dangerous action, while a low-confidence classification may be harmless if it only changes an internal label that is easy to undo. The gate belongs before the first material side effect.
| Action profile | Default control | Example |
|---|---|---|
| Low consequence and readily reversible | Automate within explicit limits; sample outcomes | Suggesting an internal ticket category |
| Moderate consequence or ambiguous evidence | Review before execution or use an exception queue | Drafting a customer response using account history |
| High consequence, privileged or hard to reverse | Named authorized approver plus deterministic policy checks | Issuing a refund, changing access or sending a contract |
| Prohibited or outside policy | Block regardless of model recommendation | Exporting restricted data to an unapproved destination |
| Unclear ownership or missing evidence | Stop and route to triage | Changing a supplier bank account without verified documentation |
Design a review a person can actually perform
A reviewer should not have to reconstruct the task from a chat transcript. Present the requested action in plain language, affected records, proposed changes, policy basis, source evidence, uncertainty and the consequence of approval. Show what the automation did and did not verify. Where the model extracted a fact from a document, link to the relevant page or passage. Where data came from a system of record, identify the source and retrieval time.
The available decisions should match the real process: approve, reject, return for correction, edit within allowed bounds or escalate. Require a reason for exceptions and high-impact approvals, but avoid forcing meaningless comments on routine decisions. The reviewer must be able to disagree without fighting the interface. Preselected approval, countdown pressure and visually dominant accept buttons create automation bias rather than thoughtful oversight.
- Identify the human decision and the policy authority behind it.
- Show evidence and provenance beside each material claim.
- Separate model suggestions from verified facts and policy checks.
- State exactly which side effects approval will authorize.
- Provide rejection, correction, escalation and safe-cancel paths.
- Make conflicts of interest and delegated authority visible before response.
- Keep the request stable while it is under review or clearly show any revision.
Make the approval state durable
Approval is a long-running workflow state, not a modal dialog. Store a canonical request ID, immutable action proposal, policy and model versions, approver requirements, creation and expiry times, evidence references and current status. The workflow should survive process restarts and delayed responses. It must reject duplicate callbacks, prevent a stale approval from executing a revised proposal and apply the authorized side effect idempotently.
Define timeout behavior explicitly. Silence should not imply consent. On expiry, either cancel safely or escalate to a named role. If evidence changes while review is pending, invalidate the old request and create a new version. If execution fails after approval, retain the approval but move the workflow to a visible recovery state; do not ask the model to guess whether the action succeeded. A reconciliation job should compare workflow state with the destination system.
| Failure mode | Required behavior | Evidence to retain |
|---|---|---|
| Duplicate approval response | Accept one valid state transition and ignore later duplicates | Callback identity, timestamp and idempotency result |
| Proposal changes during review | Invalidate the request and require review of the new version | Old and new hashes plus change summary |
| Approver becomes unauthorized | Re-evaluate authority at response time and reassign | Directory and policy versions used |
| Timeout | Cancel or escalate according to policy; never infer approval | Reminders, expiry and escalation events |
| Downstream write fails | Retry idempotently, surface recovery work and reconcile | Approved payload, attempts and destination response |
Enforce security and accountability outside the model
The workflow service should authenticate the reviewer, authorize the action and enforce separation of duties. The agent should receive only the tools and data needed for the task, with restrictive scopes and deterministic limits on amounts, destinations and operations. Microsoft guidance for agentic risk similarly emphasizes approval for high-risk or irreversible actions, system-level interruption, intelligibility and least privilege. These protections must remain effective even when the model produces an unexpected tool call.
Record who proposed, reviewed, approved, rejected, changed and executed the action; when each event occurred; the versions of prompt, model and policy; the evidence references; and the destination result. Protect audit integrity and minimize sensitive payloads. NIST SP 800-53 provides a broader control catalog for least privilege, separation of duties and audit records. ISO/IEC 42001 places these system controls within an organizational cycle of ownership, risk treatment, performance review and continual improvement.
Example: an AI-assisted refund workflow
Consider a support workflow that proposes refunds. The model summarizes the conversation and extracts order ID, reason, requested amount and relevant policy text. Deterministic services verify order ownership, payment state, refund history, allowed amount and the staff member's authority. A low-value, clearly eligible case may proceed under an approved straight-through policy. A case above the threshold, with conflicting evidence or an exception request pauses before the payment API.
The reviewer sees the exact refund amount and destination, verified transaction facts, cited conversation excerpts, matched policy clause and any unresolved discrepancy. Approval authorizes one immutable command with a short validity window. The payment service uses an idempotency key, and reconciliation confirms the provider's final state. If the customer updates the request or the order changes, the old approval becomes invalid. This arrangement uses AI to reduce reading while preserving policy and financial authority in controlled services and named roles.
Measure whether oversight works
Queue speed alone can reward careless review. Measure decision quality and operating burden together. Useful signals include override rate by use case, reviewer disagreement, policy-exception frequency, correction reasons, approval latency by risk tier, expired requests, downstream reversals, incidents, complaint resolution and manual bypasses. The NIST AI RMF Playbook specifically points to overrides, reported errors, adjudication activity, exceptions and accountable go or no-go decisions as oversight evidence.
- Segment metrics by action type, risk tier, model version and reviewer role.
- Review false approvals and false escalations, not only average acceptance.
- Inspect unusually fast approvals for signs of rubber-stamping.
- Treat repeated reviewer edits as product and policy feedback.
- Reassess thresholds after incidents, policy changes and model updates.
- Give operations a kill switch and a tested manual fallback.
Roll out approval automation in controlled stages
Start with one bounded action whose policy owner, evidence sources and outcome are known. Map current decisions and exceptions, then build the durable state machine and deterministic controls before connecting a model. Run the AI in shadow mode: produce proposals without changing the live route. Compare them with actual decisions and investigate disagreements. A small pilot can then expose proposals to trained reviewers while all side effects remain human-authorized.
Promote only a defined low-risk class to straight-through processing after the team has representative evaluation evidence, reliable monitoring, tested rollback and an owned exception queue. Keep higher-risk actions gated. Revalidate after changing the model, prompt, data source, policy or tool permissions. Expansion should be a risk decision made by accountable owners, not an automatic reward for a high aggregate score.
Key takeaways
- Put the gate before the material side effect and base it on consequence, reversibility and policy.
- Give reviewers verified facts, source evidence and a clear statement of the action they authorize.
- Persist approvals as versioned workflow state with expiry, idempotency and reconciliation.
- Enforce identity, authority, least privilege and prohibited actions outside the model.
- Measure overrides, errors, exceptions and reviewer burden before expanding autonomy.
Frequently asked questions
Which AI actions should always require human approval?
There is no universal list, but high-consequence, privileged, legally reserved, externally binding or hard-to-reverse actions are strong candidates. Examples include changing access, moving money, disclosing sensitive data, accepting exceptions and making commitments. Applicable laws, contracts and internal policies may impose additional requirements.
How can a team prevent approval fatigue?
Use deterministic validation to remove obvious invalid cases, group low-risk items only when policy allows it, route requests to the correct role and show concise evidence. Measure queue volume and unusually rapid decisions. If nearly every request is approved unchanged, reassess whether the gate is well placed or the policy can safely define a narrower automated path.
Should reviewers be able to edit an AI proposal before approving it?
Often yes, within explicit boundaries. Save the edited proposal as a new version, record who changed it and validate it again before execution. Material changes should not inherit approval from the original proposal. Repeated edits are valuable evidence that the prompt, policy, data or interface needs improvement.
Does adding a human approval step make an AI system compliant?
No. Meaningful oversight depends on authority, competence, timing, information, workload and the ability to intervene. Compliance also depends on the use case, jurisdiction and other obligations. A button cannot substitute for risk assessment, technical controls, documentation, monitoring and accountable governance.
Conclusion
Human approval should be a designed control in an end-to-end system. The strongest implementations pause before consequential actions, present evidence that supports independent judgment, enforce authority outside the model and recover safely from timeouts, duplicates and failed writes. Start with a narrow workflow, measure the quality of intervention and increase automation only when operating evidence supports the change.