Artificial intelligence capabilities implementation starts by matching a specific task to evidence, controls and operating ownership. “Use AI” is not a requirement, and a model's broad demonstration capability does not establish fitness for a consequential workflow. This checklist helps teams decide what to build, buy, constrain or reject. Read it with Edilec's AI capabilities business guide, AI capabilities FAQ and AI services delivery plan.
The NIST AI RMF is voluntary and organizes work through Govern, Map, Measure and Manage. Its Core says capabilities, targeted use, benefits, costs, context and human oversight should be documented. Use that discipline continuously: capability and risk can change when data, model, prompt, integration, population or operating context changes.
Define the task and decision rights
Write the task as an observable transformation: classify a document, retrieve approved evidence, forecast demand, draft a response or recommend a route. Name inputs, outputs, user, affected people, time available, baseline method and downstream action. Separate advisory output from autonomous action. Record prohibited uses and the person who can approve expansion. If the team cannot state what happens after the output, it cannot evaluate useful capability.
Classify the system by context, data, model, task and output. The OECD AI classification framework helps distinguish systems according to characteristics and impact. A summarizer for public material, an employment ranking model and a safety controller should not share one approval path. Determine applicable law, sector rules, contracts and labor duties with qualified owners before selecting architecture.
| Capability | Suitable evidence | Important limitation | Human role |
|---|---|---|---|
| Classification | Labeled representative cases | Class drift and ambiguous cases | Review exceptions |
| Prediction | Temporal validation and calibration | Changing populations | Use with context |
| Generation | Grounded task-level evaluation | Unsupported fluent output | Verify before action |
| Retrieval | Relevant-source judgments | Missing or permission-wrong source | Inspect citation |
| Automation | End-to-end outcome tests | Cascading irreversible error | Approve or stop |
Approve data and component provenance
Inventory training, tuning, retrieval, evaluation and live-input data with source, authority, purpose, license, consent, quality, representation, retention and access. Track third-party models, datasets, libraries, prompts and safety services. Minimize personal and confidential data. Test whether deletion, correction or source withdrawal propagates to indexes, caches and derived artifacts. A model card without an enterprise data flow is incomplete.
For purchased services, document whether prompts and outputs are retained, used for provider improvement, exposed to subprocessors or transferred across regions. Verify contract against technical configuration. Define model and API versions, capacity, rate limits, update notice, security evidence and exit. The organization remains accountable for the workflow it deploys even when a supplier owns the model.
Build task-specific evaluation
Create an evaluation set before optimization. Include ordinary, difficult, rare, subgroup, adversarial, missing-data and out-of-scope cases. Label expected answer, acceptable variation, evidence, refusal and severity. Use quantitative and qualitative review, with domain experts for consequential outcomes. Measure uncertainty and calibration where relevant. Report distributions and worst cases rather than a single blended accuracy number.
For generative systems, the NIST Generative AI Profile identifies risks and actions across the lifecycle. Evaluate supported facts, harmful content, privacy, intellectual property, prompt injection, data leakage and over-reliance according to the use case. Test the complete system including retrieval, tools, policies, user interface and human review, not an isolated model endpoint.
| Release gate | Evidence | Approver | Stop condition |
|---|---|---|---|
| Purpose | Task, users and prohibited use | Business owner | Unbounded authority |
| Data | Provenance and rights review | Data owner | Unknown critical source |
| Performance | Representative evaluation report | Domain owner | Material subgroup failure |
| Control | Access, oversight and incident tests | Risk owner | No safe fallback |
| Operations | Monitoring, support and exit | Service owner | Unowned production path |
Design meaningful human oversight
Specify what reviewers see, what they may change, how disagreement is recorded, how much time they have and when escalation is mandatory. Sample whether review actually catches errors. Avoid confirmation-heavy interfaces that make acceptance easier than inspection. For high-volume low-risk cases, route uncertain or novel items to humans; for high-consequence decisions, require independent evidence and preserve appeal or recourse.
Secure the AI system and its tools
Threat-model access, prompts, retrieval, tool calls, plugins, training pipelines, secrets, logs, outputs and supply chain. Enforce least privilege and allowlisted actions outside the model. Validate tool parameters, confirm consequential changes and limit rate and blast radius. Treat retrieved documents and user content as untrusted input. Monitor abuse, extraction, leakage and unexpected action sequences while protecting the monitoring data itself.
Connect governance to current obligations
ISO/IEC 42001 specifies an organizational AI management system for responsible development, provision or use. It can structure policy, objectives, risk and continual improvement but does not decide a use case's legal status. The European Commission AI Act overview explains a risk-based regime and current application timeline. Organizations operating in or serving the EU should verify role, system category and applicable dates rather than relying on a static checklist.
Deploy, monitor and retire by version
- Approve task, baseline, affected users, authority and prohibited outcomes.
- Record data, model, prompt, integration and supplier versions with provenance.
- Pass representative capability, safety, security and human-factor evaluation.
- Pilot with bounded users, visible fallback, incident handling and stop authority.
- Monitor inputs, outputs, outcomes, overrides, drift, cost and affected-group feedback.
- Revalidate material changes and retire access, data, integrations and records deliberately.
Define change classes and re-evaluation triggers before launch. A new model snapshot, prompt, retrieval corpus, tool permission, population or regulation can invalidate evidence. Keep a rollback version and test provider unavailability. Retirement includes disabling endpoints and credentials, exporting required records, deleting data according to policy and informing dependent teams; removing a user interface alone leaves risk behind.
Conduct an AI capability commissioning review
Commissioning should resemble a service readiness decision, not a model demonstration. Bring together the business owner, affected-user representative, domain expert, data owner, evaluator, security, privacy, legal and operations. Trace one output through input, preprocessing, model, retrieval, policy, tool, human review and downstream action. Ask which actor owns each failure and who can stop the system immediately.
Challenge the evaluation boundary. Compare pilot users and data with production populations, channels, languages and edge cases. Identify cases that were excluded because labels were unavailable or reviewers disagreed; those may represent the most consequential ambiguity. Re-run a blinded sample with independent reviewers and record inter-rater disagreement. A precise score can conceal a subjective or poorly specified target.
Test knowledge limits and abstention. Present missing, contradictory, stale and adversarial inputs and observe whether the system asks for clarification, refuses or fabricates certainty. Measure how users respond to uncertainty language. Calibrate thresholds by consequence and provide an alternate path with realistic capacity. A fallback queue that immediately overwhelms staff is not a working control.
Review supplier and component change. Determine what notice is provided for model retirement, safety-policy updates, context limits, pricing, regional processing and subprocessors. Freeze critical versions where the service permits and maintain an evaluation harness for replacements. Verify data export and deletion. Avoid architecture that grants a general-purpose model permanent credentials merely because future capabilities are unknown.
Run a tabletop for harmful output, personal-data leakage, model outage, prompt injection, compromised tool and public complaint. Confirm detection, containment, evidence preservation, communication, rollback and reporting. Include a case where monitoring fails. Afterwards, close findings or assign dated owners and restrict scope. Commission only the capability demonstrated under these conditions, not every adjacent task the model appears able to perform.
Review environmental and resource constraints when they are material to the use case. Measure latency, energy or compute demand, data movement and peak capacity under the selected architecture rather than a vendor benchmark. Determine whether batching, caching, a smaller model or a non-AI rule can meet the task more reliably. Resource efficiency is both an operating and governance concern: excessive cost can force later sampling or control reductions that invalidate acceptance. Record the chosen tradeoff and the conditions for revisiting it.
Commissioning should also produce a concise evidence index. Link purpose, owners, classification, data flows, model and prompt versions, evaluations, risk decisions, controls, incident plan, monitoring and retirement. Mark superseded artifacts and restrict sensitive ones by role. Select one production claim and trace it through the index during the review. If the trace depends on several people remembering where files live, the capability is not yet supportable at enterprise scale. Set an owner and review date for the index itself, and test access with the support team.
- Trace the full human-AI decision path.
- Challenge population and label assumptions.
- Test abstention and fallback capacity.
- Plan supplier version change and exit.
- Exercise harmful output and tool compromise.
Key takeaways
- Define AI capability at task and decision level.
- Evaluate the complete human-AI workflow on representative and adverse cases.
- Keep data, model, prompt, tool and supplier provenance versioned.
- Make oversight usable and externalize authority from the model.
- Monitor outcomes and revalidate every material context change.
Frequently asked questions
Is a proof of concept enough for production?
No. A proof of concept can test feasibility. Production also needs representative evaluation, security, privacy, integration, human oversight, monitoring, support, continuity, cost and exit evidence.
Should we choose the most capable model?
Choose the smallest architecture that meets the task, risk and operating requirements. Broader capability can add cost, latency and unwanted behavior. Compare models under the same task-level evaluation and controls.
Who owns an enterprise AI capability?
A named business or product owner should own purpose and outcomes, with data, domain, risk, security, legal, engineering and operations responsibilities assigned. A committee can govern standards, but it cannot replace service ownership.
Conclusion
Artificial intelligence capabilities become enterprise capabilities only when task fit, evidence, authority and operations remain connected. Start with the decision, govern components and data, evaluate real cases, constrain actions and monitor outcomes by version. That approach makes it possible to scale useful AI while stopping systems whose confident output exceeds their proven competence.