An AI and data implementation succeeds when a team can connect a business decision to governed data, tested model behavior and an accountable operating process. The model is only one component. Source records, retrieval logic, prompts or features, permissions, user experience, human review and downstream actions all shape the result. This AI and data implementation checklist turns those dependencies into evidence a release owner can inspect.
Start with the broader AI and data practical guide and use the companion AI and data FAQ to settle architecture questions. For a model-facing delivery sequence, compare the AI application production checklist. The goal here is not to approve AI in the abstract. It is to determine whether one named use case is useful, lawful, secure, measurable and supportable.
Define the AI decision and its boundary
Write the use case as a decision contract: who uses the output, what action follows, how soon it is needed, what an error can affect and who may override it. Separate advisory outputs from automated actions. A summarization assistant that drafts text for review has a different consequence profile from a system that changes a credit limit, blocks an account or schedules maintenance. This boundary determines the required data, evaluation rigor, oversight and incident authority.
Use the four functions in the NIST AI Risk Management Framework as a practical cross-check: Govern establishes accountability, Map establishes context, Measure establishes evidence and Manage prioritizes treatment. The functions are not sequential project phases. Each should remain visible as the system changes. If the service will reach the EU market, perform a role and risk classification against the EU AI Act with qualified counsel instead of inferring obligations from a model label.
| Decision field | Minimum record | Release question |
|---|---|---|
| Outcome | Baseline, target, population and measurement window | Would a better model result change a business outcome? |
| Consequence | Effects of false positive, false negative, delay and unavailable service | What harm or cost can each failure mode create? |
| Authority | User, reviewer, approver, override owner and incident owner | Who may act, stop or reverse the output? |
| Boundary | Included channels, regions, data classes and prohibited uses | Can the team detect use outside the approved context? |
Create governed data contracts
For every source, name the system of record, owner, lawful basis or permission, sensitivity, refresh expectation, retention rule and acceptable quality thresholds. Record how identifiers are joined and where labels or reference answers came from. Training data, retrieval corpora, operational context and evaluation sets need separate lineage because they can change on different schedules. A useful contract also states what happens when a required field is late, duplicated, out of range or unavailable.
Minimize data before optimizing a model. The NIST Privacy Framework treats privacy as enterprise risk management, which helps teams ask more than whether data is encrypted. Determine whether collecting, inferring, retaining or exposing information can create adverse consequences for people. Mask direct identifiers in non-production environments, restrict sensitive attributes, test deletion and correction workflows, and prevent logs from becoming an uncontrolled copy of prompts, documents or model outputs.
- Inventory source tables, documents, APIs, event streams and externally licensed data.
- Assign an owner and consumer-facing contract to every production input.
- Version transformations, feature logic, retrieval filters, prompts and model configuration together.
- Keep representative evaluation cases separate from tuning examples to reduce leakage.
- Document exclusions and coverage limits so users understand where evidence is weak.
- Test retention, access revocation, correction and deletion across derived stores and logs.
Evaluate the complete workflow
A single accuracy score is rarely enough. Build an evaluation set around real tasks and important subgroups, including routine cases, difficult cases, abstention cases, malformed input and known misuse. Select metrics from the decision consequence: precision may matter when false accusations are costly; recall may matter when a missed event is dangerous; calibration may matter when a confidence threshold controls escalation. For generative systems, use explicit rubrics for groundedness, completeness, prohibited content and correct tool use.
The NIST Generative AI Profile identifies risks that require lifecycle treatment, not a one-time demonstration. Evaluate output against authoritative references, test indirect prompt injection in retrieved material, and verify that untrusted model text cannot silently become a privileged command. Human review is itself a control that needs testing: reviewers require enough source context, time, authority and interface support to catch material errors rather than merely approve them.
Build security and operating controls into the path
Apply conventional software controls to the whole AI stack. The NIST Secure Software Development Framework calls for protected code, verified components, security criteria and vulnerability response. Extend the software bill of materials with model, dataset and service dependencies. Pin versions where reproducibility matters, scan images and libraries, isolate secrets, use workload identities instead of embedded credentials, and restrict model tools to narrow allowlisted actions with server-side authorization.
Design a safe failure state. If retrieval is stale, a model endpoint times out or a confidence rule fails, the workflow should queue for review, return a bounded message or use an approved deterministic path. Never let a model decide whether it has permission to perform the action it proposes. Log input and output references, policy decisions, tool calls, reviewer changes and final business action using privacy-aware identifiers. Give support teams a runbook for disabling one feature, source or model without shutting down unrelated work.
| Gate | Evidence to retain | Example acceptance criterion |
|---|---|---|
| Data readiness | Contracts, lineage, profiling and privacy review | Critical fields meet thresholds for the measured population |
| Model readiness | Versioned evaluation set, results and limitations | All critical failure classes remain below approved limits |
| Workflow readiness | Authorization tests, human-review study and failure drills | Unsafe actions are blocked and overrides are attributable |
| Operational readiness | Dashboards, alerts, rollback and supplier contacts | On-call staff restore or disable the feature within the objective |
| Governance readiness | Risk decision, approvals and user communication | Named owners accept residual risks and review dates |
Release gradually and monitor decisions
Begin with shadow evaluation or a limited advisory release where outputs cannot directly alter records. Compare AI recommendations with actual outcomes and reviewer decisions. Expand by user group, workflow or risk tier only when the evidence supports it. A percentage rollout is useful only if the team can identify affected decisions, compare cohorts and roll back cleanly. Capture model, prompt, data and policy versions for each production event so an incident can be reconstructed.
Monitor the service and the decision. Technical indicators include latency, failure rate, token or compute cost, retrieval misses and tool errors. Quality indicators include correction rate, abstention, groundedness, subgroup performance and escalation. Outcome indicators include cycle time, completion, loss avoided and user appeal. The updated OECD AI Principles emphasize accountability, transparency, robustness, security and safety across the lifecycle; those properties require owners and thresholds, not a values statement.
Implementation example: invoice exception triage
Consider an accounts-payable team that wants AI to classify invoice exceptions and draft a routing recommendation. The approved boundary is advisory: the model cannot approve payment, change supplier master data or post to the ledger. Source contracts cover the invoice image, purchase order, receipt and supplier record. Retrieval is filtered to the user's legal entity. The evaluation set includes duplicates, credit notes, tax differences, missing receipts, altered bank details, handwritten annotations and documents containing instructions aimed at the model.
Release begins in shadow mode. Reviewers compare the proposed category and evidence with their normal decision. Critical measures include missed bank-detail changes, unsupported claims, correct source citation, review time and percentage routed without rework. Low-confidence cases and unavailable sources go to the existing queue. A server-side policy blocks all write tools. After limited release, operations monitor source freshness, correction by exception class, reviewer overrides and cost per completed case. Weekly review examines whether suppliers, languages or invoice formats are underrepresented and adds approved cases to the regression set. This narrow design creates evidence for expansion without giving an uncertain component financial authority.
Change review uses the same evidence chain. A new model, revised extraction prompt or supplier-data source cannot move directly to production because a benchmark improved. The owner records the reason, affected populations, evaluation comparison, privacy and security impact, rollout cohort and rollback trigger. Production observations remain comparable by version. If correction rises for one document class, the team can disable that path, route affected invoices to manual handling and preserve the rest of the service while investigation proceeds.
Key takeaways
- Approve a decision boundary, not a general technology purchase.
- Treat data, prompts, retrieval, model configuration and policy as versioned production dependencies.
- Choose evaluation metrics from the consequences of error and test important subgroups and misuse.
- Keep authorization deterministic and place human review where a reviewer can materially intervene.
- Release in observable stages with a tested fallback, rollback and incident owner.
- Measure business decisions and user corrections alongside model and infrastructure health.
How much data is needed before implementation?
There is no universal row count. The team needs enough representative, permitted and well-understood data to evaluate the intended population and important failure modes. A narrow retrieval assistant may rely on a small curated corpus, while a predictive model may require substantial historical coverage. Start by proving source authority, label quality and evaluation coverage. More data does not correct a poorly framed decision or biased collection process.
Can a vendor's model evaluation replace application testing?
No. Vendor results describe a model under the vendor's conditions. Your application adds instructions, retrieval data, tools, permissions, users and consequences. Test the assembled workflow with your representative cases, including dependency failures and adversarial input. Supplier documentation remains useful evidence for model selection and dependency risk, but application owners retain responsibility for the deployed use.
Conclusion
A disciplined AI and data implementation makes uncertainty visible. It links a bounded decision to governed inputs, reproducible evaluation, enforceable controls and monitored outcomes. Teams that preserve that chain can improve models without losing accountability, and they can stop or reverse a release when the evidence no longer supports safe use.