An AI service is a business capability operated through software, data, models, controls, people, and suppliers. A demo proves that a model can produce an interesting result. Implementation must prove that the complete service can deliver a bounded outcome repeatedly, protect data, stop outside its authority, support affected users, recover from failure, and remain economical as models and sources change. This AI services implementation checklist uses those proofs as delivery gates.
Use the checklist with Edilec's AI services scope and delivery plan, AI services FAQ, and agent governance implementation checklist. It is deliberately vendor-neutral. Applicable law, sector rules, employment obligations, contracts, and internal policy should be reviewed by qualified owners for the deployment context.
Key takeaways
- Define one user outcome, baseline, authority boundary, and fallback before selecting a model or platform.
- Document data provenance, purpose, rights, quality, retention, residency, and access across prompts, retrieval, logs, feedback, and evaluation.
- Evaluate the complete service by representative workflow slice, including security, human factors, latency, cost, and recovery.
- Release through shadow use and constrained pilots with explicit go, narrow, pause, and stop criteria.
- Operate versions, suppliers, incidents, change, recourse, and retirement as first-class service responsibilities.
Stage 1: Prove the use case and boundary
Write a service brief that names the user, job, current process, pain, baseline, permitted input, expected output or action, prohibited action, affected people, accountable business owner, service owner, and manual fallback. Estimate value from time, quality, risk, capacity, or customer outcome, then include review and correction effort. Prefer a narrow workflow with a stable source of truth and recoverable effect. Avoid starting with a general assistant whose purpose changes with every stakeholder conversation.
| Discovery check | Evidence to collect | Gate question |
|---|---|---|
| User need | Observed workflow, volume, failure points, and user interviews | Is the problem important without AI? |
| Baseline | Time, quality, delay, cost, rework, risk, and user outcome | Can improvement be compared honestly? |
| Authority | Allowed advice, decision, tool, and prohibited consequence | Can the service stop safely? |
| Affected people | Direct users, subjects, customers, workers, and reviewers | Are impact and recourse understood? |
| Fallback | Manual route, capacity, service level, and trigger | Can work continue during model or supplier failure? |
| Economics | Full build, inference, integration, review, support, security, and exit cost | Is the useful outcome worth operating? |
Classify consequence early. A drafting aid, recommendation, automated communication, access change, financial action, employment input, or safety-related decision needs different oversight. Apply the NIST AI RMF to organize governance, context mapping, measurement, and management. The point is not to label a system high or low risk once. It is to identify concrete harms, owners, controls, evidence, and conditions that change the classification.
Stage 2: Establish the data and supplier boundary
Inventory data that may enter prompts, retrieval indexes, fine-tuning, evaluation, telemetry, feedback, support, and vendor systems. For each source, record owner, provenance, permission or legal basis, purpose, classification, quality, effective date, access, geography, retention, deletion, correction, and contractual restriction. Separate user-provided content from approved knowledge and system instructions. Apply the NIST Privacy Framework to connect data processing with privacy risk rather than assuming a vendor setting resolves the organization's obligations.
| Supplier question | What to verify | Exit evidence |
|---|---|---|
| Data use | Training, retention, human access, subprocessors, region, deletion, and incident terms | Export and deletion test |
| Model lifecycle | Version notice, deprecation, behavior change, and pinned options | Alternative model or compatibility plan |
| Security | Identity, encryption, isolation, logging, vulnerability handling, and assurance | Current assessment and issue route |
| Reliability | Rate limits, quotas, status, support, recovery, and dependency regions | Fallback exercised under outage |
| Economics | Input, output, cache, tool, storage, throughput, and support pricing | Cost model at expected and stress volume |
| Portability | Prompt, evaluation, embedding, index, logs, and configuration formats | Rebuild time and retained business records |
Minimize data before it reaches a model. Use retrieval and scoped tools rather than copying whole repositories into prompts. Enforce the user's permissions before retrieval; post-filtering a generated answer is too late. Keep secrets in a broker, not model context. Define which traces can be retained for debugging and evaluation, redact protected fields, and control access to them. Build deletion and correction flows across source, index, cache, evaluation corpus, and vendor rather than promising deletion that touches only one store.
Stage 3: Design the service and controls
Document the request path: authentication, authorization, input validation, content handling, retrieval, model routing, prompt assembly, output validation, policy, human review, tools, transaction execution, user response, logging, and feedback. Mark trust boundaries and data classes. Keep deterministic rules deterministic. Use structured outputs for machine consumption, validate every field, and encode or constrain output at its destination. For actions, separate recommendation, preview, confirmation or approval, idempotent execution, and verification.
| Control objective | Implementation example | Acceptance proof |
|---|---|---|
| Least privilege | Per-tool credentials and resource-scoped authorization | Unauthorized cases reveal no data and perform no action |
| Prompt-injection resistance | Treat retrieved instructions as data, isolate policy, constrain tools | Adversarial corpus cannot override authority |
| Reliable output | Schema validation, business rules, citations, and safe fallback | Malformed and unsupported outputs are contained |
| Human oversight | Eligible reviewer sees evidence, consequence, uncertainty, and alternatives | Reviewer can disagree without hidden pressure |
| Transaction safety | Idempotency key, expected version, limits, receipt, and reconciliation | Timeout and duplicate tests create one verified effect |
| Operational containment | Feature, tool, model, tenant, and cohort kill switches | On-call can narrow harm without full outage |
Threat-model user input, retrieved content, files, tool output, model supply chain, and administrative interfaces. The OWASP LLM application project offers a practical risk inventory, and the NCSC secure AI development guidelines cover secure design, development, deployment, and operation. Convert each relevant risk into a control owner, test, monitor, and incident response. A policy paragraph without enforcement and evidence is not an implementation.
Stage 4: Build release evidence
Construct a versioned evaluation set from real workflow slices: routine, ambiguous, incomplete, conflicting, adverse, out-of-scope, multilingual where relevant, and previously failed cases. Document permitted use, annotation rubric, reviewer disagreement, and expected behavior. Test components and the complete service. Measure task quality, critical failures, groundedness, safe refusal, policy compliance, subgroup behavior, human correction, latency, availability, cost, and recovery. Define thresholds before evaluating the candidate.
- Register model, prompt, retrieval snapshot, embeddings, tools, policy, code, configuration, evaluator, and dataset versions.
- Validate deterministic transforms and access controls before subjective quality tests.
- Run security, privacy, tool-failure, timeout, duplicate, stale-state, rate-limit, and supplier-outage scenarios.
- Observe representative users completing work; measure correction effort, automation bias, accessibility, and exception load.
- Compare with the current process and production version by critical slice, not aggregate average alone.
- Record limitations, accepted risks, compensating controls, owner, expiry, and trigger for review.
The NIST AI Resource Center provides material for AI testing, evaluation, verification, and validation. The NIST Generative AI Profile adds generative-AI-specific considerations. Use them as inputs to a release decision tied to the local service. Evidence should answer whether the bounded workflow is useful and controlled, not whether the underlying model appears capable in general.
Stage 5: Release through constrained operation
Start in shadow mode where the AI result does not affect live decisions. Compare it with actual outcomes and inspect disagreement. Move to a small, trained user group with read-only or drafting assistance. Add reversible actions only after policy, confirmation, idempotency, and recovery work. Expand by use case, cohort, language, data source, and action class so each step has interpretable evidence. Publish user guidance about capability, limitations, data handling, feedback, and recourse without forcing users to understand model internals.
| Release gate | Evidence | Decision options |
|---|---|---|
| Workflow value | Better outcome after review and correction effort | Continue, redesign, or stop |
| Critical risk | No unacceptable policy, security, privacy, or safety failures | Block, narrow, or add control |
| Operational fit | Latency, cost, capacity, fallback, and support meet target | Tune route or restrict population |
| Human factors | Users understand role, can challenge output, and are not overloaded | Change interface, training, or review |
| Recovery | Kill switch, rollback, reconciliation, supplier fallback, and communication work | Approve only recoverable scope |
| Ownership | Service, data, model, security, and business owners accept duties | Do not launch an ownerless pilot |
Prepare support and on-call before exposure. Runbooks should distinguish source defect, retrieval failure, model behavior, policy denial, integration fault, user misunderstanding, and supplier outage. Set action thresholds for harmful output, leakage, tool misuse, cost spike, latency, queue growth, and drift. Give operators controls to disable a tool, model route, tenant, or population while preserving unaffected service. Define who communicates with users and who assesses contractual or regulatory obligations.
Stage 6: Operate, change, and retire
Monitor user outcome and system behavior together. Track accepted tasks, corrections, abstentions, escalations, policy denials, complaints, incidents, latency, tool errors, model turns, cost per accepted outcome, and performance by meaningful slice. Sample cases with lawful, minimized retention. Review sources for freshness and permissions. Reevaluate after material changes to model, prompt, retrieval, policy, tool, user population, supplier, or downstream system. Canary changes and retain a tested rollback or restriction path.
Retirement is part of implementation. Define triggers such as supplier deprecation, weak value, unacceptable risk, unsustainable cost, source loss, policy change, or a better non-AI process. Preserve required decisions and audit records, revoke credentials, delete data according to policy, remove indexes and caches, notify users, redirect integrations, and verify that no background task continues. Conduct an exit exercise before dependence becomes critical; portability claims are meaningful only when the team has measured rebuild and reconciliation.
Final implementation checklist
- The use case, baseline, value, prohibited actions, affected people, owners, and fallback are explicit.
- Data and supplier records cover provenance, rights, purpose, quality, access, retention, location, deletion, change, and exit.
- Architecture enforces identity, authorization, validation, policy, oversight, transaction safety, logging, and containment outside model text.
- Evaluation represents real and adverse work, uses decision-linked metrics, and applies predeclared gates by critical slice.
- Release begins with shadow or constrained use and expands only when value, risk, operations, human factors, and recovery pass.
- Production monitoring has owners and action thresholds; feedback and incidents enter a controlled evaluation and change process.
- Support, fallback, supplier outage, rollback, correction, recourse, and retirement have been exercised end to end.
Frequently asked questions
How long should an AI pilot run?
Long enough to cover representative volume, hard cases, user learning, and at least one realistic operational change. Set evidence and a decision date before starting. An open-ended pilot often accumulates production dependence without production ownership.
Should a company build or buy the AI service?
Usually both: buy model or platform capabilities and build the workflow, data contracts, controls, evaluation, integrations, and operations that create differentiation. Compare full lifecycle control, cost, security, portability, support, and exit rather than model quality alone.
Who owns an AI service in production?
One accountable service owner should coordinate business outcome, technology, data, model, security, privacy, support, and supplier duties. Specialist ownership remains distributed, but incidents and changes need a clear final decision route.
Conclusion
AI services implementation is complete only when the organization can explain the purpose, prove the outcome, enforce the boundary, support users, detect harm, recover, change suppliers or models, and retire the service cleanly. A stage-gated checklist makes those obligations visible before scale. It also creates freedom to experiment: teams can try new capabilities without confusing an impressive model response with a dependable service.