Data and AI solutions turn organizational data into decisions, predictions, automation or generated content. They range from governed reporting and forecasting to document classification, recommendations and assistants. The right starting point is a decision or workflow that can be improved, not a preferred model. A production solution also needs suitable data, evaluation, privacy and security controls, human responsibility, integration, monitoring and a plan for change when data or model behavior shifts.
Choose a use case with a measurable decision
Describe who will use the output, what decision or action follows and what happens when the output is wrong. A forecasting tool may influence staffing; a classifier may route requests; a retrieval assistant may help an employee find approved guidance. Define the current baseline, frequency, delay and cost of errors. If no owner can state how an output changes work, a polished model is unlikely to create durable value.
Compare deterministic rules, search, workflow changes, conventional analytics and machine learning. AI is justified when patterns or language cannot be handled adequately by simpler methods and the organization can evaluate the result. Prioritize use cases by value, feasibility, risk and reuse of data or platform capabilities. Avoid starting with a high-impact automated decision merely because it offers a visible demonstration.
| Use-case question | Decision | Evidence |
|---|---|---|
| What changes? | Named decision, task or user outcome | Current workflow and baseline |
| How are errors different? | Cost of false positive, false negative or unsupported output | Reviewed example set |
| Can a simpler method work? | Rules, search, process or analytics alternative | Small comparison experiment |
| Is data permitted and suitable? | Purpose, quality, rights and representativeness | Data profile and governance approval |
| Who remains accountable? | Output reviewer, process owner and risk owner | Responsibility and escalation map |
| How will value persist? | Adoption, quality, cost and monitoring measures | Benefits and operating plan |
Build the data foundation before scaling the model
Inventory source systems, owners, collection purpose, access, quality, retention and change history. Define the unit of analysis and target carefully: a label created by a historical process may encode delay, policy or unequal access rather than the outcome the team intends to predict. Split training, validation and test data in a way that prevents leakage across time, people or related records. Keep transformations reproducible and versioned.
For retrieval and generative systems, document which sources are authoritative, who can access them and how quickly updates or deletions must appear. Preserve source links and document versions so users can inspect evidence. W3C PROV-O offers concepts for representing entities, activities and responsible agents; an implementation need not adopt the ontology to benefit from explicit provenance. The essential question is whether a team can explain which data and processing produced an important output.
| Data layer | Control | Failure to detect |
|---|---|---|
| Source | Owner, permitted purpose and extraction contract | Unauthorized or unexplained use |
| Quality | Completeness, validity, timeliness and duplicate checks | Systematic bad inputs |
| Transformation | Versioned code and lineage | Irreproducible features or metrics |
| Training and evaluation | Leakage-resistant splits and dataset record | Inflated test performance |
| Knowledge retrieval | Access-aware indexing and source freshness | Exposure or outdated answers |
| Feedback | Reviewed labels and abuse protection | Self-reinforcing errors |
Govern AI risk across the lifecycle
NIST's AI Risk Management Framework organizes work around Govern, Map, Measure and Manage. Governance sets policies, roles and risk tolerance; mapping establishes context and affected people; measurement evaluates trustworthiness; management prioritizes and treats risk. Use these functions throughout design, acquisition, testing and operation. A one-time review before launch cannot address model, data, supplier or usage changes.
Classify the use by impact and applicable obligations. Consider safety, rights, privacy, financial loss, access to services, employment and whether people can understand and challenge outcomes. The EU AI Act uses a risk-based legal framework with obligations that vary by role and use; applicability and timing require qualified legal assessment, not assumptions from a product label. Maintain an inventory that records purpose, owner, model and data dependencies, affected groups, approvals and current status.
The NIST Privacy Framework helps teams examine how data processing may create problems for individuals, not only whether data is confidential. Apply data minimization, purpose limitation, access controls, retention and deletion. For generative AI, also test prompt injection, disclosure, harmful content, misleading output, over-reliance and third-party model changes. NIST's Generative AI Profile and SSDF community profile provide lifecycle actions for these distinct risks.
Design evaluation around the intended use
Create an evaluation set from representative, difficult and high-impact cases, with documented selection and review. Select metrics that reflect the error tradeoff: precision and recall for classification, forecast error across relevant periods, ranking quality for retrieval, or rubric-based review for generated answers. Break results down by meaningful groups and conditions where lawful and appropriate. Report uncertainty and examples, not one aggregate score.
Generative outputs need repeatable task tests, source-grounding checks, safety tests and human review criteria. Automated model-based graders can assist but should themselves be validated for the use. Test attempts to override instructions, retrieve unauthorized material and provoke unsupported answers. Set release thresholds, but also define conditions that require abstention, escalation or a deterministic fallback. Human review must provide time, evidence and authority to disagree; a decorative approval button is not oversight.
Estimate full lifecycle cost
AI project cost is not only model training or an API charge. Discovery, data access, labeling, governance, experimentation, integration, evaluation, security and change management often dominate. Recurring cost includes inference, storage, retrieval, monitoring, vendor minimums, human review, incident response and repeated evaluation. Forecast by a business driver such as documents, interactions or predictions, then test sensitivity to input and output size, concurrency, retention and growth.
| Cost area | Main drivers | Evidence to gather |
|---|---|---|
| Use-case discovery | Stakeholders, decisions and risk uncertainty | Workflow map and baseline |
| Data preparation | Sources, quality, rights, labeling and lineage | Profiled sample and ownership |
| Model work | Build, buy, tune or configure complexity | Comparison on representative tasks |
| Evaluation | Impact, test coverage and expert review | Evaluation plan and scored set |
| Integration | Workflow, identity, interfaces and exception handling | End-to-end proof |
| Operation | Inference, monitoring, review and support | Volume model and service objectives |
| Change and exit | Retraining, supplier change and migration | Version and portability plan |
Use a staged investment case. Fund enough work to test data suitability and the central value assumption, then decide whether to proceed, change approach or stop. For external models, record service terms, data-use settings, geographic processing, retention, rate limits, version policy and exit options. A cheaper unit price can be outweighed by review labor, weak observability or an expensive migration from proprietary interfaces.
Example: assisting service-request triage
Consider a service team that receives free-text requests and manually assigns a category and queue. A bounded first release suggests a category and retrieves the relevant routing guidance; an employee confirms or changes it. Historical categories are reviewed for inconsistency before training. The evaluation set emphasizes ambiguous requests, rare high-impact categories and language variation. Success includes routing accuracy, correction rate and handling time without an increase in reopened work.
The system shows the suggested category, confidence or abstention state, and the guidance source. It cannot close requests or make eligibility decisions. A limited group starts with shadow recommendations before suggestions affect routing. Corrections are reviewed rather than automatically becoming labels. Monitoring covers category distribution, abstention, override reasons, latency, source freshness and supplier version. A rules-based route remains available during outage or unacceptable drift.
Manage data and AI risks explicitly
| Risk | Control | Early signal |
|---|---|---|
| Wrong problem | Baseline and simple alternative comparison | Output has no clear downstream action |
| Data leakage or misuse | Purpose review, minimization and access-aware pipelines | Unexpected sensitive content |
| Unrepresentative evaluation | Curated slices and independent review | Large subgroup or edge-case gaps |
| Automation bias | Evidence, abstention and meaningful human authority | Reviewers rarely challenge outputs |
| Model or data drift | Version tracking, monitoring and scheduled reevaluation | Quality or distribution change |
| Supplier dependency | Abstraction, export and tested fallback | Unannounced model behavior change |
| Unbounded cost | Usage limits, unit measures and anomaly alerts | Tokens or review time rise per task |
A staged data and AI delivery plan
- Frame the decision, affected users, baseline, owner, risk tier and simpler alternatives.
- Assess data purpose, rights, quality, representativeness, lineage and access.
- Prove value with a small comparison and a documented evaluation set.
- Design integration, security, human review, fallback, monitoring and supplier controls.
- Pilot in shadow or advisory mode with a limited group and active support.
- Release progressively against quality, safety, adoption, cost and service thresholds.
- Operate with version records, drift review, incident handling, reevaluation and retirement criteria.
A launch decision should show performance by relevant conditions, unresolved limitations, privacy and security findings, fallback behavior, monitoring owners and user readiness. Define a pause mechanism and preserve the ability to identify outputs from each model, prompt, retrieval index and policy version. When the use or impact changes, return to mapping and evaluation rather than assuming the original approval still applies.
Key takeaways
- Start with a measurable decision or workflow and compare simpler alternatives.
- Treat data purpose, quality, provenance and access as product requirements.
- Evaluate representative tasks, error tradeoffs and human oversight before automation.
- Estimate data, review, integration, monitoring and change costs alongside model usage.
- Release in bounded stages with fallback, traceability, reevaluation and accountable owners.
Frequently asked questions
Should we build or buy an AI model?
Compare task performance, data sensitivity, control, speed, operating skill, cost and exit options. Many teams should integrate an existing model first, but a representative evaluation matters more than a general benchmark or vendor claim.
Why do AI pilots fail to reach production?
Common gaps include unclear workflow value, unsuitable data, no integration owner, weak evaluation, unplanned review effort and absent monitoring. Design the proof to test production constraints rather than only model capability.
Does human review make an AI system safe?
Not automatically. Reviewers need understandable evidence, enough time, training, authority to override and an escalation route. Measure overrides and missed errors to learn whether oversight works in practice.
Make AI an accountable operating capability
A durable data and AI solution links a justified use case to governed data, representative evaluation and a workflow people can challenge. Staged investment and progressive release keep uncertainty visible. With clear ownership, traceable versions and continued monitoring, the organization can improve the system while retaining the ability to pause, change or retire it.