{"id":"KM-AI-0021","slug":"how-engineering-teams-should-think-about-ai-agents","title":"How Engineering Teams Should Think About AI Agents","excerpt":"AI agents should be engineered as bounded services with explicit goals, tools, identities, approvals, observability, recovery paths, and evidence for every consequential step.","kind":"Guide","category":"ai","tags":["ai agents","Artificial Intelligence","AI operations","governance","product delivery"],"seoKeywords":["ai agents","ai agents guide","ai agents operations","ai agents architecture","ai agents evaluation"],"authorId":"krishnam-murarka","publishedAt":"2026-06-24","updatedAt":"2026-09-09","readingTime":"10 min","image":"/social-images/blog/edilec-photo-km-ai-0021-7e05d7812f18.jpg","featured":false,"trending":true,"sourceCredits":[{"title":"AI Risk Management Framework","url":"https://www.nist.gov/itl/ai-risk-management-framework","author":"National Institute of Standards and Technology"},{"title":"Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile","url":"https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf","author":"National Institute of Standards and Technology"},{"title":"OWASP Top 10 for Large Language Model Applications","url":"https://owasp.org/www-project-top-10-for-large-language-model-applications/","author":"OWASP Foundation"},{"title":"MITRE ATLAS","url":"https://atlas.mitre.org/","author":"MITRE"},{"title":"Building effective agents","url":"https://www.anthropic.com/engineering/building-effective-agents","author":"Anthropic"}],"researchSources":[{"title":"AI Risk Management Framework","url":"https://www.nist.gov/itl/ai-risk-management-framework","author":"National Institute of Standards and Technology","reason":"Supports the controls, evaluation, and operating guidance in this ai agents guide."},{"title":"Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile","url":"https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf","author":"National Institute of Standards and Technology","reason":"Supports the controls, evaluation, and operating guidance in this ai agents guide."},{"title":"OWASP Top 10 for Large Language Model Applications","url":"https://owasp.org/www-project-top-10-for-large-language-model-applications/","author":"OWASP Foundation","reason":"Supports the controls, evaluation, and operating guidance in this ai agents guide."},{"title":"MITRE ATLAS","url":"https://atlas.mitre.org/","author":"MITRE","reason":"Supports the controls, evaluation, and operating guidance in this ai agents guide."},{"title":"Building effective agents","url":"https://www.anthropic.com/engineering/building-effective-agents","author":"Anthropic","reason":"Supports the controls, evaluation, and operating guidance in this ai agents guide."}],"mediaAssets":[],"status":"published","body":[{"type":"paragraph","text":"Ai agents should be treated as a software component that can select actions or tools toward a goal within constraints set by the application and the organization, not as a free-standing model feature. A useful implementation starts with the work item that must improve, the person accountable for the result, and the evidence that proves the result is safe enough to use. That framing keeps design conversations concrete: which inputs are allowed, what the system may propose, what it must not decide, and how a user can see the basis for an output. It also makes room for operational reality. A system can sound capable in a demonstration yet create new queues, hidden data flows, and unreviewable exceptions when it is placed in routine work."},{"type":"heading","id":"define-the-operating-boundary","text":"Define the ai agents operating boundary"},{"type":"paragraph","text":"The first operating decision for ai agents is the boundary. Teams should make the agent's permitted goal, data scope, tool verbs, spending or mutation limits, and required approvals explicit outside its natural-language instructions. Write this boundary as a short case contract that names the initiating event, permitted inputs, authoritative systems, expected output, prohibited action, human owner, and recovery route. The contract is not bureaucracy for its own sake. It gives engineers a testable behavior, operators a reason to stop a case, and reviewers a shared answer when a plausible-looking output conflicts with policy or source evidence. Change requests should update the contract before they expand permissions or scope."},{"type":"table","columns":["Control question","Practical decision","Evidence to keep"],"rows":[["Outcome","Name the work result and its accountable owner.","Case contract, baseline, and success threshold."],["Authority","State what the capability may recommend, read, or change.","Permission decision and approval rule."],["Sources","Identify the records that can support an output.","Source owner, version, date, and access scope."],["Exceptions","Define when to abstain, hold, or escalate.","Reason code, queue, and service target."],["Recovery","Specify how to pause and reconcile a faulty path.","Incident record, affected cases, and restart approval."]]},{"type":"heading","id":"engineer-the-action-boundary","text":"Engineer the action boundary"},{"type":"paragraph","text":"A dependable design preserves the task request, agent identity, plan or selected action, tool inputs and outputs, authorization decision, human approval, state transition, and compensation or rollback record. The service should be able to reconstruct a completed case without relying on a person's memory or a chat transcript that has already scrolled away. In practice, that means stable identifiers, versioned configurations, timestamps, and an auditable connection between evidence, recommendation, approval, and outcome. Give agents narrow tools with structured inputs, independent policy checks, idempotency where possible, timeouts, rate limits, and a durable workflow state rather than an open-ended tool loop. The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) is useful here because it frames trustworthy AI as a lifecycle concern: governance, mapping, measurement, and management are activities to make visible in the work, not a compliance label added at the end."},{"type":"image","src":"/social-images/blog/edilec-photo-km-ai-0021-7e05d7812f18.jpg","alt":"A laptop on a train table shows an agent run waiting for approval before sending an update.","caption":"Fictional editorial scene: AI agents need externally enforced tool scope, approvals, observable evidence and boundaries around consequential steps.","width":1200,"height":750},{"type":"heading","id":"run-the-service-with-signals","text":"Run the service with signals"},{"type":"paragraph","text":"Operations decide whether ai agents remains useful after launch. Measure task completion with valid evidence, tool error rate, policy denials, unauthorized-action attempts, human approval latency, rollback frequency, and cost per completed workflow. These measures need owners and thresholds, not just a dashboard. A rising correction rate may indicate source drift, a changed user population, or a confusing interface; it does not automatically justify a model swap. Review results by meaningful slices such as task type, business unit, data source, impact level, and exception route. Pair quantitative signals with sampled case review so the team can distinguish a genuine service improvement from a metric that improved because difficult work was diverted elsewhere."},{"type":"table","columns":["Signal","What it can reveal","Operational response"],"rows":[["Outcome quality","Whether useful work is actually improving.","Sample cases and compare with the baseline."],["Exception pattern","Where policy, data, or model behavior is weak.","Route a named owner and add a durable test case."],["Source or input freshness","Whether evidence remains fit for use.","Refresh, retire, or restrict the affected source."],["Human intervention","Whether review capacity and authority are adequate.","Adjust routing, service targets, or staffing."],["Cost and latency","Whether the service can scale responsibly.","Optimize the expensive path without lowering the quality gate."]]},{"type":"callout","tone":"warning","title":"Control to preserve","text":"For ai agents, do not allow a confident output to substitute for authorization, evidence, or an accountable decision. Make the abstention and escalation path as deliberate as the successful path."},{"type":"heading","id":"roll-out-with-a-fallback","text":"Roll out with a fallback"},{"type":"paragraph","text":"For rollout, deploy in read-only or proposal mode first, then introduce one reversible action with a clear receipt and a named operator who can pause the capability. Establish a baseline before enabling the new capability, decide what result would pause expansion, and retain a reliable fallback. Start with a limited audience and a named support path. Releases should include a simple runbook: how to identify an affected case, how to inspect its trace, who can disable the capability, and how to reconcile downstream effects. This creates evidence for a real product decision rather than forcing the organization to infer quality from anecdote."},{"type":"list","items":["Map normal cases, uncomfortable edge cases, and requests the service must decline.","Name the business owner, technical owner, reviewer group, and incident contact.","Version the configuration, sources, prompts, tools, and evaluation set used for each release.","Set release criteria for quality, permissions, latency, cost, and support readiness.","Give users a visible way to report an incorrect result or a missing source.","Review the evidence after each expansion before granting broader data access or action authority."]},{"type":"heading","id":"prevent-predictable-failures","text":"Prevent predictable failures"},{"type":"paragraph","text":"The recurring failure is giving a general-purpose model a broad credential and treating a prompt instruction as the authorization boundary for money, records, communications, or production systems. This is why [agent tool permissions](/blog/gen-ai-0008/agent-tool-permissions-a-practical-guide-for-technical-decision-makers/) is a useful adjacent design problem: the interface is only one layer of a system that also needs ownership, access controls, evidence, and recovery. Use pre-mortems with operators and reviewers to identify the moment when a bad output could become a bad decision. Then convert that moment into a deterministic check, a review gate, an explicit abstention, or a compensation path. A model should never be the only place where a material control exists."},{"type":"heading","id":"improve-with-verified-cases","text":"Improve with verified cases"},{"type":"paragraph","text":"Agent reliability improves when each tool call is treated like a production API request with an identity, schema, authorization decision, timeout, and receipt. Use simulations and a sandbox to test malformed tool outputs, stale state, duplicated requests, interrupted handoffs, and adversarial content in retrieved material. A successful demo path is not an action model. The engineering question is whether the agent can fail boundedly: stop, record what it attempted, avoid repeating an external effect, and leave the next operator enough context to recover."},{"type":"paragraph","text":"Review agent traces as a product and security practice. Look for repeated plan revisions, unnecessary tool calls, denied permissions, unexpected parameter values, and actions that succeeded technically but did not solve the user task. Link those observations to a small set of approved improvements: tighten a tool schema, add a policy rule, improve case state, or change the approval route. Broadening autonomy should be a deliberate release decision supported by those traces, never an accidental side effect of a new prompt or credential."},{"type":"paragraph","text":"Separate orchestration from authority. An agent may decide which bounded step to attempt next, but the service hosting the tool should decide whether that step is permitted in the current state. Encode high-impact constraints in schemas and policy engines: allowed destinations, amounts, record fields, environments, and approval identities. Give tools narrow return types so the agent cannot smuggle instructions through unstructured output into another privileged action. For long-running work, store state durably and make resumption explicit. This reduces the chance that a retry after a timeout repeats an external effect or that an operator cannot tell what has already happened."},{"type":"paragraph","text":"Run a periodic permission review for every agent identity and tool. Remove unused grants, verify approval routes, and replay a sample of consequential traces. This simple operational habit catches authority creep before a new feature or integration turns a bounded agent into a broadly privileged service."},{"type":"heading","id":"takeaways","text":"Key takeaways"},{"type":"list","items":["Ai agents needs a bounded job and a named accountable owner.","Evidence, permissions, and approval should be inspectable outside model instructions.","Measure quality and operational burden by meaningful case slices, not a single average.","Keep a fallback, a pause authority, and a reconciliation procedure before scaling.","Use verified failures and reviewer corrections to improve the workflow and its evaluation set."]},{"type":"heading","id":"faq","text":"Frequently asked questions"},{"type":"paragraph","text":"**When is ai agents ready for production?** It is ready for a limited production release when the permitted task, source scope, evidence record, accountable owner, quality threshold, exception route, and rollback path are all explicit and exercised. **What should be automated first?** Choose a repeated, reversible step that reduces preparation work while preserving human authority over consequential decisions. **How often should it be reviewed?** Review after material changes to users, data, tools, policy, model configuration, or observed incident patterns, and set a regular operating cadence for the service."},{"type":"heading","id":"conclusion","text":"Conclusion"},{"type":"paragraph","text":"Ai agents earns trust when it improves one bounded task while leaving responsibility and evidence legible. Keep the first release narrow, measure the work rather than the novelty, and expand only after the team can explain what happened in normal cases, exceptions, and recovery. That is the practical path from an impressive capability to an operation people can rely on."},{"type":"heading","id":"sources-and-practice-notes","text":"Sources and practice notes"},{"type":"paragraph","text":"Anthropic's engineering guidance usefully distinguishes workflows from more autonomous agent patterns; in either case, external authorization and operational controls remain necessary. The [NIST Generative AI Profile](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf) and the [OWASP Top 10 for LLM applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) are complementary references: one helps structure lifecycle risk decisions, while the other keeps common application-level failure modes in view. Read them against the actual workflow and applicable obligations; neither replaces a careful assessment of local data, users, and consequences."},{"type":"image","src":"/attachments/article-media/editorial/edilec-ai-agents-bounded-action-control-loop.svg","alt":"ai agents: accountable operating path","caption":"A six-stage operating path for ai agents, from a bounded work item to measured improvement."}],"faqs":[],"relatedIds":["KM-AI-0022","KM-AI-0028","KM-AI-0040","KM-AI-0146"],"relatedArticleIds":["AI-1024","GEN-AI-0008","GEN-AI-0024","KM-AI-0022","KM-AI-0028","KM-AI-0040"]}