Durable agent execution means a run can pause, lose its worker or cross a deployment boundary and still resume from an authoritative checkpoint without repeating unsafe effects. Durability is not keeping a process alive. It is persisting enough typed state to decide what already happened, what remains valid, which authority still applies and which transition may occur next. This becomes essential when an agent waits for approval, external work, a human answer or a retry window.
A durable design wraps the model loop in workflow semantics. The OpenAI Agents SDK results guide exposes run results and state that can continue later, while the A2A specification models long-running tasks, status updates and artifacts. Those primitives help, but the application still owns persistence, business invariants, expiry and recovery. Edilec's agent workflow architecture guide provides the surrounding loop.
Define the durable agent execution model
Give every run a stable identifier, task type, owner, tenant, current state, state version and terminal outcome. Separate workflow state from transient model context. Workflow state records facts needed to resume: accepted objective, completed actions, pending approvals, evidence references, deadlines and next allowed transitions. Model context can be reconstructed from approved facts and selected history. If a run can resume only by replaying an entire transcript, its state model is implicit and fragile.
Choose terminal states such as completed, failed, canceled and expired, plus explicit waiting states for user input, approval, external event and retry. “Running” is not enough for operations because it cannot reveal whether a worker is active or work is merely eligible. Store lease and heartbeat separately from business state. A crashed worker should release its lease without changing a task from waiting to failed unless policy says the task itself cannot continue.
Persist a versioned checkpoint contract
A checkpoint should contain state-machine position, state version, input and output references, completed tool-call records, pending actions, approvals, policy versions, model and prompt configuration identifiers, trace link, attempt count and timestamps. Encrypt sensitive fields and prefer references to authoritative systems over copied records. Include a schema version and migration path so a deployment can read older checkpoints or deliberately quarantine them.
Commit checkpoints at semantic boundaries: before requesting approval, before dispatching a side effect, after reconciling that effect and before entering a wait. Do not persist arbitrary in-memory objects whose serialization changes with a library upgrade. The checkpoint is an application contract, not a framework memory dump. Test round-trip recovery in a fresh process and a newer compatible deployment.
| Checkpoint field | Purpose | Resume validation |
|---|---|---|
| State and version | Selects permitted transitions | Schema and migration supported |
| Evidence references | Rebuilds verified context | Source exists and access remains valid |
| Action ledger | Prevents duplicate side effects | Outcome reconciled or explicitly unknown |
| Approval record | Proves bounded human authority | Digest matches and approval has not expired |
| Runtime configuration IDs | Explains behavioral dependencies | Approved model, prompt and tool set available |
Separate replayable computation from side effects
Model calls, parsing and deterministic policy evaluation may be repeated if inputs and versions are recorded, though a model may produce a different answer. External effects such as payment, email, ticket creation and configuration change cannot be assumed replayable. Wrap each in an operation record with a stable idempotency key, requested effect, dispatch state and authoritative outcome. Resume queries that record before deciding to invoke again.
Use an outbox or workflow command record when a database change and external dispatch must remain coordinated. The agent commits intent; a worker dispatches it; a reconciler records the observed effect. Exactly-once message delivery is rarely available end to end, so design for at-least-once attempts with deduplicated business effects. The agent control plan adds action tiers and approval controls around these boundaries.
Bind approval to the state that will resume
When pausing, create an approval request with a digest of consequential fields, evidence references, policy reason, approver role, expiry and allowed decisions. The checkpoint enters waiting_approval only after the request is durable. The worker then releases resources. On response, verify approver authority and write the decision as a new state transition using optimistic concurrency. A late duplicate response returns the current state instead of applying twice.
Before executing an approved action, compare the request digest with current action fields and revalidate material facts. Price, account status, inventory, sanctions status or policy may have changed during the wait. Define which changes invalidate approval, which require refreshed evidence and which are immaterial. Approval is not a reservation of external state unless the domain system explicitly created one.
Revalidate stale state before resuming
Every waiting state needs an expiry and freshness policy. On resume, check task deadline, user and workload access, referenced record versions, policy version, tool availability and downstream compatibility. If a material dependency changed, transition to refresh, replan, reapprove or expire. Do not silently feed old evidence back to the model and let it decide whether it remains safe.
Use optimistic concurrency on task state so two workers cannot resume the same version. Acquire a short lease, compare the expected state version, perform bounded work and commit the next state atomically. If commit fails, discard the stale computation and reload. Model output generated from an old version is advisory data, not permission to overwrite newer state.
Distinguish timeout, cancellation, and failure
A timeout means a caller stopped waiting or an operation exceeded its window; it does not prove the underlying work stopped. Cancellation is an explicit request to prevent future work and, where possible, interrupt current work. Failure means a defined attempt cannot continue. Persist each separately. After a dispatch timeout, query the authoritative system before retrying. After cancellation, reconcile in-flight effects and report any action that could not be undone.
Define cancellation scopes: current model turn, pending tool call, child tasks or entire business task. Propagate cancellation tokens where supported, but treat them as requests rather than guarantees. Compensation is a new authorized business action, not database rollback by assumption. A refunded payment, closed ticket or restored configuration needs its own identity, evidence and possible approval.
| Interruption | Persisted transition | Safe next step | Never assume |
|---|---|---|---|
| Approval wait | waiting_approval | Release worker and await durable decision | Approval will arrive before facts change |
| Worker crash | State unchanged, lease expires | Another worker reloads checkpoint | Last external call did not happen |
| Tool timeout | outcome_unknown | Query authoritative record | Timeout equals failure |
| User cancellation | cancel_requested | Stop new work and reconcile in-flight effects | All effects were reversed |
| Deployment change | resume_validation | Migrate or quarantine checkpoint | Serialized runtime objects remain compatible |
Observe transitions, not just model calls
Emit events for state transition, checkpoint write, lease acquisition, resume validation, action dispatch, reconciliation, approval and terminal outcome. Include task ID, state version, attempt, policy version and parent task. The OpenTelemetry generative AI conventions help correlate model and agent spans; workflow events should add controlled business fields without recording sensitive prompt content.
Measure age by waiting state, resume failures, duplicate suppression, unknown-outcome duration, checkpoint migration errors, cancellation latency and terminal completion. Alert on tasks with no valid next event, not merely long total duration. A legitimate three-day approval wait differs from a three-day task whose worker lease never cleared. Operations need a queryable queue and authorized controls to retry, cancel, reconcile or escalate, backed by investigation-ready events from the audit log design guide.
Prove recovery before long-running production work
Start with a read-only workflow that pauses for input. Kill workers after each checkpoint and verify fresh-process recovery. Then introduce an approval with bound request fields, followed by one reversible idempotent action. Inject duplicate events, delayed approvals, policy changes, deployment upgrades and ambiguous responses. Advance only when every run reaches a known terminal or owned recovery state.
Apply secure development discipline to checkpoint schemas, migration code and recovery tools. NIST's SP 800-218A extends secure software practices for generative AI systems and their producers and acquirers. Protect state stores, minimize retained context, review administrative resume capability and test dependency changes. Durable execution increases retained value and therefore increases the importance of access control and deletion.
Document regional ownership as well. Only one region should advance a task version at a time, while failover verifies the latest committed checkpoint and operation ledger before acquiring work. Disaster recovery is incomplete if restored tasks can repeat effects that survived in an external system.
Key takeaways
- Persist explicit workflow state rather than relying on a live process or transcript replay.
- Checkpoint at semantic boundaries and version the application-owned state contract.
- Record side effects in an idempotent action ledger and reconcile unknown outcomes before retries.
- Bind approvals to request digests, then revalidate authority and material facts on resume.
- Treat timeout, cancellation, worker failure and task failure as different transitions.
Frequently asked questions
Can a conversation database serve as the checkpoint?
Not by itself. Conversation history lacks explicit transitions, completed-action records, approvals and concurrency controls. It can be one input to reconstruct model context, while a typed workflow record remains authoritative for resumption.
Must the same model resume the run?
Not always. Record model and prompt versions, then define compatibility policy. A changed model may require re-evaluation or approval for consequential work. Resume from verified task state rather than assuming behavior will be identical.
How long should checkpoints be retained?
Set retention from task duration, legal obligations, investigation needs and data minimization. Delete transient model context earlier where possible while retaining compact action and decision evidence. Terminal tasks should enter a separate archival and deletion lifecycle.
Conclusion
Durable agent execution makes interruption an expected state transition instead of an exceptional mystery. A task can pause without holding a worker, resume without trusting stale context and recover without repeating an unconfirmed effect. That reliability comes from explicit state, not from a more capable model.
Design the checkpoint and recovery console before enabling long waits or consequential tools. Then repeatedly terminate the runtime at inconvenient moments. When each restart leads to a valid next state, preserved authority and reconciled outcome, the agent is ready to outlive a request-response cycle.