API idempotency design answers a precise distributed-systems question: when a caller cannot tell whether an operation completed, how can it repeat the request without repeating the business effect? The answer is not simply to use PUT, add a UUID header, or rely on a broker's delivery promise. Payments, job submissions, and webhook consumers have different identity boundaries, result lifetimes, and side effects. Each needs a durable operation identity tied to atomic state transitions and observable replay behavior.
Google's request identification guidance says that providing a request ID must guarantee idempotency and recommends returning the response for a previously successful request. That guarantee requires server-side storage and concurrency control. A key only becomes meaningful after the service defines its scope, validates that retries represent the same intent, records a terminal or in-progress result, and prevents two workers from committing the effect concurrently.
Build the six-stage idempotent effect path
Define the business operation before choosing a key. A payment capture is not the same operation as creating a payment intent; a job submission is distinct from rerunning a failed job; a webhook delivery attempt is distinct from the domain event it carries. Write the effect invariant in one sentence, such as one captured charge per merchant order and capture purpose. The invariant identifies which database writes, external calls, notifications, and ledger entries must not repeat.
Scope keys to a caller or tenant and operation family. The same random key from two tenants must not collide, and the same key used for refund and capture must not alias. Store a canonical request fingerprint so a retry with changed amount, destination, or parameters is rejected instead of receiving an unrelated cached success. Avoid fingerprints built from unstable JSON text; canonicalize semantically relevant fields and exclude transport metadata that legitimately changes.
| Workload | Operation identity | Stored result | Retention driver | Special risk |
|---|---|---|---|---|
| Payment command | Tenant plus key plus operation; often linked to order | Outcome, resource ID and safe response | Maximum client retry and reconciliation period | External processor call and ledger write diverge |
| Job submission | Tenant plus key plus job definition fingerprint | Job ID and accepted state | Submission retry horizon and job lookup policy | Same job may be intentionally rerun later |
| Webhook consumption | Provider or source plus event ID | Processed marker and outcome class | Provider redelivery and replay horizon | One event triggers several side effects |
| Resource creation | Parent plus request ID | Created resource name and response | Documented request-ID validity window | Resource changes after original success |
| Batch command | Caller plus batch key and item identities | Batch status and per-item results | Longest item retry and support window | Partial completion obscures replay behavior |
Persist an atomic idempotency state machine
Use a record with a unique scoped key, request fingerprint, state, lease or owner, timestamps, response reference, and failure classification. Insert or claim it atomically before starting the effect. A first request moves from absent to in progress; a concurrent duplicate observes that state and either waits, polls, or receives a documented response. Success stores the durable result. A retryable internal failure releases or expires the claim carefully; a permanent business rejection can be stored and replayed if repeating evaluation would be misleading or expensive.
The idempotency record and local business mutation should share a database transaction when possible. If the effect crosses systems, a local record alone cannot atomically commit a remote charge. Use the remote provider's idempotency facility with a stable derived key, or model a durable workflow that reconciles uncertain outcomes before retrying. Never mark the local operation failed merely because the network response was lost; query by the external operation identity first.
Resolve concurrent duplicates deterministically
Two identical requests can arrive before either sees a stored response. Enforce uniqueness in durable storage rather than checking then inserting in application code. Decide what the loser receives: a short wait for the winner, 202 Accepted with a status resource, or a conflict/in-progress problem. Bound waits so a stalled worker does not consume every connection. If leases are used, fencing tokens or compare-and-set transitions must prevent an expired owner from committing after a replacement takes over.
Concurrency tests should force interleavings around claim, validation, external call, local commit, response storage, and timeout. Kill workers after each boundary and retry from another process. Verify one business effect, one authoritative result, and explainable state. Ordinary sequential duplicate tests miss the race that idempotency is meant to control.
Define response replay and payload mismatch behavior
Return the original successful result when feasible, including the resource identifier clients need. The current resource may have changed since creation, so decide whether replay returns the historical response or current representation and document the choice. Preserve status and essential headers safely, but avoid caching volatile credentials or personalized fields longer than required. For asynchronous jobs, replaying the accepted response with the same job ID is usually clearer than synthesizing the latest job state.
A repeated key with a different fingerprint is a client error, not a request to overwrite the prior operation. Return a stable problem that states the key was reused with different parameters without disclosing the original sensitive payload. This protects callers from accidental key caching and prevents an attacker from probing details by guessing keys. Key entropy, tenant scoping, authorization, and rate limits still matter; idempotency storage is not an access-control mechanism.
| Failure point | Observed state | Unsafe reaction | Safer recovery |
|---|---|---|---|
| Before claim commit | No durable operation | Assume effect occurred | Retry claim using the same key |
| After claim, before effect | In-progress record | Create a second operation | Recover or expire with fenced ownership |
| After remote effect, before local commit | External outcome uncertain | Call remote system again with a new identity | Query or retry using the same remote key |
| After local commit, before response | Business success exists | Return failure and let client create anew | Replay stored success for the key |
| During multi-side-effect webhook | Some effects recorded | Restart handler from the top blindly | Checkpoint or make each effect independently idempotent |
| After retention expiry | No key record remains | Promise duplicate prevention indefinitely | Document expiry and reconcile with business identifiers |
Choose retention from retry and business horizons
Retention must exceed realistic ambiguity, not merely the HTTP client timeout. Consider mobile offline retries, queue backlog, provider webhook schedules, support-assisted replay, disaster recovery, and legal or financial reconciliation. Publish the honored window when callers depend on it. Expiry should remove or archive both key and fingerprint consistently. If a late retry could create severe harm, use a durable business uniqueness constraint such as merchant plus order number in addition to the temporary idempotency record.
Storage cost scales with request volume, response size, and retention. Store references or compact outcomes instead of full payloads where possible, partition by expiry, and monitor insert conflicts and lookup latency. Encryption and access controls apply because fingerprints and responses may reveal customer activity. Do not silently shorten retention under load; that converts an operating problem into duplicate effects.
Make webhook consumers idempotent at every side-effect boundary
Use the provider's stable event ID when its contract guarantees uniqueness and replay identity. Scope it by provider account or source. Verify authenticity before recording success, then persist receipt and processing state. If one webhook updates a database, sends email, and calls another service, a single processed flag set at the end is insufficient: a crash after email can send it twice. Use an outbox, workflow checkpoints, or idempotent downstream commands so every effect has recoverable identity.
Acknowledge only according to the provider's delivery contract and your durable state. Distinguish poison data from transient dependencies. Operators need a safe replay tool that preserves the original event ID and records who initiated replay. Rewrapping a dead-lettered event with a new identity defeats consumer deduplication unless the domain explicitly intends a new operation.
Measure the guarantee in production
Track first-seen requests, matched retries, payload mismatches, in-progress collisions, stale leases, result replays, expired-key retries, and reconciliation discrepancies. Correlate without placing raw idempotency keys in broadly searchable logs; hash or tokenize where needed. A falling duplicate count is not automatically good because a broken client may have stopped retrying. Pair service metrics with business invariants such as duplicate charges, repeated notifications, and duplicate jobs.
Operational runbooks should explain how to inspect an operation safely, determine whether an external effect occurred, repair state, and replay without changing identity. Support staff should not be told to try again with a fresh key when the first outcome is uncertain. That advice discards the one mechanism able to reconcile the ambiguity.
Key takeaways
- Define the exact business effect and operation identity before accepting a key.
- Scope keys by tenant and operation, and reject changed payloads with the same key.
- Claim keys atomically and protect lease takeover with fencing or compare-and-set transitions.
- Carry the same identity into remote systems or reconcile uncertain remote outcomes.
- Set retention from actual retry and reconciliation horizons, backed by business uniqueness where harm is high.
- Make each webhook side effect recoverable and independently idempotent.
API idempotency design FAQ
Are only POST requests relevant to idempotency keys?
No. HTTP methods have defined semantics, but any operation whose effect can be repeated across retries deserves analysis. Keys are especially useful for non-idempotent creation or action methods, while consumers and workflows also need durable effect identity beyond HTTP.
Does an idempotency key provide exactly-once delivery?
No. Messages or requests may still arrive many times. The design aims for one accepted business effect for one operation identity. That requires atomic state, deduplication, and idempotent downstream boundaries, not a claim that transport delivered once.
What happens after the deduplication window expires?
The service may treat the key as new unless a longer-lived business uniqueness rule prevents it. Document the window and use durable domain identifiers for consequences that must never repeat across the life of the business record.
Conclusion
Reliable idempotency is an operation ledger, not a decorative request header. Scope identity, fingerprint intent, claim atomically, replay results, reconcile remote uncertainty, retain records for the real retry horizon, and measure business duplicates. Applied this way, the same principles protect payment commands, asynchronous jobs, and webhook side effects while preserving each workload's distinct semantics.