Dead letter queue replay is a production write operation disguised as queue administration. Moving messages back to a source may repeat charges, emails, inventory reservations, workflow transitions, or downstream publications that succeeded before the original handler failed. Managed redrive makes message movement convenient; it does not prove the consumer is idempotent, preserve every ordering relationship, repair malformed data, or show which side effects already occurred.
AWS documents configurable SQS redrive velocity and notes that redriven messages can interweave with new traffic; Google Pub/Sub describes dead-letter forwarding as best effort and wraps the original message with source-subscription attributes. Those platform details matter, but a safe process starts higher: preserve evidence, classify the failure, choose correction semantics, prove effect safety, replay a bounded cohort, observe outcomes, and reconcile the business state.
Build the six-stage replay control path
Treat the dead-letter destination as quarantine with access controls, retention monitoring, encryption, and an owner. Capture original message body or reference, headers, source topic or queue, partition and offset where applicable, event identity, schema version, first and last failure times, delivery attempts, consumer release, error class, and trace or correlation ID. Do not rely on application logs whose retention may expire before triage.
Preserve immutability. Analysts can attach diagnosis and proposed correction, but changing the quarantined artifact destroys evidence. If replay requires a transformed message, generate a new controlled artifact that references the original, records the transform version and approver, and states whether identity is preserved. Access to payloads should follow data classification; a DLQ is not an unrestricted debugging warehouse.
| Class | Example | Replay readiness | Preferred action |
|---|---|---|---|
| Transient dependency | Database or downstream service unavailable | Ready after dependency and capacity recover | Replay original identity with rate limit |
| Consumer defect | Null handling or release regression | Ready after fixed version is deployed and tested | Canary replay against corrected consumer |
| Schema incompatibility | Producer added unsupported field or enum | Ready after consumer or transform compatibility | Deploy tolerant reader or governed transform |
| Invalid business data | Impossible state or missing required reference | Not automatically replayable | Correct source/business state or reject permanently |
| Authorization/configuration | Expired credential or missing ACL | Ready after scoped control repair | Verify least privilege, then canary |
| Permanent policy rejection | Event must not cause the requested effect | Not replayable as success | Record disposition and reconcile upstream |
Select replay scope by root cause and consequence
Never begin with replay all. Query or export a manifest of candidate message identities and group them by root cause, schema, producer, time range, tenant, partition, and side-effect class. Exclude messages whose business outcome has already been reconciled, whose retention or authorization no longer permits processing, or whose failure differs from the fixed defect. Freeze the manifest so approval and post-run counts refer to the same cohort.
Account for dependencies among messages. A later event may require an earlier one; a cancellation may make a failed creation obsolete; replaying only one partition can violate cross-entity assumptions. When ordering is essential, pause or isolate live traffic, replay by key and sequence, or rebuild state from an authoritative source. Do not claim a global order from a broker that guarantees only partition or ordering-key order.
Choose repair, transform, skip, or compensate
A transient failure usually needs unchanged replay. A consumer bug needs a deployed fix. An old schema may need a tolerant reader or deterministic upcaster. Invalid business data may require upstream correction or a permanent rejection. A side effect that partially completed may need compensation before retry. Record the choice per cohort and test it against copies. Editing messages manually during redrive creates unaudited new facts.
Preserve original event identity when retrying the same occurrence so consumer deduplication can recognize completed work. If payload semantics change materially, create a corrective event with new identity and explicit causation instead of reusing an ID for different data. Keep original occurrence time distinct from replay time; otherwise downstream windows, audits, and latency metrics interpret historical events as new business activity.
| Gate | Evidence | Stop condition | Owner |
|---|---|---|---|
| Root cause removed | Fixed release, dependency health or config test | Failure reproduces in staging or canary | Service owner |
| Identity preserved | Manifest maps original and replay identities | Transform assigns accidental fresh IDs | Event owner |
| Effects idempotent | Duplicate and crash-injection test results | Any consequential effect repeats | Application owner |
| Capacity bounded | Rate, concurrency and downstream headroom plan | Lag, errors or saturation exceed threshold | Platform owner |
| Ordering addressed | Key and sequence policy for cohort | Preconditions or sequence checks fail | Domain owner |
| Reconciliation ready | Expected counts, ledger queries and rollback/compensation | Outcomes cannot be independently measured | Operations owner |
Prove side-effect idempotency before redrive
Inspect each effect, not just the handler's final processed flag. Database mutation, payment call, email, webhook, search update, and emitted event need stable operation identity. Use an inbox record atomically with local effects, idempotency keys for remote APIs, outbox records for downstream publication, and unique business constraints where duplicates are unacceptable. A crash-injection test should stop after each effect and replay the same message.
If proof is impossible, build reconciliation and compensation before replay. For example, query the payment processor by merchant operation ID, compare ledger entries, and suppress a second charge; check notification delivery records before resending. Choosing to accept possible duplicates is a business risk decision with named approval, not an implicit consequence of a console button.
Canary at controlled velocity and preserve stop control
Replay a tiny representative cohort first. Include common cases and the riskiest edge classes, not only messages expected to pass. Set a custom maximum velocity below spare downstream capacity, account for live traffic, and watch processing latency, retries, dependency errors, and business outcomes. AWS SQS can move at a custom maximum rate, but destination queues ingest redriven and new messages as they arrive, so redrive order does not isolate the workload.
Define abort thresholds before starting and ensure operators can stop the move and consumer processing. A transport-level cancel may not recall messages already delivered. Use a replay campaign identifier in safe metadata or an external manifest so telemetry can separate campaign effects from live events without changing original event identity. Expand cohorts only after reconciliation confirms the canary.
Observe transport and reconcile business outcomes
Track selected, moved, delivered, successfully processed, deduplicated, failed again, permanently rejected, compensated, and unresolved counts. Counts must reconcile by immutable message identity; queue depth alone is insufficient because retention expiry, forwarding wrappers, or new IDs can change it. Measure oldest age and redrive duration, and retain the manifest, configuration, approvals, release versions, and stop events.
Verify domain outcomes independently: one payment per order, expected inventory totals, one workflow transition, correct notification policy, and no unexplained downstream events. Sample successful messages and review every exceptional class. A replay is complete only when transport counts and business records agree or each difference has an owned disposition.
Keep campaign evidence long enough for delayed consequences to appear. Notifications can generate complaints later, settlement systems may close the next day, and downstream analytics may reconcile on a weekly cycle. Define the observation window from the slowest material effect rather than declaring success when queue depth reaches zero. During that window, keep the campaign identifier, release versions, dashboards, and accountable owner available for investigation and compensation.
Use replay findings to reduce future quarantine
Fix the route that created the DLQ population. Add schema compatibility tests, bounded retries with jitter, dependency timeouts, poison-message classification, consumer idempotency, key-aware ordering, and actionable alerting. Review maximum delivery attempts against expected transient recovery; too few sends recoverable work to quarantine, while too many can amplify load or block ordered processing.
Define retention from detection and response time. Google notes dead-letter delivery attempts are approximate and require correct configuration and permissions, so monitoring must verify forwarding itself. Test DLQ permissions, alarms, replay tooling, and evidence capture before an incident. A dead-letter path that nobody owns is delayed data loss, not a recovery system.
Key takeaways
- Preserve original messages and failure context as immutable, access-controlled evidence.
- Classify root cause and business consequence before selecting a replay cohort.
- Preserve identity for the same occurrence; use a new corrective event when semantics change.
- Prove each consequential side effect idempotent or prepare reconciliation and compensation.
- Canary at bounded velocity with predefined abort thresholds and live-traffic headroom.
- Close the campaign only after transport counts and business outcomes reconcile.
Dead letter queue replay FAQ
Can we replay immediately after deploying a fix?
Only after confirming the cohort shares that root cause, the fixed release is active on every processing path, effect idempotency is proven, capacity is available, and a canary plus reconciliation plan is ready.
Does redrive preserve original ordering?
Do not assume it. Service behavior varies, redriven messages can interleave with live traffic, and dead-letter forwarding may weaken order. Design replay around the actual key and broker guarantees.
Should a replay get a new event ID?
Not for another attempt of the same occurrence. Preserve identity for deduplication. If an operator changes the event's business meaning, issue a new corrective event linked to the original.
Conclusion
Safe dead-letter replay combines forensic care with release discipline. Immutable evidence, root-cause cohorts, explicit correction, idempotent effects, bounded canaries, stop controls, and business reconciliation turn redrive into a controlled recovery. Without those controls, an easy message move can amplify the original defect and repeat its most expensive consequences.