Transactional outbox CDC solves one specific consistency gap: an application must update its database and publish an event, but a normal database transaction cannot atomically commit to an independent broker. Writing business state first can lose the event if publication fails; publishing first can expose an event for a transaction that later rolls back. The outbox pattern writes business state and an event record in one local transaction, then change data capture relays the committed record asynchronously.
Debezium's Outbox Event Router expects insert-oriented outbox records and can map event ID, aggregate key, type, payload, timestamp, and additional fields into broker messages. That convenience does not remove architecture decisions about transaction boundaries, identity, ordering, cleanup, connector ownership, replay, and consumer idempotency. The relay can deliver a record more than once around failures, so the business outcome still needs deduplication.
Build the six-stage database-to-broker path
Start with the state transition that deserves an event. Name the aggregate or business entity, invariant, transaction, event meaning, intended consumers, and acceptable publication delay. Insert the outbox row in the same transaction and database as the authoritative mutation. A separate database call, asynchronous ORM hook, or best-effort post-commit callback recreates the dual-write gap. Test rollback to prove no event row survives a failed business transaction.
Create the event payload from transaction-consistent state. Decide whether it carries full state, changed fields, or a notification. Avoid a relay that rereads current tables later; the entity may have changed, producing a payload that does not describe the committed occurrence. Persist enough event content or a versioned reference to reconstruct the intended fact without racing subsequent updates.
| Field | Purpose | Design rule | Failure prevented |
|---|---|---|---|
| Event ID | Stable delivery identity | Generate once in the transaction and preserve through relay | Consumer cannot deduplicate redelivery |
| Aggregate ID | Ordering or routing key | Use the business boundary requiring order | Related events land on unrelated partitions |
| Aggregate or event type | Topic routing and semantic class | Use governed stable values | Infrastructure routes on payload parsing |
| Payload | Consumer-facing event data | Serialize with pinned schema and occurrence semantics | Relay reread reports later state |
| Occurred timestamp | Business event time | Generate from a defined clock and meaning | Broker time is mistaken for occurrence time |
| Schema/version metadata | Payload interpretation | Reference immutable or governed schema identity | Historical replay uses today's schema |
| Trace/correlation context | Causal observability | Persist only approved portable context | Post-commit worker loses causal link |
Configure CDC and routing with narrow ownership
Capture only intended outbox tables with the connector and apply the transformation selectively. Debezium recommends an outbox connector capture outbox records only, and tables captured together must share the expected structure. Pin connector and transformation versions, store configuration as code, and make topic routing explicit. A permissive regex or broad database capture can publish internal tables or expose sensitive payloads.
Separate ownership: application teams own event meaning and transactional insertion; the platform team owns connector availability, offsets, credentials, routing guardrails, and broker policy. Both own the interface schema. Define what happens when an unknown aggregate type, malformed payload, update to an insert-only outbox row, or missing key appears. Failing closed may stop all publication; routing bad records aside may violate ordering. Choose and test the policy.
Preserve event identity and the required ordering scope
Use the outbox event ID as stable message identity and the aggregate ID as the broker key when events for one aggregate must be ordered. Ordering is typically guaranteed only within a partition, and consumers can still retry. Do not use event type as a key if the invariant is per order or account; that creates hot partitions and does not colocate one aggregate predictably. Conversely, using tenant ID may serialize an entire large tenant unnecessarily.
Database commit order, change-log order, broker append order, and consumer effect order need explicit analysis. Multi-table or multi-transaction events may not form one global business order. Add aggregate sequence or version when consumers need to detect gaps and stale transitions. Consumers should reject or defer an event whose precondition is absent rather than assuming broker position proves domain validity.
| Failure | Durable evidence | Expected recovery | Required protection |
|---|---|---|---|
| Application transaction rolls back | Neither business change nor outbox row commits | No publication | Same-database atomic transaction |
| Connector stops before reading row | Committed outbox row remains in log/table | Resume from connector offset | Log retention and monitored lag |
| Broker acknowledgment is lost | Connector may retry record | Duplicate delivery possible | Stable event ID and consumer deduplication |
| Consumer commits effect before acknowledgment | Broker redelivers | Replay same consumer operation | Atomic inbox/effect or idempotent downstream call |
| Bad outbox payload blocks relay | Row and error evidence remain | Quarantine or repair under governed procedure | Schema validation before commit and runbook |
| Cleanup races replay | Source row may be gone while log remains or not | Use proven retention watermark | Separate publication evidence from table deletion |
Design consumers for repeated delivery
A consumer can store an inbox record keyed by source and event ID in the same local transaction as its effect. If the insert conflicts, it recognizes prior processing. This works for local database effects; remote calls need their own idempotency identity or a downstream outbox. A processed marker written before the effect can lose work, while one written after can repeat work. Atomicity or independently idempotent effects close that gap.
Define duplicate handling, schema errors, out-of-order events, missing predecessors, and poison messages. Acknowledgment should follow durable acceptance according to the client and broker contract. Keep original event identity through dead-letter and replay paths. If an operator edits payload to correct it, preserve provenance and create a governed corrective event identity rather than disguising changed data as the original occurrence.
Operate relay lag as user-visible consistency
Outbox publication is asynchronous, so connector lag is part of product behavior. Define service objectives from business tolerance: how long after commit may search, notifications, fulfillment, or analytics lag? Monitor database log position, oldest unpublished event age, broker acknowledgment, connector restarts, rejected records, and topic throughput. Record counts alone can look healthy while one old partition or transaction remains stuck.
Capacity-plan database logs, connector tasks, broker partitions, and downstream consumers together. Large payloads increase transaction logs and broker cost; bursty commits can create relay spikes. Backpressure should protect the database without silently discarding events. If publication delay must block the user transaction, the outbox's asynchronous model may not meet the requirement; redesign the workflow rather than polling aggressively.
Clean up outbox data only behind a proven watermark
Outbox tables grow indefinitely unless archived, partitioned, or deleted. Cleanup must not outrun CDC. A row's presence in a connector offset or database log is not automatically proof that all needed downstream systems accepted it. Define the cleanup watermark from connector progress and operational replay requirements, leave a safety margin, and test connector rebuild. Partitioning by insertion time can make removal efficient, but only if transactions and connector behavior support it.
Database log retention is a separate control. A long connector outage can cause required log segments to disappear, forcing resnapshot or recovery. Alert well before that horizon. Keep immutable event archives only when the business needs replay or audit, with access, encryption, schema, and deletion policy. The operational outbox table need not become the permanent event store.
Govern replay and schema evolution
Choose whether replay republishes original bytes, reserializes the original logical event, or emits a new corrective event. Original replay best preserves historical meaning but requires consumers to support retained schemas. Upcasting can ease consumption but must be deterministic and audited. Never rebuild historical events from today's mutable business rows without labeling them as new state snapshots.
For schema change, deploy tolerant consumers before producers, validate the outbox row at write time, and keep schema identifiers immutable. Changes to table columns, router mappings, topic names, or keys are infrastructure migrations as well as payload changes. Rehearse with copied log records and a nonproduction broker; do not discover after cutover that the connector omitted a new field or repartitioned aggregates.
Key takeaways
- Insert business state and immutable event data in one local database transaction.
- Generate stable event identity and a partition key aligned to the true ordering invariant.
- Limit connector capture and make routing, bad-record, and ownership policy explicit.
- Assume relay redelivery and make every consumer effect idempotent or atomic with an inbox.
- Monitor oldest unpublished age and log-retention headroom, not only throughput.
- Clean up behind a proven CDC watermark and define replay semantics before incidents.
Transactional outbox CDC FAQ
Does the outbox guarantee exactly-once processing?
No. It atomically records business state and publication intent. The CDC relay and broker can redeliver, and consumers can fail after effects. Stable identity plus idempotent or atomic consumer handling is still required.
Must the relay use CDC?
No. A polling publisher can claim rows safely and publish them, but must solve concurrency, retry, ordering, and cleanup. CDC avoids application polling and follows the transaction log, while adding connector and log-retention operations.
Can rows be deleted immediately after broker publication?
Only under a proven connector and recovery model. Cleanup should follow a durable progress watermark and safety window, and it should preserve any replay or audit evidence the business requires.
Conclusion
Transactional outbox CDC closes the local database-to-publication gap by making event intent part of the business commit. Reliability then depends on immutable event data, stable identity, correct keys, governed connector behavior, observable lag, safe cleanup, and idempotent consumers. Treating those as one operating system produces dependable events without pretending a cross-system transaction exists.