The Outbox Pattern's Hard Parts Start After the Commit

The transactional outbox pattern has the best sales pitch in distributed systems: write your business row and your event in the same database transaction, poll the table, publish to Kafka, mark it sent. One commit, so the published fact can never disagree with the row that caused it. It's four sentences, it's obviously correct, and it's the standard answer to the dual-write problem — microservices.io canonized it years ago.

The pitch is also where almost every writeup stops. A recent incident report from developer Qihu Zhang, who ran a polling outbox between two real services on PostgreSQL and Kafka, documents what happens after sentence four: a poison message that silently jammed the relay, distributed traces that evaporated at the queue boundary, and duplicate deliveries that double-counted inventory. None of these are exotic. All three are structural consequences of the pattern that the canonical description simply doesn't mention — and two of them will follow you even if you swap the polling loop for proper change-data-capture.

The atomic commit is the easy 10%

Here's the reframe worth internalizing: the outbox pattern isn't a database trick, it's a message broker you're now operating. The INSERT ... same transaction part — the part every tutorial covers — is trivially correct and takes an afternoon. Everything hard lives in the relay, which is a tiny custom middleware with all of a broker's classic policy problems: retry limits, dead-lettering, ordering under failure, and observability.

Zhang's first incident is the cleanest example. To preserve per-partition ordering, the relay stopped on the first row that failed to publish rather than skipping past it. Reasonable — until one row failed permanently (a serialization error, not a flaky broker), and the relay retried it forever while everything behind it queued up. The tests all passed, because the tests only modeled transient failure. The fix was an attempts column, a max-retry threshold, and a terminal FAILED status the relay skips and a human gets paged about.

That's not a bug fix; that's implementing dead-letter semantics from scratch. Head-of-line blocking versus skip-and-quarantine is a genuine design decision every ordered queue system has to make — Kafka consumers, SQS FIFO, and every outbox relay ever written. If your outbox table's schema has id, payload, status, and nothing else, you haven't made that decision yet. Production will make it for you.

Two of these problems aren't even about the outbox

The more interesting lesson hiding in the report is which failures the pattern actually caused. The poison-message jam is a relay problem. The other two incidents are properties of any asynchronous boundary, and the outbox just made them visible.

Trace propagation broke because HTTP carries context in headers automatically, while a row sitting in a table for 200ms does not. The checkout request's trace ended at the insert; the consumer started a fresh, orphaned trace. Nobody notices this class of failure until 2 a.m., because a missing trace fails silently — it doesn't error, it just stops. The fix is mechanical but must be built deliberately: persist the trace ID into the outbox row, forward it as a Kafka header, restore it into the consumer's logging context. OpenTelemetry defines exactly this header-based propagation for messaging, but no CDC tool or framework will thread it through your outbox table for you.

Duplicates, likewise. The outbox gives you at-least-once delivery by design — the relay can crash after publishing but before marking the row sent, and that's before Kafka's own producer retries and consumer rebalances redeliver messages as a matter of routine. Zhang's team treated duplicates as a rare edge case until "order-paid" ran twice and the stock ledger drifted. The correct fix is worth spelling out because half the implementations get it wrong: insert into a dedup table with ON CONFLICT DO NOTHING and run the side effect in the same transaction, only if the insert took. Split those into two transactions and you've rebuilt the exact dual-write window the outbox exists to close — plus a unique constraint on the ledger as a backstop for when the dedup logic itself has a bug.

Does Debezium save you? Partially

The standard objection to a hand-rolled polling relay is "just use CDC." It's half right. Debezium tails the transaction log instead of polling, which genuinely fixes real problems: no query load spikes, sub-second latency, and — as Gunnar Morling has argued — events emitted in exact commit order, which polling under concurrent transactions can't strictly guarantee. Postgres users can go further with pg_logical_emit_message() and skip the outbox table entirely, writing events straight to the WAL.

But look at the scorecard against the three incidents. CDC eliminates the polling loop, so the specific jam Zhang hit disappears — and reappears one hop downstream as a stuck Kafka Connect task or a consumer wedged on a poison record. Trace propagation: still your problem. Consumer idempotency: still your problem, because CDC is at-least-once too. Debezium solves the transport; it doesn't touch the two failure modes that live at the edges. Meanwhile it costs you a Kafka Connect cluster, connector monitoring, and WAL-slot management — real operational weight that Zhang's polling relay, running fine on a 6 GB box, didn't need.

So the honest decision rule: poll if you're doing hundreds of events per second or less and multi-second latency is acceptable, because a relay you fully understand beats a connector you don't. Go CDC when latency or throughput demands it, or when you already operate Kafka Connect. Either way, the checklist is identical and non-optional: retry cap and terminal failure state, alerting on stuck rows, trace context persisted through the table, and transactional dedup on every consumer.

The outbox pattern deserves its reputation — dual writes without it are genuinely worse, and the alternatives (Kafka-first "read yourself" flows, durable-execution frameworks like Temporal) trade these problems for different ones rather than eliminating them. But budget honestly. The four sentences are the interview answer. The attempts column, the trace header, and the dedup transaction are the job.