Implementing the Outbox Pattern to Eliminate Event Loss at Acko
Identified a dual-write reliability gap in production and implemented the outbox pattern to guarantee at-least-once event delivery.
Suraj Nanda
SDE 2 at Acko
From their time as
SDE 2
Acko • 2025
Overview
Suraj identified a reliability problem in Acko's production systems: a dual-write issue where events were going missing whenever Kafka went down or downstream services were unavailable. During those windows, the system had no guarantee that events would be delivered, which created silent data inconsistencies.
The Story
Suraj identified a reliability problem in Acko's production systems: a dual-write issue where events were going missing whenever Kafka went down or downstream services were unavailable. During those windows, the system had no guarantee that events would be delivered, which created silent data inconsistencies.
He proposed and implemented the outbox pattern as the solution. The outbox pattern works by writing events to a local database table atomically with the business transaction, then having a separate process relay those events to Kafka. This decouples the event publish from the availability of Kafka, ensuring no event is lost even during downstream failures.
Suraj owned the implementation end to end. The fix eliminated the event-loss window and established at-least-once delivery guarantees across the affected services.
The work required understanding the failure modes of the existing system, designing the outbox table and relay mechanism, and validating the fix in production without disrupting live traffic.