Dhrishti: Scaling GPU-Backed Camera Processing

Dhrishti processed camera streams through custom models running on our GPU clusters. As the service moved beyond its internal deployment, the architecture needed to distribute work across tenants and recover when a processing node failed.

I made the core architectural decision to separate tenants and plan spare capacity by configuration. A typical customer might have dozens of cameras spread across two nodes. We kept at least one additional node ready in a passive state, then adjusted backup capacity according to workload volume and observed failure rates.

Reliability had a direct infrastructure cost, so we exposed that trade-off in the product. Standard plans used a configuration we selected, while enterprise customers could choose additional backup capacity through a custom plan.

The system ran lifecycle checks against active nodes. After five consecutive failed checks, it moved affected camera processing to the backup node in stages. A workload of roughly 30 cameras could be serving from the new node within about five minutes, depending on how many cameras the failed node handled.

Failed work remained in queues rather than being discarded. A separate failure queue retained unsuccessful processing for later repair, which meant the recovery path did not lose data.