Managing a Live System Outage on Night Shift
Acted as the sole risk representative during an ISV outage on night shift, clearing order books to prevent unintended fills in the Brazilian market.
Rishal Naik
Analyst - Risk Monitoring and Control at Futures First
From their time as
Analyst - Risk Monitoring and Control
Futures First • 2024
Overview
Rishal was the only member of the central risk team on shift during a night rotation when the firm's ISV platform went down. A major international market was live at the time, and with the system outage, the firm's order book was no longer visible through the platform. The risk was immediate: orders sitting in the exchange could get filled without the relevant market analysts knowing their position or the price at which they were being executed.
The Story
Rishal was the only member of the central risk team on shift during a night rotation when the firm's ISV platform went down. A major international market was live at the time, and with the system outage, the firm's order book was no longer visible through the platform. The risk was immediate: orders sitting in the exchange could get filled without the relevant market analysts knowing their position or the price at which they were being executed.
He had to act without escalation. No senior team member was available. He pulled the full order book for the entire firm, reconciled it against exchange records, and communicated the position status to the relevant branch managers so they had an accurate picture of their exposure.
Where the risk of unintended fills was high, he made the call to clear the order book for the affected products, preventing ghost orders from triggering at the exchange. This ensured that market analysts knew exactly what positions they held and at what prices, rather than discovering unexpected fills after the fact.
He documented the discrepancies identified during the outage and flagged them for correction in the next reconciliation cycle.