A Citi popup showed a trader 711 alerts: They clicked it away – and generated $196 billion in trading orders by mistake
90% of controls on one trading desk were manual and key processes including trade pricing were "conducted manually, predominantly, in Excel spreadsheets..."
A Citi trader rather than the bank’s algorithms or risk functions spotted their $196 billion “fat finger” error, said a UK market watchdogs – even as the bank this week played down the incident as having arisen “from an individual error that was identified and corrected within minutes.”
The May 2022 incident saw a trader click through a warning pop-up with no fewer than 711 alerts on it – simply overriding 646 “soft blocks.” They subsequently placed orders worth $196 billion for execution in the bank’s CitiSmart electronic trading platform before realising their mistake.
The trader was trying to make just $58 million-worth of trades. But despite their efforts to undo the “fat finger” error, over $1.4 billion of sell orders were executed across various European exchanges by mistake.
That was the second colossal “fat finger” error at Citi in 24 months after an August 2022 incident in which the bank transferred $894 million of the bank’s own funds (instead of a planned $7.8 million interest payment) to several lenders; sending them the principal and the interest for a loan.
(It blamed a “clerical error” and “out-of-date software” for that incident. The Stack looked in more detail at how its Oracle Flexcube deployment and inadequate internal training allowed that to happen here.)
Citi fined: Says it's investing more
Citi has now been fined a modest £33.8 million by the Prudential Regulation Authority over the incident – one of several between 2018 and 2022 that UK regulators say represent a failure of its risk management.
The bank has since highlighted investment in risk technology and controls both at a 2022 investor day and, more recently, said it was intensifying “regulatory remediation” efforts during its April 2024 Q1 earnings.
(A spokesperson told The Stack that in the wake of the 2020 incident “We immediately took steps to strengthen our systems and controls, and remain committed to ensuring full regulatory compliance.”)
It’s just the latest PRA report to give a window into how poorly over the years many banks have resourced risk, with a 2022 “final notice” on Metro Bank revealing that in 2017 its Regulatory Reporting Team (RRT) had “only one permanent member of staff (who was relatively junior)” and that that its analysis of Risk Weighted Assets (RWA) was “largely manual… [and had] created key-person dependencies on a small number of individuals familiar with spreadsheets that were not scalable…” (Metro Bank says it has now overhauled and resourced these functions properly.)
The PRA’s Citi investigation is more recent but, like the Metro Bank investigation, shows that Excel spreadsheets still power critical functions even in a global multinational bank like Citi. The watchdog’s report, published on May 17, reveals how in November 2020 Citi’s Internal Audit (IA) team investigated its own Delta One trading unit, which sells financial instruments to pension funds, hedge funds and corporate clients.
The audit found that its sales and trading desk “relied heavily on manual processes and workarounds” for “key processes including trade pricing, booking, and rebalances [that were] conducted manually, predominantly, in Excel spreadsheets…” and testing 31 controls (an astonishing 90% of which were manual) said that “only 61% were assessed as effective.”
“The Firm’s risk management functions failed to provide effective real-time monitoring of those trades which generated suspensions and information alerts” the PRA said on May 17. A team “monitoring for Firm trades to be executed on external venues, was not resourced on 2 May 2022 due to scheduled leave and therefore arranged a scheduled handover to another team to provide cover,” the PRA investigation found.
“The team providing cover failed to react to 284 real-time information alerts generated by CitiSmart, relating to 284 orders each in excess of the maximum notional value of US$25mm. Consequently, neither of these two risk functions alerted the trader to their error on 2 May 2022. It was in fact the trader who discovered the error and cancelled the order, approximately 15 minutes after they had entered the order,” said the PRA.
See also: Watchdog's reports, fines reveal reliance of banks on spreadsheets
IT system fragmentation was also flagged by the PRA as a challenge for Citi: “Running multiple systems and processes could create an operational burden, resulting in control gaps,” it wrote, adding that this was “more egregious because, during the Relevant Period, the PRA repeatedly gave supervisory feedback to the Firm regarding the poor state of its trading controls, and the Firm’s internal risk and compliance functions also repeatedly identified weaknesses in those controls.”
By early 2022 when the PRA reviewed a “self-assessment” from Citi it noted that this “made no reference” to items the regulator had flagged earlier, including, for example, “measuring and monitoring the extent of preventive trading controls relative to manual controls, on a front to back basis, and prioritising those in need of automation by reference to internal history of real operational risk events and near misses.”
The bank subsequently added “enhancements into its planned programme of work to address the feedback provided” the PRA said.
Want to talk to The Stack about the challenges of modernising trading controls and/or GRC software more broadly? Have an axe to grind or a success story? Get in touch for a friendly chat, either on- or off-the-record.
