A $1bn fat finger helps drive Citi's $13.6bn IT spending plans

Citigroup plans to spend $54 billion in 2023 on an ongoing transformation as it overhauls data, risk, and technology processes and infrastructure – after mistakenly sending $900 million to clients in August 2020.

“Approximately 25% of the investments in those [transformation] programs are related to technology” said CFO Mark Mason on a January 13 earnings call. The figure is up 6% on Citigroup spending of $51.3 billion in 2022.

The figure means Citi technology spending will hit around $13.6 billion in 2023.

(Citi's Technology and Operations lead Stuart Riley explained in March 2022 that his $11 billion IT budget that year was "split roughly equal between change the bank and run the bank activities". The bank had migrated around 25% of applications to a containerized architecture, served around eight billion API calls across 300 APIs and was employing some 30,000 software engineers, he said at an investor day event in early 2022.)

A sweeping Citigroup transformation now has has a small army of 11,000 people working on it, according to the firm’s Q4 investor presentation. (“Ironically, it's initially quite a manual process to build an agile data architecture” as Citigroup’s Chief Administrative Officer Karen Peetz said drily at a 2022 Investor Day.)

“As our investment in transformation and control initiatives mature, we expect to realize efficiency as those programs transition from manually intensive processes, to technology-enabled ones… and, ultimately, make our company more efficient” said Citigroup Chief Financial Officer (CFO) Mark Mason on the earnings call.

Citigroup technology spend in 2023 follows fines, howling error

In October 2020, regulators hit Citigroup with multiple consent orders, including a $400 million fine for a “long-standing” issue of inadequate internal controls related to compliance, data, and risk management.

Some of those deficiencies dated back to 2013 and 2015, but Citigroup’s error in August 2020 cannot have helped. That month the bank transferred $894 million of the bank’s own funds (instead of a planned $7.8 million interest payment) to several lenders; erroneously sending them the principal and the interest for a loan.

Credit Suisse aims tackle 100,000 servers, 90,000 workstations, 37,000 relational databases, 3,000 apps and 13 data centres

Citigroup blamed a “clerical error” and “out-of-date software” for the incident, which triggered a lawsuit after several of the recipients declined to return the funds and a district judge initially backed them, saying that “to believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake that had never happened before, to the tune of nearly $1 billion - would have been borderline irrational.”

(All recipients ultimately returned the funds after a two-year legal battle.)

Citi's $900 million error: Flexcube boxes not checked...

Legal filings as the case proceeded illustrated precisely what had happened.

Citibank’s Asset-Based Transitional Finance (“ABTF”) team, which was focused on processing and servicing asset-based loans, was tasked with executing the rtransaction on Oracle’s Flexcube software.

As one filing noted: “On Flexcube, the easiest (or perhaps only) way to execute the transaction [and] to pay [the lenders] their share of the principal and interim interest owed as of August 11, 2020… was to enter it in the system as if paying off the loan in its entirety, thereby triggering accrued interest payments to all Lenders, but to direct the principal portion of the payment to a ‘wash account’”. (An internal Citibank account used to account for internal cashless fund entries and . . . to help ensure that money does not leave the bank.)

“Placing the principal into the wash account would allow the loan to then be rebuilt.”

(i.e. In the software it would now reflect the smaller remaining principal balance.)

Citigroup controls: Six-eyes, none of them reading the manual....

As the court documents showed: "When entering a payment, the employee is presented with a menu with several ‘boxes’ that can be ‘checked’ along with an associated field in which an account number can be input”.

Citigroup manuals meanwhile explained that to prevent funds being sent out of the bank “ALL of the below field[s] must be set to the wash account: FRONT [;] FUND [; and] PRINCIPAL.”

The bank had a “six-eyes” sign-off process on such payments. Two sets of eyes were at IT outsourcer Wipro in India. The third was a Citigroup senior manager in the US. None of them realised that although they had clicked the “PRINCIPAL” box (to send the principal to the wash account) they had to click the other two buttons as well.

All now ancient history perhaps, but grist to the mill for the Citigroup transformation.

Citigroup transformation: What's the priority for CAO Karen Peetz?

Citigroup CAO Karen Peetz came out of retirement to help tackle the bank's data and controls challenges. She told investors last year that "redundant data needs to be eliminated. Data sources need to be validated and prioritized. We also need to  update our systems and make sure the teams are appropriately trained on them…

"We're looking at what processes exist today and determining opportunities to do things in a more consistent way. Then, we're looking at each of these steps. Can they be simplified? Do we have enough controls for each step, or too many? Aer  they the right controls? Building the right end-to-end processes is an essential step to move forward from manual to automated ways of doing things. Everyone intuitively understands the benefit of automation, but automating bad or overly-complex processes won't achieve our desired outcomes... once we have more and better end-to-end systems, we'll have better data that we can use more effectively. These efforts, like virtually all of our transformation work, are closely linked and connected. We're also strengthening Citi's project management capabilities to create centrally-led discipline and structure around how we get things done."

Speaking about the Citigroup transformation, CEO Jane Fraser added at the investment day: "This will be a multi-year program, but it's one that we are now firmly down the execution path in.

"[It's] something that I spent a lot of my days involved in -- we put a lot of time into looking really maniacally at the details of what's working, how the investments going, where is it we've got the right technology investments, what are the resources that are needed? It's got attention from the whole management team. Our board is constantly challenging us as to making sure that we are making the progress that should be expected."

Citi on January 13, 2023, reported net income for Q4 of $2.5 billion, on revenues of $18.0 billion.

For the full year 2022, Citigroup reported net income of $14.8 billion on revenues of $75.3 billion, down sharply from net income of $22.0 billion on revenues of $71.9 billion for the full year 2021."