The Big Interview: Fidelity's DevOps boss Topo Pal on bad shifts left, "love letters", more.

Tapabrata "Topo" Pal, Vice President, Architecture at Fidelity Investments, has been innovating across enterprise architectures since 1996. At Fidelity, which serves 43 million individual investors and has $11.7 trillion in assets under administration, he runs DevOps domain architecture and its open source programme office – but cut his teeth in the industry setting up point of sale systems at Circuit City Stores.

Pal, who prior to joining Fidelity in 2021 spent a decade at Capital One (the last four years as a distinguished engineer) is credited with restructuring Capital One’s software delivery processes/leading its CI/CD/DevSecOps approach and a developer transformation that saw it become an active contributor to a range of open source projects.

He sat down with The Stack at DevOps World in New Jersey to talk about his career, “love letters”, his issues with “shift left” and more.

His approach to secure software development started at Capital One: “I first joined Capital One as an enterprise architect, and then I moved over to overseeing software engineering principles; DevOps and CI/CD. Since Capital One was a bank, all these engineering principles were good only if they met certain regulatory or compliance requirements.

"That included security, which is why I started to form the notion of bringing it into the DevOps process, and that was the notion of DevSecOps taking form for me” he said (referring, to those not in this space, for the need to bake security early into software delivery.)

See also: "We have an endemic problem" OpenSSF director warns over secure development

DevSecOps has been long associated with a "shift left" or moving security to the earliest point in the development process)

But despite being an early advocate of bringing security early into development processes (having written compellingly about Capital One’s “16 Gates”, or guiding design principles) Pal is not convinced about the state of the shift left in today's tech landscape.

"When we as an industry conceptualised the shift left, this [the current day modus operandi] is not what we wanted it to be perceived as,” he says: "The whole idea is that certain processes that people needed to meet before going to the right (production) are moved to the beginning of the line. It didn't mean that developers needed to learn all those rules and skills and techniques; it was just about shifting processes."

"Somehow the shift left has now come to mean that the developer on the starting stage needs to learn all these security and compliance rules. The original intent of moving the process would mean that we bring the people who are the owners of the practices, the subject matter experts right at the beginning of the pipeline with the developers and break down that wall..." he says; a view that may resonate with developers now expected to deliver great applications at a rapid pace and also be security and compliance framework experts.

See also: MGM ransomware attack puts Okta configurations in the spotlight

Pal's gripe with the pressures on developers facing the burden of a shift-left isn't merely anecdotal: DevOps communities have been vocal about the excess workload, unsupportive cultures and uneven testing environments that plague most shift left scenarios within enterprise.

(With the average software engineer turnover rate already high and the room for error on security lower than ever, getting this right is critical.)

"Just like DevOps didn't mean that developers had to be Ops or vice versa, DevSecOps shouldn't mean that developers have to be security experts” says Pal, adding that “they won't be, they can't be – and in fact it is a very bad idea to even have this expectation."

Pal also links this issue to a broader problem within tech culture.

"We technologists are quite bad when it comes to one thing: We keep everything within ourselves and then complain about everything that is out of our own domain. If you build a house the way a civil engineer wants it, without consulting anyone else, the house will fall apart.

“Similarly when it comes to software delivery, it is not just engineering; there are other variables at play and they are much much bigger. You are creating what customers need and what will make your company flourish in the future. But we technologists [often] forget about the customers – and think that the technology is the key thing."

"We talk all the time about automation, shifting left and all these specific technologies. The thing is that the people on the other side, customers and auditors; they also don't care about what we are doing."

It’s time for a love letter

One of his propositions:  A love letter from DevOps to auditors.

"The love letter is basically a realisation and a reaching out to audit compliance and other regulatory bodies, and saying that, hey we are sorry we did not involve you from the beginning, but we want to talk to you and show you what we have – and you, in turn can teach us what you know so that we can make our technology better for the human."

Despite being a true blue technologist, Pal admits that fundamentally, his job comes down to people management: “The biggest challenge to DevOps success is not tools, not technologies, not engineers- it is about the culture. What we are aiming for is the right culture of collaboration –  because the origination of DevOps came from the fact that Dev and Ops are separate and they don't trust each other and they always fight.

“The only way to reduce that fight is to have a culture change and cultivate collaboration” – but when it comes to integrating security, Pal grimaces at the DevSecOps term, admitting he doesn’t like it.

"Why? Because the whole notion of DevOps creation was about how I deliver high quality software faster  and that whole premise means including everyone in there. It's not just about the Devs and the Ops teams; it is about everyone who has a role to play. That not only includes security, but non-engineering people; risk, audit, compliance.

“We do not need to create new terms to say the same thing over and over again. New terms don't help, they only confuse people."

At Fidelity he started tackling bottlenecks in DevOps by conducting a listening tour to understand pain points across the organisation.

These included the fact that onboarding was onerous due to too many processes. In fact, “too much” of nearly everything was a critical issue: Too much documentation, too many tools, technologies, and platforms, leading to too many abstraction layers and frameworks…

That resulted in the establishment of a DevOps Council, which broke into small work groups aimed at tackling some of these key problems. The final goal, he earlier explained, was a unified developer experience to include tools standardisation, contextual metrics and continuous compliance.

It’s been making great progress (helping drive a great surge in commits to open source projects supporting teams across the 46,000-employee business, for example) but there’s always room for improvement across the industry. And at the DevOps World event he is acerbic about a lack of clarity that still pervades the industry. That includes metrics (e.g. deployment frequency: “What is a deployment? How do you count them? Who are we measuring and when should we measure?”) and the access component of DevSecOps.

He proposes a budget for “production access debt”for example. (“Each persistent production access costs 25 points. Each break glass/temporary read access costs 1 point” etc.), proposing that organisations set a budget and aim to reduce debt before feature deployment to reduce risk.

Both “DevOps” and “DevSecOps” may be terms that have been done to death. At the coal face, there's a lot of work to do to improve developer workflows and boost application security across the industry. And for Pal, yet another portmanteau is clearly not the answer. Listening, simplicity, collaboration and clearly articulated design principles may just be.

Join peers following The Stack on LinkedIn