The Big Interview: Jinhong Brejnholt, Chief Cloud Architect, Saxo
"We used Istio for our service mesh. Now we're using Linkerd. We are already considering..."
Jinhong Brejnholt, Chief Cloud Architect at Saxo Bank, has a role that straddles avant-garde cloud-native technologies, developer experience, and cybersecurity for the Danish investment bank – which provides some 850,000 clients including investors and traders with regulated access to over 70,000 financial instruments.
Brejnolt, a self-described “hands-on cloud architect/developer with a strong security focus” speaks about her role with unbridled enthusiasm, after getting “hooked” on cloud native technologies in 2016 as a developer.
At the bank she takes a leading role in ensuring new products are delivered securely and scalably – working in recent years to “modernise our tech stack to make everything run in containers, run in Kubernetes.”
The bank is API-led she says. Customers can use its FIX API or build bespoke APIs via OpenAPI to entire capital markets infrastructure, including its instrument and client portfolio data, charts, quotes, order placement, lead sign-up and event notification. It offers a wide range of technology solutions including trade event notification systems, client services management, end-of-day reporting and business intelligence and more.
Many of these have been ‘Kubernetified’ and that process is a constant evolution.
“Components swap out all of the time”
“So many building blocks to go into Kubernetes” she muses. “That’s my biggest passion: How do we run a super solid and secure Kubernetes platform which can take the load when it counts. Because when the market opens, you can see our workloads start spinning up as it caters for all of the newly logged-on clients as they start trading. We also have to be able to scale down seamlessly, as we need to be cost-effective…”
There’s a limited pool for really experienced Kubernetes talent out there – and also a very rapidly evolving technology ecosystem. What kind of challenges has she faced in building this resilient and very scalable infrastructure? One of them is simply the pace of change, she says: “Things happen so fast in the Kubernetes world; components are swapping out all the time, so you have to keep an eye on the trends and tools and not just stick with the same components or you can get outdated. I can give you an example on how we implement service mesh in Kubernetes.”
“Just three years ago we used Istio for our service mesh. On our new platform right now we're using Linkerd. And we are already considering a third service mesh candidate. We need to stay up-to-date; understand which is most lightweight, future-proof and fits our needs best. There are so many moving parts. So one of my roles is helping the platform team stay on top of all the components in the cluster…”
Jinhong Brejnholt: “It’s interest that drives people…”
When it comes to platform engineers there is a particular shortage of enterprise-ready people in the market, she says: “Many people can spin up a ‘Hello World’ Kubernetes platform. But if you want enterprise-grade requirements, it’s quite difficult to find the right candidates. I think that’s an international issue right now…”
Brejnolt herself did not start coding meaningfully until her 20s, she says, focussing more on mathematics and other subjects. Her first degree had more of a hardware focus. Partly as a result of this and partly due to the vagaries of the market when she hires, she says, she looks for enthusiasm and affinity for a subject rather than years of experience: “When you see someone with great interest in something, great passion and a real desire to learn, that carries you very far and very fast. I was super late to programming but I picked it up fast. When you look at containers, at Kubernetes, it really is that interest that drives people. That’s what I look for.”
Operating at the sharp end of the spear in a highly sensitive and heavily regulated industry means a tight focus on security and again, that means thinking differently about how developers, architects, product owners, cybersecurity teams work together, she suggests; not least because if some of the ingredients of modern architecture are novel for those building them, there is an education piece here for security teams and CISOs too. This needs to be balanced with ensuring the developer experience and product focus isn’t ruined by endless security-driven interventions that introduce friction. It’s a fine balancing act to try and pull off.
There are two elements she likes to focus on here, she suggests; one being a “shift-left” type approach that aims to bake security in as early in the build process as possible and then sustain that through real-time visibility of every process in production (she notes that Saxo Bank uses end-to-end cloud security specialist Sysdig’s platform here for runtime visibility, Kubernete in part the collective responsibility of all stakeholders who play a role in the software delivery lifecycle) and another being regular testing of disaster recovery processes.
See also: Ransomware attack cost Expeditors $60m in remediation, lost business
On the former, she muses, this becomes an issue around the “maturity of developers”.
“[With cloud-native] you need to learn not just how to package your code, but configure all infrastructure requirements together to deploy into our environment – which is super secure, meaning you cannot just ‘exec’ into the pod and start investigating [issues]. Our production environments need to be locked down way better than that! But developers are very used to thinking ‘I have a server; I have an issue; I’m going to login and check the logs.’
“[Taking a different approach to this] is a lot to take in for developers! Because they have a huge cognitive workload. They have to learn new ways of deploying and running things on top of existing jobs, and they need to be secure; it’s a lot!”
A mindset shift on cybersecurity
“We also use so much open source software in our tech stack” she adds.
“When we build Docker images, a lot of base images are also open source. All those dependencies make up a very complex environment to deal with. Many security architects are very familiar with, say, how to protect Windows Server, or how to protect a data centre. But now, it's totally different. And it's no longer IP-bounded, if you have a Kubernetes cluster, your [IP] range is huge. Embedding a security mindset here is critical.
“Those [are among the] kinds of things we spend most time on -- understanding how we can [be secure] with the least pain for developers, and staying fully compliant: To understand 'OK, I shouldn't use this package anymore because I don't want a critical vulnerability in my dependency chain..."
An internal framework team tries to take out some of the heavy lifting here: “[This team] builds libraries for developers that cover most cases, but not all. We also have reference architecture showcase applications
On resilience, testing disaster recovery processes regularly is critical, she adds.
“A lot of companies take a lot of backups. Some of them never delete their backups!” she says, but “very few test their backups. I have been to so many roundtable discussions where when you ask people how many of them have tested their backups, most of them have just once or twice on the first setup of their system. It’s rare that people do, so for our critical infrastructure we do regular backup testing for full disaster recovery.”
Jinhong will be joining The Stack's founder Ed Targett and Sysdig CEO Suresh Vasudevan for a conversation on scaling and securing cloud native applications live on June 6, 16:00 GMT+1. Join us.
