Netflix's problem is everyone's problem now: How to eliminate database trade-offs
“A lot of cloud databases have turned into a compromise vote..."
For growing numbers of enterprises data is at the heart of what they do: simply, it’s where the value is. So thinking about how to handle that data is one of the most consequential decisions any organisation can make: “The problems that were Facebook's problem and Netflix’s problem, and Apple's problem in 2012 are now turning into everyone's problem today,” Patrick McFadin, VP for Developer Relations at Datastax, tells The Stack.
“Any team that's building a new application has the trade-off conversation" he adds: "I've been in that whiteboard discussion we sit around and go 'we would really like the gold version of this, but that's really hard, or it's going to be slow so we're not going to go that direction, we'll take the compromise version'.”
Two of the major compromises are ability to scale and a risk of lock-in, versus ease and speed of deployment. “A lot of cloud databases have turned into a compromise vote, because well, it's a lock in, but it's fast. So we'll use it. And as a result a lot of organisations find themselves stuck – data is a hard thing to move,” says McFadin on the latter, noting this is one of the key reasons behind the rise in popularity of open-source database platforms.
To avoid the issue of lock-in, McFadin emphasises the value of APIs: “Being API based, and this is one of the really important tenets of being cloud native, is using REST, GraphQL, gRPC. Thinking in terms of how do you stay portable that way, always try to API your data calls, turn everything into a data service.
“I've worked as a consultant for years, helping companies migrate databases by first putting an API layer in front of it, and then migrating the underlying data store. It's where data services are.”
Hornet's Nate Mitchell on repairing Cassandra, serverless, and K8s
As for scalability, this has become increasingly critical, in an age when demand can surge in an instant. “Like, Wordle, where did that come from? And now there's a tonne of Wordle clones out there. And how many different versions of Candy Crush are there?” McFadin muses. But factoring in scalability from the start has – traditionally – had inherent costs, particularly in time-to-market. As McFadin notes on our call: “Five years ago, if I wanted to build a really high scaling app, it would take six months, maybe – if I was really quick.”
The go-to database for scalability has long been Cassandra; the open-source project, initiated by Facebook and now under Apache, is used by dozens of major enterprises, including Apple, Netflix, Walmart, ING, Sky and eBay. DB Engines ranks Cassandra as the most-popular wide-column database management system, and the 11th most popular system overall. But, as McFadin says: “Even though it's probably the database you need, you may skip it, because it's too hard.” Traditionally Cassandra has been one platform which could not be deployed as a service on the cloud, with any cloud-based implementations relying on provisioning servers and the complexity which comes with this, or using a managed service and the costs which come with that."
This has now changed, with Cassandra-as-a-Service available for the first time. DataStax’ offering, Astra DB, is the only native-Cassandra DBaaS on the market – and McFadin says the aim of the project was to eliminate a lot of the issues which forced organisations to compromise.
The developer on Astra DB can develop and deploy an application in a single day and scale up when they need it. It's an enormous game-changerPatrick mcfadin
“The developer on Astra DB can develop and deploy an application in a single day. And then, because not only can they deploy it on a single day, but the second day, the third day, and the fourth day, they’re covered, because they know it'll scale up when they need it.
“So that is an enormous game changer. And when you hear people talking about building cloud native applications, that's all they're talking about: I just want to go faster. They’re not talking about, ooh I want to use Kubernetes or API. Nope. They just want to go faster. And cloud’s just how they're getting there,” says McFadin.
“If you shorten the amount of time that a developer requires to be productive – because there's nothing more important than how fast a developer can put something in production – that's the most important thing, right? Everything else is details.”
See also: Troy Hunt’s Azure bill shock is a cautionary tale
For enterprises looking to migrate their database to another platform, or from a self-managed to a cloud-based system, there will always be at least some work – especially for applications built directly on a database.
“The closer you are to the database you're using, the less portable you get. So, if you're using very specific syntax like for Oracle, then if you want to migrate away from Oracle – good luck, you're going to have to rewrite some code,” says McFadin.
But the effort of migration can bring big dividends. McFadin cites the example of esports firm ESL Gaming, which migrated from on-prem Cassandra clusters to Astra DB: “Companies like ESL, they are going to have a distinct advantage over their competitors.”
ESL had to consider moving Cassandra to the cloud as part of a wider migration strategy. However, their implementation relied on functions such as Time To Live and Materialized Views which weren’t supported by any cloud DB providers. This would have required significant rewriting of the firm’s code to migrate. Astra DB is fully compatible with Cassandra and supported these functions which meant that no rewrites were needed. ESL completed the migration in October 2021. This has resulted in a significant reduction in its overheads, having eliminated the need to monitor and update its Cassandra clusters.
McFadin also talks about the social network Hornet, which similarly moved from on-prem Cassandra to Astra DB last year: “When I talked to Nate [Mitchell, lead DevOps engineer at Hornet], he just started at Hornet recently, and he said: ‘I got a lot of other things going on. I don't want to have to care and feed for my database. But I believe in this database.’
“So by them moving to Astra DB now, they've shifted a lot of that burden over. And I know Nate's sleeping better night now. But their developers now don't have to go through a conduit whenever they want to add new features and things like that - they have direct access to the database, and they can make those choices themselves.
“Hornet is a good example of how Cassandra, the open source database from a few years ago, is now maturing into a cloud native database,” says McFadin.
He compares the growing availability of on-demand services to the start of the shift to grid energy in the 20th century: “At the time, if factories needed reliable electricity, they built their own power plant, they staffed it with their own engineers. Eventually, all of that decentralised and most, if not all large industrial customers buy in grid electricity now. And that's what's shifting right now in cloud, and that’s what cloud native is.”
Sponsored by DataStax
