Rethinking your approach to storage platform development
A unified hardware and software platform can...
The data storage market continues to rapidly evolve in dimensions of both scale and functionality, offering CTOs and CIOs a wide array of solutions – each with their advantages and drawbacks. Broadly speaking, there are two different thought processes when it comes to storage infrastructure – build versus buy, which can also be looked at as software-defined storage versus the appliance market. Both approaches aim to solve the same challenges.
Differing approaches to storage platform development
Following-on from the IT infrastructure explosion that began in the 1990s and the massive growth in data storage requirements this gave rise to, different mindsets evolved around the best approaches to the challenges it presented - especially for virtualization solutions.
At one end of the spectrum there was an emphasis on software defined storage strategies, with the rationale being that with the right software one could manage any storage issue. At the other end of the spectrum were those who placed their bets on hardware-based solutions, aided by software. The advent of the internet and later the public cloud dramatically changed the dynamics of the data storage market, with differing sets of non-functional and functional requirements for NFS, S3 and block protocols, writes Alex McMullan, CTO International, Pure Storage.
Together, they drove a whole new set of capabilities and aspirations that neither the software defined or hardware led storage platform approaches could completely address, and were the driving force behind a new wave of innovation in the market. They also created the conditions for companies focused on rapid innovation to thrive, while others chose the path of iterative change and essentially the repackaging of existing solutions.
This brings us to the present day and a storage market that continues to evolve at a rapid pace, both on the software and hardware fronts. Innovation in the sector is closely tied to the current and anticipated needs of CIOs and CTOs, who are looking to invest in platforms that can be evolved over time, with inbuilt flexibility to readily adapt to changing needs for traditional and VM-based apps evolving to containers and cloud-native deployments.
However, in general, there is still some difference of opinion on the optimal way to approach platform selection, software-based solutions, or hardware focused approaches.
See also: The benefits of Infrastructure-as-Code for modern CIOs
Given the competing approaches to data storage platforms, engineering teams are faced with the challenge of finding a balance of iteration where software and hardware are deeply intertwined, without leaving each other behind - this gave rise to the hardware compatibility list (HCL) concept. Backward compatibility is also required in order to avoid sunk investments. A common way of dealing with this issue is to ensure that hardware sets can accommodate iterative change and that software models can integrate at the various stages of those same hardware platform changes.
It’s entirely possible though to sidestep the software versus hardware platform selection debate by adopting an integrated software and hardware approach, which has numerous advantages, more on that to come.
Addressing the challenges of storage platform development
Designing and creating a new platform is a major challenge that in the early development stages, will be dictated largely by its target, required functionality and the timeline. For example, if a new platform needs to be able to run containers and virtual machines, or to ship data to the public cloud. Knowing this will help determine the functional requirements of the platform and answer the question of which hardware capabilities can be built and which bought.
To achieve the desired outcome, compromise will likely be required – at least in the early days of platform development. Key challenges include integrating off-the-shelf components and ensuring that software can keep-up with the pace of development. This stage of platform development is often referred to as minimum viable product (MVP) – that is developing a product that is good enough to capture the attention of the market and secure some early customers. At this stage, in terms of storage devices, there’s a strong argument that the primary requirement should be durability and reliability. These challenges must be addressed on day one, or the product is doomed from the outset.
Having a single type of card substantially reduces the number of tests that must be performed to maintain the quality level if different types of network cards are in different PCIe slots or storage nodes...
The second phase of product development follows the development of the initial version of the platform. This is focused on redesigning the compromises made in initial product development and in many cases, replacing off-the-shelf components with something bespoke that is more efficient and elegant. What’s most important is for the VP of engineering to have a clear view of the strategy and the product development roadmap.
There are clear benefits to having a consolidated hardware and software development approach.
Chief amongst these is having a platform that can evolve, grow and adapt over time, future proofing the technology investment and offering cost efficiencies. Additionally, with a sufficiently agile software development capability, it is possible to work around the supply chain constraints that we've seen in the technology sector in recent years. Having visibility into hardware supply chains and the ability to flex avoids co-dependencies and delays.
Streamlining storage platform development through a unified platform approach
The choice CIOs and CTOs face in building-out a storage platform involving multiple hardware and software vendors, or going with a single vendor offering a unified platform (the one-stop-shop approach) can be challenging.
There may be solid reasons for choosing either path. Increasingly though, there are many good reasons to consider a unified hardware and software platform approach when creating a storage platform, but a particularly compelling one that often gets overlooked is quality. A self-designed and managed hardware platform gives full transparency into its construction and components. Using the example of network cards, having a single type of card substantially reduces the number of tests that must be performed to maintain the quality level if different types of network cards are in different PCIe slots or storage nodes. Having fewer permutations of hardware that need to be supported with the software, results in higher quality and greater uptime because every possible code path can be tested in a much more comprehensive manner. A one-stop platform approach also yields supply chain benefits as well as support efficiencies, and avoids the blame shifting that can occur in troubleshooting, when working with a range of vendors.
The emergence of subscription and consumption models...
Historical utilisation of storage capacity (actual data written as opposed to allocated) has always been a target for CIOs, due to the low rates achieved where 60% would be considered as very efficient. Coupled with an increasing focus on energy efficiency and increased demand, every enterprise has looked for alternative approaches to manage these challenges whilst lowering long-term unit costs.
We’ve seen the emergence of subscription and consumption models in storage, which is a positive development. Storage as a Service (STaaS) is a natural extension of the combined hardware and software platform development approach, which is bringing further innovation to the storage market, offering greater flexibility, reliability and ensuring storage technology is always up to date. Best in class STaaS offerings will even offer non-disruptive upgrades.
However, for STaaS to truly deliver on its promises, it must be backed by robust SLAs and built on an architecture that is focused on simplicity, as opposed to being based around rebadged legacy tools and on-demand portals as cloud consoles. These types of offerings can’t deliver a sustainable, non-disruptive, as-a service business.
The increasing popularity of storage subscription models is giving rise to another notable industry change related to risk management and mitigation. Before now, CIOs and CTOs faced an array of challenges when working to deliver on a storage platform creation project, with high levels of risk. Subscription models are mitigating many of these risks because of their flexibility. Now, when a project flourishes, it can be easily scaled, but at the same time, less successful ones can be quietly shelved or adapted, without wasting vast sums of money. This is a positive development that actually favours the risk taker and drives innovation. It also establishes a clear path forward for storage platform creation.
Storage innovation will continue to be fuelled by combined hardware and software
In recent years, the greatest innovations in storage platforms have taken place at the intersection of hardware and software. This creates a compelling case for seeking a balanced solution that values hardware and software equally to address ever-evolving data storage demands of organisations. The exponential growth of unstructured data continues to create new challenges in data storage, ranging from energy consumption through security to systems heading towards exabyte scale. These challenges will require giant strides to be made in data protection, resilience and materials science to welcome the next wave of applications, with the rapid innovators ideally placed to be at the forefront.
