Tackling large-scale transformation of legacy systems with automation
From tackling 25m lines of mainframe code to 10.7b rows of data...
It’s not uncommon for IT modernisation programmes to fail, writes Tim Jones, Managing Director of Application Modernisation EMEA at Advanced, and a lack of planning or a skills gap is often to blame. There has to be a better way, and automation can be used as a key component to completing large-scale transformation of legacy systems – on time, within budget, and with reduced risk.
With a decreasing pool of specialist support for critical systems that carry mounting risk and technical debt, a common strategy for tackling the legacy skills gap is to take a refactoring approach which involves transforming legacy applications and databases into equivalent modern ecosystems using automated tooling. This enables companies to reap the rewards of legacy modernisation within relatively tight timeframes, minimising costs, as well as reducing risk through the universal adherence to their standards and practices.
Follow The Stack on LinkedIn
What does that mean in practice? As an example, the Department for Work and Pensions (DWP) recently completed a large-scale programme of work to move its entire legacy application estate from the VME mainframes to a modern, open systems platform. This included 11 major systems that pay out £150 billion in key benefits ranging from Jobseeker’s Allowance to the State Pension, some of which are over 40 years old.
Around 25 million lines of code, and supporting definitions, were refactored using an automated approach. The code was successfully translated to a modern code set with an integrated development environment.
Additionally, 10.7 billion rows of citizens’ data contained within around 250 mainframe databases were migrated from Integrated Data Management System (IDMS) to Oracle and verified.
Advanced refactored all of the code and set up the data migration process for each system within the first 12 months. It took a ‘factory line’ approach, using high levels of automation to work through each stage in parallel systems to meet challenging timelines. The key enabler here is automation, code refactoring, environment builds, code deployment, data conversion, data verification, testing and so on. Much higher levels of quality and consistency can be achieved, reducing human error and improving predictability.
Modernising legacy IT systems through automation
Three major areas of automation can create a particularly significant impact:
1. Automation of code refactoring
Using a code refactoring process can pull hundreds of thousands of lines of data dictionary content, source code, copybooks and refactoring parameters from repositories like Git, before executing the code transformation and compilation, finally deploying the converted code into the test environments. This delivers the following benefits:
- Accurate, consistent and repeatable code refactoring of millions of lines of code.
- Retention of the existing business logic and end-user view so that testing is simplified with no requirement for end-user retraining.
- Inventory management – nothing gets left behind or undone.
- Unattended overnight builds.
- Reduced reliance on key staff – enabling small teams of specialists, together with limited numbers of customer staff to massively upscale the migration effort to fit within limited timescales.
In turn, this can form the basis of an organisation’s own DevOps based approaches which enable the rapid upscaling of development and testing capability, and ensures that code is quickly and safely deployed and tested into production environments.
2. Automation within development and support
In most organisations running legacy systems, code development, maintenance and support tends to use highly bespoke mainframe-based approaches. These can be replaced by a modern Git-based DevOps style, underpinned by cloud infrastructures such as AWS, Google Cloud, or Microsoft Azure. Standard builds can be made available in repositories like Git and can automatically be pulled down to make and test code changes.
Organisations that are modernising legacy systems generally embrace a DevOps approach to code development, testing, and deployment. This is largely because it delivers access to industry standard processes, toolsets and readily available staff. Furthermore, version control and tracing is usually several-fold better than on legacy systems.
Not only does this have beneficial impacts on the migration project, but it places customers into a modern future-proofed environment ready to capitalise on further development opportunities without building new technical debt.
3. Automation of data conversion
Most mainframe estates have complex data architectures which have often evolved to compromise the needs of resilience, performance and costs. Their architecture is often excellent for their design purpose, but inflexible when challenged by modern agile business requirements.
A vital part of any legacy modernisation is to efficiently migrate the legacy data into modern platforms while maintaining its validity and integrity and improving both flexibility and access. This often forms the critical path at cut-over and therefore must be both fast and reliable. The high levels of accuracy required also force any processes to be repeatable, calling for a need to test, test, and test again.
With legacy data often distributed widely throughout the mainframe estate, and being updated at different points in the business cycle, it is essential to correlate the transfer and conversion at the correct time. Automation is a powerful tool that can reduce the burden of a stressful, high-risk cut-over to a relaxed and predictable monitoring exercise.
The successful migration of IDMS databases to Oracle or SQL Server, for example, requires the use of a meta-driven process that can:
- Validate the completeness of the input data.
- Process the input data, converting it from IDMS to Oracle and validate the integrity of each line.
- Create the target tables and import the converted data into Oracle, applying referential integrity to ensure ongoing protection of the data.
- Validate the Oracle data to ensure that all rows have been successfully converted and loaded.
Read this: DataStax on evolving enterprise data strategies, a new CEO, and its renewed commitment to open source
This process can then be wrapped in code to suit the mainframe and conversion architecture to ensure there is little or no human intervention in the data transfers (other than quality checking). It will also ensure contention on the mainframe, conversion servers, and networks is minimised, while maintaining maximum throughput, and that network, server and database redundancy can be built in to provide resilience to the cut-over process.
Furthermore, the metadata-driven approach enables organisations to make configuration changes and to test them quickly and safely. They can, for example, adjust the data verification intensity dynamically – in testing and rehearsals, every field in every record is cross-matched between the IDMS and relational data sets. However, the resource and time costs of this can be significant and can impinge cut-over itself – so it is possible to tune the sampling frequency to arrive at the best quality/time permutation.
A practical benefit of this level of automation was highlighted recently by the Covid-19 pandemic that forced changes in working. Although no longer able to congregate in teams on-site, existing high levels of automation allow for rapid switches to remote working. In the case of DWP, the need to implement the replacement Job-Seeker’s Allowance system became more urgent. The number of claims was increasing at a time when the old system was approaching capacity. Again, automation enabled us to bring the implementation date forward, proceeding very smoothly with 6.3 billion rows of IDMS data converted to Oracle in just over 24 hours. The new services are supported by 4 million lines of converted legacy code that now deliver services to the UK public at a greater scale and faster speed than had previously been possible on the mainframe.
More than a year on, and automation plays an increasingly significant role with the introduction of DevOps as standard within organisations. The same IDMS automation is now playing a key part in the repeated migration of code and data sets for several large-scale IDMS and Adabas projects throughout Europe and the US.
In summary, for organisations faced with the challenge of modernising their critical and complex legacy systems, using proven automation tools can help to reduce project risk, accelerate project timelines and ensure that all of the valuable pre-existing business logic and data is safely retained. By its very nature, automation has a very predictable and repeatable outcome, mitigating the risks of overruns and spiralling costs that have been seen so many times in alternative modernisation approaches the world over. Get the strategy and approach right, find the right partner with a demonstrated track record, and success will follow.
