This government department pays out £3 billion weekly: What if its IT systems went down?
New DWP tender aims to make sure that that never happens...
The UK’s Department of Work and Pensions (DWP) makes over 2.5 million payments every day, serving 34% of the British population. For too many recipients, this money is the thin buffer between being able to eat and penury. The cost of crippling IT disruption that prevents such payments does not bear thinking about.
As the largest single payment system provider in Europe, DWP is, however, rightly thinking about it and this month is preparing to launch a hunt for “payment infrastructure to address a catastrophic event which prevents DWP from preparing and issuing payments files” – it plans to tender as early as June 23, 2022.
DWP has the ambitious aim of taking this new system (dubbed the “Disaster Emergency Payment Service”, or DEPS) live on 1 December 2022, the department recently revealed in a public interest notice (PIN).
Join colleagues following The Stack on LinkedIn
The project is the latest addition in what has been a complex overhaul of the department’s legacy IT estate – hosting of which has bounced between in-house in the ‘90s, through to outsourced hosting with EDS and later Hewlett Packard, back in-house some six years ago and now to primarily public cloud IaaS.
(Beyond the hosting element, DWP has launched a major push to refactor payments and other applications to be event-based rather than batch-based; underpinned by a microservices architecture, with more use of noSQL databases -- it has doubled down on MongoDB -- and with legacy COBOL mainframe applications refactored to run on Linux servers. Its Digital Payments Service primarily works in Java, and Node JS; using the Drop Wizard framework for building APIs and web applications and open source monitoring tools like Prometheus and Grafana "so we're not dependent on some commercially procured software" as lead architect Nick Cutting puts it.)
Department of Work and Pensions’s Disaster Emergency Payment Service plans revealed
“The Department could be vulnerable to external risk such as cyber security threats which could result in the estate environment not being able to access the data and potentially unable to process and create the payment files. The potential solution which will be operating outside of DWP estate is crucial for the Department's risk mitigation strategy” DWP said on June 1.
The tender comes amid what has been as a billed as a “once-in-a-generation transformation of the UK's biggest government department” as DWP modernises one of central government’s most complex, sprawling IT estates – one which has long been rife with “legacy” technology and ageing applications: the department continues to run over one billion lines of code in 90+ programming languages. Its APIs receive 173 million requests every month and it now spins up 280,000 AWS servers every month, one job advert for a site reliability engineer suggests.
The public interest notice this month came in the wake of DWP’s award-winning Virtual Machine Environment Replacement (VME-R) programme which saw it migrate 26 million lines of COBOL to MF COBOL (A COBOL version and compiler developed by Micro Focus to run on non-mainframe operating systems) and transitioned 10.6 billion rows of citizen data from IDMS to a new Oracle RAC database running on Red Hat Enterprise Linux.
The Department of Work and Pensions has been transparent about its digital transformation efforts, in a 2021 podcast for example detailing how it is working with banks and UK financial regulators on broader industry transformation of the payments sector ("as the largest payer in the UK, we've been able to help shape the direction of these changes") and with Nick Cutting from the Digital Payments Services team waxing lyrical about how shifting architecture to the cloud has improved flexibility et al: "They're providing all the wraparounds, sort of the security frameworks, the operational frameworks... my trick as the lead architect is making sure that we're using the right services, that we're getting the right information, monitoring the right things and then again, from a security perspective, making sure we're implementing the right security framework to make sure that our services are secure."
It seems highly likely as a result that DWP's Disaster Emergency Payment Service is likely to involve some form of isolated mirror of the existing AWS environment as a failover. As DWP's Natalie Weir, also from the digital payments team, spelled out in 2021: "We're working towards our vision to create an industry leading and intelligent payment service... so citizens who are entitled to welfare payments can confidently depend on our modern and resilient payment system. It aims to deliver exceptional service to its users and it will be responsive, intuitive and adaptable to the changes that we see coming in the industry over the coming years."
