A RECOMMENDED software update crashed 20 critical IT systems
The update in question...
The installation of a “recommended update and security patch to an operating system issued by an international software provider” resulted in a major IT incident that took down 20 NHS IT systems.
It’s the nightmare of most in IT – and one of the many reasons systems often don’t get patched promptly, despite the exhortations of vendors and security agencies. In December 2021, unfortunately that nightmare became a reality for Sandwell and West Birmingham NHS Trust, resulting in a major incident.
It got worse: “On attempting to remove the patch there was system and data loss” the trust admitted.
The revelation came in the minutes of the NHS trust’s board meeting, published January 5, 2022.
The “in-house IT team and their suppliers have worked to recover and restore the majority of systems engaging with a specialist data recovery company”, the minutes noted, without naming the provider.
Some operations and procedures were postponed as a result "on assessment of clinical risk."
“At the time of writing [the report was published January 5, 2022] the recovery of the full data set for the BMEC [Birmingham and Midland Eye Centre] patients is still underway” the minutes noted, adding that “no patient data has been exfiltrated and the Information Commissioner is aware.”
The paper adds: “There is no evidential behaviour that leads us to believe this incident was caused by a cyber-attack. This was also not caused by any individual clinical systems or suppliers, and the response from our suppliers and the Informatics team, to this unprecedented event, has been well managed and professional.
The recommended software update crashed systems just after...
The minutes do not name the software provider or the recommended update in question.
In possibly unrelated news, however, the trust is a significant VMware customer.
An earlier business case for a systems update published by the trust notes that its “virtual server platform is VMWare.
And in mid-2021 the trust's head of infrastructure posted on social media that they were "very pleased to now have core Infrastructure services operational in the new VMC [VMware Cloud] on AWS cloud datacentre and the first wave of clinical services due to migrate shortly. This is a turning point for Sandwell and West Birmingham Hospitals Trust which reduces risk and opens up possibilities."
Just weeks before the recommended software update that crashed the trust's systems, VMware had withdrawn the latest full release of vSphere (7.0 Update 3) however, citing the need to “protect our customers from potential failures” — after the software update triggered system crashes, backup failures, and fury from users at a cascade of issues, as reported by The Stack here.
VMware blamed critical “partner driver interoperability problems” – with issues 8600 (“may cause multiple [ESXi] hosts in an high availability cluster to fail with a purple diagnostic screen”); 85982 (“Rollup upgrade of ESXi hosts… fails”); 86069 (“VAMI backup fails when using SMB as protocol”); and 86191 (“HA can no longer be successfully enabled”) all reported to have affected thousands of customers who installed the software.
VMware’s Paul Turner said at the time: “Unfortunately, our quality testing and certification process missed this issue [sic]. We investigated options to address it with patches, however, due to certain operational complexities for our customers we removed the ESXi 7 Update 3 release from our download site.
VMware had billed vSphere 7.0 update 3 as “the ultimate update release to vSphere 7, making it the best vSphere ever”. In its last update to a Q&A on the issue, published November 19, 2021, the company noted that “VMware does not currently have a date confirmed for the next vSphere 7.x release."
The Stack has asked the trust for more detailed comment on the software update questions.
