VMware pulls flawed update that triggered purple death crashes
"Unfortunately, our quality testing and certification process missed this..."
VMware has withdrawn the latest full release of its server virtualisation product vSphere, citing the need to "protect our customers from potential failures" -- after the release triggered system crashes and backup failures.
VMware pulled vSphere 7.0 Update 3 late last week after a flurry of issues reported by users, saying "vSphere is used in the most mission-critical customer environments, and we take any escaped defects seriously."
"After careful consideration, we have removed the ESXi 7 Update 3 release from our Product Downloads site. We made this decision to protect our customers from potential failures which may occur as they upgrade to ESXi 7 Update 3", the $50 billion by market capitalisation firm told customers late on Friday November 19, 2021.
The company is "looking at further increasing transparency for subsequent releases by publishing quality metrics that are accessible both internally and externally" it said, while claiming in a blog on the removal it (already) has "stringent Root Cause and Corrective Action (RCCA) process to continually identify quality improvements."
VMware withdraws vSphere 7.0 update 3: What's the problem?
VMware's vSphere 7.0 Update 3 (its latest major vSphere release) triggered what the virtualisation specialist described in one update as a series of critical "partner driver interoperability problems".
Among these were issues allocated the codes 8600 ("may cause multiple [ESXi] hosts in an high availability cluster to fail with a purple diagnostic screen"); 85982 ("Rollup upgrade of ESXi hosts... fails"); 86069 ("VAMI backup fails when using SMB as protocol"); and 86191 ("HA can no longer be successfully enabled").
Users can find a Q&A here that suggests contacting support for most issues.
Unfortunately VMware had billed vSphere 7.0 update 3 as "the ultimate update release to vSphere 7, making it the best vSphere ever". Its withdrawal and the issues around it drew predictable frustration from users.
(Sample comment on Reddit/sysadmin: "7.anything has been a massive fuck up. Issues I've had: VMs or hosts not responding to commands. VMware's response: 'sd card issue. Apply these fixes.' Fixes don't help! A couple weeks later: VMware says booting from sd no longer supported. Fuuuuuuuuuuuuuu.... Now I have to reinstall to all hosts. Then.. can't use lifecycle manager to update. VMware's response: 'update manually using CLI....' Guess what!! FAILS!! Vmwares reply : oh delete this vib, reboot, updated, reboot, update again, reboot again! What they didn't tell me was the vib reapplied with the update so I had to do it again. Now, vmtools doesn't like to auto update. Had to do a repair install then manual update. Get your shit together vmware!!")
VMware's Paul Turner said: "Unfortunately, our quality testing and certification process missed this issue [sic]. We investigated options to address it with patches, however, due to certain operational complexities for our customers we removed the ESXi 7 Update 3 release from our download site.
"Note that the VMware vCenter Server 7 Update 3 release remains stable and available. Customers may continue to download and upgrade to this latest vCenter Server release. For customers who have already successfully upgraded to ESXi 7 Update 3, you may remain on this release and will have full support from VMware.
"We will reissue ESXi 7 Update 3 as soon as we have the driver issues fully resolved–which involves working with our partners. Customers will be notified when the release is available via our normal release notifications, including a post on this vSphere Blog site" he added in the November 19 blog.
