xz-utils Github repository disabled as Linux maintainers assess blast radius of backdoor, earlier commits
Incident suggests a state actor exploiting overstretched maintainer of an "unpaid hobby project"
Github has disabled the entire repository of xz-utils, a data compression utility widely used in Linux, after it was backdoored and used to sneak malicious code onto Kali Linux, Fedora 40, Debian testing and openSUSE Tumbleweed, amongst other largely beta/experimental releases.
The CVSS 10 backdoor, allocated CVE-2024-3094, was added to the upstream tarballs, or compressed files, of xz-utils, in version 5.6.0.
The Kali Linux team said: “The backdoor could potentially allow a malicious actor to compromise SSHD authentication, granting unauthorized access to the entire system remotely… the severity of this vulnerability poses a threat to the entire Linux ecosystem.”
Sysdig's Michael White explained: "Once SSHD loads the now malicious library, the authentication flow is redirected during the RSA key checking. With control over the authentication flow, the library can grant access based on the criteria set by the attacker. This is most likely their RSA keys or some other data which only the attacker knows. None of the password or PKI based authentication would be effective at this point...."
State actor exploiting an "unpaid hobby project"?
The malicious changes were submitted by a developer going as Jia Tan or JiaT75; who has contributed significantly to the xzu-utils project for two years and who appears to have done so from behind a VPN.
Security researcher The Grugq said on X that if the backdoor had not been spotted and instead had successfully percolated down into production deployments of major Linux distributions “the end game would be the ability to login to every Fedora, Debian and Ubuntu box on the internet…”
wild stuff re: xz/liblzma backdoor https://t.co/aBenGeQlLk pic.twitter.com/BNSjjXuNxm
— Dominik Penner (@zer0pwn) March 29, 2024
Like so many critical upstream open source projects (most of The Stack’s readers will be familiar with the “Nebraska” issue) xz-utils has long been an “unpaid hobby project” as its original creator Lasse Collin said in 2022.
In what is possibly a case of collateral damage, Collin – the only and overstretched maintainer of the utility prior to Jia Tan joining the project – has also had his Github account suspended as the spotlight of suspicion started to shine onto the entire project over the past 24 hours.
See also: This JavaScript library is EVERYWHERE, from Paypal to Pornub. Its sole maintainer is broke
Facing questions over a lack of maintenance in 2022, Collin had noted that he hadn’t “lost interest [in the project] but my ability to care has been fairly limited mostly due to longterm mental health issues…
“Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future,” he posted in 2022.
The xz backdoor was the final part of a campaign that spanned two years of operations. These operations were predominantly HUMINT style agent operations. There was an approach that lasted months before the Jia Tan persona was well positioned to be given a trusted role.
— thaddeus e. grugq [email protected] (@thegrugq) March 31, 2024
The incident has triggered deep concern about other commits by the developer who appears to have made them – and the risk of bad actors playing the long game and exploiting overstretched upstream maintainers by building trust and becoming core contributors to upstream libraries.
See also: Linux Foundation urges "more cooperation" from US gov't on 0days
Joey Hess, a longstanding Debian contributor, said he counted “a minimum of 750 commits or contributions by Jia Tan, who backdoored it.
“This includes all 700 commits made after they merged a pull request in Jan 7 2023, at which point they appear to have already had direct push access, which would have let them push commits with forged authors.”
The xz backdoor is, well, setting a fire under the entire Linux ecosystem... but I'm also so impressed with how it was set up: 2-yr maintainership, oss-fuzz, etc.
— Danny Lin (@kdrag0n) March 30, 2024
...and who knows how long it would've stayed undetected if the injected sshd code ran faster (<600ms)
Highlights:
Linux distributions like Debian and others are urgently exploring how far back they should roll back their xz-utils versions, as although the backdoor was found in a recent update, given the developer who slipped it in has been prolific, the entire project has now become suspect and commits they or their associates made in other projects suspect.
See also: Feds to CIOs: Actively ask your vendors if they’ve done a SQLi audit...
Debian’s Hess noted: “Reverting the backdoored version to a previous version is not sufficient to know that Jia Tan has not hidden other backdoors in it. Version 5.4.5 still contains the majority of those commits.
"Commits by them such as 18d7facd3802b55c287581405c4d49c98708c136 and ae5c07b22a6b3766b84f409f1b6b5c100469068a show that they were deep into analyzing the security of xz. They were well placed to insert a buffer overflow that could allow eg, a targeted xz file to cause arbitrary code execution. The impact of such a security hole could be much more stealthy and bad than the known backdoor since it would allow chaining xz with other unrelated software releases on an ongoing basis.
"The package should be reverted to a version before their involvement… or their early commits vetted and revert to a later point, but any arbitrary commit by a known bad and malicious actor almost certainly has less value than the risk that a subtle change go unnoticed. I'd suggest reverting to 5.3.1. Bearing in mind that there were security fixes after that point for ZDI-CAN-16587 that would need to be reapplied," he added.
Wiz has a good quality technical breakdown of the attack chain here. Kevin Beaumont also has an excellent overview here.
