How a volunteer stopped a backdoor from exposing Linux systems around the world

Linux, the world's most widely used open source operating system, narrowly escaped a massive cyberattack over the Easter weekend, all thanks to one volunteer.

The backdoor is included in a recent version of the Linux compression format called XZ Utils, a tool little known outside the Linux world but used in almost every Linux distribution to compress large files, making them easier to transfer. If the virus had spread more widely, countless systems could have remained vulnerable for years.

And as Ars Technica noticed in Comprehensive summaryThe perpetrator was working on the project in public.

The vulnerability, which was introduced into Linux remote login, only exposed itself to a single key, so it could hide from public computer scans. like Ben Thompson writes at Strachry. “The majority of the world's computers will be vulnerable and no one will know.”

The story of the discovery of the XZ backdoor begins in the early morning of March 29, as San Francisco-based Microsoft developer Anders Freund posted on Mastodon and I sent an email To the OpenWall security mailing list with the title: “xz/liblzma upstream backdoor leads to ssh server compromise.”

Freund, who volunteers as a “supervisor” at PostgreSQL, a Linux-based database, noticed some strange things over the past few weeks while running tests. Encrypted logins to liblzma, part of the XZ compression library, were consuming a significant amount of CPU. None of the performance tools he used revealed anything,” Freund wrote in Mastodon. This immediately aroused his suspicions, and he recalled a “strange complaint” from a Postgres user a few weeks earlier about Valgrind, a Linux program that checks for memory errors.

After some investigation, Freund eventually discovered what was wrong. “XZ Warehouse and XZ Tar Balls have closed back,” Freund noted in his email. The malicious code was present in versions 5.6.0 and 5.6.1 of xz tools and libraries.

Shortly after, open source software company Red Hat sent a message Emergency security alert For Fedora Rawhide and Fedora Linux 40 users. Ultimately, the company concluded that the Fedora Linux 40 beta contains two affected versions of the xz libraries. It's possible that Fedora Rawhide versions received versions 5.6.0 or 5.6.1 as well.

Please immediately stop using any FEDORA RAWHIDE products for business or personal activity. Fedora Rawhide will be rolled back to xz-5.4.x soon, and once this is done, Fedora Rawhide instances can be safely redeployed.

Although the beta version of Debian, a free Linux distribution, contains packages compromised by its security team I acted quickly To return to them. “At this time, no stable versions of Debian are affected,” Debian's Salvatore Bonaccorso wrote in a security alert to users on Friday evening.

Freund later identified the person who sent the malicious code as one of two lead xz Utils developers, known as JiaT75 or Jia Tan. “Given the activity has been going on for several weeks, the perpetrator was either directly involved or there was a severe compromise of their system. Unfortunately the latter seems to be the least likely explanation, given that they spoke in different lists of the ‘fixes’ mentioned above,” Freund wrote. In his book. analysisafter linking several solutions made by JiaT75.

JiaT75 was a familiar name: they had worked alongside the original developer of the .xz file format, Lasse Collin, for a while. As programmer Ross Cox pointed out in his book timetableJiaT75 began sending seemingly legitimate patches to the XZ mailing list in October 2021.

Other arms of the scheme were revealed a few months later, as two other identities, Jigar Kumar and Dennis Ince, Complaints have begun to be sent via email To Colin about the mistakes and slow development of the project. However, as noted in reports Evan Buhs Others, “Kumar” and “Ins” have never been seen outside of the XZ community, leading investigators to believe that they are both fakes who only exist to help Jia Tan access his location to deliver the backdoor code.

“I'm sorry about your mental health issues, but it's important to be aware of your limits. “I realize this is a hobby project for all contributors, but the community wants more,” Ince wrote in one message, while Kumar said in another: “Progress will not happen.” Until there is a new supervisor.”

Amid the back and forth, Collins wrote: “I have not lost interest but my ability to care has been somewhat limited due to long-term mental health issues but also some other things,” and suggested that Jia Tan take on a larger role. “It's also good to keep in mind that this is an unpaid hobby project,” he concluded. Emails from Kumar and Ens continued until Tan was added as a moderator later that year, to be able to make modifications and attempt to introduce the backdoor package into Linux distributions with more authority.

The xz backdoor incident and its aftermath are an example of the beauty of open source and the incredible vulnerability in the Internet's infrastructure.

A developer of FFmpeg, a popular open source media package, has highlighted the problem In a tweet“The xz fiasco showed how relying on unpaid volunteers can cause major problems. Trillion-dollar companies expect free, urgent support from volunteers. They brought receipts indicating how they handled a 'high-priority' bug affecting Microsoft Teams.

Despite Microsoft's reliance on its software, the developer wrote: “After politely requesting a support contract from Microsoft for long-term maintenance, they offered a one-time payment of a few thousand dollars instead… Investments in maintenance and sustainability are unattractive and a middle manager probably won't get it.” For his promotion, he will even pay him a thousand times over the course of many years.

Details of who is behind JiaT75, how their plan will be carried out, and the extent of the damage have been revealed by an army of developers and cybersecurity professionals, both on social media and online forums. But this happens without direct financial support from the many companies and organizations that benefit from the ability to use secure software.