Backdoor found in widely used Linux utility breaks encrypted SSH connections
Backdoor found in widely used Linux utility breaks encrypted SSH connections
Malicious code planted in xz Utils has been circulating for more than a month.
The malicious changes were submitted by JiaT75, one of the two main xz Utils developers with years of contributions to the project.
“Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system,” an official with distributor OpenWall wrote in an advisory. “Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the ‘fixes’” provided in recent updates. Those updates and fixes can be found here, here, here, and here.
On Thursday, someone using the developer's name took to a developer site for Ubuntu to ask that the backdoored version 5.6.1 be incorporated into production versions because it fixed bugs that caused a tool known as Valgrind to malfunction.
“This could break build scripts and test pipelines that expect specific output from Valgrind in order to pass,” the person warned, from an account that was created the same day.
One of maintainers for Fedora said Friday that the same developer approached them in recent weeks to ask that Fedora 40, a beta release, incorporate one of the backdoored utility versions.
“We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added),” the Ubuntu maintainer said.
He has been part of the xz project for two years, adding all sorts of binary test files, and with this level of sophistication, we would be suspicious of even older versions of xz until proven otherwise.
Dude seems like a foreign asset
Jia Tan, University of Hong Kong in China. He’s been the sole maintainer of the package for almost two years.
From the article...
Will Dormann, a senior vulnerability analyst at security firm Analygence, said in an online interview. “BUT that's only because it was discovered early due to bad actor sloppiness. Had it not been discovered, it would have been catastrophic to the world.”
Is auditing for security reasons ever done on any open source code? Is everyone just assuming that everyone else is doing it, and hence no one is really doing it?
EDIT: I'm not attacking open source, I'm a big believer in open source.
I'm just trying to start a conversation about a potential flaw that needs to be addressed.
Once the conversation was started I was going to expand the conversation by suggesting an open source project that does security audits on other open source projects.
Please put the pitchforks away.
Edit2: This is not encouraging.
You're making a logical fallacy called affirming the consequent where you're assuming that just because the backdoor was caught under these particular conditions, these are the only conditions under which it would've been caught.
Suppose the bad actor had not been sloppy; it would still be entirely possible that the backdoor gets identified and fixed during a security audit performed by an enterprise grade Linux distribution.
In this case it was caught especially early because the bad actor did not cover their tracks very well, but now that that has occurred, it cannot necessarily be proven one way or the other whether the backdoor would have been caught by other means.
Also they are counting the hits and ignoring the misses. They are forgetting that sneaking a backdoor into an open source project is extremely difficult because people are reviewing the code and such a thing will be recognized. So people don't typically try to sneak back doors in. Also, backdoors have been discovered in an amazing amount of closed source projects where no one was even able to review the code.
It's maybe possible, but perhaps even unlikely still.
Overwhelmingly thorough security review is time consuming and expensive. It's also not perfect, as evidenced by just how many security issues accidentally live long enough to land Even in enterprise releases. That's even without a bad actor trying to obfuscate the changes. I think this general approach had several aspects that would made it likely to pass scrutiny:
- It was in XZ, which was likely not perceived as a security critical library. A security person would recognize any thing as potentially security critical, but they don't always have the resources and so are directed to focus on obviously security related and historically security incident magnets.
- it was carried out by someone who spent years building up an innocuous reputation. Investigation may even show previous "test samples" to be malicious but not caught, or else it was a red herring to get people used to random test samples getting placed in the project.
- The only "source code" he touched was "just build scripts". Even during a security audit, build shell scripts are likely going to be ignored, they are just build scripts and maybe you run some tests on all scripts, but those tests aren't going to catch this sort of misbehavior.
- The actual runtime malicious code was delivered as portions of ostensibly throw away test sample xz files. The malicious code is applied by binary patch of the build output. A security audit won't be thinking too hard about a sea of binary files that are just throwaway samples as fodder for test.
So while I see the point about logical fallacy about it accidentally not getting far enough to see if the enterprise release process would have caught it, I think we know track records well enough to deem this approach likely to get through. Now that it has been caught, I could see some changes that may mitigate this in the future. Like package build scripts deleting all test samples and skipping tests when building for release, as well as more broad scrutiny.
There's also the reality that a lot of critical applications deem themselves too cool to settle for "old crusty enterprise distributions". They think that approach is antiquated and living on the edge is better. Admittedly I doubt theyd go as far as arch, tumbleweed, or rawhide, but this one could have easily made it to Debian testing, fedora release, or an Ubuntu release.
Have those audits you allude to ever caught anything before it went live? Cuz this backdoor has been around for a month and RedHat is affected, too. Plus this was the single owner of a package who is implicitly trusted, it's not like it was a random contributor whose PRs would get reviewed.
The code being open source helps people track it down once they try to debug an issue (performance issue and crashes because in their setup the memory layout was not what the backdoor was expecting), that's true. But what actually triggered the investigation was the bug. After that it's just a matter of time to trace it back to the backdoor. You understimate reverse engineers. Or maybe I'm just spoiled.
How long until US bans code from developers with ties to CN/RU?
You’re making a logical fallacy called affirming the consequent where you’re assuming that just because the backdoor was caught under these particular conditions, these are the only conditions under which it would’ve been caught.
No, I'm actually making that comment based on a career as a software developer, who has actually worked on a few open source projects before.
Having once worked on an open source project that dealt with providing anonymity - it was considered the duty of the release engineer to have an overview of all code committed (and to ask questions, publicly if needed, if they had any doubts) - before compiling and signing the code.
On some months, that was a big load of work and it seemed possible that one person might miss something. So others were encouraged to read and report about irregularities too. I don't think anyone ever skipped it, because the implications were clear: "if one of us fails, someone somewhere can get imprisoned or killed, not to speak of milder results".
However, in case of an utility not directly involved with functions that are critical for security - it might be easier to pass through the sieve.
I don’t think anyone ever skipped it, because the implications were clear: “if one of us fails, someone somewhere can get imprisoned or killed, not to speak of milder results”.
However, in case of an utility not directly involved with functions that are critical for security - it might be easier to pass through the sieve.
I've actually seen people checking in code that doesn't get reviewed properly on mission critical apps before (like in the health industry).
My understanding is basically the same as yours, and in theory I agree with you. However, the problem is we all tend to hand-wave away any possibility of bad things happening, because it's open source, and don't take into account human nature, especially when it comes to volunteer versus paid work.
Auditing can be done only on open source code. No code = no audit. Reverse engieneering doesn't count.
True, but does it actually get done, or just everyone just assuming gets done, because it's open source?
Bystander effect, yes.
The answer is the same as closed source software: sometimes.
But that's beside the point, a security audit is not perfect. Plenty of audited codebases are the source of security vulnerabilities in the wild. We know based on analysis that the malicious actor's approach would have a high chance of successfully hiding from a typical security audit.
Thankfully this was discovered before hitting stable distros but I'm hoping it increases scrutiny across the board. We dodged a bullet on this one.
Across the board indeed. Scrutiny in code is one thing, where this story, as far as is known right now, really went south is the abuse of a trusted, but vulnerable, member of the community.
I know the (negative) spotlight is targeting Jia Tan right now (and who knows if they (still) exist), but I really hope Larhzu is doing okay. Who's name is mentioned in the same articles.
Mental health is a serious issue, that, if you read the back story, is easily ignored or abused. And it wasn't an unknown in this story. Don't only check the code, check up on your people too.
This is why I run debian oldstable.
Or maybe it's because I'm too lazy to do a dist-upgrade.
Long game supply chain attacks, pretty much going to be state actors. And I wouldn't chalk it up to the usual malicious ones like China and Russia. This could be the NSA just as easily.
I honestly think the NSA has changed. If you look at the known backdoors they haven't got caught making any new backdoors since like 2010. Their MO also seems to be more hardware and encryption (more of an observational charter) than manipulation.
There's also evidence US Congress acted to stop the NSA from doing these underhanded tacits at least once https://www.wired.com/story/nsa-backdoors-closed/
They're not idiots, lots of smart people there that surely understand the risk of something like this to US national security interests. It's not the NSA that's been asking for encryption to be broken in recent years. They've been warning about quantum threats and ... from what I'm aware of actually been taking on the defensive role they were conducted to perform https://gizmodo.com/nsa-plans-to-act-now-to-ensure-quantum-computers-cant-b-1757038212
This seems like something that could actually be weaponized against predominantly western technology companies so I'd be very surprised if it was them and very surprised if they used someone that appears to be a Chinese born resident to do it.
I really can't believe they've stopped. Their mentality is "national security has no morals". They'll do everything they can do to facilitate that mission, though not getting caught is a big part of the facade they need to put on to keep or renovate their image to do this.
Maybe they're being more careful, and doing simple things like putting in timestamps that emulate working hours in other timezones are certainly the first thing they're going to think about. That one has always cracked me up, security researchers point to it like it's proof of something, which is ridiculous. Just like our people are smart, I don't think the foreign actors are dumb either.
And before you say it, I'd be all over not being paranoid if it hadn't been proven to me time and again that these agencies won't change, that they don't give a shit about what's right if it gets in the way of their mandate. The only thing that might change is how well they hide things now and intimidate their people into staying quiet. Because potential whistleblowers have seen the examples that have been made.
That's not true, Shadow broker leaks for example contained 0-day found by the NSA well after 2010. And that's only what got published, there's probably more !
It's not the NSA that's been asking for encryption to be broken in recent years.
I remember 2013 backdoored crypto by NSA. If they get caught less doesn't mean they make less backdoors.
EIDT: it was discovered in 2007 and revoked as standard in 2014
Also they owned corporation that made backdoored crypto algos till 2018. And the only reason they stopped is FOIA.
I don't know man. Imagine you could have ssh access to every Debian and fedora server on the planet, and all you had to do was write tests for some compression library for 2 years and sneak in a clever patch. I'd guess such an exploit is worth millions. You wouldn't work 2 years for millions of dollars?
This is sophisticated but it doesn't have to be a state actor.
Yup. I think it's an independent hacker, probably hired by a state actor, but not a state actor themselves.
My understanding is that state actors generally look for exploits, not create them. I also think they'd be a little more clever than this.
If you throw enough money at the right person you can get shit done.
I think you are greatly underestimating FSB incompetense.
There are no known reports of those versions being incorporated into any production releases for major Linux distributions
...
A stable release of Arch Linux is also affected.
... BTW.
The malicious code is only thought to have affected deb/rpm packaging (i.e the backdoor only included itself with those packaging methods). Additionally, arch doesn't link ssh against liblzma which means this specific vulnerability wasn't applicable to arch. Arch may have still been vulnerable in other ways, but this specific vulnerability targeted deb/rpm distros
I liked the joke, but ya arch is not compromised. Check out this user's detailed comment.
This is really bad.
The backdoor has existed for a month at least. Yikes.
The backdoor appears to specifically target RSA public key authentication, so they must have had a target in mind that they know uses RSA keys.
I would highly recommend Curve25519, etc., just because such keys are faster and less common than RSA public-private keys in today's world. RSA 2048-bit keys are considered weak today, while the Curve25519 256-bit keys remain stronger. Also, the ChaCha20-Poly1305 cipher has an interesting backstory and doesn't necessarily need hardware acceleration (which, in theory, could be borked by the HW-vendor) to obtain good performance.
Unfortunately, some SSH front-ends don't play nice with Curve25519 public-private keys yet... (I'm pointing at the putty SSH client, but that may have improved from the last time I had to use it)
It would be Ed25519 for SSH:
$ ssh-keygen -t ed25519
Would another less complex answer simply be that many (most?) people and organizations use RSA because it was first and elliptic signing is not yet as prevalent?
Going with Occam’s Razor here…
A stable release of Arch Linux is also affected. That distribution, however, isn't used in production systems.
Shots fired!
It seems WSL Ubuntu and Kali are safe with versions 5.2.5 and 5.4.4 installed respectfully.
Damn, I installed mine disrespectful.
Is that when you install it in a VM, as if to show it that it’s not a good enough little distro to boot directly on your hardware?
I suddenly feel so guilty!
I thought about the same (disrespectful) as I typed it. I've already laughed with you!
Don't forget about openSUSE Tumbleweed! It's actually affected AFAIK.
I think the AI that wrote the article misunderstood.
Arch doesn't build from release tar balls, but straight from git. Arch also doesn't link sshd against liblzma. So while they've shipped the dirty version of xz utils, at least sshd is not affected.
It's possible that the dirty version affected some of the other things that link liblzma. Like a handful of kde components for example.
Also, the malicious code only activated if it detected being built as dpkg or rpm.
Here is a more detailed FAQ about what happened:
This is also a good summary of the timeline: https://boehs.org/node/everything-i-know-about-the-xz-backdoor
Please help me as a novice Linux user- is this something that comes preinstalled with Mint Cinnamon? And if so, what can I do about it?
As the other person said it’s likely that
xzis already installed on your system, but almost certainly a much older version than the compromised one. It’s likely that no action is required on your part assuming you’ve not been downloading tarballs of bleeding edge software.As the other person said, just keep doing updates as soon Mint recommends them (since it’s based on Ubuntu LTS, it’s a lot less likely to have these bleeding edge vulnerabilities).
Much appreciated, thank you.
The library itself is very common and used by a lot of things (in this case it seems that the payload only activated when used by specific programs, like SSH).
What you can do about it is keep your system up-to-date using your distribution update mechanisms. This kind of thing, when found out, is usually fixed quickly in security updates. In Mint (which I don't use, but I believe is based on either debian or ubuntu, which uses dpkg/apt) security updates are flagged differently anc can be installed automatically, depending on your configuration.
tl;dr: keep your system up-to-date, it will keep known vulnerabilities away as much as it can;
dpkg --list | grep xz
should return what version of xz package is on your system. Likely 5.4, in which case you should be okay.
It says "5.2.5-2ubuntu1." So I'll have to see about updating it.
EDIT: However, this says I should be safe: https://forums.linuxmint.com/viewtopic.php?t=416756
openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.
It’s all systemds fault, got it.
Do it all? Then you did it all, motherfucker.
Had redhat's patch write
READY=1into $NOTIFY_SOCKET instead linking to libsystemd to do this, this backdoor would not be possible.
Technically it breaks libsystemd encryption, which is not used in upstream openssh. There are unofficial redhat patches that use this library instead of reading one enviroment variable and writing to one file.
This is the best summary I could come up with:
Researchers have found a malicious backdoor in a compression tool that made its way into widely used Linux distributions, including those from Red Hat and Debian.
An update the following day included a malicious install script that injected itself into functions used by sshd, the binary file that makes SSH work.
So-called GIT code available in repositories aren’t affected, although they do contain second-stage artifacts allowing the injection during the build time.
In the event the obfuscated code introduced on February 23 is present, the artifacts in the GIT version allow the backdoor to operate.
“This could break build scripts and test pipelines that expect specific output from Valgrind in order to pass,” the person warned, from an account that was created the same day.
The malicious versions, researchers said, intentionally interfere with authentication performed by SSH, a commonly used protocol for connecting remotely to systems.
The original article contains 810 words, the summary contains 146 words. Saved 82%. I'm a bot and I'm open source!
So-called GIT code available in repositories aren’t affected
I wonder what convinced the model to treat git as an acronym
I imagine many aren't familiar with British slang and therefore assume git must stand for something, especially considering software devs love their acronyms.
It was like that in the original article. Or are you saying that the original was written by an AI too (it might be).
Who wants to bet he received a nice lump sum deposit of cash from a five eyes state to make an "accident"...
No...
Just read the story...
And that's why you cannot trust open source software blindly.
And yet with closed-source software you have no choice but to trust it blindly. At least open source software has people looking at the code.
Bet you anything there were more pairs of eyes on SolarWinds code than this. Sick of this open source bystander effect.
Code scanners check for vulnerabilities not malicious code. Ain't no one running full coverage dynamic scanners to trigger all branches of code on this thing, otherwise this would've been caught immediately
You are an idiot. It's not blind. That's how it was found.
Not having world accessible SSH is the real fix here.
You are an idiot. It’s not blind. That’s how it was found.
From the article...
Will Dormann, a senior vulnerability analyst at security firm Analygence, said in an online interview. “BUT that's only because it was discovered early due to bad actor sloppiness. Had it not been discovered, it would have been catastrophic to the world.”
Yeah I nearly panicked for a second there, then I remember noone's getting near that anyway. Back to my relaxing weekend.
Not really. The most important admin interfaces are the ones you can't lock behind an IP whitelists.
"whitelists good IPs" - OK but what if I need to manage the "good ip" infra, etc
As opposed to what? If you said "thats why you cannot trust any software blindly" it would have been not that wrong.
And precisely that's why the exploit was found. If it was a closed source programme, a lone threat actor modifying the code and passing it in a release can happen, and no one will find it out. In that case everyone trusts the internal security team of a closed source company blindly. I really don't see this as an open source issue. These are malicious actors.
This is one hell of a take.
Certainly one of the takes of all time.
Imagine trying to make a helpdesk of a proprietary company take your "it's taking 0.5 seconds longer to login" complaint seriously...
So many vulnerabilities were found due to time to login that one of the security features was to take longer to respond to a bad login so they couldn't tell what part failed. Here's an article I found about one such vulnerability.
s/open source software/anything/
Reflections on Trusting Trust
- paper by Ken Tompson, 1984(not that one)
You can trust blindly whatever software you like. Most of us, even those that can code, trust blindly whatever software we use because we have other priorities. But what you can do only with open source software, is open your eyes if you choose.
Ftfy: And that's why you cannot trust people blindly.
Just because we cant observe the code of proprietary software and is sold legally doesn’t mean its all safe. Genuinely I distrust anything with a profit incentive.
What an oblivious attitude
Yep, you're not wrong.
This is why I don't use Linux. Insecure as fuck.
This. Everyone knows that windows is a perfectly safe and secure environment with no exploits and vulnerabilities whatsoever.
Unironically, this is true. They are features
Tell me you dont know shit about software development without telling me you dont know shit about software development
Windows user?
If it was closed source software you wouldn't even know this was happening.