▲ 280 points
back
128 comments
Note that NixOS and reproducible builds did not detect the xz backdoor, and in fact NixOS shipped the malicious builds of xz (though they didn't do anything because the malware didn't target NixOS):
> I am a NixOS developer and I was surprised when the backdoor was revealed to see that the malicious version of xz had ended up being distributed to our users.
As always theory and reality are different, and the thing that made xz possible was never a technical vulnerability with a technical solution—xz was possible because of a meatspace exploit. We as a community are very very bad at recognizing that you can't always just patch meatspace with better software.
> NixOS and reproducible builds did not detect the xz backdoor
Nix declarativeness is quite useful to increase protection against exploits in a number of ways. Unfortunately, there is still a lot of untapped potential. My number one priority would be to implement fine-grained ephemeral containers. Guix has these already.
This would make it convenient to run every single process with restricted privileges, including no access to ~/, except those directories that are needed by the task. That would prevent e.g. a rogue pip package from stealing SSH keys.
Still, I think the xz backdoor did not work on NixOS because its unusual non FHS-compliant filesystem structure.
I still like the blogpost, because NixOS bills itself as a technical solution to prevent build artifacts that are decoupled from the source code (i.e. not reproducible), and the xz backdoor was hidden in build artifacts.
Right. The title seemed to be suggesting that the Nix way of doing things might have detected the backdoor. It's actually intending to suggest ways that Nix could be changed in order to detect the backdoor.
> As always theory and reality are different
Thank you very much for citing that! along with highlighting the fact that the exploit was in fact, not detected by reproducible builds prior to other means of discovery.
In recent times, actual reality is often maligned when compared to how people feel about objective reality and how it meshes with their individual value systems.
I have personal values too, but I don't hold the opinion that actual reality is less significant than how I feel about it. It's not a popular perspective 8-/
I've always liked to say: The difference between theory and reality is that, in theory they're the same, and in reality they're not.
I hope the realization that the reproducible builds of NixOS _could_ have detected the xz exploit, but didn't, will lead to new advances in the analysis of those reproducible builds to detect other exploits sooner in the future.
The reason is a bit funny - the NixOS bootstrap downloads it's source code which is a xz compressed tarball.
I feel the author is a bit tunnel visioned by what happens to happen this time. The Jiatan incident has a sample size of one, it'd be a bit short sighted to think that's the only way it could happen. You can imagine various scenarios where the defenses suggested here will not have worked.
Also I (as a nix user myself) think it's unlikely NixOS would have caught it. As evidenced by the fact that it didn't. (Yeah I realize I just said next time it might happen differently but it'd be foolish to put faith in nix without evidence).
NixOS is really irrelevant here because the xz backdoor specifically targeted RedHat and Debian. It's equally relevant to say the xz backdoor didn't affect Windows (ironically the backdoor was ultimately found by a Microsoft employee, an oft-overlooked detail).
Article says that distributions should get source code directly from the VCS (for instance Github) rather than the traditional installation tarball.
I don’t see what this solves though. Couldn’t a malicious maintainer simply add binary blobs directly to the source code repository?
The author suggests Github is trusted, as though Github validates code in some way. Which of course it does not.
So the argument hinges on the fact that the XZ maintainer hid malicious code in the tarballs that were not checked into Git.
The author demonstrates that Nix can be configured to generate the tarballs from git that go into building the binaries.
What I don't see, however, is how is this a feature that requires Nix or NixOS?
Any build system out there (including the stuff that goes into RPMs and Debs) can be configured to generate tarballs as a intermediate step.
In fact making reproducible builds is a major thing that Debian has been working on for some time now.
If we want to focus on a thing that NixOS could have prevented, we should focus on the CrowdStrike incident. Being able to boot to yesterday's config because today's config isn't working would've mitigated most of the problems.
Yes, if you use a trusted framework then you are safe from things until that framework is attacked. The xz backdoor might have been detected, but the xz backdoor wasn't crafted with the goal of working against the Nix ecosystem. When a nix core developer ends up being a spy or whatever then there will end up being an attack against the nix ecosystem. Don't reply to this with some claim that Nix is inherently secure unless you want me to track you down and make you admit you were wrong when Nix ends up getting successfully exploited in a year or two.
While NixOS goes a bit further with it, most other distrubtions also compile everything from source, cryptographically verify that the sources they use are not tampered with, and have versioned dependencies between packages. Debian also has reproducible builds.
The problem is just that the build systems did not strip pre-compiled object files before building from source. Even with that fixed, if nobody checks the source code then you can add all the backdoors you want, and there is nothing in NixOS or any other distro that would protect against that.
Excellent descriptive analysis. Wrong, misleading title, perhaps "technically correct," but at best with a "backdoored" meaning.
It points out the need and use for build-manager tools that go a step beyond union file system layers, but track then enforce that e.g. tests cannot pollute build artifacts. Take a causal trace graph of files affecting files, in the build process, make that trace graph explicit, and then build a way to enforce that graph, or report on deviations from previous trace graphs.
Yeah it certainly would have made hiding the backdoor more difficult. But far from impossible. You can always hide backdoors in source code if you want, it just takes more effort to make a plausible bug, and probably has a higher chance of detection.
If Jia Tan's PR was approved, malicious artifacts could go to github releases just as easily as in a tarball. Struggling to understand the point made about github releases being a security mitigation.
"Could have" means unproven here, and actually... They shipped it
> the release tarball being different than the source is
> the maintainer provided tarball was honestly generated from the original source code.
How, then? What about differing versions, etc. or has it been mentioned and I just missed it?
Just make sure the generated tarball can be generated from the source code itself, do not exclude anything, git add & commit everything. Can't we do that? We would still have to look at commit history in this case, I believe, and again, he said it himself, it was harmless to the naked eye, so even then, how could we verify? Maybe I don't understand what he meant by verification, but if maintained tarballs are generated from the owner's source code and is not on GitHub (or anywhere else, just a git repo), that is a problem in itself.
Of course there was more to it than just pushing poisoned test files, but still. I do not see how Nix would have prevented it, if the git repo has those test files and with seemingly harmless code (and is reproducible).
Perhaps what we can do is: if an (in)famous project has changed its main lead, then pay closer attention to the commits and check who it is? I don't know, TBH.
Did I misunderstand the article, or am I missing something?
Why is nobody questioning this:
> To build xz from sources, we need autoconf to generate the configure script. But autoconf has a dependency on xz!
Both directions of this seem crazy to me.
1. Why the heck should a build configuration tool like autoconf be unable to function without a compression tool like xz? That makes no sense on its face.
2. For that matter, why the heck should xz, a tool that is supposedly so fundamental, have a hard dependency on a boilerplate generator like autoconf?
At the end of the day all autoconf is doing is telling you how to invoke your compiler. You ought to have a way to do that without the tool, even if it produces a suboptimal binary. If you care about security, instead of taking a giant tarball you don't understand and the running another tool in it, shouldn't you just generate that command line somehow (even in an untrusted fashion), review it, and then use that human-verified script to bootstrap?
And if you need a (de)compressor that low on the dependency tree so that literally the entire world might one day rest on it, surely you can isolate the actual computation for bootstrapping purposes and just expose it with just the open/read/write/close syscalls as dependencies? Why do you need all the bells and whistles?
I think it's a vuln to think of OSS in terms of 'a community'. It's an abstract thought construct that does not represent reality (though it helps to make sense of it in a rather specific manner)
xz happened because of the absence of community.
It could happen inside this abstract thought of a community as well but here it did not.
xz targeted deb and rpm. The vast majority of what is facing the world.
Nix did not stop it.
I believe this article feeds the possible vuln rather than prevent it.
could have / should have => being smart retrospectively
This article suggests that Nix could have prevented xz backdoor, only to conclude that backdoor could be avoided by building from a git tag rather than source tarball.
This is true for every distro and it grinds me that Nix is even mentioned.
What's with all the baseless claims about NixOS lately? Is Sam Altman invested in it or something?
This kind of argument is like how Gentoo used to be better than anything else, because everyone knows how little piece is built, and here we are two decades later, who still is wasting nights compiling everything.
https://news.ycombinator.com/item?id=41486565 -> possible targets due to high impact and low maintainers
Learned a lot reading this article!
it's somehow immensely funny to me that some state probably had an entire project to land this backdoor in xz, spend literal years to make it happen. And then it was immediately detected and all effort was for nothing.
Is the massive number of spam messages on this thread an attempt to suppress the article / discussion around it?
I've not seen this many from multiple but evidently related green accounts before. Given the implications about nation state actors in play, it's tempting to jump to conclusions here.
I don't like how this site causes my headphones to crackle...
there's a disappointing amount of firm conviction here by people who are clueless about NixOS and/or the xz attack...