It took a while though until this was understood. In 2007 when pointing out on debian-devel that this is needed, I was still told what huge waste of time this would be. And indeed it took a huge amount of work by many people to get there, but it is well worth it.
"Well worth it" is not correct. And it just ups the the contribution barrier to Debian higher, I already heard a lot of people complaining that contributing to Debian is hard and while in past I defended it by "they need all the checks and bounds to make sure packages play with eachother nicely", this is just step that makes it hard for no reason and little benefit.
https://reproducible-builds.org/
Could you perhaps respond to the argumentation here?
Another thing to consider is that Debian has quite a few derivatives who may also rebuild packages from source, so you have a multiplier there.
Forcing devs to use hardware keys to sign commits/CI requests would be actual security improvement, thwarting many supply chain attacks that only worked coz the attacker got to developer credentials. Hardware keys at least have option to make some operations require physically pressing the key so there is chance developer will notice.
But thanks to reproducible builds, at least someone can... validate that the binary code of vulnerable package can be reproduced. Very fucking useful.
I am not saying it is useless. I am saying it is one of highest hanging fruits on security tree.
With reproducible builds, you do not need to trust that the system that build the binary was not compromised, because this would be detected immediately.
Source compromises are still an issue, but there is a much bigger change that they are detected. Also if there is a compromise, reproducible builds allow you to later track it to the source. For an infected binary it is much more difficult to understand how it got there and what else might be compromised.
> Also if there is a compromise, reproducible builds allow you to later track it to the source.
They do not. Git log and build logs allow for that.
Reproductive builds only have value after the source. They protect build servers from being compromised (and then only if some other uncompromised environment is also running verification passes), if the bug is at source reproductive builds are exactly as valuable as writing commit that was used for build in app's code/package metadata.
Not being reproducible is a bug.
There is no reason for a build to not be reproducible, but somehow we let the built binaries become infested with timestamps, login names and file system paths. We recently moved to reproducible builds at work, and discovered that our login names and local home directory paths were being shipped in every release. No one was was very happy about leaking PPI like that.
You may not consider it worth the effort, but you aren't the one putting in the effort so I'm not sure why that matters to you. It is very much worth the effort to those people doing the work. Debian is a do-orcracy and so the people doing the work get to make the decisions.
Anyone having to maintain a code base or a distributed fleet of devices will gain from this decision, immensely, as their operational periods come and go.
Reproducible builds are about longevity as much as they are about security.
Please don’t make bold claims about ‘no reason and little benefit’ while demonstrating ignorance of this hard fact: reproducible builds should have been the norm, in computing, from the get-go.
Why not? Debian has a fantastic track record of providing old versions, for instance here's the build tools from Debian 2.0 from 1998:
https://mirrors.accretive-networks.net/debian-archive/debian...
Just baking in build ID and commit is enough. What you think reproducible builds add ?
> Please don’t make bold claims about ‘no reason and little benefit’ while demonstrating ignorance of this hard fact: reproducible builds should have been the norm, in computing, from the get-go.
So far not a single person in the thread gave me concrete example (as in existing project, existing problem, no other solution can solve it). Just claiming it's better based on their feelings. Come on, be the first one.
Think industrial applications, such as rail and heavy industry transportation. We use reproducible builds here as part of a wider safety-critical protocol which guarantees that what we are running is what we expect to run - nothing more, nothing less.
Reproducible builds are certifiable. They can be relied on in environments where certification costs millions and takes years.
Think outside your consumer box for a minute.
Have many organizations produce the binaries independently and post the arifacts.
Once n of m parties agree on the arifact hash, take that as the trusted build.
If every party reaches a different hash then we cannot build consensus.
Obviously, it would be a ton of work to make such a system resistant to gaming by malicious actors (see GNU Guix for useful efforts), but it would provide valuable diversity in architecture and (political or other) control.
It would be even cooler if we could have independent projects that could run on various distros and OS, and build packages for any of them. Having packages for bsd verified on linux and vice-versa with statistical logging (this code has been verified x times on y OSes) would be reassuring.
(It was caught before being promoted into a stable Debian release, yes, but this sort of relied on a happy accident, too close for comfort)
Still, lots of good non-security benefits to reproducible builds too.
The backdoor relied first on a difference between building a package in a packaging environment versus building the package on your own. And also, it relied on the very common practice of checking in unreviewable artifacts into the source tree (e.g., the configure script, malicious binary test artifacts).
Reproducible builds guarantee that two people can follow the same instructions and get the same, bit-identical outcome. It does nothing to guarantee that those instructions have not been compromised, and all of the great packaging security failures of my lifetime that I can think of have relied on those instructions being compromised (e.g., xz utils, Debian OpenSSL keygen issues).
At the time of xz attack, the package was already reproducible.
Similarly for package managers, the biggest issues are typo-squatting or maintainer credentials compromise. And in neither case does the attacker have any incentive to take advantage of it in a way that breaks reproducibility--they can be completely honest about what they're doing. Now even if I were an attacker who had compromised a maintainer's machine, I'd still probably go for compromising the source rather than compromising the final artifact-generation process... simply because compromising the source makes the exploit live longer.
As xz shows, once you have a compromised maintainer, there's basically nothing you can do to fix it except by having someone else discover the compromise and locking out the maintainer.
Defense in depth obviously is a good thing
Like what exactly?
If anything it will make attacker's job easier, as Ubuntu package will have same files structured exactly same way as Debian one.
As opposed to what? If Ubuntu uses the same source, of course they get the same binaries. And if Ubuntu applies patches, they'll get something different. And that's still true.
Those people do not care about quality in opensource at all. For longliving software this is very important.
Of course, all those javascript and kubernetes packages which are irrelevant in a few years again, might complain, but let them complain.
I'm reading this as a suggestion that the reproducible builds effort was an ineffective deterrent.
However, note that your observation could also be explained by the opposite: the reproducible builds effort was an effective deterrent, so nobody bothered with attempts.
> And it just ups the the contribution barrier to Debian higher
Until yesterday, the package just got flagged in the tracker, and you could either ignore it, or fix it yourself, or the kind people behind the reproducible builds effort supplied a patch themselves.
Now, you can no longer ignore it. But fixes are often trivial. Use a (stable) timestamp provided by the build, seed RNGs with some constant (instead of eg: time), etc. These are best practices anyway.
There was no attack that reproducible builds would help protect from before 2007 either.
> Until yesterday, the package just got flagged in the tracker, and you could either ignore it, or fix it yourself, or the kind people behind the reproducible builds effort supplied a patch themselves.
> Now, you can no longer ignore it. But fixes are often trivial. Use a (stable) timestamp provided by the build, seed RNGs with some constant (instead of eg: time), etc.
that's the entirety of the problem. App developers don't want to be package experts or build experts.
> These are best practices anyway.
They are not. They are best practices if you want reproducible builds. They are entirely useless waste of time if you don't care.
App developers and Debian package maintainers are already separate groups.
Or if you're writing a test suite, and you want failing test results to be actionable.
Or you have any other type of behavior that you'd like to reproduce somehow.
One of the first things app developers ask for in bug/issue templates are the steps to reproduce something. I wonder why you'd think that they would suddenly be opposed to the concept when thinking of a build peocess.
The class of bugs would be extremely small as the stuff that makes build hard to reproduce are 99% of the time stuff irrelevant to runtime like some build time embedded in binary, some file metadata having different timestamp, or maybe linker putting stuff in a bit different order.
> One of the first things app developers ask for in bug/issue templates are the steps to reproduce something. I wonder why you'd think that they would suddenly be opposed to the concept when thinking of a build peocess.
I think you will find amount of people that had problems reproducing because of non-100% exact build is vanishingly small, possibly non-existent.
And that is because if you get package version and want to reproduce it, you get the package, install it and try to reproduce it. The package WILL be 100% the same as the one you got in bug report because you both downloaded the same artifact from same mirror network. You don't need reproducibility to get same binary to reproduce bug