back
160 comments
I find it fascinating that, from a security standpoint, it's best for the static representation of a program (the executable and the package) to be perfectly predictable, while the dynamic representation of a program (once it's loaded into memory and running) should be as unpredictable as possible while maintaining correctness (using ASLR and other methods to slow down attacks on vulnerable code). I guess running a program is like setting off an explosive: you need to know exactly what you're handling before you release its power.

Anyway, many kudos for this work!

What do you mean by "as unpredictable as possible"?

From the perspective of an application developer I want that dynamic representation to also be predictable.

You are probably talking about different kinds of representations.

hathawsh meant representation in the sense of memory layout and other low-level details of how application state is encoded. You seem to be talking about the programmer-visible abstractions built out of that layer. Behaviour at that level has to be predictable in order to "maintain correctness", but predictability at the lower level is mostly a help maintaining for the correctness of malware.

For someone who's found a way to inject data into your app, you don't want them to know the address of your symbols and data structures.
Wow, I'm impressed by the consistent progress.

There is so many things based on Debian and adding reproducibility in Debian is a massive security improvement.

I have a lot of respect for people working on this, it must be hard to respect release freezes when you've been working on this for years.

Slightly fluid numbers aside, half the packages for Buster still sounds like good progress to me.
It's certainly the right direction! I applaud them for pushing this.

From the article, it looks like many of the others are reproducible on a code level, but the release system is using older binaries, which haven't been rebuilt yet.

It's too bad it won't be fixed in time for this release, but bodes very well for future releases.

What's the importance of reproducible builds?

Edit: found this nice overview by clicking a few links from the original post https://reproducible-builds.org/

It allows you to verify that the source code was unaltered when the original build was produced. (By building it a second time with known good code.)
One thing that gets me excited about reproducible build systems in general, which has less to do with trust and verifiability, is cacheability/content addressable storage of build artifacts.
Its extremely valuable for debugging. When you've got a bug that is in a specific build you can collect the set of dependencies that produced that build and BAM you've got an env that should replicate the bug.
one thing that I think got lost culturally in the move to linux was the emphasis on open source. yes, not open source in some abstract warm fuzzy way, but actual useful open source.

if I cant run the build process, and I dont really know which patch sets have been applied on my system, and which version of the source was used, and I dont really have a handle on the dependencies. doesn't make it substantially more difficult to debug and work with the resulting system?

its really quite odd that the dominant open source platform settled on opaque and unreproducible binaries as the distribution mechanism early on.

Not much. Mostly a fad. Repo security plus secure transport handles most of the risk there. Whereas, most black hats will use bad configurations, social engineering, or app/OS vulnerabilities. There's enough of those that vulnerability brokers aren't paying high for hitting that OS.

So, best use of time would be one of three activities:

1. Gradually rewriting apps or kernels in safe languages. Can even make stuff reproducible as a side effect of rewrites.

2. Improving tools like SAFEcode snd Softbound+CETS that make C code safe. Esp usability and packaging. If not that, then more mitigations in OS like OpenBSD does. Can even port theirs.

3. Running static analyzers, fuzzers, etc on all that code out there. Then, fixing the vulnerabilities. Id say prioritize highly-priveleged and newest code.

That would actually improve the security of Debian against the kind of attacks that hit it the most. There's some people doing this. Not enough by far.

BTW I was just looking at diffoscope - developed as part of the "reproducible builds" Debian project.

https://try.diffoscope.org/

We need a body/service that verifies that the code-binary combo being distributed by a developer is actually reproducible. Because, most users don't have the time or resources to verify each package they use. But its useful to society if distributed packages are reproducible. So a trusted third party should provide this service.

I would envisage it working in the following way. The developer submits links to their code and links to their package. The service builds from code and checks if the final package matches what is being distributed by the developer. If yes, it then publishes the hashes of the code and package, so users can quickly check that they are using a reproducibly built package from the correct source.

This is being actively worked on and already has had quite a bit of work. This project is also submitted to Google summer of code under the Debian project.

https://ssl.engineering.nyu.edu/blog/2019-01-18-in-toto-pari...

https://salsa.debian.org/reproducible-builds/debian-rebuilde...

https://github.com/in-toto/apt-transport-in-toto

https://reproducible-builds.org/docs/sharing-certifications/

The way F-Droid (android app distribution service) does this is all builds are done server-side (devs only update the publicly available source) and then anyone can quickly setup a verification server that builds+diffs the packages https://verification.f-droid.org
For those of us unfamiliar with the term "reproducible", yet in the business of software - what does it mean and why is it desirable or needed?
If you build the package correctly, you get the exact same binary.

This allows users to verify that the binary packages you see were actually gotten by compiling the source code (presuming your compiler isn't compromised). This means that auditing the source-code is almost as good as auditing the package (the exception being compiler mistakes). Thus, we need less trust in the software packagers.

Just wanted to add that besides the security gains there's a real performance gain. If you've got a slow-to-build codebase where you've made a small modification you can use a remote artifact cache so that all the parts that don't change just get downloaded from when someone built it. Then you have faster incremental compilation.

You can do this with Bazel and Buck and with a few assumptions and some configuration even with Gradle.

By definition, for open-source software, you can compile it yourself. But most users of most distros don't do that, instead they use pre-built binary packages.

Which means someone has to build and upload those binary packages.

How can you tell that the person who built or hosts the binary packages didn't change the source code (for example, putting in backdoors or other malware)?

It helps with this problem if anyone else can build the same source code, and get a byte-for-byte identical copy of the binary package.

This sounds trivial, but it actually requires some dedicated support from build tools and build scripts. (You have to have the exact same version of the compiler and everything else, the compiler wants to automatically put the build timestamp in the executable, the order the OS lists files in a directory can sometimes change, the timestamps in files contained in tar/zip archives can't be set to the current time, etc.)

Some people in the Debian community have been making an effort to update all of Debian's packages to be byte-for-byte reproducible.

This seems to be the issue https://issues.apache.org/jira/browse/MNG-6276 for reproducible builds with Maven. I'm not sure what the current status is given that all the issue links are closed.

Go in and vote for it anyway. I'm sure some of the Debian packages are written in Java. ;-)

I definitely understand the merit of making it super easy to verify, by simply hashing the build ISO. But that also shifts the burden of forcing deterministic results to the build process. I wonder if for a little more complexity in verification, if a lot less complexity would be needed on the build side?

For example the ISO file system, typically ISO 9660 or UDF, will have a volume UUID from a random number. Sure you can code a flag for mkfs to specify a fixed number, that's easy. But then next that ISO typically contains a payload in the form of a squashfs file. And quite a bit of work on squashfs has happened to make sure file timestamps can be set to a known value. However, if one build process uses xz level 3 compression, and another build process uses xz level 7 compression - hashing will of course fail. The point is, be it inode UUIDs and timestamps and compression levels, there's a lot being measured that we don't really care about, just to have a simple verification method at the back end.

Can anyone recommend a good way to install/maintain/remove new software on distros like Debian or Ubuntu LTS? (By 'new' I mean versions newer than the ones available through apt).

I never figured out a maintainable scheme. Some of my packages are in /opt, some are in ~, some in ~/local, and some under GNU stow. And other than GNU stow's half-assed uninstall method, it's always a pain in the neck when I have to remove or upgrade any of these packages.

One thing that works many times is to download the deb source package from a newer distro and rebuild it in the desired system.

If there’s no newer package available you can try to reuse the debian directory from the older source package to build from the newer source.

Sometimes this works in the first attempt, but if you need this for a lot of packages, it’ll be a lot of work.

Why do you find GNU stow's uninstall method to be half-assed? Just curious.
Helps having Google as a client wanting reproducible builds.
Eli5?
One has to step back and ask: "What made you decide to introduce non-determinism into your compilation process in the first place?"
Nobody sat down and said "yes, I'll make this nondeterministic". If you look at the wiki page's sampling of different issues -- https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_... -- you'll see it's a mix of various things:

* output is deterministic but dependent on some aspect of the build environment (locale, hostname, etc)

* output is accidentally non-deterministic (eg "we put all the .html files into a tarball with a shell glob pattern, which gets you an order dependent on your filesystem implementation and the phase of the moon")

* a wide array of "output contains a timestamp" issues

"Let's put the timestamp into our version string/a generated file so the user knows when it was built" is a really common thing, and it seemed like purely a nice convenience feature until the concept of 100% binary-reproducible builds became a current concern.

You don't really 'introduce non-determinism'.

That's like saying, "why did you introduce entropy?" It's there. It happens, and it's difficult to get rid of in any complex system.

https://en.wikipedia.org/wiki/Reproducible_builds

https://wiki.debian.org/ReproducibleBuilds

https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html

Debian was one of the first large distributions to use a build farm for many different architectures. Maintaining deterministic builds (even as far as just file timestamps) in a distributed environment is very challenging.

If you aren't thinking about reproducibility things like putting timestamps in the binary seem reasonable. Which is what lots of tools did. Hence the problem.
This is true and it's also the model I use to avoid bugs. Any time I'd add a bug to the codebase, I just look at it and decide not to write the bug. In this way, no bugs are ever introduced.
What software currently has 100% reproducible builds?
What software? Like, individual packages? Many of them - here's the ones that do so on Arch: https://tests.reproducible-builds.org/archlinux/archlinux.ht...

If you mean which distribution has 100% of its packages reproducible, probably none yet. But Arch and Debian are both making progress.

Package management-wise, Nix.

OS-wise, NixOS.

Build system-wise, there are lots of options: Blaze, Buck, Pants, Please (AFAIK)

Any program with a build system designed in such way it doesn't introduce anything beyond the source code into the binary should be.

If you build the same code on two different machines, using the same compiler, with the same options, then the generated binaries should be exactly the same.

The vast majority of code written at Google does, for one.
GuixSD, NixOS
I believe Solaris is as well.
My experience at Dave & Buster's is around 80% reproducible, so I think progress is possible.
FYI, Debian codenames are all based off Toy Story characters. See https://unix.stackexchange.com/questions/222394/linux-debian...
Might be good progress but it still sounds very low to me as I didn't know anything below 100% was possible... it sounds crazy to me (almost like something that was introduced to be able to inject backdoors undetected).
Lots of problems come from things like timestamps, or race conditions in concurrent build systems giving slightly different bytes on disk. These generally aren't "trusting trust" level problems, since they do not and cannot affect program behaviour; but they do screw up things like digital signing, cryptographic hashes, etc. which are useful for automatically verifying that self-built artefacts are the same as distro-provided ones.

These problems can also cascade, if component A embeds the hash of another component B, e.g. to verify that it's been given a correct version. If that hash comes from an unreproducible upstream, and building it ourselves gives a different hash, then we'll need to alter component A to use that new hash. That, in turn, changes the hash of component A, which might be referenced in some other component C, and so on.

Nope, loads of build tools were never built with reproducibility in mind.

Look at windows. Even if you fix the compiler and linker, you still non-reproducibility by design, the PE header contains a timestamp.

People also like to stick non-reproducible stuff into builds directly, like timestamps.

Compilers don't have any reason to lay down data in a specific order, so if they are threaded in the backend they just don't.

IDL tools might stick in the timestamp of when a file was generated, for convenience.

and on and on and on.

Every significant project I've worked on embedded the build host and build time in the resulting executable or firmware image. This was along with other static build information, like version number, compiler version and build flags.

Once you make the sensible choice to include build time in the result you've broken reproducibility. Fixing this means tracking down every package that does this and removing the timestamp.

My guess is that making code reproducible involves some kind of change that hasn’t been applied or all of the code or build files.