At the Mountains of Madness

Nix, and NixOS, are designed for those of us who have to clean 10,000 proverbial litter boxes every day. I use Nix fairly extensively at work; I use it very little at home, where I don't need to worry about what dependency someone took on a specific version of Python five years ago, etc.

It's like k8s, imo - it solves some real problems at scale but is rarely going to be a good idea for individual users.

You are an excellent writer.

I have not use Nix, but I have had a similar bad experience with a similar but different automatic litter box. Your “small predictable chore” point is spot on: how much human grief is created by elaborate, expensive, unreliable solutions to minor annoyances?

If you enjoyed that litter box experience you should consider owning an older high mileage and high end German luxury car… ideally something with a double digit number of cylinders, and a computer controlled adjustable… everything

A lot of Linux users are tinkerers. In my experience, Nix provides almost an endless sink for tinkering. There are escape hatches to "just make stuff work" but it becomes a fun challenge to do it the nix way.

This also describes DIY home automation for me. I spend hours setting up some fancy automation routines that work great. Then, a few months later something fails and I spend more time fixing it than the automation affords me in the first place.

In my experience it kind of just works. Takes a bit more maintenance than arch, around an hour a month at most. I had to reinstall it once and it was way easier than my arch, since way more stuff is in text files I can commit.

Also, since it's very very easy to rollback to a previous version, managing unpredictable issues is easy. I have a colleague that lost lots of time to an arch kernel panic during an update that required a reinstall. On NixOS I can reboot, choose the last derivation, work and fix that if/when I want.

What you’re describing is called the law of conservation of misery. Misery is like a water-filled party balloon. If you squeeze it in one place it expands in another, often between your fingers.

And then some Go fanatic will come along and use this experience to explain why it makes perfect sense to static link bits of the standard library into every single binary ever produced, and dynamic linking was the worst idea since mankind's ancestors crawled out of the ocean and past the whales crawling back in.

Why am I sensing systemd vibes from your message lol

I think it solves a problem that shouldn't exist: managing complex state in an operating system.

OSes should be mostly immutable. Apps should own their state. Everything else should be in a neat tidy box that is portable. Mobile almost gets this right.

The idea of installing things "on" the OS needs to die, badly. It's a security and privacy nightmare because it means everything more or less has root, and it makes every OS install a special snowflake that is under the hood a giant mixed pile of shit.

> Every time I look at NixOS, I think that it perfectly solves a problem that I only have once every 5 years, when buying a new computer. I think I even looked into it once to automate that exact process, but that idea fell apart at the first line of Nix syntax. I'll stick with OSX and `brew bundle` I guess...

To each their own!

With two Mac laptops, each with a Linux VM, plus five Raspberry Pi and a Mac Mini under Asahi on NixOS it's been a godsend to have a consistent management system and setup with reusable bricks, that is also able to remote build on the Mini for the Pis.

That plus shell.nix and direnv, and you can pry Nix from my cold, dead hands.

Some generators like Automatic1111 embed the prompt in the image metadata.

I’m using Nix for development and generally I agree.

The first catch is that I want to be able to update my system on a regular basis, and keep using exactly the same dependencies in my project after an update. Maybe I’m in the middle of working on a change.

The second catch is that sometimes my development environment is really weird, and the packages I need aren’t in Debian. At least, not the versions I want. Nix can handle cross-compilation environments and you can use it for embedded development. You stick your entire development toolchain (arm-none-eabi-gcc, whatever) inside your development environment.

> Why would I want to roll a system back to an (definitely insecure) state of a few months ago?

Periodically, I want to update everything in my development environment to the latest version of everything. Sometimes, something will break. Maybe a new version of GCC reveals previously undiscovered bugs in my code. Maybe a function gets removed from a library (I’ve seen it happen). In Nix, it’s pretty easy to pin my entire development environment to an old version, while I’m still updating the rest of my system. I can also get the same environment on either Linux or macOS with relatively minimal hassle (with the note that I’ve run into several packages that just don’t run on macOS, which required me to make “fixed” versions).

Also keep in mind when I say “Nix”, I’m talking about nixpkgs. I’m not using NixOS and I just don’t care about NixOS.

Nix also has its pain points. I think of it as being like a coarse-grained Bazel with a ton of packages.

I run into DLL hell any time I try to install some software that isn't in some sort of package repository or other happy path. Most recent example I can think of was about a year ago when I helped my father-in-law install Klipper on an RPi4 so that he could do input shaping on his 3d prints. All of the guides and documentation seemed to assume that you were using a specific version of Linux on the RPi, and if you weren't (like my FIL) then welcome to dependency hell. Took several hours of pulling my hair out to resolve them all, for a non-developer it would have been impossible.

You don't need to run bleeding edge versions unless you feel like it; there's a stable release with rolling security patches every 6 months (current is 24.05, next will be 24.11).

You don't need to keep multiple copies of each library - but you _can_ when you find out that an update broke something you care about while still updating everything else on your system. You aren't rolling back your entire system state, just the...light-cone of the one tool that has issues.

The problem with SO numbers is that your Python/Ruby/Java/NodeJS packaging and tooling doesn't respect that at all. If you can satisfy all of your dependencies using the Debian-maintained repositories great! When you can't, Nix provides a harm-reduction framework.

Nix also makes certain hard things trivial - like duplicating the exact system state that someone else used to build a thing some months/years ago, or undoing the equivalent of a `dist-upgrade` gone awry.

> And then when there's a security problem, who goes and checks that every version of every dependency of every application has actually been patched and updated?

The nixpkgs maintainers, same as the Debian maintainers. Repology's down right now but nixpkgs seems to do quite well on a CVE level.

> Why would I want to roll a system back to an (definitely insecure) state of a few months ago?

Insecure is sometimes preferable to down. Being able to inspect an older/insecure state with new/secure tools is neat.

> I have many of the same questions about Snap and even Docker.

Snap and Docker solve similar problems that most people don't have. Same with k8s. You might just not have these problems - I have a screwdriver on my desk that's specifically for opening up GameCube consoles (so it's longer than the one I use to open up N64 cartridges, even though it's the same shape); unless you have that specific need, it'd be completely pointless in your toolbox and cause you trouble every time you tried to use it.

> I've been using Debian/Ubuntu for 15+ years and never really experienced dependency hell.

Debian (or derivatives) too here and once in a very rare while I encounter some dependency hell.

IIRC the last problematic one was trying to add JPEG XL support to Emacs. Emacs uses ImageMagick under the hood to display pictures (for example from image-dired) if I'm not mistaken. But the version of ImageMagick shipped with the latest Debian stable (Bookworm) doesn't support JPEG XL yet.

Something like that.

It does happen but I do agree that it's highly uncommon.

> Why would I want to roll a system back to an (definitely insecure) state of a few months ago?

A just question!

Nix does have tools for running stuff in an FHS container. Something I have considered but not yet attempted is to use this to wrap the build such that building the binary happens in the FHS container (using the unwrapped versions of the compiler and associated tooling).

Yeah I wish the ld-linux.so interpreter would actually indicate when a file it's trying to link wasn't found. Something like "unable to locate shared object BLAH" would go a long way. It's like a rite of passage the first time you debug something like that.

I've only encountered this once and I've not forgotten it.

Downloaded a release of some binary for Linux, but I'd downloaded the FreeBSD built binary. Was lost until I explored in the same fashion as the author.

I would like to half-seriously recommend that you overwrite a different character than the first when mangling the environment variable name. Specifically one beyond the third, so as to stay within the LD_ namespace (not yours exactly, but at least easier to keep track of and more justifiably excludable from random applications) and deny someone ten years from now the exciting journey of figuring out why their MD_PRELOAD environment variable is overwitten with garbage on some systems. How do you feel about LD_PRELOAF?

Also it's probably better to leave LD_PRELOAD properly unset rather than just null if it was unset before; in particular I wonder if empty-but-set might still trip some software's “someone is playing tricks” alarms.

There are probably other ways this is less than robust…

(hi, I kind of have a Thing for GNU and Linux innards sometimes)

As an aside, a lot of people don't know about ldd, and introducing it to them is very cool, but it should almost always come with a warning - maybe add a note that people should be careful with ldd - ldd _may_ execute arbitrary code. This is in the ldd man page, but most people never read documentation. It is unsafe to use on any binary you're not otherwise believed safe.

> This minimal meta-loader will totally work if you invoke it directly like `$ meta_loader.sh foo`, and it will totally not work if you hardcode its path (or a symlink to it) in the ELF headers of a binary.

why not have `foo` be a shell script which invokes the meta loader on the "real" foo? like:

``` #!/bin/sh # file: /bin/foo

# invoke the real "foo" (renamed e.g. ".foo-wrapped" or "/libexec/foo" or anything else easy for the loader to locate but unlikely to be invoked accidentally) exec meta_loader.sh .foo-wrapped "$@" ```

it's a common enough idiom that nixpkgs provides the `wrapProgram` function to generate these kinds of wrapper scripts during your build: even with an option to build a statically-linked binary wrapper instead of a shell-script wrapper (`makeBinaryWrapper`).

No questions about Madness, but I really enjoyed the article tone and playfulness. Thank you.

Love it. I came to this same insight about nix and containers being two approaches to a dynamic linking work around, but via a different path, of building my own little container runtime.

Feels like we are building things who's original purpose is now holding us back, but path dependence leaves us stuck wrapping abstractions in other abstractions.

Why did you want to use Nix to make impure binaries for other distros? Much of the appeal of it for me is using it to distribute software in a more reliably portable way, but that of course always means shipping a whole chunk of /nix/store, once way or another.

What made your team/company want to use Nix to build binaries and then strip them down for old-fashioned, dependency hell-ish distribution? Why not install Nix on your target systems or use Nix bundle, generate containers, etc.?

It's a shame that DLL hell was never resolved in the obvious way: deduplication of identical libraries through cryptographic hashes. Containers basically threw away any hope of sharing the bytes on disk - and more importantly _in ram_. Disk bytes are cheap, ram bytes are not, let alone TLB space, branch predictor context, and so on.

There was a middle ground possible at one point where containers actually were packaged with all of their dependencies, but a container installer would fragment this assembly into cryptographically verifiable share dependencies, but we lost that because it was hard.

I'd like to express that while this article is WAY outside my wheelhouse, but I liked the writing style and the AI illustrations felt like they were emotionally additive to the section rather just a distraction (to me). Also, my head hurts trying to still understand this cursed thing: https://github.com/antithesishq/madness/blob/main/pkgs/madne...

Somehow your website has semi-broken scrolling, which is impressive since that normally only happens when Javascript is enabled.

Also, please QEFS (quote every string) in your shell script fragments.

Also not a question, just want to say that "crt glow" (or maybe "Cerenkov glow"?) effect upon hovering over a link is awesome.

Hi Will, I'm curious what your thoughts are about the Nix uniqueness problem, and the characterization of failures, or lack thereof under undefined behavior's failure domains. Exception handling generally requires a defined and deterministic state which can't be guaranteed given design choices to resolve DLL hell under Nix (i.e. its a stochastic process).

I mention this since it is a similar form of the problem you mention in writing this piece of software, that can lead to madness.

Also, operationally, the troubleshooting problem-space of keeping things running, segments nicely into deterministic and non-deterministic regions; which the latter ends up costing orders of magnitude more as a function of time to resolve since you can't perturb individual subsystems to test for correct function, without determinism and time in-variance (as system's properties), testing piecemeal has contradictions in stochastic processes.

Hashing by rigorous definition is non-unique (i.e. its like navigating a circle), and there is no proof of uniformity. So problems in this space would be in the latter region.

While, there are heuristics from cryptography that suggest using factional cubic roots to initialize the fields brings more uniformity to the examined space than not, there is no proof of such.

When building resilient systems, engineers often try to remove any brittle features that promote failures.

Interestingly, as a side note, ldd output injects non-determinism into the pipe by flattening empty columns non-deterministically (i.e. if you ldd ssh client, you'll see the null state for each input to output has more than a single meaning/edge on the traversal depending on object type, this ends up violating the 1:1 unique input-output state graph/map required for determinism as a property, though it won't be evident until you run use it as an input that problematically maps later in automation (i.e. grepping the output with RegEx will silently fail, providing what looks like legitimate output if one doesn't look too closely).

PaX ended up forking the project with the fix, because the maintainers refused to admit the problem (reported 2016, forked in 2018), the bug remains in all current versions of ldd (to my knowledge).

While based in theory, these types of problems crop up everywhere in computation and few seem to recognize them.

Working with system's properties, and whether they are preserved; informs on whether the system can be safely and consistently used in later automated processes, as well as maintained at cheap cost.

Businesses generally need a supportable and defensible infrastructure.

Been there, done that. In my case, I symlinked myself out of this mess rather than modify ELF.

Dig the purple anteater pix.

Static linking essentially freezes not only the ABI to the kernel, but many implementation details of the linked libraries as well. Including, for example, how library client code talks to daemons, or formats of files directly read in by the libraries. The timezone configuration would be one instance, or things related to NSS.

It's really not viable in a lot of cases, unless you like rebuilding (or at least relinking) with every system software update.

And then there's of course the memory savings. macOS and iOS for example have giant "shared caches" which are mapped into all processes and comprise of all the system libraries. (Other OSs often do this on the individual shared library level.) With static linking, you'd instead have many copies of lots of potentially-but-not-necessarily identical library code pages in DRAM.

IIRC the underlying implementation may be different on other systems. I think in particular, DNS resolution.

Linux is the only system where static linking all the way really makes any sense. For most systems, you don’t get a stable syscall ABI. Instead, you get a stable ABI to the library which does syscalls for you… Windows has kernel32.dll, macOS has libSystem.

Note that on Linux, the vDSO is dynamically linked.

Compilation speed is a big plus. For large projects, linking time can easily dominate the time needed for incremental rebuilds.

Due to tooling issues, PIE is a lot easier with dynamic linking, and this gives you better ASLR. These issues are solvable, but it’s a lot easier to get PIE if you use dynamic linking. If you want static PIE, you need to compile all your static libs as PIE—doable, but you don’t get it out of the box.

Biggest advantages I know of for dynamic linking:

* You can use the LD_PRELOAD trick to override behavior at runtime.

* You can run with entirely different implementations of the dynamically linked library in different places.

* Software can pick up interface-compatible upgrades to its dependencies without being re-compiled and distributed again.

We use all three of these tricks in our SDKs, FWIW. But it is still a giant pain in the ass.