Item — HN Reader

Can someone provide context for this? I am not familiar with Debian and do not know the back-story.

Also, I assume this is not directly related to the recent resignation[0] of Heen. But does the resignation of two people (whose resignations were deemed worthy of being voted to near the top of the HN front page) within a few day span suggest concernening structural issues or is this just poisson statistics at work?

[0] https://news.ycombinator.com/item?id=8617874

3 comments

You missed some:

https://lwn.net/Articles/621895/ https://lwn.net/Articles/621003/ https://lwn.net/Articles/620879/ https://lwn.net/Articles/620878/ https://lwn.net/Articles/619749/

They are related in the sense that are all fall-out of the struggle to integrate or block systemd in Debian and/or how it exposed some problems with the Debian constitution.

It's a shame to see that so much talent is lost and frustrated over what in my eyes seems to be a small issue. systemd and Upstart are both a large improvement over the System V init scripts. So, it's quite logical that it should be replaced at some point. Since systemd seems to have the most traction in other major distributions and upstream software, it makes sense to make that the default, while making it still possible to switch to Upstart et al. if someone wants to (at the risk of losing support for some upstream software).

If it turns out in three years that systemd is a dead-end, rip it out, and use whatever is better then. Distributions did that with devfs, the old hotplug scripts, egcs (which was merged back in mainline), etc.

I get the impression which is admittedly informed by very cursory examination of the issues that ripping systemd out may be far more difficult than ripping say, Upstart. Systemd seems to have taken the approach of owning far more than just the Systemv init stuff and that removing it might entail sweeping changes to huge parts of a linux distribution. Systemd does seem to be pushing somewhat of a lock-in strategy and for many in the linux community that rubs them the wrong way.

Personally I don't really care much either way.

I get the impression which is admittedly informed by very cursory examination of the issues that ripping systemd out may be far more difficult than ripping say, Upstart.

Definitely. But this is probably true for many things at a low-level in the stack. E.g., it would also be hard to replace D-BUS or the Linux kernel (as evidenced by the fact that Debian/KFreeBSD will probably not be in Jessy).

I agree that it is a worry. But you can never make progress without taking any risks. We have tried 'fully isolated init' for years and it does not really work for modern systems/applications.

    We have tried 'fully isolated init' for years and it 
    does not really work for modern systems/applications.

I keep hearing this but I haven't seen any concrete reasons why it would be true. I'm curious what those reasons are. As I said I don't personally care that much as I work far enough up the stack to not be affected for the most part.

I think it's not a problem primarily with "init" itself, but for example that systemd provides you with the capability of starting a service "after network is up" so it needs to be able to tell "when network is up". That's much easier to do if there's a canonical (from systemds POV) way to tell "what's networks state". That again leads to "bah, let's do that ourselves" so there's now an optional systemd component that does this, that, this and people start relying on those - systemd-logind is a requirement for gnome. You could write a replacement for systemd-logind that offers the same functionality, but as long as nobody does, you're stuck with systemd.

The way I see it is that the systemd developers just plowed a way through the undergrowth and produced a suite of components that provide functionality that's appealing for people to build on. That's an offering that was accepted and now other people complain that the "systemd people are forcing their system on us and I want sysV init". The right course of action would be "meh, don't like systemd, I'll build a better offering." Talk is cheap, code is what counts.

I very much like systemd. It's documentation is actually long, but clear and well structured. The init component is much simpler than the mess of shell scripts used before. There certainly are problems and issues with it, but I've met problems and issues with about any kind of software I had to deal with so far (including my own), so I take that as a given.

Thanks for the detailed response. It makes a certain amount of sense.

    The right course of action would be "meh, don't like
    systemd, I'll build a better offering." Talk is cheap,
    code is what counts.

This statement seems to ignore Upstart though. Which is again from a very cursory examination exactly that. A better and more modular init without some of the complaints of systemd and yet is not gaining the mind share.

Why isn't it considered a viable alternative?

> systemd provides you with the capability of starting a service "after network is up" so it needs to be able to tell "when network is up"

That's not that hard once you have proper dependency-driven init, Gentoo's done it for a long time. You just need some way for the networking scripts to tell init that they're started now. Traditionally they'd do it by forking into the background once they're done starting just like any other init script. It's a bit harder if the network might not become available until after startup but not that much so.

Socket activation:

http://0pointer.de/blog/projects/socket-activation.html

Socket activated containers:

http://0pointer.de/blog/projects/socket-activated-containers...

Resource management:

http://0pointer.de/blog/projects/resources.html

Better service information:

http://0pointer.de/blog/projects/systemctl-journal.html

In fact, the whole series is a recommended reading. Anyway, all these features requirer tighter integration with other components than SysV init. E.g. Linux' cgroups, services that support socket activation, integration with the logging system for status information, etc.

Also, a lot of techniques in systemd were already pioneered in launchd on OS X since Tiger.

Why make apples and oranges comparisons to the kernel? It isn't that hard to replace sysvinit. Why should sysvinit's replacement regress in this area?

Because systemd is, although people argue this point, essentially monolithic. Sure, in theory it's modular, but the module dependencies are so intertwined that good luck using any of the modules without the entire thing - or writing so many compatibility shims that you might as just write the module yourself. And that's even assuming a compatibility shim can be written without essentially reimplementing the entire module. It prevents incremental replacements. And it keeps expanding to pull more and more essentially unrelated functionality into itself.

It's "you're stuck with this architecture unless you redo the whole darn thing". And, again, systemd keeps getting larger and larger. As you say, it isn't that hard to replace sysvinit. But to replace systemd you need to replace / write shims for far more things. (Your window manager? I wish I was joking...)

I'm starting to wonder if this is a problem with modern culture in general -- all over the place I'm seeing less "let's try our different approaches and see who has the best end result" and more "I will rape you to death with a flaming donkey for not running your project my way". It's like nobody wants to do any work to demonstrate that their way is superior, they just want to sit back and bitch about how the other way is inferior, until the other team gives up and our overall result is nothing :-/

(See also: both sides on pretty much any political issue in the past few years)

Both sides did lots of work to make their solution better and each side had a sizable list of legitimate advantages over the other. The ultimate decision came down to a subjective value judgement about which advantages were more important and inevitably wound up ignoring about half of those honestly held priorities (not to mention the work that went into Upstart). I don't see how it could have happened any other way.

It's sad that people resigned over it but the pain itself seems unavoidable from where I'm sitting.

> the pain itself seems unavoidable from where I'm sitting.

If there's enough time and energy in the community to be throwing death threats around, surely there is enough energy to maintain debian/systemd and debian/upstart as separate distros, both of which live happily ever after, sharing the 99.999% of code and infrastructure that they have in common? Then if one side really IS technically superior, people will naturally end up moving to that one and the other will die a gentle natural death, no personal attacks and resignations needed.

(For example, see what happened when Canonical wanted Debian to be more apple-like -- they didn't send death threats to Debian developers, they just did the work that they wanted to see done, and now both projects are a great success)

That was the Linux community of old. These days it seems it is the old hats that is required to fork if they want to maintain their tried and true ways rather than risk their systems to a massive switchover.

A better political analogy would be comparing the talking heads on propaganda TV channels vs the lobbyists and corporate donors who actually run the .gov.

There's some strife at both levels, the level of discourse is quite a bit different as is the level of publicity, and the level of actual impact.

You could be correct about modern culture and public non participant trolling culture. But there are more groups involved than just that (loud) group.

It all comes down to the fact that talk is cheap while concrete action is expensive.

"rape you to death with a flaming donkey"

Dude ...

Downvote all you want, but this expression needed a reaction !

Of course English being not my mother tongue I didn't have enough vocabulary to express my distress at the mental picture that exploded in my brain when I read those words.

I thought "dude" conveyed best my disgust, puzzlement and amazement :-)

I don't hate you though.

Systemd is seemingly deliberately designed to be difficult to replace. And it causes problems on some Debian platforms right now - as someone who was about to install Debian/kFreeBSD on a server, I am now very worried about the project's future.

Yep. right around the announcement of the GR there was a bug reported where installing one package would result in pam-systemd getting pulled in, and then it would cascade until sysv was replaced by systemd. I do wonder how many such bugbears there will be before there is a collective "fuck this" and just a blanket mandate of requiring systemd components everywhere...

It's not a problem to have systemd installed; on Debian it doesn't actually run as init unless you install the systemd-sysv package.

Debian has ports to FreeBSD and HURD; systemd doesn't support anything but Linux. I would assume that this would make systemd a non-starter in Debian (especially since there are no technical reasons systemd couldn't be ported).

There are also other issues. Yes, sysvinit doesn't parallelize scripts or do automatic dependencies. If that's all systemd fixed, I'm sure everyone would be for it. But systemd seems to be consuming everything in its path (logging, cron, etc), ignoring the well-proven and time-tested design tenet of "do one thing well".

The reason why sysvinit runs on FreeBSD and HURD kernels is that those kernels have implemented the Linux procfs file system.

https://teythoon.cryptobitch.de/posts/on-portability-of-init...

Besides, none of these are release architectures, so why should their limitations restrict what features >99.9% of Debian users have available?

I think Debian's supported parallel script execution and dependency resolution on top of sysvinit for a while anyway. It's not a true dependency-based init because the dependencies are resolved at configuration time rather than runtime, but whether that matters depends on what you're doing.

I look forward to seeing what these people will all create now they're not involved in Debian. It's not a good situation, but it does mean there's talent free to tinker with new ideas.

>Since systemd seems to have the most traction

Windows has even more traction, how about replacing all distributions with windows then?

How does another OS have anything to do with the need of replacing init system?

Systemd has been very good for me in Arch and in OpenSUSE for me. I feel like it is a much better tool for my needs.

>systemd and Upstart are both a large improvement over the System V init scripts.

Userland utils like wget depending on a specific init package is a large improvement over sysvinit? Nope.

How is wget depending on systemd?

So I had to look this up.

On Arch at least, wget depends on util-linux (package containing kernel tools like mount, dmesg etc.): https://www.archlinux.org/packages/extra/x86_64/wget/ And util-linux depends on systemd: https://www.archlinux.org/packages/core/x86_64/libutil-linux...

I have to assume this is Arch-specific, because I can't anything that basic as util-linux including a dependency on a specific init tool (and given there are still distros using other inits than systemd and still working fine...)

The systemd is a make dependency only. Unless you plan on compiling it yourself, you don't need systemd to run it.

    ~ $ y -Qi libutil-linux | grep Depends
    Depends On     : None

EDIT: conversely, these are the packages depending on systemd:

    ~ $ y -Qi systemd | grep Required | fmt
    Required By    : accountsservice  chromium  colord  crda  cups
    device-mapper  gnome-session  lib32-systemd  libgdm  libgusb  libpulse
    libusb  libvirt  libwacom  lvm2  media-player-info  mesa  mkinitcpio
    netctl  polkit  procps-ng  qt5-base  qtwebkit  rtkit  spotify  subversion
    systemd-sysvcompat  udisks2  upower  xf86-input-evdev  xf86-video-ati
    xf86-video-modesetting

As a sysadmin and developer I don't understand this systemd-hate. If you ever have written a /etc/init.d script for a python non-daemoning script running in a virtualenv, or something to the same effect, you'd notice the improvement over SysV init scripts.

Upstart is nice too, but as a Linux desktop user I love socket activation: I can have cups, avahi and other optional/non-critical daemons "active" but not actually using any resource, until effectively used by some other process.

That's probably because util-linux has a command, logger, which can optionally log to systemd's journal. Take a look at http://man7.org/linux/man-pages/man1/logger.1.html.

Probably accidentally, due to a poorly abstracted dependency chain.

Debian:

[wget]-depends-[libuuid1]-recommends-[uuid-runtime]-depends-[libsystemd0]

Ph’nglui mglw’nafh Cthulhu R’lyeh wgah’nagl fhtagn

Proposal: https://lwn.net/Articles/616571/

Vote: https://lists.debian.org/debian-devel/2014/11/msg00891.html

I get that there's a lot of politics in Debian. I also get that systemd is a controversial topic, but is this really a vote that just came out with the result that no vote was needed?

How many votes will there be about systemd in Debian that will turn out inconclusive?

Also, I have yet to see a single article of somebody running into all the various horror scenarios envisioned by adversaries of systemd. I mean: Multiple distros have already switched and there still aren't that many blog posts about systems dying in flames because of the switch.

I would so love for this discussion to happen based on technical merits and (by now actually available) real world experiences than FUD (on both sides)

I think this is the last vote related to systemd for a while.

There can only be a single default init system obviously, so a vote was required there. It was a hot topic, no need to rehash this.

The second vote, initiated by Ian Jackson, was controversial from the start. A lot of people were tired about the init topic and wanted to move on, and didn't want the GR. And they got the majority here: the result of the vote is "General Resolution is not required". This is the majority clearly expressing it's now time to move on. And to be honest I'm happy about this, although I'm just a Debian user and do not vote. With this, it's now unlikely to be another vote.

And this desire to move on can be now felt on all sides of the argument, hence the resignations and people moving to other things. I just want to point out that except for Joey Hess (really sorry about that) all are still remaining part of Debian. They just move out of the TC (or systemd maintainership) but are still DD, working on some parts of Debian.

And now, hopefully, everyone can cool off and move to other things too.

If adopting systemd actually broke everything, then of course they'd be doing something else instead. The anti-systemd crowd looks at this issue the way you might look at someone who took a working piece of software, and then did a completely correct transformation of all loops and conditionals to use bare gotos.

It's not incorrect to do that. Nothing crashes. It might actually even be faster or use less memory if you were sufficiently clever about it. However, the argument is that you just bought yourself a future full of pain and misery for not a big enough present gain.

Look at poor technical decisions from the past. The earliest browsers were encouraged to be liberal in what they accepted (an adaptation of Postel's law to HTML parsing). I think a lot of people today regard that as a decision that probably held the internet back by a decade -- think of the time we wasted writing compatibility layers and artificially restricting what our web pages could do. But it wasn't immediately obvious that that was the case. It took 10 or 15 years for the implications of that decision to really hit home.

What happens next is not a collapse of the world. It's a slow accrual of follow-up decisions that are needed to keep the lights on, until one day you look at the Eldritch horror you've had to build over 10 years and go, "Jesus, that was a bone-headed idea." Maybe they're completely wrong in opposing it, but systemd isn't anywhere near old enough to have decided that just yet.

Someone needs to dedicate an academic thesis to documenting how systemd advocates pulled off their coup, and what improvements are needed to OSS governance structures to prevent a repeat. By the time the pain becomes visible, no one will remember how the actual decision got made.

I don't think there's any vast conspiracy to understand or try to prevent. sysvinit has actual problems, and systemd solves many of them. I'm one of the ones who finds it ugly/inelegant/gross aesthetically, but it did address issues people wanted addressed.

And there wasn't a faultless alternative either. It's not like upstart or sysvinit was so good that you need to invent anything to explain why systemd won out. It won out because people either don't share my mild aversion to its design or because they chose functionality on Linux over elegance/portability/fill-in-your-own-thing. And hey, that's a Pareto-optimal choice as well.

A bunch of devs scratching itches, Just so happened that their scratching sticks where tired together and covered in super glue.

Yes definitely, and the thesis should discuss in detail the role of the Illuminati in getting systemd adopted, and provide a by-the-minute protocol of what went down at Area 51.

The sole voice of truth in the entire thread, down voted for telling an uncomfortable bit of it.

The outcome of the vote was something roughly like, "The General Resolution (GR) process is not the right venue to resolve this issue". The GR process has traditionally been used rarely, and mainly for "meta" things (deciding how Debian Developers are approved, that kind of thing). So it's not a huge surprise that sentiment against involving the GR process in the "init wars" was high.

All GRs also have a meta-option of "more discussion needed". Sort of a "no confidence" vote on all the candidates/options although in actual use unless it comes in first, when ranking it creates a border something like "absolutely positively no way". Technically the "no GR needed" people were abusing the system and should have just put "more discussion needed" as their highest rank. Its the same concept but this is a "funny" rephrasing of the same concept.

Perhaps the only thing everyone on all sides can agree on is funny, is that "further discussion" didn't get many votes. Somewhere last night I read a quote about there's hundreds on each side but only nine masochists who voted for "further discussion"

I keep seeing the odd one over at G+.

One i found some months back when the whole systemd issue got my attention was of someone that ended up doing a parallel arch install to read the journald logs.

Going from memory the problem seemed to be that he could not run the required command to get the logs dumped into readable form without systemd running as pid1. And systemd was refusing to boot for no apparent reason.

Thats the kind of thing that will get seasoned admins, that are used to cat being all you need to get at the logs, backing away with a "no way, no how".

This “binary logs!” thing is approaching FUD-like levels. “The logs” will not be binary. The logs will be where they have always been, in the format they have always been in. Sure, journald has some binary logs of its own, but it puts all the log messages also into the normal syslog daemon, so nothing changes.

As far as I can tell, the whole brouhaha started when there was a bug in the packaging of syslog-ng so it would not start properly after installing systemd. Since syslog-ng was not running, the only logs left were journald’s binary logs, and people assumed that “systemd ate my log files!”. No. It was a bug, the bug was not even in systemd, your log files will always be where you want them.

Unless you uninstall your syslog daemon, but then you are on your own.

Either I'm using journald to log to a binary format, or I'm having journald forward everything to syslog. If I'm trying to avoid binary logs in the first place, then there's really no point for journald to be running--just forward directly to syslog in the first place. Journald is just dead weight in this use-case.

Ian Jackson has been involved with Debian for 20 years and is not one to back down easily in an argument or quit over trivia.