Replacing Kubernetes with systemd (2024)

I share the author's sentiment completely. At my day job, I manage multiple Kubernetes clusters running dozens of microservices with relative ease. However, for my hobby projects—which generate no revenue and thus have minimal budgets—I find myself in a frustrating position: desperately wanting to use Kubernetes but unable to due to its resource requirements. Kubernetes is simply too resource-intensive to run on a $10/month VPS with just 1 shared vCPU and 2GB of RAM.

This limitation creates numerous headaches. Instead of Deployments, I'm stuck with manual docker compose up/down commands over SSH. Rather than using Ingress, I have to rely on Traefik's container discovery functionality. Recently, I even wrote a small script to manage crontab idempotently because I can't use CronJobs. I'm constantly reinventing solutions to problems that Kubernetes already solves—just less efficiently.

What I really wish for is a lightweight alternative offering a Kubernetes-compatible API that runs well on inexpensive VPS instances. The gap between enterprise-grade container orchestration and affordable hobby hosting remains frustratingly wide.

Systemd gets a lot of hate but it really solves a lot of problems. People really shouldn't dismiss it. I think it really happened because when systemd started appearing on distros by default people were upset they had to change

Here's some cool stuff:

  - containers

    - machinectl: used for controlling:

      - nspawn: a more powerful chroot. This is often a better solution than docker. Super lightweight. Shares kernel

      - vmspawn: when nspawn isn't enough and you need full virtualization

    - importctl: download, import, export your machines. Get the download features in {vm,n}spawn like we have with docker. There's a hub, but it's not very active

  - homed/homectl: extends user management to make it easier to do things like encryption home directories (different mounts), better control of permissions, and more

  - mounts: forget fstab. Make it easy to auto mount and dismount drives or partitions. Can be access based, time, triggered by another unit (eg a spawn), sockets, or whatever

  - boot: you can not only control boot but this is really what gives you access to starting and stopping services in the boot sequence. 

  - timers: forget cron. Cron can't wake your machine. Cron can't tell a service didn't run because your machine was off. Cron won't give you fuzzy timing, do more complicated things like wait for X minutes after boot if it's the third Sunday of the month and only if Y.service is running. Idk why you'd do that, but you can!

  - service units: these are your jobs. You can really control them in their capabilities. Lock them down so they can only do what they are meant to do.

    - overrides: use `systemctl edit` to edit your configs. Creates an override config and you don't need to destroy the original. No longer that annoying task of finding the original config and for some reason you can't get it back even if reinstalling! Same with when the original config changes in an install, your override doesn't get touched!!

It's got a lot of stuff and it's (almost) all there already on your system! It's a bit annoying to learn, but it really isn't too bad if you really don't want to do anything too complicated. But in that case, it's not like there's a tool that doesn't require docs but allows you to do super complicated things.

I've run my homelab with podman-systemd (quadlet) for awhile and every time I investigate a new k8s variant it just isn't worth the extra hassle. As part of my ancient Ansible playbook I just pre-pull images and drop unit files in the right place.

I even run my entire Voron 3D printer stack with podman-systemd so I can update and rollback all the components at once, although I'm looking at switching to mkosi and systemd-sysupdate and just update/rollback the entire disk image at once.

The main issues are: 1. A lot of people just distribute docker-compose files, so you have to convert it to systemd units. 2. A lot of docker images have a variety of complexities around user/privilege setup that you don't need with podman. Sometimes you need to do annoying userns idmapping, especially if a container refuses to run as root and/or switches to another user.

Overall, though, it's way less complicated than any k8s (or k8s variant) setup. It's also nice to have everything integrated into systemd and journald instead of being split in two places.

The next step to simplify this even further is to use Quadlet within systemd to manage the containers. More details are at https://www.redhat.com/en/blog/quadlet-podman

I created skate (https://github.com/skateco/skate) to be basically this but multihost and support k8s manifests. Under the hood it’s podman and systemd

We went back to just packaging debs and running them directly on ec2 instances with systemd. no more containers. Put the instances in an autoscaling group with an ALB. A simple ansible-pull installs the debs on-boot.

really raw-dogging it here but I got tired of endless json-inside-yaml-inside-hcl. ansible yaml is about all I want to deal with at this point.

The article is more than one year old, systemd now even has specialized officially supported OS distro for immutable workflow namely ParticleOS [1],[2].

[1] ParticleOS:

https://github.com/systemd/particleos

[2] Systemd ParticleOS:

https://news.ycombinator.com/item?id=43649088

From what I read, I think you can replace this all with a docker compose command and something like Caddy to automatically get certs.

It's basically just this command once you have compose.yaml: `docker compose up -d --pull always`

And then the CI setup is this:

  scp compose.yaml user@remote-host:~/
  ssh user@remote-host 'docker compose up -d --pull always'

The benefit here is that it is simple and also works on your development machine.

Of course if the side goal is to also do something fun and cool and learn, then Quadlet/k8s/systemd are great options too!

I am of the opinion that deploying stuff to a single server shouldn't be this complicated, and I wrote a tool to deploy the way I wanted:

https://harbormaster.readthedocs.io/

Harbormaster uses a YAML file to discover repositories, clones and updates them every so often, and runs the Docker Compose files they contain. It also keeps all state in a single directory, so you can easily back everything up. That's it.

It's by far the easiest and best tool for container orchestration I've come across, if all you need is a single server. I love how the entire config is declared in a repo, I love how all the state is in one directory, and I love how everything is just Compose files, nothing more complicated.

I know I'm tooting my own horn, I just love it so much.

Reading the comments here makes me feel old.

Doesn't anyone just use ssh and nginx anymore? Cram everything onto one box. Back the box up aggressively. Done.

I really don't need microservices management for my home stuff.

I think you are only looking at Kubernetes for running and updating container images. If that’s the use-case then I guess it’s overkill.

But Kubernetes does much more in terms of providing the resources required for these containers to share state, connect to each other, get access to config or secrets etc.

That’s where comes the CPU and memory cost. The cost of managing your containers and providing them the resources they need.

> basically acts as a giant while loop

Yep. That’s the idea of convergence of states I guess. In a distributed system you can’t always have all the participating systems behave in the desired way. So the manager (or orchestrator) of the system continuously tries to achieve the desired state.

I never moved to containers and seeing the churn the community has went through with all of this complicated container tooling, I'm happy orchestrating small-scale systems with supervisord and saltstack-like chatops deployments - it's just stupid simple by comparison and provides parity between dev and prod environments that's nice.

Why on earth would you run services on a server with --user and then fiddle with lingering logins instead of just using the system service manager?

Funny, because when we built Fleet (https://github.com/coreos/fleet/) Kubernetes didn't exist.

I don't know if someone knows a better stack for my fleet of self hosted applications, maybe moving to quadlet would simplify stuff ?

Right now I have an Ansible playbook responsible for updating my services, in a git repo.

The playbook stops changed services, backups their configs and volumes, applies the new docker-compose.yml and other files, and restarts them.

If any of them fail to start, or aren't reachable after 3 minutes, it rolls back everything *including the volumes* (using buttervolume, docker volumes as btrfs subvolumes to make snapshots free).

I am looking into Kubernetes, but I didn't find a single stack/solution that would do all that this system does. For example I found nothing that can auto rollback on failure *including persistent volumes*.

I found Argo Rollback but it doesn't seem to have hooks that would allow me to add the functionality.

At some point I tried to run a few small websites dedicated for activism (couple of Wordpress instances, a forum and custom PHP code) using docker. It was time sink as updating and testing the images turned out to be highly non-trivial.

Eventually I replaced everything with a script that generated systemd units and restarted the services on changes under Debian using the Wordpress that comes with it. Then I have a test VM on my laptop and just rsync changes to the deployment host and run the deployment script there. It reduced my chores very significantly. The whole system runs on 2GB VPS. It could be reduced to 1GB if Wordpress would officially support SQLite. But I prefer to pay few more euros per month and stick to Mariadb to minimize support requirements.

For anyone interested in quadlet, I've discovered a very usefull tool to convert compose files and manifests into quadlets: podlet.

It dramatically speeds up the process of converting the usual provided files into quadlets.

https://github.com/containers/podlet

I also use systemd+podman. I manage the access into the machine via an nginx that reverse proxies the services. With quadlets things will probably be even better but right now I have a manual flow with `podman run` etc. because sometimes I just want to run on the host instead and this allows for me to incrementally move in.

https://github.com/coreos/fleet I feel like fleet deserved more of a shot than it ultimately got.

Kamal is also a decent solution if you just have a static set of webapps that you want to easily deploy to static set of systems but still want 'production-like' features like no-downtime-deploys.

And I'm pretty familiar with Kubernetes but, yeah, for small tasks it can feel like taking an Apache to the store to buy a carton of milk.

I’m here just doing docker compose pull && docker compose down && docker compose up -d and it’s basically fine.

This is cool, but it doesn't address the redundancy/availability aspect of k8s, specifically, being able to re-schedule dead services when a node (inevitably) dies.

Well, if you're planning to run a single-node container server, then K8s is probably an overkill compared to Podman Quadlets. You just choose the lightest solution that meets your requirement. However, there was a noteworthy project named Aurae [1]. It was an ambitious project intended to replace systemd and kubelets on a server. Besides running containerized and baremetal loads, it was meant to take authenticated commands over an API and had everything that was expected on K8s worker nodes. It could work like K8s and like Docker with appropriate control planes. Unfortunately, the project came to an abrupt end when its main author Kris Nova passed away in an unfortunate accident.

[1] https://aurae.io/

Do you actually need the container at that point?

I host all of my hobby projects on a couple of raspi zeros using systemd alone, zero containers. Haven’t had a problem since when I started using it. Single binaries are super easy to setup and things rarely break, you have auto restart and launch at startup.

All of the binaries get generated on GitHub using Actions and when I need to update stuff I login using ssh and execute a script that uses a GitHub token to download and replace the binary, if something is not okay I also have a rollback script that switches things back to its previous setup. It’s as simple as it gets and it’s been my go-to for 2 years now.

I just deployed a couple containers this way, was pretty easy to port the docker-compose. However, I then tried to get them to run rootless, and well, that turned out to be headache after headache. Went back to rootful, other than I'm pretty happy with the deployment.

I know some large AWS environments that run a variation of this

Autoscaling fleet - image starts, downloads container from registry and starts on instance

1:1 relationship between instance and container - and they’re running 4XLs

When you get past the initial horror it’s actually beautiful

I do that too, I run everything in rootless podman managed by systemd units it's quite nice. With systemd network activation I could even save the cost of user space networking, though for my single user use case, it's not really needed and for now I could not bother.

I also have Quadlet on my backlog, I'm waiting the release of next stable version of Debian (which I think should be released sometimes this year) as the current version of Debian has a podman slightly too old which doesn't include Quadlet

A couple years ago I upgraded my desktop hardware, which meant it was time to upgrade my homelab. I had gone through various operating systems and methods of managing my services: systemd on Ubuntu Server, Docker Compose on CentOS, and Podman on NixOS.

I was learning about Kubernetes at work and it seemed like such a powerful tool, so I had this grand vision of building a little cluster in my laundry room with nodes net booting into Flatcar and running services via k3s. When I started building this, I was horrified by the complexity, so I went the complete opposite direction. I didn't need a cluster, net booting, blue-green deployments, or containers. I landed on NixOS with systemd for everything. Bare git repos over ssh for personal projects. Git server hooks for CI/CD. Email server for phone notifications (upgrade failures, service down, low disk space etc). NixOS nightly upgrades.

I never understood the hate systemd gets, but I also never really took the time to learn it until now, and I really love the simplicity when paired with NixOS. I finally feel like I'm satisfied with the operation and management of my server (aside from a semi frequent kernel panic that I've been struggling to resolve).

When a service is updated like this:

- is there downtime? (old service down, new service hasn't started yet)

- does it do health checks before directing traffic? (the process is up, but its HTTP service hasn't initialized yet)

- what if the new process fails to start, how do you rollback?

Or it's solved with nginx which sits in front of the containers? Or systemd has a builtin solution? Articles like this often omit such details. Or no one cares about occasional downtimes?

At https://controlplane.com we give you the power of Kubernetes without the toil of k8s. A line of code gets you a tls terminated endpoint that is geo routed to any cloud region and on-prem location. We created the Global Virtual Cloud that let's you run compute on any cloud, on premises hardware or vm's and any combination. I left vmware to start the company because the cognitive load on engineers was becoming ridiculous. Logs, metrics, tracing, service discovery, TLS, DNS, service mesh, network tunnels and much more - we made it easy. We do to the cloud what vmware did to hardware - you don't care what underlying cloud you're on. Yet you can use ANY backing service of AWS, GCP and Azure - as if they merged and your workloads are portable - they run unmodified anywhere and can consume any combination of services like RDS, Big Query, Cosmos db and any other. It is as if the cloud providers decided to merge and then lower your cost by 60-80%.

Check it out. Doron

Whoever thought running your personal blog on Kubernetes is a good idea? Kubernetes is good for large scale applications. Nothing else. I do not get why even mid-sized companies take the burden of Kubernetes although they have low infrastructur needs.

Looking forward to the follow up "replacing systemd with init and bash scripts."

I remember seeing a project in development that built k8s-like orchestration on top of systemd a couple years ago, letting you control applications across nodes and the nodes themselves with regular systemd config files and I have been unable to find it again. IIRC it was either a Redhat project or hosted under github.com/containers and looked semi-official.

Anyone knows what I’m talking about? Is it still alive?

EDIT: it’s not CoreOS/Fleet, it’s something much more recent, but was still in early alpha state when I found it.

Wrt to the updates: Does it mean it maintains the original environment variables you passed in separate from the ones set by the container?

E.g. when updating a container with watchtower:

You deploy a container 'python-something'.

The container has PYTHON=3.11.

Now, the container has an update which sets PYTHON=3.13.

Watchtower will take the current settings, and use them as the settings to be preserved.

So the next version will deploy with PYTHON=3.11, even though you haven't set those settings.

FYI the link to Flux in

> Particularly with GitOps and [Flux](https://www.weave.works/oss/flux/?ref=blog.yaakov.online), making changes was a breeze.

appears to be broken.

EDIT: oh, I hadn't realized the article was a year old.

Soon back to chroot with a busybox cgi shell script as an API.

I am working on proot-docker:

https://github.com/mtseet/proot-docker

OK it does not need process namespaces, but that's good enough for most containers.

Article about podman, what systemd has to do with anything? What's wrong with docker + watchtower? How is is possible to both use gitops and not "remember" which flags you used for containers?

So many questions...

I've had a really great experience with docker compose and systemd units. I use a generic systemd unit and it's just as easy as setting up users then `systemctl enable --now docker-compose@service-name`.

Couldn’t it be argued that this is a bug in Kubernetes?

Most of what it does is run programs with various cgroups and namespaces, which is what systemd does, so should it really be any more resource intensive?

As a wannabe home lab hacker, but never had the energy or knowhow to get started, this thread is a goldmine of information!