Go is portable, until it isn't

I ran into this issue when porting term.everything[0] from typescript to go. I had some c library dependencies that I did need to link, so I had to use cgo. My solution was to do the build process on alpine linux[1] and use static linking[2]. This way it statically links musl libc, which is much friendlier with static linking than glibc. Now, I have a static binary that runs in alpine, Debian, and even bare containers.

Since I have made the change, I have not had anyone open any issues saying they had problems running it on their machines. (Unlike when I was using AppImages, which caused much more trouble than I expected)

[0] https://github.com/mmulet/term.everything look at distribute.sh and the makefile to see how I did it.

[1]in a podman or docker container

[2] -ldflags '-extldflags "-static"'

  # Build the Go project with the static library
  if go build -o $PROG_NAME -ldflags '-extldflags "-static"'; then
    echo "Go project built with static library linkage"
  else
    echo "Error: Failed to build the Go project with static library"
    exit 1
  fi

  # Check if the executable is statically linked
  if nm ./$PROG_NAME | grep -q "U "; then
    echo "Error: The generated executable is dynamically linked"
    exit 1
  else
    echo "Successfully built and verified static executable '$PROG_NAME'"
  fi

  // #cgo LDFLAGS: -L. -lfoo

> We did not want to spend time maintaining a backward compatible parser or doing code archaeology. So this option was discarded.

Considering all of the effort and hoop-jumping involved in the route that was chosen, perhaps this decision might be worth revisiting.

In hindsight, maintaining a parser might be easier and more maintainable when compared to the current problems that were overcome and the future problems that will arise if/when the systemd libraries decide to change their C API interfaces.

One benefit of a freestanding parser is that it could be made into a reusable library that others can use and help maintain.

Once you use CGO, portability is gone. Your binary is no longer staticly compiled.

This can happen subtley without you knowing it. If you use a function in the standard library that happens to call into a CGO function, you are no longer static.

This happens with things like os.UserHomeDir or some networking things like DNS lookups.

You can "force" go to do static compiling by disabling CGO, but that means you can't use _any_ CGO. Which may not work if you require it for certain things like sqlite.

Was there not a third option: Calling the journalctl CLI as a child process and consume the parsed logs from the standard output? This might have avoided both the requirement to use CGO and also to write a custom parser. But I guess I am missing something.

So you can’t pull in c libraries built for different distributions and expect this to work.

If you use pure go, things are portable. The moment you use C API, that portability doesn’t exist. This should be apparent.

I think this is true for nearly all compiled languages. I had the same fun with rust and openSSL and glibC. OP didn’t mentioned the fun with glib-c when compiling on a fairly recent distro and trying it to run on an older one. There is the “many Linux” project which provides docker images with a minimum glib c version installed so it’s compatible with newer ones. The switch to a newer open ssl version on Debian/Ubuntu created some issues for my tool. I replaced it with rust tls to remove the dynamic linked library. I prefer complete statically linked binaries though. But that is really hard to do and damn near impossible on Apple systems.

You hit this real quick when trying to build container images from the scratch. Theoretically you can drop a Go binary into a blank rootfs and it will run. This works most of the time, but anything that depends on Go's Postgres client requires libpq which requires libc. Queue EFILE runtime errors after running the container.

All of this, every last bit of complexity and breakage and sweat, is downstream of this:

> Journal logs are not stored in plain text. They use a binary format

And it was entirely predictable and predicted that this sort of problem would be the result when that choice was made.

And a set of people rediscovered why cross compiling only works up to certain extent, regardless of the marketing on the tin.

The point one needs to touch APIs that only exists on the target system, the fun starts, regardless of the programming language.

Go, Zig, whatever.

Has nothing to do with go. You added a dependency which is not portable. It is well known that systemd project only targets Linux.

Vendorise systemd and compile only the journal parts, if they are portable and can be isolated from the rest. Otherwise just shell out to journalctl.

Use dlopen? I haven’t tried this in Go, but if you want a binary that optionally includes features from an external library, you want to use dlopen to load it.

Interesting that it uses the C API to collect journals. I would’ve thought to just invoke journalctl CLI. On platforms like macOS where the CLI doesn’t exist it’s an error when you exec, not a build time error.

If you really need a portable binary that uses shared libraries I would recommend building it with nix, you get all the dependencies including dynamic linker and glibc.

Hashicorp's Vault go binary is a whopping 512Mb beast. Recently considered using its agent mode to grab secrets for applications in containers but the size of the layer it adds is unviably big. And they don't seem interested into making a split server/client binary either...

I’ve had some success using Zig for cross compiling when CGO is required.

FWIW I maintain an official implementation of the journal wire format in go now.

https://github.com/systemd/slog-journal so you can at least log to the journal now without CGO

But that's just the journal Wire format which is a lot simpler than the disk format.

I think a journal disk format parser in go would be a neat addition

I tink the title is a bit misleading. This is about very low level metrics collection from the system which by definition is very system dependent. The term “portable” in a programming language usually means portability for applications but this more portability of utilities.

Expecting a portable house and a portable speaker to have the same definition of portable is unfair.

This seems to imply that Go's binaries are otherwise compatible with multiple platforms like amd64 and arm64, other than the issue with linking dynamic libraries.

I suspect that's not true either even if it might be technically possible to achieve it through some trickery (and why not risc-v, and other architectures too?).

Systemd. Binary logs are wonderful aren't they?

Go was never truly portable on Linux unfortunately due to its dependency on libc for DNS and user name resolution (because of PAM and other C-only API). Sure, pure Go implementation exists, but it doesn't cover all cases, so, in order to build a "good" binary for Linux you still needed to build the binary on (oldest supported) Linux distro.

If your production doesn't have any weird PAM or DNS then you can indeed just cross-compile everything and it works

This is an (organizational) tooling problem, not a language problem - and is no less complicated when musl libc enters the discussion.

This article reminds me of the days before LLMs ruled the world, when the word "agent" was most commonly used in the DevOps area, representing the program that ran on a remote machine to execute dispatched jobs or send metrics. Now I wonder how many developers would look at "agent" and think of this meaning.

Cross-compiling doesn't work because you're not defining your dependencies correctly and relying on the existence of things like system libraries and libc. Use `zig cc` with Go which will let you compile against a stub Glibc, or go all the way and use a hermetic build system (you should do this always anyhow).

Well, that was pretty obvious that the portability is gone, especially when you start linking into systemd, even on the host system you have to link with the shared libs into systemd, you cannot link statically.

Cgo is terrible, but if you just want some simple C calls from a library, you can use https://github.com/ebitengine/purego to generate the bindings.

It is a bit cursed, but works pretty well. I'm using it in my hardware-backed KMIP server to interface with PKCS11.

Go is portable until you have to deploy on AS/400

Basically everything is portable unless it isn't. Java - the same. We fly in abstractions unless you need to delete a file

There's no such thing as a portable application; only programs limited enough to be lucky not to conflict with the vagaries of different systems.

That said, in my personal experience, the most portable programs tend to be written in either Perl or Shell. The former has a crap-ton of portability documentation and design influence, and the latter is designed to work from 40 year old machines up to today's. You can learn a lot by studying old things.

From the article:

> In the observability world, if you're building an agent for metrics and logs, you're probably writing it in Go.

I'm pretty unconvinced that this is the case unless you happen to be on the CNCF train. Personally I'd write in Rust these days, C used to be very common too.

I really hate this type of blog. It pollutes the world with this attitude of “I messed up, how I have to frame the problem in a way, and write a blog that lets my ego stay intact”, which results in blogs like this showing in decision making process as “why you should not use go”. And mostly people never look past the title.

The fact is go is portable, it provides the ability to cross compile out of the box and reasonably executed on other platforms it supports. But in this case, a decision that had little to do with go, the desire to use c code, a non go project, with their go project made things harder.

These are not “just a set of constraints you only notice once you trip over them”, this is trivializing the mistake.

Entire blog can be simplified to the following.

We were ignorant, and then had to do a bunch of work because we were ignorant. It’s a common story in software. I don’t expect everybody to get it right the first time. But what we don’t need is sensational titled blogs full of fluff to try to reason readers out of concluding the obvious. Somebody in charge made decisions uninformed and as a result the project became more complicated and probably took longer.

The portability story for Go is awful. I've blogged about this before: https://blog.habets.se/2022/02/Go-programs-are-not-portable....

It's yet another example of Go authors just implementing the least-effort without even a slight thought to what it would mean down the line, creating a huge liability/debt forever in the language.

i wonder, for their use case, why not just submit journal in binary format to the server and let the serve do the parsing?

more like C is portable, until it isn't

This stuff is out of my frame of reference. I've never used Go before and have never had the need to go this low level (C APIs, etc); so please keep this in mind with my following questions, which are likely to sound stupid or ignorant.

Can this binary not include compiled dependacies along side it? I'm thinking like how on windows for portable apps they include the DLLs and other dependant exes in subfolders?

Out of interest, and in relation to a less well liked Google technology, could dart produce what they are after? My understanding is dart can produce static binaries, though I'm not sure if these are truly portable compile once run everywhere sense.

Well now you've gone and linked to a fascinating tool which I'm going to have to dive into and learn: https://kaitai.io/

Thanks.

so like every other language