back
222 comments
I'm probably the main person responsible for making journald usable at all.

But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.

Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.

My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size.

Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.

The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.

I've met this unwarranted love for mmap() many times in the developers who never professionally worked on storage projects. Especially common with C++ programmers for some reason. There are people who think they found a "trick" to make I/O go faster and never consider why filesystems or databases don't use it... Like, obviously, those losers who wrote eg. Ext4 never bothered to look at the system interface, right?

On the other hand, if I was ever to advise anyone on how to do I/O when they are working with an (unknown) filesystem... It's really hard. And I'd probably default to saying "do as few tricks as possible" because filesystems today are very elaborate, with a lot of optimizations that are very difficult to predict from user-space. It's quite possible that someone trying to outsmart a filesystem will end up harming themselves in the process.

Doing as few tricks as possible would allow the administrator to configure the filesystem independently of the program writing to it to match the nature of the workload instead of locking the program into a specific pattern of operation that might be impossible to rectify with administrative tools. Not an ideal situation by any means: storage-heavy user-space applications s.a. databases usually do the opposite: they try to optimize for the specific filesystem, its version and quirks... but it takes a lot of effort, obviously.

Thank you for your work. ISTM the workload is naturally LSM-shaped.

> If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.

Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple.

> I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory).

If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily.

journald uses hash tables, I think it update it on every new log line, although I didn't debug it in depth yet.

https://github.com/systemd/systemd/blob/199f75205b9c0625bf56...

> I'm probably the main person responsible for making journald usable at all.

Thank you for your service!

Why not just have a SQLite file and call it a day?

Also, why mmaped file?

Something must have happened along the way, because this was not the original design intent of the database (emphasis mine):

"""

The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that.

"""

See https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG...

If it ever worked like that, then gradual accretion of (mis)features and misguided enhancements pretty clearly broke it. Based on my years and years and years of reading about and using the output of the Systemd Project, there's really clearly no Linus Torvalds on the project to hold the line on software quality.

Edit: Looks like someone who did a ton of work attempting to get journald even vaguely usable has chipped in with additional information. [0] My hunch is that the current set of people working on the Systemd Project are going to be supremely disinterested in fixing the problem... and might even be entirely unable to fix it. A project this large and sprawling that runs for this long without a solid commitment to quality doesn't tend to retain many very highly-skilled individuals.

[0] <https://news.ycombinator.com/item?id=49291376>

The traditional solution for the problem of the repetitive data included in logs is that every time when a log file grows over a certain size (or periodically in time), a new log file is created and the old file is compressed with some standard data compression algorithm, which eliminates the repetitions.

This optimally solves the problem of the space taken by logs on disk.

The only possible disadvantage is that any application that is used to scan the logs must decompress them, but in practice I have never seen any case when this caused any nuisance, even when using such a primitive solution like "zcat|grep", instead of a full-featured application.

I would argue the described format is exactly the origin of the problem.

It tries to optimize on disk footprint by deduplication and resulting in way more complex file format with many possible footguns leading to things like write amplification while also making it less robust for the actual use cases of a persistent log.

In a way, it's using a file format more useful for aggregation layer, except it doesn't do that well either, compromising immediate needs at local level.

I completely agree with one of the comments from there:

> But the fundamental conclusion is: the design was wrong. It should not have used mmapped writes. pwrite would have been far better.

It really does not make any sense to use memory-mapped files when writing logs.

Not even pwrite makes sense, because logs should normally be written by opening and using the log files as append-only sequential files.

Only when reading logs, to search for problems, accessing them as read-only memory-mapped files is OK.

Actually not only for logs, but almost always, read-write memory-mapped files are either inefficient or too complex to use (i.e. to avoid problems you must carefully use msync and/or madvise, which eliminates the simplicity that makes memory-mapped files preferable to using pread/pwrite). It is better to use memory-mapped files only for read-only accesses, using the appropriate option flags in open and mmap.

great and clear summary thank you!

I would laso add that if my design decisions or development actions lead to an issue affecting multiple linux distro defaults I would feel responsible and rush for a solid fix instead of this https://github.com/systemd/systemd/issues/15292#issuecomment...

I wonder if something like https://github.com/open-telemetry/otel-arrow would be a better fit instead
journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that has zero impact to the user whatsoever. You almost need to keep a script tracking all the journal floods for every new service to make sure it's not treating your system log as its dumping ground. To be fair, the kernel and usb peripherals can also have a bad day and spam 3 million lines an hour into it, think input irq status -75.

It's too much of a chore to keep up with all the program-level configs (if they have them) and service files, but LogFilterPatterns in systemd can help in an unintended way: you can make one log blacklist with a .conf file in /etc/systemd/system/service.d/, and put in there all the patterns that spam your journal one by one, don't even have to chase misattributed loglevels. It just looks something like:

[Service]

LogFilterPatterns=~I am a completely useless log entry

LogFilterPatterns=~I am another useless log entry

But it doesn't pick up on identifiers and doesn't do anything for kernel spam. It's only great to make some messages shut up. Also, I'd consider any btrfs install that does not have nocow on cache, journal etc. to be defective.

That was the task for years for syslog services that dealt with it without issue.
> I'd consider any btrfs install that does not have nocow on...to be defective.

You're getting COW on the extents if you're snapshotting anyway.

systemd-journald also has rate limits that you can configure ;-)
I'm using a certain object storage implementation post minio enshitification. The software itself is great don't get me wrong but I've noticed their logs are basically unreadable. Its metrics and traces in json data meant to be rendered on a dashboard instead of being read by humans. It's also extremely verbose even at an INFO level.

Maybe just get these on an open telemetry endpoint instead? I also don't get why people send by default json logs to journald as it's clearly meant to be a replacement to syslog which is already a good standard.

The cherry on the cake is that you practically cannot filter journald. The only option is limiting by severity (e.g. errors and higher) or switch to non persistent journald storage and forward to rsyslog and filter there.

Am a bit vague on the details but sometimes a driver goes bezerk and starts logging many times per second, e.g. a bug in amdgpu after resume from suspend. Took a while to get that filtered which luckily was only possible because it were kernel messages (dmesg), but for a while I had to disae persistent kernel logging which is dat from ideal.

I get that for certain core parts simplicity is more important than features. But journald is just too basic to enable persistent storage but I also don't want to switch it off.

If you have systemd>=253 you can make use of LogFilterPatterns[0] (in .service files), but it's really unpredictable, cumbersome to work with, and does not work with user services or non-service log sources.

[0]: https://www.freedesktop.org/software/systemd/man/latest/syst...

journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
I would much rather that they had used an existing database file format. Sqlite3 is robust and already present in the default installation of most Linux distributions. Querying system logs with SQL would be cool and likely faster than using the sd_journal API with all it's weird quirks.
I recently put a lot of effort into reducing logging because of excessive writes. It was so much easier when everything had its own log and you could just look at which files were growing.
Systemd is touted as being highly modular. So it should be easy enough to replace the logging module journald.

Why hasn't this been done if it's that terrible?

I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedom

Will try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.

Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately.

It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years).

If you're interested, here are my findings this time so far:

    - workrave: 60 second stat sync https://github.com/rcaelers/workrave/pull/717
    - kde klipper: saves to disk on every copy, even if permanent storage is disabled https://bugs.kde.org/show_bug.cgi?id=501030
    - kde plasmashell: saves qt shader cache each time notification popup disappears https://bugs.kde.org/show_bug.cgi?id=523805
    - bitwarden firefox extension: tries to connect to desktop application every 10 seconds, writes about every failure to browser's WebStorage 14+ KB https://github.com/bitwarden/clients/issues/22192
    - firefox datareporting/glean: very chatty .mozilla/firefox/xxx/datareporting/glean/db/data.safe
    - ipfs: writes every received DHT announce to disk, 20 GB in 3 hours https://discuss.ipfs.tech/t/constant-writes-to-datastore-log/20316
    - mailcow: redis saves data every 5 minutes https://github.com/mailcow/mailcow-dockerized/pull/7405
systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.

I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.

How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?

The first thing I do on a Linux system is install a proper syslog daemon.

Ooh, mmapped writes. I make that mistake once, years ago. :) I posted a comment in that GH issue.
Someone should implement a new operating system that can efficiently handle text processing.

It could have some simple tools that let you generate reports, display them on screen, and compose tools for that sort of thing in a natural way.

We could call it UNIX.

What does the (currently latest, https://github.com/systemd/systemd/issues/40262#issuecomment...) comment mean? Who are the "large folio people"?
Cool to see @ValdikSS here as well. The guy never sleeps or he is AI in disguise ;-)
Okay, so how do I disable journald and switch to something else, without getting rid of systemd completely?
My hunch having looked at the journald code as an amateur is that this write amplification is coming from scattering, with a few possible sources:

1. Writes try to compress away duplicate metadata at the application layer, which causes them to issue scattered writes when new metadata shows up.

2. Indexing is also surprisingly log-line/application-layer aware, such that index writes might also be scattering.

3. The indexes themselves seem like they could benefit from an append-mostly write model with periodic compaction rather than a mutate-in-place model.

4. I was surprised that the journal’s “WAL” doesn’t seem to be a major concern of a lot of the code. For a database, supporting reads “through” the WAL with periodic application back to the data files (“checkpoints” in RDBMS) seems like something I’d expect to see more of here. But I don’t really have deep understanding of the code, so I may be missing that it’s doing that already.

The choice of mmap instead of regular file writes here isn’t, as others have proposed, a design flaw. I think that makes sense given what journald is (a database) and how significant its durability concerns are. And it looks like the code does spend a lot of time trying to be careful about which blocks/pages are dirtied. But this is a famously hard-to-get-write (ha!) area so perhaps defects are present at that layer.

The systemd developers are talented in their area; I am not a systemd hater. However, “talented at low-level OS design” is not the same as “talented at building a database from scratch”, and I think that shows here.

I strongly feel like this system could be a wrapper around SQLite, which is definitely something that could be integrated everywhere journald is used (license-wise and compatibility-wise). I’m puzzled as to why that wasn’t chosen as an approach: a SQLite vfs implementation that handled compression and online rotation seems like it would have resulted in a design that’s both more interoperable and less prone to flaws like this one.

I also think that a per-log-emitter setting that doesn’t eagerly persist to disk (wait for page cache flush) would be very useful to have available—perhaps even as a default—for user-level/init6 level logs that are OK with a potential for data loss on kernel panic.

journald has never been of great quality. It somehow manages to be visibly slower than grepping gzipped text logs.
years ago, I set Storage=volatile on almost all the journalD configurations I have. This largely solved this kind of problem.
hello ValdikSS! nice to see you alive
Systemd should just use DuckDB. It's perfect for this job.

"But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?"

Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data.

BTW: no, you don't need O(N*log(N) writes for DuckDB. No, you're not doing a whole block-group write for every message. No, Parquet is not a magical solution. I mean, maybe it's fine, but DuckDB is already columnar, and arguably better at it.

Seems like there are a lot of mistaken impressions about DB storage engines out there.

Oh how I love totally predictable poetterings reply to previous bug report that got closed because "measuring it wrong" and "this is not a support forum".
Systemd things being horse shit as usual because it was vibecoded even before LLM existed. And there are still people that said that systemd and tools are awesome because they never encountered any of the countless ridicule bugs.
For 99% of installs the basic assumption that local logging (with local reading) is the primary mode is just wrong.
This issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.