`apt` installation could be easily sped-up with `eatmydata`: `dpkg` calls `fsync()` on all the unpacked files, which is very slow on HDDs, and `eatmydata` hacks it out.
back
4 comments
Really if you could just disable fsync at the OS level. A bunch of other common package managers and tools also do. Docker is a big culprit
If you corrupt a CI node, whatever. Just rerun the step
CI containers should probably run entirely from tmpfs.
Can tmpfs be backed by persistent storage? Most of the recent stuff I've worked on is a little too big to fit in memory handily. Ideally about 20GiB of scratch space for 4-8GiB of working memory would be ideal.
I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster
tmpfs is backed by swap space, in the sense that it will overflow to use swap capacity but will not become persistent (since the lack of persistence is a feature).
We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching:
We're having some success with doing this at the block level (e.g. in-memory writeback cache).
Why do it at the block level (instead of tmpfs)? Or do you mean that you're doing actual real persistent disks that just have a lot of cache sitting in front of them?
The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.
You can probably use a BPF return override on fsync and fdatasync and sync_file_range, considering that the main use case of that feature is syscall-level error injection.
edit: Or, even easier, just use the pre-built fail_function infrastructure (with retval = 0 instead of an error): https://docs.kernel.org/fault-injection/fault-injection.html
I'd love to experiment with that and/or flags like `noatime`, especially when CI nodes are single-use and ephemeral.
noatime is irrelevant because everyone has been using relatime for ages, and updating the atime field with relatime means you're writing that block to disk anyway, since you're updating the mtime field. So no I/O saved.
atime is so exotic you shouldn't need to consider disabling it experimental. I consider it legacy at this point.
> Docker is a big culprit
Actually in my experience with pulling very large images to run with docker it turns out that Docker doesn't really do any fsync-ing itself. The sync happens when it creates an overlayfs mount while creating a container because the overlayfs driver in the kernel does it.
A volatile flag to the kernel driver was added a while back, but I don't think Docker uses it yet https://www.redhat.com/en/blog/container-volatile-overlay-mo...
Well yeah, but indirectly through the usage of Docker, I mean.
Unpacking the Docker image tarballs can be a bit expensive--especially with things like nodejs where you have tons of tiny files
Tearing down overlayfs is a huge issue, though
This is a neat idea that we should try. We've tried the `eatmydata` thing to speed up dpkg, but the slow part wasn't the fsync portion but rather the dpkg database.
No need for eatmydata, dpkg has an unsafe-io option.
Other options are to use an overlay mount with volatile or ext4 with nobarrier and writeback.
unsafe-io eliminates most fsyncs, but not all of them.
"`dpkg` calls `fsync()` on all the unpacked files"
Why in the world does it do that ????
Ok I googled (kagi). Same reason anyone ever does: pure voodoo.
If you can't trust the kernel to close() then you can't trust it to fsync() or anything else either.
Kernel-level crashes, the only kind of crash that risks half-written files, are no more likely during dpkg than any other time. A bad update is the same bad update regardless, no better, no worse.
Let's start from the assumption that dpkg shouldn't commit to its database that package X is installed/updated until all the on-disk files reflect that. Then if the operation fails and you try again later (on next boot or whatever) it knows to check that package's state and try rolling forward.
> If you can't trust the kernel to close() then you can't trust it to fsync() or anything else either.
https://man7.org/linux/man-pages/man2/close.2.html
A successful close does not guarantee that the data has been
successfully saved to disk, as the kernel uses the buffer cache to
defer writes. Typically, filesystems do not flush buffers when a
file is closed. If you need to be sure that the data is
physically stored on the underlying disk, use fsync(2). (It will
depend on the disk hardware at this point.)
Of course none of this matters much on an ephemeral Github Actions VM. There's no "on next boot or whatever". So this is one environment where it makes sense to bypass all this careful durability work that I'd normally be totally behind. It seems reasonable enough to say it's reached the page cache, it should continue being visible in the current boot, and tomorrow will never come.
Writing to disk is none of your business unless you are the kernel itself.
Huh? Are you thinking dpkg is sending NVMe commands itself or something? No, that's not what that manpage means. dpkg is asking the kernel to write stuff to the page cache, and then asking for a guarantee that the data will continue to exist on next boot. The second half is what fsync does. Without fsync returning success, this is not guaranteed at all. And if you don't understand the point of the guarantee after reading my previous comment...and this is your way of saying so...further conversation will be pointless...
"durability" isn't voodoo. Consider if dpkg updates libc.so and then you yank the power cord before the page cache is flushed to disk, or you're on a laptop and the battery dies.
> before the page cache is flushed to disk
And if yank the cord before the package is fully unpacked? Wouldn't that just be the same problem? Solving that problem involves simply unpacking to a temporary location first, verifying all the files were extracted correctly, and then renaming them into existence. Which actually solves both problems.
Package management is stuck in a 1990s idea of "efficiency" which is entirely unwarranted. I have more than enough hard drive space to install the distribution several times over. Stop trying to be clever.
> Wouldn't that just be the same problem?
Not the same problem, it's half-written file vs half of the files in older version.
> Which actually solves both problems.
it does not and you would have to guarantee that multiple rename operations are executed in a transaction. Which you can't. Unless you have really fancy filesystem.
> Stop trying to be clever.
It's called being correct and reliable.
> have to guarantee that multiple rename operations are executed in a transaction. Which you can't. Unless you have really fancy filesystem
Not strictly. You have to guarantee that after reboot you rollback any partial package operations. This is what a filesystem journal does anyways. So it would be one fsync() per package and not one per every file in the package. The failure mode implies a reboot must occur.
> It's called being correct and reliable.
There are multiple ways to achieve this. There are different requirements among different systems which is the whole point of this post. And your version of "correct and reliable" depends on /when/ I pull the plug. So you're paying a huge price to shift the problem from one side of the line to the other in what is not clearly a useful or pragmatic way.
> You have to guarantee that after reboot you rollback any partial package operations.
In both scenarios, yes. This is what dpkg database is for, it keeps info about state of each package: whatever is it installed, unpacked, configured and so on. It is required to handle interrupted update scenario, no matter if it was interrupted during package unpacking or in the configuration stage.
So far you are just describing --force-unsafe-io from dpkg. It is called unsafe because you can end up with zeroed or 0-length files long after the package has been marked as installed.
> This is what a filesystem journal does anyways.
This is incorrect. And also filesystem journal is irrelevant.
Filesystem journal protects you from interrupted writes on the disk layer. You set some flag, you write to some temporary space called journal, you set another flag, then you copy that data to your primary space, then you remove both flags. If something happens during that process you'll know and you'll be able to recover because you know in which step you were interrupted.
Without filesystem journal every power outage could result in not being able to mount the filesystem. Journal prevents that scenario. This has nothing to do with package managers, page cache or fsync.
Under Linux you do the whole write() + fsync() + rename() dance, for every file, because this is the only way you don't end up in the scenario where you've written the new file, renamed it, marked package as installed and fsynced the package manager database but the actual new file contents never left the page cache and now you have bunch of zeroes on the disk. You have to fsync(). This is semantic of the layer you are working with. No fsync(), no guarantee that data is on the disk. Even if you wrote it and closed the file hours ago. And fsynced package manager database.
> There are different requirements among different systems which is the whole point of this post.
Sorry, I was under assumption that this thread is about dpkg and fsync and your idea of "solving the problem". I just wanted to point out that, no, package managers are not "trying to be clever" and are not "stuck in the 1990s". You can't throw fsync() out of the equation, reorder bunch of steps and call this "a solution".
Like I said.
Pretty sure kernel doesn't have to fsync on close. In fact, you don't want it to, otherwise you're limiting the performance of your page cache. So fsync on install for dpkg makes perfect sense.
I didn't say it synced. The file is simply "written" and available at that point.
It makes no sense to trust that fsync() does what it promises but not that close() does what it promises. close() promises that when close() returns, the data is stored and some other process may open() and find all of it verbatim. And that's all you care about or have any business caring about unless you are the kernel yourself.
I would like to introduce you to a few case studies on bugzilla:
https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slo...
Get involved, tell and show them you know better. They have a bug tracker, mailing list etc
> Kernel-level crashes, the only kind of crash that risks half-written files [...]
You can get half-written files in many other circumstances, eg on power outages, storage failures, hw caused crashes, dirty shutdowns, and filesystem corruption/bugs.
(Nitpick: trusting the kernel to close() doesn't have anythign to do with this, like a sibling comment says)
A power outage or other hardware fault or kernel crash can happen at any unpredicted time, equally just before or just after any particular action, including an fsync().
Trusting close() does not mean that the data is written all the way to disk. You don't care if or when it's all the way written to disk during dpkg ops any more than at any of the other infinite seconds of run time that aren't a dpkg op.
close() just means that any other thing that expects to use that data may do so. And you can absolutely bank on that. And if you think you can't bank on that, that means you don't trust the kernel to be keeping track of file handles, and if you don't trust the kernel to close(), then why do you trust it to fsync()?
A power rug pull does not worsen this. That can happen at any time, and there is nothing special about dpkg.
this is actually an interesting topic and it turns out kernel never made any promises about close() and data being on the disk :)
and about kernel-level crashes: yes, but you see, dpkg creates a new file on the disk, makes sure it is written correctly with fsync() and then calls rename() (or something like that) to atomically replace old file with new one.
So there is never a possibility of given file being corrupt during update.
What's your experience developing a package manager for one of the world's most popular Linux distributions?
Maybe they know something you don't ?????
> Kernel-level crashes, the only kind of crash that risks half-written files, are no more likely during dpkg than any other time. A bad update is the same bad update regardless, no better, no worse.
Imagine this scenario; you're writing a CI pipeline:
1. You write some script to `apt-get install` blah blah
2. As soon as the script is done, your CI job finishes.
3. Your job is finished, so the VM is powered off.
4. The hypervisor hits the power button but, oops, the VM still had dirty disk cache/pending writes.
The hypervisor may immediately pull the power (chaos monkey style; developers don't have patience), in which case those writes are now corrupted. Or, it may use ACPI shutdown which then should also have an ultimate timeout before pulling power (otherwise stalled IO might prevent resources from ever being cleaned up).
If you rely on sync to occur at step 4 during the kernel to gracefully exit, how long does the kernel wait before it decides that some shutdown-timeout occurred? How long does the hypervisor wait and is it longer than the kernel would wait? Are you even sure that the VM shutdown command you're sending is the graceful one?
How would you fsync at step 3?
For step 2, perhaps you might have an exit script that calls `fsync`.
For step 1, perhaps you might call `fsync` after `apt-get install` is done.
This is like people that think they have designed their own even better encryption algorithm. Voodoo. You are not solving a problem better than the kernel and filesystem (and hypervisor in this case) has already done. If you are not writing the kernel or a driver or bootloader itself, then fsync() is not your problem or responsibility and you aren't filling any holes left by anyone else. You're just rubbing the belly of the budda statue for good luck to feel better.
You didn't answer any of the questions proposed by the outlined scenario.
Until you answer how you've solved the "I want to make sure my data is written to disk before the hypervisor powers off the virtual machine at the end of the successful run" problem, then I claim that this is absolutely not voodoo.
I suggest you actually read the documentation of all of these things before you start claiming that `fsync()` is exclusively the purpose of kernel, driver, or bootloader developers.
So... the goal is to make `apt` to be web scale?
(to be clear: my comment is sarcasm and web scale is a reference to a joke about reliability [0])