back
2 comments
This is interesting:

> https://en.wikipedia.org/wiki/Virtual_memory_compression#His...

When I worked at MS I heard some chatter about people doing it with NT circa 2010 (I guess per this wikipedia article it didn't ship until five years later). Then after I left I heard there were Linux patches from before that (wikipedia cites 2008, with it being in the mainline tree in 2013). And then later, that Apple had also done it.

It's interesting that a bunch of major operating systems decided to do this roughly the same time.

There was various adhoc hackish support for compressed RAM and swapping to various RAM-like things (eg. VRAM) in Linux since at least 2004. But around the 2010 doing such stuff started to be somewhat worthwhile for somewhat normal use cases and not only neat trick that had some questionable benefit for live-CDs and nothing much else.
> It's interesting that a bunch of major operating systems decided to do this roughly the same time.

I'm guessing it's due to the popularity of 64-bit address space. There are lots of pointers in RAM, and while they use a lot of bits, they don't have much entropy. So they probably compress extremely well, and with compression, give one less reason to stick to 32-bit architecture. (I don't fully understand why, but I hear that people run 32-bit OSes on 64-bit machines to save memory. As long as one process uses less than 4G of RAM, maybe it's the right optimization. But memory compression certainly changes that calculus and gives the OS vendors one less use case to have to support a 32-bit OS for.)

One alternative would be to define an ABI for using the x86-64 processor with 32-bit pointers. This would let everyone leave the processor in the more modern 64-bit mode without increasing pressure on the memory subsystem.

Such a thing exists:

https://en.wikipedia.org/wiki/X32_ABI

It's been available in the mainline Linux kernel and in glibc for ages. It also breaks any code which assumes #ifdef __x86_64__ means 64-bit pointers, and the concept doesn't seem to attract a lot of excitement:

https://www.phoronix.com/scan.php?page=news_item&px=MTU1MjE

Or it's the latency chasm between RAM and disk. A cursory googling to refresh my own memory, SSD is at least 100x slower than peeking at RAM and possibly much more, depending on who's numbers you trust.

It seems at least plausible to save time by compressing pages in memory rather than going to disk. Certainly for low end or consumer systems not blessed with gobs of RAM.

It is available in win 10: https://www.howtogeek.com/319933/what-is-memory-compression-...
It may sound so weird, but it's really a relatively straightforward equation. If the CPU is compressing fast relative to disk operations, the expected compression ratio is good (not hard to imagine for a lot of cases), and both memory and CPU overhead are tolerable, compressing RAM is favorable to paging to disk.

I wonder how much of an effect SSDs have on that balance.

Random late reply, but: compression can compete surprisingly often. Latency can get less attention than overall IOPS numbers and such, but SSD service times for individual 4K random reads are often something like 0.1ms. You can pack a compressible page faster than that for sure.

(Intel's selling their new NVDIMMs as providing much better latency figures than SSDs, especially if you read a cache line out at a time instead of a whole page. Be nice to see the things and their pricing.)

Instead of a general LZ compressor Apple's stuff uses a word-at-a-time algorithm called WKdm, and there are others (there's one on GitHub called centaurean/density for example). Hardware can help--Samsung had a memory compressor in some chips and Qualcomm's server ARM chips used compression to avoid bottlenecks in memory bandwidth (but not increase capacity). Fun stuff, seems like somewhere more progress could be made.