Item — HN Reader

Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test.

In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though obviously you need a 2TB server for full Kimi k3 and 1M context.

5 comments

The really big thing is the model weights, which are a read workload not a write one, it won't affect an SSD's write endurance.

The write workloads are just the context and any K/V cache - llama.cpp does not mmap those to disk, so they would remain in memory or VRAM as space affords.

I plan to give it a try in a day or two with llama-server from the main branch compiled today, when my Q8 GGUF download of K3 finishes, on a system with 256GB (should be more than ample for context and KV cache and a moderate chunk of the whole 1.6TB). If it works it's going to be sloooooooow as hell, but it'll be an interesting data point to see just how slow.

> Letting the kernel use SSD based swap space for something this big

Why would it swap? If stuff gets evicted from page cache is just gets reread?

The behavior from llama-server I've seen in the past is that it fills the RAM, then completely fills the swap when the GGUF won't fit in available CPU-connected + GPU RAM. I plan to do some further testing watching iostat live and other metrics for level of constant ongoing writes to the swap, to see just how detrimental it could be to SSD write life.

You should better not use swap at all, which eliminates all problems, especially on any system that has a decent amount of DRAM.

I have stopped using swap a quarter of century ago, and it was for the better.

I have seen swap advocates, but I do not agree with any of their arguments. I have encountered workloads for which the amount of memory in a computer was insufficient, so the OOM was invoked, but in all such cases I preferred to learn immediately about the existing problem and solve it by various means, e.g. increasing the amount of physical memory or reducing the amount of concurrent jobs, than to waste a lot of time because of not knowing why the performance was inadequate.

Ever since I stopped using swap everywhere I haven't had to deal with that really annoying situation where a server is still online but limping and you're hoping ps and kill respond quickly enough before the server falls over

It is much better to not use any swap in Linux, so you will never have this probem, or any other problem caused by swap.

I stopped using swap on Linux about a quarter of century ago, when it was a great improvement, and since then I have never seen a case when swap would have been useful, and I use Linux on a variety of laptops, desktops and servers.

Even if you do not use swap, you can have memory-mapped files that are much bigger than your physical memory. The LLM weights files must be mapped as read-only. In this case, the pages that have not been used recently will be freed when memory is needed to load other pages from the files.

The weights files must be mapped using huge pages, otherwise an excessive amount of physical memory would be wasted and reading new pages would be very slow.

It is likely that it is not possible to reach a good enough performance with a memory-mapped file without using carefully "madvise", with which it is possible to force the reading of the pages that you know that they will be needed in the future and also the freeing of the pages that you know that they will not be needed soon.

On Linux, it is possible to execute "madvise" asynchronously (with liburing). An alternative to liburing is to execute "madvise" from a concurrent thread, which synchronizes with requests to do "madvise" from the orchestrating thread.

When you mmap a file read only, the OS fills it in on a page fault by reading. There's no writes.

> Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months.

Optanes are a good option here, right?

I bought mine for $100 for each 128GB DDR4 stick. I believe write performance is off-the-charts on these, besides the fact they're c-h-e-a-p.