The behavior from llama-server I've seen in the past is that it fills the RAM, then completely fills the swap when the GGUF won't fit in available CPU-connected + GPU RAM. I plan to do some further testing watching iostat live and other metrics for level of constant ongoing writes to the swap, to see just how detrimental it could be to SSD write life.
back
1 comments
You should better not use swap at all, which eliminates all problems, especially on any system that has a decent amount of DRAM.
I have stopped using swap a quarter of century ago, and it was for the better.
I have seen swap advocates, but I do not agree with any of their arguments. I have encountered workloads for which the amount of memory in a computer was insufficient, so the OOM was invoked, but in all such cases I preferred to learn immediately about the existing problem and solve it by various means, e.g. increasing the amount of physical memory or reducing the amount of concurrent jobs, than to waste a lot of time because of not knowing why the performance was inadequate.
Ever since I stopped using swap everywhere I haven't had to deal with that really annoying situation where a server is still online but limping and you're hoping ps and kill respond quickly enough before the server falls over