Show HN: Getting GLM 5.2 running on my slow computer

After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way.

Some things that give it away to me:

- "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)"

- "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires."

- "Honest caveat from the same measurement: ..."

It's the new "It's not X, it's Y". I have no issue with this, I just found it amusing.

Cool project btw!

My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.

0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.

edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".

Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.

I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.

To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.)

In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it.

Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower.

[1] https://arxiv.org/abs/2401.10774

This is exactly the kind of technology that I expect Apple to ship anytime soon given the RAM prices and their HW/SW integration skills:

- ship super fast SSD (tbh they are already top notch)

- add a specific cache layer for tokens

- keep the amount of unified memory reasonable

I've taken a similar strategy w/ image/video gen at https://github.com/cretz/thinfer (see video branch for a ton of work).

Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coupled with some pipelining/double-buffering, I almost always end up compute bound not memory bound. Granted I use way smaller models heh.

llama.cpp supports a wide variety of 4-bit and smaller quants and mmap's models by default, so you dont need to be able to hold the weights in memory (the OS will handle bringing them in from storage as needed)

Its cool to see this implemented in a tiny amount of code without dependencies, but does it actually bring more performance?

Pretty cool! I've also been playing around with GLM 5.2 this week and was equally impressed. At work we're running it locally on some crazy expensive hardware as a test before starting another project so it's great to see people taking this massive FOSS model release and running it on an average machine, even if it's not terribly practical at this point.

Nice work!

Looking at the progress with DS4 project from Antirez and Colibri, are we probably going in the direction of having a stripe RAID of SSDs instead of RAM to get decent performance without losing a kidney to the RAM mafia? I really hope so.

The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD.

[0] https://github.com/JustVugg/colibri#ssd-wear-warning

This is the hacker spirit

I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2

I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.

>SSD Wear Warning

> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.

Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?

This is something that would benefit from Intel Optane memory. Too bad it was killed at the time.

I must say, this is incredible, nice work! I've been tinkering with local LLMs for a while now and experimenting with different tech, and I'd never thought it'd be possible to run such a large model on consumer-grade compute. It still feels like these days, you need to buy a $10K DGX Spark or something crazy expensive just to get to that level of performance but with experiments like yours, it looks like there's still a lot of room for improvement and innovation.

What I'd like to see personally is Small Language Models giving similar performance eventually (because do we really need that many hundred billion parameters?). Researchers have shown that a lot of attention heads in the LLMs we're using these days are barely even active, so the MoE architecture seems like a step in the right direction. I still wonder how far we can push things back towards a 1B or 3B parameters model while trying to reach similar performances as double-digit or triple-digit B params models.

“answering correctly on a machine that costs less than one H100 fan”

really love such comparison.

The best ideas are the ones that seem obvious. This is one of those ideas.

I'm not fully understanding this business of MoE so please forgive me if this is a dumb question, but would it be possible to use MPI with a small cluster to distribute the load?

How much time is spent interfacing between userland and the kernel? Can you try to get it to run as a kernel module? :)

Also in case your CPU is old enough, did you try disabling CPU bug mitigations?

This sort of thing is a lot of fun.

I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.

It's been interesting learning about the balance of factors for performant metal kernels on unified memory.

Should have a repo up on github in the next few weeks.

I'm curious but don't know much about the internals of LLMs - could you use a similar architecture with other models that have "layers"? I mean, could you have one layer do its work, then remove that layer from RAM, load the next layer from disk, and have that layer activate on the result of the first layer?

Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?

I have prototyped something similar with ollama some months ago.

Do you mmap or issue reads on demand? Also do you use io_uring to interleave compute with io or do you spawn extra threads?

I also tried predicting which experts get reused and I managed to beat a simple LRU very slightly.

EDIT: That was on Kimi 2.5 but even worse quant than 4bit. IIRC it was 2.6 or so

I wonder if you could replicate this in a Colourful GeForce RTX 50-series GPU, they ship it with 2 NVMe drive slots.

I only have 16GB of RAM in my laptop, but I would love to run a powerful model like that even at 0.5 tokens/s. For the questions I am asking, such a speed is more than enough. If anyone has any suggestions, please let me know! I am not an expert on LLMs, just a humble mathematician.

I believe you could use some sort of GPU-direct (or BAM-like approach, see https://dl.acm.org/doi/10.1145/3575693.3575748) to stream weights on-demand from the GPU.

I just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.

I love it but where do you find that NVMe SSD for less than the price of an h100 fan let alone the memory

I love seeing that kind of tinkering

Well under 1tps, but the fact it runs at all on a 16gb system is incredibly impressive!

I’m truly impressed by your work ! I don’t know if this is planned for near future, but how about adding energy efficiency benchmarks ? Because running locally is a great feeling, but the electricity bill should not be forgotten

So when I clone that repo, am I downloading > 370 GiB, or when and how is that being pulled in?

And can I leave out the web part? I cannot in good conscience run npm on my machine (it's not even installed).

I'm running an archaic Dell PowerEdge T710 dual-Xeon E5690 (6/12 CPU, 3.73GHz turbo) with 192GB DDR3-800 (was 288GB but now PassMark 14,328).

Token rate is 0.091 per second.

Good for an overnight job.

This is great stuff, I feel like fast disk (SSD) is a somewhat solvable problem, if you have many disks with the same content and a fast controller (?).

Is this similar to fastllm?

https://github.com/ztxz16/fastllm

I am curious if it's possible to use coli on ubuntu as coding agent, as i've got a machine with 32GB RAM and 2T SSD.

was lucky enough to snag the Olares One from Kickstarter just before this whole AI induced memory chip price gouging started.

specs are Intel Core Ultra 9 275HX (24 Cores, 5.4GHz),96GB of DDR5 5600MHz RAM, NVIDIA GeForce RTX 5090 Mobile GPU with 24GB of GDDR7 VRAM, 2TB NVMe PCIe 4.0 SSD.

going to see if I can wring at least 5 tok/s.

Why did you use Claude to help you make this rather than GLM? (It’s listed as a contributor in your repo)

Curious for what an MTP only result would look like, both in terms of output quality & tk/s ?!

Question to the OP, have you tested this on a machine where the entire model and context fit in RAM ?