Item — HN Reader

Seems it’s not fully supported in mainline llama.cpp yet https://github.com/ggml-org/llama.cpp/pull/25165

In the huggingface link they mention building for CPU and for CUDA, does anyone know if that means it wouldn’t be possible to build targeting Vulkan?

1 comments

Replying to myself, seems this PR was merged into main and it the model does work with a Vulkan backend on my Framework desktop, I’m getting about 220 tok/s prompt processing and 21 tok/s output on the 4-bit quant. This is really a sweet spot imo on this machine between maximizing ram use and still having decent speed due to the expert size. This looks really promising.

What's your hardware? I've got 64GB RAM and a 4090 here, wondering if it's worth a play.

It’s the Strix halo (AMD) with 128 GB shared memory. The 4bit quant is ~75GB.

Unfortunately I don’t know about the best way of running on an Nvidia gpu, you could try llama.cpp and offloading as many layers as possible into the gpu and using RAM for the rest, not sure if that would slow it down too much though.

I got usable token rates (10-20tps from memory, so marginal) with the Qwen A10B a while back, well before all the new speculative speedups landed in llama.cpp. There's an unmerged branch which allegedly supports this, but I don't know how well yet. Looks worth investigating but I might give it a few days to see what bugs get shaken loose.