> I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
Whats up with focusing on the active param count? Do yall fiddle with the weights or something?
back
2 comments
Total param count decides how much vram you need to run it. Active param count decides how fast it runs. My 10 year old GPU can load quantized 35B or 27B, but it can’t process 27B parameters per token faster than 2-4tok/s, while it can do A3B at >40tok/s
Thank you Kenny
You can run these on CPUs at a somewhat reasonable speed.
Or (somewhat) low TDP GPUs for that matter, like workstation ones, that might have enough total VRAM but not the best bandwidth/compute.