back
138 comments
Coincidentally I've been playing with small (~30B) self-hostable models for coding tasks today -- specifically plugging them into Cloudflare OS (which I work on) and asking each to build a collaborative whiteboard.

I'm finding that the Mixture-of-Experts (MoE) models (Qwen 3.6-35B, and Nemotron 3.5 Lightning) are, well, terrible at this. They just couldn't get the job done at all. Went way off the rails. They are really fast though!

Whereas ~30B dense models (not MoE) are pretty decent. I tried Muse Glimmer, Gemma 4-31B, Qwen 3.6-27B, and Laguna XS[0]. They were all able to build a working collaborative whiteboard app, without any guidance (other than feeding back error logs to the model). I also asked each to then draw a monkey by calling the API of the whiteboard it has just built. Laguna drew random scribbles but the rest all managed to produce something monkey-like.

(Frontier models in comparison will write the app in one shot with no errors at all.)

Note that both Qwen 3.6 and Gemma 4 each have both MoE and dense variants. I find this very confusing, because e.g. ollama's model index typically only distinguishes variants by their size, but MoE vs. dense makes a huge difference in how they actually perform. IMO they should use a suffix, like Qwen 3.6-moe vs. Qwen 3.6-dense, or maybe Qwen 3.6-fast vs. Qwen 3.6-smart...

[0] EDIT: Turns out Laguna XS is MoE, I misunderstood. It performed similarly to the dense models. But maybe this explains why it couldn't write code and think about monkey shapes at the same time!

> ollama's model index typically only distinguishes variants by their size

Don't use ollama. The entire project is just a series of stupid decisions like this.

There is an old rule of thumb that says the quality of an MoE is equivalent to a dense model with the geometric mean of its total and active parameters. So, for example, the Qwen3.6 would be equivalent to a dense model with approximately sqrt(35×3) ≈ 10.25B parameters.

Both MoEs and dense models are always getting better, so I don't think this comparison is meaningful across generations. But still for a first approximation, this tends to hold (you wouldn't expect a lot from a 10B model in coding yet).

I've been pretty impressed with Laguna. I downloaded their coding agent and have used it for a task here and there (the larger variant). Obviously it's nothing like a frontier LLM, but it surprised me with how good it was. And I think the model personality and way it talks is pretty pleasant
I’m confused about the naming suggestion. Seems like the AxB bit differentiates pretty clearly, no?
Laguna XS is MoE, however.
Ollama uses "consumer UI guidelines", it's the second worst AI tool to use for anything more than writing fanfic.
What’s good for data extraction? I have a hard time getting models to just pull names and titles from a blob of text.
One major consequence of the ramapocalypse, I think, is an even higher focus on small efficient models. I personally believe that the multi-trillion parameter models are fundamentally missing things and the push to smaller, more efficient will drive evolutionary structural changes that will lead to future gains
I'd assume the closed weight models are all working on shrinking their parameter counts anyways. They too benefit from smaller models. It'd be foolish for these SOTA labs to not be working at reducing parameter counts.
I would say even without rampocalypse there would still be the strong incentive to innovate at the edge and under more extreme constraints. The incentives are just even stronger now.

I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.

This is just temporary though, right?

With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.

I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on some good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the cost.

LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.

Maybe we'll have model weights being shared like game cartridges.

I am literally betting my company on this being true.
Let's not forget the Bitter Lesson. Small models sound really nice but at some point you're just fighting the laws of information theory. Efficiency gains on the small model side are nice, but efficiency gains + giant model tends to be even better...
I think the path of least resistance will end up being the cheapest and that is scaling up the parameters a ridiculous amount until you get highly capable models that can develop/distill/design the RAM efficient models. Going straight for low param is foolish and just a cope by smaller labs because they don't have the compute/talent to train the large ones.

This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.

Many on HN are in deep denial about this imo.

I hope ssd streaming gets more popular, maybe more breakthroughs like that will help change things
Right now there is a stark difference between what smaller models can do and bigger models can do.

Smaller models are suitable for simple tasks like classification / summarisation while larger models are better in agentic capabilities.

I honestly hope to see this across all applications, games, services, operating systems, etc. We've been in a period of wasteful RAM usage for over a decade. Constraints, whatever their origin, can be a good thing.
Really? I feel like because nobody has RAM they're being pushed to the cloud frontier models. If we could all have our own 64GB+ VRAM GPUs, I feel like the open weight model scene would be even stronger.
> NeMo Switchyard, an open source library for smart routing

> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job

How do routers like this handle prompt caching when you send the second request?

Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.

I've seen ones that are configurable to pick a trade off point between lower cost (cache stickiness) and routing performance (best model for that turn).

But yeah I'm skeptical all this overhead is worth it.

https://github.com/NVIDIA-NeMo/Switchyard#routing-strategies

Looks like your great question doesn’t have an answer, but looking at the routing strategies things get even more confused, since the proposed ones tend to rely on extra llm calls to determine which model to pick.

The nice thing is that it makes sense for specific setups, less conversation oriented.

As an example, you need to classify batches of data, and have many fine tuned models. Or you need to do speed to text and need to pick which whisper to use.

You can write your own strategy, in that case an harness with subagents would be able to leverage this, picking the right model and then keeping its session sticky, but overall the lack of concern for caching points towards use cases where you do not gain much from it.

The repo is probably a better entrypoint to it, bit more concise description than the press releases: https://github.com/NVIDIA-NeMo/Switchyard (Notably: "Experimental software. Not for production use."). Unclear if they actually want you to deploy it or not, press release says yes, README says no, do with that what you will.

Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.

Caching should be possible as long as all the models use the same shared cache. The models don't even need to be running on the same server if the shared cache is distributed.

I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.

I would like to propose something:

- problem: massive deluge of information because of AI

- solution: human beings should adopt a minimalist style of communicating in writing.

- e.g. this entire website page can be ten bullet points.

There's a famous Pascal quote, "I have only made this letter longer because I have not had the time to make it shorter."

Communicating an idea concisely is difficult. Most people struggle to get ideas across at all, asking them to do it well with fewer words is often out of reach.

While I agree with the spirit, I don't think the solution to bad prose is slightly less bad prose. We can write good prose instead.
Why use more words when few do trick?
k
Judging by how my nieces and nephews text this has already happened!
In my opinion: the only way forward is zero-knowledge-proof authenticated social media.

We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate.

If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.

it takes more effort to write with conciseness
hell yeah
They conveniently decided not to include the Qwen range of models in the Artificial Analysis graph, except the out-of-league Max variant. At least be brave and honest.
24 comments so far about Nemotron on this earlier submission: https://news.ycombinator.com/item?id=49257947
I love the wave of new small model releases. Pleasantly surprising that an NVIDIA model runs so well on Apple Silicon using MLX! I was using nemotron-3.5-lightning:30b-mlx with OpenCode on my old (cheap) Mac this morning and no bad experiences except for running slowly.
An open source model from Nvidia is a free drug to later buy their chips.
Nemotron 3.5 Lightning runs on how little GPU vram? Can q4 run on 16gb?
Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?"

Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).

Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).

The new Meta 30B models seems A LOT better:

https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...

Muse Glimmer 30B seems to be on par with Qwen 3.6 27B (4 months old)

but

Qwen 3.8 27B is dropping this week...

The top 4 models on that site are all variants of Gemini Flash? That does not match my experience at all.
big week for open models... seems like companies are noticing the 26-35b sweet spot... though I think a 12b-a1b-MoE model would be helpful for the 16gb folks
Good luck getting it to run with NVFP4 on a DGX Spark, the very architecture Nvidia created that format for.