My understanding is that the chip carries the neural network itself while the weights are loaded on top, so any model that uses that same network topology should be loadable on demand.
back
2 comments
No I think you are thinking of Etched/Sohu.
Taalas' approach (at least for their demo'd product) is to bake the whole thing in completely. IIUC the optimiser can even see the weights while generating RTL. It's like there's an "uint8_t weights[] = " in the source code.
They are loaded on top in the sense that they are contained in the upper layer(s) of the chip. So when you want to change the weights, you have to produce fewer masks for fabrication, which reduces cost and time to market.
Oh... That's not great - it'd be nice if it had a way to push updates without building a new chip.
OTOH, maybe because of this our future cyberdecks will have cartridge ports.
I believe the fully baked-in nature is pretty important for the perf they get. With a cartridge port you now have a bus between the weights and the compute and the weights and that bus can become a bottleneck.
So I think we're looking at a spectrum here:
- Fully fixed function - i.e. Taalas
- Fixed function transformer unit (or whatever other AI architecture) with a "cartridge" for weights etc - i.e. Etched
- Flexible TPU/GPU type stuff
I think the middle of the spectrum is a bit of a dead space ATM because new models have recently been coming with significant updates to the architecture (like MoE, MTP) so by the time you have new weights you wanna load, you also want to replace the compute too. So really you probably either say "I can tolerate an old model, but I want it fast as FUCK" and go for Taalas-style, or you say "I want a near-frontier model" and you have to use flexible compute anyway.
But, caveat: this comment seems to be making me sound more knowledgeable than I actually am. Take this with a grain of salt.