Hmm. No MoE or active params weights means this will run _slow_