The Brainformer building block is designed using neural architecture search.

"Brainformer consistently outperforms the state-of-the-art dense and sparse Transformers, in terms of both quality and efficiency. A Brainformer model with 8 billion activated parameters per token demonstrates 2× faster training convergence and 5× faster step time compared to its GLaM counterpart."

Note that the 8B model mentioned above has 158B total parameters. The authors compare the training time to other sparsely activated models, but seemingly not to dense models.