When distilling models for speed, you get a better win from removing decoder parameters, since they are run in serial, than encoder parameters. For example see this work https://arxiv.org/abs/2006.10369

- paper co-author

They don't justify it explicitly, but they do talk about using the distilled model as an assistant for the original. With the encoder precisely the same for both you only need to additionally load the distilled decoder layers for a 2x speedup with the same accuracy as the original.