Presents a method for pruning Whisper’s ASR encoder (rather than the more commonly pruned decoder) by ranking layers via leave-one-layer-out change in Word Error Rate and removing the six least-impactful layers, cutting 18.5% of the encoder stack with no custom inference code required. The authors then use label-free distillation on unlabeled monolingual speech data to recover most of the resulting performance loss, bringing mean WER across four languages from 21.9% (zero-shot pruned) to 20.1%, versus an 18.2% baseline. Code and the pruned model are released publicly.
