Who Needs Every Token? : Adapting Dynamic Token Merging to Subword-level Transformers

Transformer language models apply uniform computation across input tokens, even though tokens vary in informativeness. MrT5 (Kallini et al., 2025) addressed this inefficiency in byte-level encoder-decoder models through Dynamic Token Merging (DTM), a learned deletion mechanism that reduces byte-level sequence lengths by up to 60% with minimal degradation in model quality. We investigate whether DTM generalises to subword-based Transformer architectures, where tokens correspond to information-dense subwords. To this end, we introduce MrBERT, which adapts the MrT5 delete gate to the encoder-only BERT-base architecture. Our gate fires after the first 3 encoder layers (out of 12), uses a PI controller to hit a target deletion rate δ, and introduces a novel pre-deletion blending mechanism that makes extractive QA viable under token deletion. Evaluated on four NLU task types — natural language inference (SNLI), sentiment classification (SST-2, IMDB), paraphrase detection (MRPC), and extractive QA (SQuAD, TyDi QA) — our results show: MrBERT at 30% deletion achieves 90.21% SNLI accuracy (only 0.27pp below baseline) at 1.89×A100 inference speedup; SST-2 drops by only 0.62pp (97.29%→96.67%), while MRPC shows a larger 15.00pp drop, revealing dataset-specific sensitivity; TyDi QA span EM recovers from 0.15 (no blending) to 0.39 (layer 9 + blending), nearly matching BERT baseline (0.40). A random-deletion baseline scores 2.95pp below MrBERT-30% on SNLI, confirming learned token selection. Beyond efficiency, the gate acts as an attention-sparsifying regulariser, preferentially pruning punctuation and low-information tokens while preserving content-bearing subwords. Our code is available at https://anonymous.4open.science/r/mrbert-anonymous-162A.