We thank reviewer cEPL for the thoughtful comments on our work. We appreciate that the reviewer found our paper to be clearly written, well-organized, and supported by thorough experiments. We would like to address the reviewer’s concerns and questions in detail.

Weaknesses

1. The reviewer sees our use of a regularizer with an $\alpha$ hyperparameter as a limitation, and cites the binomial loss of Nawrot et al. (2023) as an alternative method. The main problem is that Nawrot et al.'s regularizer requires binary decisions, which would not apply to our “soft” deletion method. Furthermore, we see our regularizer as a more flexible variation; without the controller, MrT5 can discover its own compression rate, and with a controller, MrT5 can be pushed toward a specific compression rate. It is also worth noting that the binomial loss of Nawrot et al. (2023) also includes a hyperparameter that must be tuned, and their boundary predictor is less lightweight, requiring 1M additional parameters compared to our 4k parameter gate. Our method is also effective with only a minimal amount of continued pre-training. We do not believe that their method will compete in the same continued pre-training setup.
2. The reviewer says that merging of contextualized tokens needs to happen later in the network. This is not the case, and we place the gate at an early layer for the most significant compute gain. For the span corruption and downstream tasks, the gates were placed at an early layer (l = 3, which is 1/4 of the ByT5 Small encoder), which resulted in deletions with minimal impact on task performance. Deleting at any reasonable layer (l >= 3) did not result in a performance impact, as shown in Figure 4. We only placed the gate at later layers for the simulations, and this was due to the simple nature of those tasks, where even a T5 model only developed non-trivial attention patterns at later layers. The synthetic experiments were designed to demonstrate that contextual deletions can be learned, rather than to optimize inference runtime at an early layer.
3. It is not the case that the delete gate only targets the correct characters when there is too much deletion. We show that MrT5 is capable of achieving the ground truth deletion rate as specified by the synthetic task. The only synthetic model that had a steep drop in accuracy was the MrT5 model with $\alpha$ = 1e–3 trained on the Sequence Merge task. This model was actually deleting more characters than specified for the task, resulting in a performance drop, as noted in the description given in Table 2c. These are copy tasks, so dropping characters that must be copied makes the task significantly harder. The other models that targeted relevant patterns did not see a significant performance drop, showing that MrT5 can be tuned to have the appropriate amount of deletion for a task.
4. We appreciate the reviewer’s suggestion to explore additional baselines. The Toucan model (Fleshman and Van Durme, 2023) speeds up the cross attention in decoding. This is negligible for improving the inference runtime of ByT5, which carries the majority of the computation in the encoder. We will be actively working on experiments adapting the boundary predictor of Nawrot et al. (2023) for encoders and will report our findings in another revision.
5. We fine-tune all models on MNLI for 4,000 steps, which is ~10 epochs. The ByT5 authors fine-tune the model for 262,144 steps, which would be ~650 epochs, given their setup. We find this to be excessive for MNLI, and this is what we intended to convey in footnote 4. Due to resource constraints, we do not train for as many epochs. However, none of the claims in our paper depend on this comparison to the fine-tuning in the ByT5 paper. Our focus is on speeding up ByT5.

We thank reviewer 91Fx for the insightful comments on our work. We are happy that the reviewer found our hard/soft deletion method to be novel, and that they found the paper to be well-written. We address the reviewer’s comments and questions below.

Weaknesses

1. We appreciate the reviewer’s suggestion to include orthogonal methods as baselines; we will be actively working on experiments using the boundary predictor/pooling of Nawrot et al. 2023 and will report our findings in another revision. We note that the boundary predictor of Nawrot et al. is less lightweight, requiring 1M additional parameters compared to our 4k parameter gate. Our method is also effective with only a minimal amount of continued pre-training. We do not believe that their method will compete in the same continued pre-training setup.
2. Our goal in this work is to enhance the efficiency of ByT5 while preserving its task performance. We do not claim to compete with or outperform subword models. Nonetheless, we provide accuracy comparisons with a subword-based T5 model on character-level tasks: under identical training settings, our model surpasses T5's performance, which achieves 47.98% accuracy on spelling correction and 72.90% on word search, as reported by Huang et al. (2023). However, we reiterate that our focus remains on optimizing ByT5’s speed rather than benchmarking against subword-level models.
3. We do not claim that the model is stronger than ByT5. We provide significant inference runtime improvements to ByT5 while maintaining its task performance. The runtime gains after the token deletion operation are provided in Table 3, and they are quite significant.

Questions

1. Consider discussing the relation between this work and GemFilter (https://arxiv.org/pdf/2409.17422) as both pertain to token deletion.

Answer: We appreciate the reviewer’s reference to GemFilter, and we will add it to the related work. We would like to note that GemFilter is concurrent work, as it was made public on September 25.

2. The proposed method seems readily transferable to decoder-only models, which is the most widely used architecture nowadays. Would like to see some experiments, or at least some discussions about this direction.

Answer: Our approach is transferable to decoder-only models, but the deleted tokens must be re-inserted in order to perform next-token prediction—a nontrivial modification to our method. We are exploring this direction, but we found this to be out-of-scope of the current work that seeks to speed up ByT5. It is a great next step, though.

3. 150-151: "(1) we want to avoid the overhead of executing the deletion algorithm multiple times;" This motivation is better discussed from the perspective of "trade-off". If executing the algorithm multiple times can reduce the number of tokens/positions to process in later layers even more without compromising generation quality, then there is no reason to not do it.

Answer: By “overhead” we are referring to additional time required to execute the algorithm. We agree that this can be discussed as a trade-off, and we will incorporate this in the revision.

4. The regularizer loss only seems to encourage the increase of the number of deleted tokens, but does not encourage the gate output to converge to the extreme values (i.e., the min or max gate value). This could make hard deletion during inference less effective because merely setting a threshold may delete some "somewhat useful" tokens.

Answer: Throughout our experiments, we thoroughly tested that loss and task metrics for hard and soft deletion matched; in other words, soft deletion during training served as a true proxy for hard deletion at inference. We found that an additional regularizer that pushes the gate to its extreme values was unnecessary, and ultimately omitted it for simplicity. Here, we provide accuracy comparisons for hard and soft deletion on our downstream tasks. The results clearly show that there is little or no difference between the two.

We also thank the reviewer for the presentation and phrasing suggestions, which we will incorporate in the revision. We are grateful for the reviewer’s comments, and we hope that our response addresses your concerns and questions.

We thank reviewer uaeo for the thoughtful comments on our paper. We appreciate that the reviewer found our method to be simple yet effective for improving the inference speed of ByT5 while maintaining its performance. Below, we address the reviewer’s comments and concerns.

Weaknesses

While tokens are deleted softly during training, our continued pre-training approach requires only a small number of steps (3,000) to effectively adapt the existing ByT5 model.

Furthermore, models are typically used far more for inference than for training, so optimizing inference runtime yields the most significant gains in practical utility. In the current paradigm, the shelf-life of LLMs is very long; they are trained just once but used extensively for inference.

Take ByT5 Small as an example. In the past month alone, it was downloaded over 2 million times. If we assume this download rate—2 million times per month—has been consistent since its release three years ago, the model would have been downloaded approximately 72 million times. However, this is likely a conservative estimate, as download rates were probably much higher during its initial release.

If each download resulted in just one inference, the model would have executed 72 million forward passes for inference—dramatically surpassing the 1 million forward passes it executed during training, even with this conservative estimate. This highlights how optimizing its inference runtime would be much more impactful for its real-world usability.

Due to resource constraints, we only performed experiments on moderate model sizes. With additional resources, we hope to scale up our approach in future work.

Questions

The MrT5 gating mechanism only adds 4,417 parameters to ByT5 Small, so both models have ~300M parameters. The additional parameter count is almost negligible, especially when compared to related approaches. For example, the boundary predictor of Nawrot et al. 2023 introduces 1M additional parameters.

We would again like to thank the reviewer for their comments, and we hope that our response addresses any concerns and questions.

We thank reviewer T4gu for their thorough feedback on our work. We are pleased that the reviewer found our work to be clear, acknowledging that it is a lightweight addition to ByT5 that significantly improves its inference runtime. Below, we address the constructive comments provided by the reviewer.

Weaknesses

1. We recognize that our model’s zero-shot performance for non-English languages has a larger drop compared to English. While perfect zero-shot transfer from English to all other languages is not expected, MrT5's deletion gate—trained on minimal English span corruption data—demonstrates impressive zero-shot transfer capabilities across several languages. Furthermore, with just a small amount of additional pre-training on multiple languages (3,000 steps), the model's cross-lingual performance improves substantially.
2. We agree with the reviewer's suggestion to provide a more detailed analysis of the variance across languages in the XNLI task experiments to improve our discussion of the trade-offs. We will incorporate a more in-depth discussion of the multilingual results in the main text.

Questions

1. There is not much difference in performance between MrT5 and ByT5 for Chinese XNLI in Appendix Table 6, and there is ~20% length reduction - why do you think there is a difference to Chinese in the top half of Figure 3 with only a ~2% length reduction? Similarly, for Swahili, there seemed to be a significant difference in CE in Figure 3 zero-shot, but a smaller gap in Table 6. Why do you think that is?

Answer: The reviewer mentions great points regarding our multilingual span corruption and XNLI task results that we clarify here. For the XNLI task discussed in Section 6, we first train the MrT5 model using multilingual span corruption data. (This is the bottom model in Figure 3). After this continued pre-training, we fine-tune the model specifically on the MNLI task, which uses only English data. This approach means that the model’s gating mechanism has been trained on multilingual data, but solely within the continued pre-training phase.

Next, we evaluate the model—fine-tuned on English-only MNLI—on the XNLI task, which spans 15 languages. This evaluation is therefore a zero-shot transfer with respect to the XNLI task, as the model has not seen these languages during fine-tuning. Our results demonstrate that effective fine-tuning on downstream tasks can be performed in English only, yielding strong zero-shot performance across other languages.

2. I suggest addressing these inconsistencies in the main text and providing possible explanations for the differences between training and downstream task performance.

Answer: In Section 6, we state that we use a MrT5 model that was tuned with multilingual span corruption data before fine-tuning it on English MNLI data. We will emphasize this more and include a more thorough discussion of multilingual results.

3. How $\alpha$ was set was not super clear to me. L196: "For most of our experiments, we set $\alpha$ by hand, which allows the model to dynamically set the deletion ratio [based on the loss]." This paragraph says $\alpha$ was set by hand most of the time, but that it is easier to allow $\alpha$ to dynamically change. Do any of the experiments in the main text allow $\alpha$ to change? (It doesn't seem so?) Why not, if it is easier?

Answer: For the downstream task experiments (XNLI and character-level tasks) we use a controller to allow $\alpha$ to dynamically change in order to target a specific deletion ratio. For the synthetic experiments and span corruption task, we set an initial $\alpha_0$ and do not let it dynamically change, which nudges the model to delete, but ultimately allows it to discover its own deletion ratio. We found this to be the appropriate approach for those exploratory experiments, to study the behavior of MrT5 without being pushed toward a specific deletion ratio.

We appreciate the reviewer for the suggested citations of Cherry et al. 2018 and Limisiewicz et al. 2024, and we will include these in the related work of the revised paper.

We would like to thank the reviewer again for their insightful comments on our paper. We hope that our response has addressed your concerns and questions.

Downstream Tasks

For the downstream tasks, all models are trained with the same hyperparameter/training configurations and for the same number of epochs, as specified in Appendix A.3 of the paper. We adjust the prior for the models that use Nawrot’s boundary predictor in order to test different compression rates. For XNLI, we first train a model with multilingual span corruption data that uses Nawrot et al.’s method before fine-tuning it on XNLI, to be consistent with the MrT5 experiments.

XNLI

For XNLI, we train a model with prior=0.5 to have a comparable compression rate to MrT5. Across all languages, MrT5 outperforms Nawrot et al.’s method. We also note that the MrT5 model has more dynamic compression rates for different languages, adapting to the specific information density of a particular language. In Nawrot et al.'s paper, their experiments required setting different explicit priors for different languages, which is a limitation in terms of flexibility when compared to our approach.

Spelling Correction with Context

For the spelling correction task, we try two settings for Nawrot et al.’s method: one with prior = 0.5, and one with prior = 0.3. MrT5 outperforms Nawrot et al.’s method in terms of both task accuracy and sequence length reduction rates.

Word Search

For the word search task, we try two settings for Nawrot et al.’s method: one with prior = 0.3, and one with prior = 0.7. Even with much lower sequence length reduction rates, Nawrot et al.’s method has very poor accuracy and is vastly outperformed by MrT5.