Don’t Throw Away Data: Better Sequence Knowledge Distillation

Thank you for your review. Below are our responses, and we hope they address your concerns.

1. The open issue means this issue has been discussed in many papers[1], yet a satisfactory solution has not been found.

2. Yes, we used the top-1 output with beam size = 4 as the KD target. We didn’t experiment with beam-n methods because, as suggested in the Seq-KD paper. We observed that beam-search outputs are often too similar, so this would provide very little learning signal.

3. The computation of MBR-N is not simply N times the cost of MBR-1. For MBR-1, a large number of outputs still need to be generated, and MBR scoring must be performed. Our method, on the other hand, considers more candidates instead of just selecting the top-1, so the time required to generate KD data remains exactly the same. Regarding training time, here we report the effect of different n values under the same update step. For the En-De task, we tested XXS student distilled from XS teacher and XXXS student distilled from XXS teacher, using 10k instances and 10k training steps. The results indicate that our method surpasses the baseline MBR, further demonstrating its runtime efficiency, with the same number of training update steps, MBR-N achieves better performance than MBR-1.

The KD data we used is not large, but this is a reasonable setting [2]. For KD, a large dataset is not necessary. In particular, for LLMs, training with a small amount of KD data can already yield excellent results.

4. We acknowledge that experimenting with additional language pairs would enhance the results; however, due to limitations in computational resources, we have not yet explored this approach. The selection of language pairs and instances follows the setup described in [2]. Furthermore, our current results already demonstrate the significant advantages of our approach.

5. Thanks for the suggestion, we have included COMET22 results in Table 5. This is important to mitigate the risk of overfitting to the metric used in MBR (Bleurt) as part of our training as well as evaluation. We observe the same overall findings using COMET as Bleurt, confirming the utility of our method.

[1] Zhang, Chen, et al. "Lifting the curse of capacity gap in distilling language models." ACL 2023.

[2] Finkelstein, Mara, et al. "Mbr and qe finetuning: Training-time distillation of the best and most expensive decoding methods.", ICLR 2024.

Thank you very much for your suggestions for our article. We hope to answer some of your questions here.

1. For novelty, we agree that the idea is straightforward. However, we show that the method is also highly effective. Added to this, we explored in depth topics such as data efficiency and the curse of dimensionality. It was previously widely believed that generating multiple distinct references for a source sentence would harm model performance. However, our method has demonstrated that this approach can actually improve the model's performance on LLM.

2. Thank you for pointing that out. We will revisit the Method Section and clarify the issue. We would greatly appreciate it if you could specify which parts are unclear, as that would be very helpful.

3. Thank you for pointing out the missing important citation. We will add it to the related work section, as it is relevant. However, this paper focuses on the question of how to train multiple students together, whereas our aim is to train a single, stronger student. Since the research questions are different, a direct comparison is challenging.

** Here are the responses to your questions below: **

Q1: This is because the largest model was not fine-tuned on the specific translation data. However, it already demonstrates strong performance, and fine-tuning such a large LLM requires significant computational resources, which is why we did not fine-tune it.

Q2: The choice of PaLM follows [1]. We agree that testing with more LLMs would lead to a stronger conclusion. However, due to time and computational resource limitations, we were unable to conduct such experiments.

Q3: Since our focus is on testing KD rather than prompt learning, we opted for zero-shot, as it already produces good translation results. Therefore, we did not experiment with different shot configurations for generation.

Q4: We have reported the runtime in Table 6, which shows the time required to achieve optimal performance. For the En-De task, we tested XXS student distilled from XS teacher and XXXS student distilled from XXS teacher, using 10k instances and 10k training steps. The table below demonstrates that, within the same number of training steps, our method outperforms the baseline MBR. Additionally, MBR decoding alone offers a significant improvement over greedy/beam/gold standard training, justifying the more resource-intensive algorithm. In contrast, our extension is relatively lightweight. Due to time constraints, we have not run experiments with Beam/Reference, but we will include them in the revision if the paper is accepted.

5. The test set is single-reference, and the improvements are substantial. It was previously believed that adding diversity on the target side would not enhance single-reference performance, but our paper demonstrates that for LLMs, this diversity can indeed lead to better results. We also analyze the diversity of student models trained with MBR-n in the Diversity Section, and find that a greater number of diverse references actually results in less diversity in the generated outputs. This insight is an important innovation and contribution of our paper.

[1] Finkelstein, Mara, et al. "Mbr and qe finetuning: Training-time distillation of the best and most expensive decoding methods.", ICLR 2024.

After reading the comments from other reviewers and the authors' responses, the authors answered most of my questions. But I am still concerned that the innovation of this work is not enough to appear in the conference, I will keep my score.

Thank you very much for your feedback, we would like to clarify a few points here.

1. For novelty, we agree that the idea is straightforward. However, we show that the method is also highly effective. Added to this, we explored in depth topics such as data efficiency and the curse of dimensionality. It was previously widely believed that generating multiple distinct references for a source sentence would harm model performance. However, our method has demonstrated that this approach can actually improve the model's performance on LLM.

2. Regarding the effectiveness of our method, we conducted the main experiments using BLEURT in the paper. A score of 0.02 represents a significant improvement. Specifically, we observed a +0.015 increase in BLEU, approximately +0.02 in chrF, and around +0.05 in COMET22, compared to the baseline beam search decoding for XXXS student distillate from XXS teacher, as shown in Table 5. These changes are considered substantial and meaningful, particularly in comparison to state-of-the-art models.

3. The "Data Efficiency" section discusses the number of candidates, with Figure 5 showing how varying candidate numbers affect performance. Additionally, generating multiple candidates is not highly resource-intensive, as it involves inference rather than backpropagation. Also, the different metrics can affect the cost of MBR, like QE would render it linear in complexity, not quadratic. In data-limited scenarios, we believe our approach can yield significant performance improvements. Table 6 showed the optimal training steps required to achieve the best performance. The table below shows that, within the same number of training steps, our method outperforms the baseline MBR. For the En-De task, we tested XXS student distilled from XS teacher and XXXS student distilled from XXS teacher, using 10k instances and 10k training steps. Moreover, MBR decoding alone provides a substantial improvement over greedy/beam/gold standard training, thus justifying the more expensive algorithm. In contrast, our extension is relatively lightweight.

Here is the response to the question:

A: We extracted top-n MBR samples from the XS teacher model for 100 source instances, then calculated the average self-BLEU (by computing the BLEU score between different references for the same source sentence and averaging the results). The self-BLEU scores are as follows: MBR-5: 63.17, MBR-10: 62.28, MBR-20: 59.83, and MBR-40: 55.77. A higher self-BLEU indicates greater similarity among the references. As we increase the value of n, the similarity decreases. A self-BLEU around 60 suggests that the references are similar but not identical. We will include this discussion in our paper.

Thank you for the response!

I appreciate the new results, and I think they make the paper more comprehensive.

However, I still share the concern regarding novelty with Reviewer WsHB and remain negative about this submission.

We greatly appreciate your insightful comments. We will thoroughly review the paper, correct any typos, and refine the subsection titles to make them more concise and precise. First, we would like to clarify two points regarding novelty and effectiveness.

1. We agree that the idea is straightforward. However, we show that the method is also highly effective. Added to this, we explored in depth topics such as data efficiency and the curse of dimensionality. We agree that runtime convergence is an important and interesting direction for analysis, and our methods exhibit better runtime efficiency compared to the baseline. Table 6 highlights the impact of varying n on converge runtime. Here we will report the effect of different n values under the same update step. For the En-De task, we tested XXS student distilled from XS teacher and XXXS student distilled from XXS teacher, using 10k instances and 10k training steps. The results indicate that our method surpasses the baseline MBR, further demonstrating its runtime efficiency. This analysis and discussion will be included in the paper.

2. According to the effectiveness. Our experimental results demonstrate the effectiveness of our method. In the distillation of the small model XXXS from XXS, we achieve a BLEURT improvement of 0.02 compared to beam search, this improvement is substantial, particularly when applied to state-of-the-art (SOTA) models. , Additionally, notable gains are also observed when using BLEU, chrF, and Coment22 metrics (see Table 5). While we acknowledge that the impact of distillation on the large model is less pronounced than on the smaller one, this is partly due to the "capacity curse" and partly because the distilled small model is already close to the observed performance limit, making further improvements challenging. This is an open question on how to do well. Employing curriculum learning to address the capacity curse is an interesting direction. Although it is not the main focus of this paper, we include the experimental results as a side exploration. We intend to further investigate Stage training as a potential solution to the capacity curse in future work.

Here are the responses to your questions below:

Q1: Thank you for the suggestion; it is indeed an interesting direction that we plan to explore in future work. Averaging the loss across tokens could potentially enhance training efficiency. Additionally, MBR with a reference-free metric (QE) has the potential to radically improve the runtime. It does however require very strong QE performance, which limits application to high resource settings. But this is the direction the field is headed, with auto raters etc.

Q2: One reason for this is that our focus is on distillation from teacher models. The only requirement for the teacher models is that they must exhibit strong translation performance; we do not emphasize the specifics of their training process. We believe this does not impact the effectiveness of our method. Since training is resource-intensive, we prefer to reuse high-quality models and only fine-tune them if their performances are poor. That said, we agree that training the teacher model using the same method can help verify whether our approach is influenced by this factor. To test this, we fine-tuned the XS teacher model using the same method as the XXS teacher, using the WMT22 en-de data. The results show that our method still leads to significant improvement. The performance of the newly trained XS teacher is 0.7724, which is lower than the reported PaLM-XS performance of 0.8008 in the paper. The second row of below table corresponds to Table 2 in the paper, representing the En-De task with an XS teacher and an XXXS student.

Q3: As mentioned in the paper, this is an interesting open question. We have some hypotheses: it may be that adding more candidates improves learning of decoding distribution, as these candidates are of high quality. In contrast, with a single target, the output may be influenced more by the original LLM knowledge, leading to a more scattered distribution.

Q4: We have discussed those in Table 4, Figure 5, and the "Runtime & Data Efficiency" section. We use training steps to assess runtime efficiency and evaluate data efficiency by testing with varying amounts of data. In the Table we show above(the first Table in point 1), our method showed the runtime efficiency is also better than vanilla MBR. If you have any additional questions, we would be happy to answer them, or if you have other metrics suggestions, we would gladly consider them.