MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods

Dear Reviewer,

Thank you for your very careful and thoughtful review of our paper. We appreciate your detailed feedback and will seek to individually address each your outstanding concerns here.

1. Some parts of the paper (e.g., the first three paragraphs of the introduction) talk about NLG in general but the paper focuses on machine translation. Reranking methods, in particular, rely on good quality estimation models that exist for MT but may not exist for other tasks. Of course, MBR finetuning can be applied to other NLG tasks (e.g., summarization) but this paper does not touch this problem. I think the authors should either revise the writing and make the scope more clear from the beginning, or perform experiments in other NLG tasks. The contribution is mainly empirical and only validated for MT.

MT is a common testbed for NLG tasks. In the abstract, we state the paper's scope: " Using the canonical NLG task of Neural Machine Translation (NMT), ...". Moreover, while the first three paragraphs of the introduction discuss NLG in general (establishing the motivation and setup in the most general terms), the fourth paragraph already narrows the focus to MT: "In this work, we focus on the NLG task of Neural Machine Translation (NMT)...". In the conclusion, we propose "extending MBR and QE finetuning to other NLG tasks besides NMT" as an avenue for future work.

Note that MBR decoding has been successfully applied to other NLG tasks including text summarization, image captioning, and data-to-text using BERTScore as the utility function, as described in https://arxiv.org/pdf/2311.05263.pdf (Model-based Minimum Bayes Risk Decoding; Jinnai et al., 2023). So applying MBR finetuning to these other tasks would be a natural extension of this work, and would likely perform well.

All of this said, given your concerns about the clarity of the scope, we have revised the first three paragraphs of the introduction, to make explicit from the very beginning that the scope is limited to MT.

2. The study is limited in terms of language coverage. They perform experiments on 2 language pairs only (mid & high resource), both for translation out of English. It’s not clear if the findings hold for lower resource languages and when translating into English. In particular, I’m expecting quality estimation models to be worse for low resource languages, which may end up affecting the quality of the translations produced with their method. Have you tried other languages? If so, what happens in that case?

The quality of MT metrics on low-resource language pairs and into-English language pairs has been evaluated in the WMT Metrics Shared Tasks. See results from WMT'22 (https://aclanthology.org/2022.wmt-1.2.pdf) and WMT'23 (https://www2.statmt.org/wmt23/pdf/2023.wmt-1.51.pdf), which show that 1) the MetricX metric (including the QE variant) generalizes well to language pairs unseen during training, including low-resource language pairs (e.g. see Hebrew-English in Table 8 of the WMT'23 Metrics Shared Task Report) and 2) BLEURT and MetricX-XXL perform well on into-English language pairs. Note that result 2) is not surprising given that most neural MT metrics, including BLEURT and MetricX-XXL, were finetuned on large language models which were pretrained mostly on English data.

While this paper focuses on mid and high-resource language pairs, existing evidence about the quality of utility functions for low-resource language pairs and into-English language pairs suggests that MBR and QE finetuning would likely work in these settings as well. This would be a fruitful avenue for future work.

3. The related work discusses other methods for training NMT models beyond MLE (e.g., RL methods) but none of them is used as a baseline.

For a correct comparison to RL baselines, we would need to implement them in our framework, to be able to run them on the same data and using the same utility function. Additionally, we would need to tune the RL settings to ensure a fair comparison. Thus, evaluating RL baselines is outside of the scope of this paper. Moreover, the MBR and QE finetuning methods proposed here use MLE, so comparison with RL methods would be tangential to this paper's focus.

(Please see the follow-up comment for individual responses to questions.)

Dear Reviewer,

Thank you for your thoughtful review of our paper. We appreciate that you recognize both the soundness and effectiveness of the proposed methods, as well as their importance for practitioners. We will seek to individually address each your outstanding concerns here.

1. The paper presents a section of the results from Palm2 finetuning and show large improvements. I think that is expected because it is a much stronger model. This set of experiments doesn't add too much value to the paper. The paper should also show the beam and MBR decoded results of the Palm2 model used for finetuning, so that the readers can understand show much of the teacher model performance can be transferred to the student via this finetuning.

As requested, we have added the performance of the en-de and en-ja PaLM-2 QE-reranked, MBR-decoded, and greedy-decoded teacher models in Table 10 and Table 12 in the new revision, respectively. Note that for en-de, the student model is strong enough that MBR finetuning brings its performance almost on par with that of the PaLM-2 teacher model (student Comet20 = 63.62 vs teacher Comet20 = 63.76). Also note that across both language pairs, the MBR-decoded teacher is strongest. Moreover, even though the PaLM-2 QE-reranked teacher is weaker than the greedy-decoded teacher, PaLM-2-QE finetuning achieves much larger gains that PaLM-2-greedy finetuning (rows 3a versus 3c in Table 2). This suggests that QE finetuning is able to smooth out bad QE translations, e.g. due to utility function failure modes (while QE reranking at inference time is not robust to bad translations), allowing the student to exceed the performance of the teacher. In particular, the en-de PaLM-2-QE-finetuned student achieves a Comet20 score of 62.35, exceeding its teacher's score of 60.86.

2. After using MBR/QE finetuning, do we still see a difference between beam decoding and MBR/QE decoding?

As requested, we have added Table 19 with MBR/QE decoding results for the MBR/QE-finetuned models, in comparison against the more efficient decoding methods of beam search, greedy decoding, and epsilon sampling. (In addition, we have also added the performance of not only QE-finetuned, but also MBR-finetuned, models to Table 4.)

We see that after MBR finetuning, QE reranking works better than MBR decoding. Similarly, after QE finetuning, MBR decoding works better than QE reranking. This suggests that using the same decoding method both to generate the finetuning data and at inference time overfits to the metric used as the utility function. Also note that the gap between MBR/QE decoding and beam search/greedy decoding is small for the MBR/QE-finetuned models, and for Phases 2 and 3 of the QE-finetuned models, as well as Phase 3 of the MBR-finetuned model, beam search or greedy decoding is actually the top-performing decoding strategy.

3. In Table 1, we see 2a perform better than 1c) and 2c) perform better than 1b). This is a bit suprising because I would have assumed that finetuning performance will be upper bounded by MBR/QE used during run-time. I couldn't find any comment on this comparison in the paper.

MBR decoding and QE reranking can produce translations with high utility score but low quality, due to honeypots/failure modes of their utility functions. While this directly degrades performance of the teacher models at inference time, the MBR and QE-finetuned students are able to smooth out these bad translations. This explains why student performance is not upper-bounded by performance of the teacher and can, in fact, exceed it.

This hypothesis is supported by work from Amrhein and Sennrich, 2022 (https://arxiv.org/pdf/2202.05148.pdf) which shows that, despite the strong performance of MBR decoding and QE reranking on average, these decoding methods suffer from failure modes which can be attributed to underlying problems with the utility function. They performed the study using Comet20 as the utility function, and showed that this metric is not sensitive enough to discrepancies in numbers and named entities. MBR and QE finetuning, on the other hand, would not necessarily be as susceptible to these same failure modes since, even though the model would be exposed to these extreme examples, its representation would be smoothed out over the course of training.

4. The paper does not have any numbers on lexical metrics like chrF. Would have been nice to include those numbers atleast in the Appendix, for interested readers.

As requested, we have added performance on chrF to Table 8 (for English-German) and Table 11 (for English-Japanese) in the Appendix.

We hope the additions we have made to our paper allay your concerns and will be taken into account when assigning your final score. Please let us know if you have any further questions or if we can provide any additional clarifications to help finalize your assessment of our paper.

Dear Reviewer,

Thank you for your constructive and thoughtful review of our paper. We appreciate your recognition of the soundness and practical importance of the proposed methods. We also appreciate your taking the time to carefully review our experiments, and your acknowledgement that they are comprehensive and rigorous. To allay your outstanding concerns, we will respond individually here to each of the weaknesses and questions that you raised.

1. The novelty of the method is unclear. Using high-quality pseudo data generated by better decoding methods is a common technique in neural machine translation. This paper does not seem to provide new insights beyond what is already known.

The review states that "using high-quality pseudo data generated by better decoding methods is a common technique in neural machine translation", but does not cite any papers to support this claim. We are not aware of any such work, so if calling into question the novelty of MBR and QE finetuning, please kindly provide evidence that "this paper does not seem to provide new insights beyond what is already known". Note: We show that self-finetuning using beam search output does not improve quality, so the decoding strategy used to generate the finetuning data is crucial to the success of this method.

2. While this paper reports significant gains with MBR and QE finetuning, the reasons for the improvements are not well analyzed. In some cases, QE finetuning performs better, while in others, MBR finetuning is superior. More details and an in-depth analysis of the advantages and limitations of each finetuning method would be expected.

This paper does provide an explanation for the improvements observed with MBR and QE finetuning in paragraph 2 of the Discussion section (Section 7), and also provides a comparison of MBR versus QE finetuning in paragraph 3. Both paragraphs are pasted below for reference:

1. Explanation for improvements (Section 7, paragraph 2): "Given the effectiveness of an external teacher model (which differs in architecture, training data, etc. from the student), the success of these finetuning methods can likely not be primarily explained by mitigation of the label exposure bias problem (Schmidt, 2019), since the data generated from an external teacher is less similar to the data the model has already seen (during previous training stages) than self-generated MBR or QE data. Instead, the effectiveness of MBR and QE finetuning seemingly can be attributed primarily to the high quality of the translations used as finetuning data."

• Also note that the experiments were designed so that each of the 3 phases answered a different question regarding the effectiveness of MBR and QE finetuning (see Section 5.4). For example, by comparing Phase 1 and Phase 2, we show that using a larger monolingual corpus to generate the MBR and QE finetuning data improves performance, and by comparing Phase 2 and Phase 3, we show that using a stronger teacher model also improves performance.
• Also see Appendix D for extensive ablation studies which isolate the effect of various source-side and target-side variables on the performance of MBR and QE-finetuned models. For instance, we show that QE finetuning on forward-translated data is more effective than QE finetuning on backtranslated data (where the QE translations are on the source, rather than target, side). All of these ablations serve to understand the reasons for the observed improvements.

2. Comparison of MBR vs QE finetuning (Section 7, paragraph 3): "When comparing MBR against QE finetuning, we see that QE tends to perform at least as well as MBR finetuning using a self-teacher, while MBR outperforms QE finetuning using the LLM teacher. We hypothesize that the sampling translations from the self-teacher do not closely approximate the true reference distribution, so using them as pseudo-references does not help (or even hurts), relative to using a reference-free (QE) metric. For higher-quality models (e.g. PaLM-2), the candidate translations are good enough approximations to references that they provide a useful signal."

• We also mention in the introduction that generating data via QE reranking is much more efficient than generating data via MBR decoding: "Despite its quality advantages, the slow inference speed of MBR decoding remains a limitation even when generating distillation data". In terms of computational cost, this is the key limitation of MBR finetuning. So the limitations of MBR relative to QE finetuning both in terms of quality (MBR finetuning requires a stronger teacher) and computational cost are included in the paper. Also see Table 7 (added to the paper based on your comment #3).

If 1. and 2. above do not satisfy the requirement for 1) an analysis of the reasons for the improvements due to MBR and QE finetuning and 2) a comparison of the advantages and limitations of MBR vs QE finetuning, please do clarify what is missing.

Dear Reviewer,

Thank you for your careful and thoughtful review of our paper. We observed that you lowered our score from 8 to 6 on November 14, and our hope is that we can address all of your outstanding concerns here.

First, to address your concerns brought up in the initial revision of the review:

Acknowledged that the experimental settings section of the paper is "dense". If there is anything unclear about the setup, please do specify and we will be happy to make changes to clarify. We have added Table 6 in the Appendix (with a reference to this table in Section 5.4) with a summary of the experimental settings (in particular, the datasets used for each experimental phase). We hope this will make the experimental settings section more easily digestible with a single pass.

When the score was revised down from 8 to 6, we observed that the diff of the revised review with respect to the original review was the following sentence: "Some missing points of comparison are approaches that leverage feedback for MT."

Could you kindly clarify which approaches that leverage feedback for MT are missing? Since no citations were provided, we are unable to comment on this further without additional information. Also, please note that leveraging feedback for MT is not the topic of this paper, so such a comparison may be out of scope.

We will now respond individually to each of your questions:

1. Is there any description of MetricX-XXL-QE in another paper? I could not find a paper describing except a paragraph in the Findings paper. How many languages does it support? This seems to be a very big QE model, 30B parameters.

The description of the reference-based counterpart to MetricX-XXL-QE (trained on the same human judgements data and with the same model architecture) can be found in https://www2.statmt.org/wmt23/pdf/2023.wmt-1.63.pdf (MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task; Juraska et al., 2023). The only difference between the original MetricX model and the QE version we use is that the QE model input is constructed as the concatenation of the source segment, rather than the reference, with the candidate translation (as described in paragraph 3 of Section 2.3 in our paper). This model has a mT5-XXL backbone with 13B parameters.

As shown in the WMT'23 Metrics Shared Task Report (https://www2.statmt.org/wmt23/pdf/2023.wmt-1.51.pdf), MetricX generalizes well to language pairs that it didn't see during training, including low-resource language pairs. See, for example, Hebrew-English performance in Table 8.

2. How much time is added to the training process when using the MBR and QE reranking? Maybe a table illustrating the added wallclock time would be interesting to contrast with the gains in performance.

To address your question, we have added Table 7 (Appendix B) in our paper revision, which shows the time and compute requirements for performing MBR-BLEURT and QE-MetricX-XXL scoring, across all three experiment phases and across both language pairs. We use TPUs for BLEURT and MetricX-XXL scoring, as described in https://dl.acm.org/doi/fullHtml/10.1145/3360307. (The TPUv4 hardware is a faster and newer generation than TPUv3.) MBR scoring takes about 7.65 TPUv4 minutes/source sentence, while QE scoring takes about 4.2 TPUv3 seconds/source sentence. So even when using faster hardware for MBR, QE scoring is 109x faster. Please do let us know if you have additional questions about this new table.

3. How much does the data used to train the QE model and the utility function used in MBR affect the final results for specific domains? Do yo have any insights on this? For example, for English-German, it seems most of the MT training data is similar to the data used to train the QE and BLEURT models. What if this is not the case? Do you observe the same improvements or does something change in the translation quality of the best models?

In the WMT'22 Metrics Shared Task (https://aclanthology.org/2022.wmt-1.2.pdf), the metrics (including BLEURT and MetricX-XXL) were evaluated on various domains, and they performed well even on the domains that they were not finetuned on (see Figure 2).

Moreover, just as the utility functions generalize across domains, the same is true for the MBR-finetuned and QE-finetuned models which use these utility functions, as shown in Table 9 in our paper. Also note that while the Phase 1 models were finetuned on past WMT test sets (with the source side partially overlapping the data used to train the utility functions), the Phase 2 and Phase 3 models were finetuned on Common Crawl data, and we do see that finetuning on Common Crawl improves generalization across domains.

We hope the additions we have made to our paper allay your concerns. Please let us know if you have any further questions or if we can provide any additional clarifications to help finalize your assessment and rating of our paper.