Learning from others' mistakes: Finetuning machine translation models with span-level error annotations

Thank you for reviewing our work. Responding to your individual points:

1. [MQM data is hard to largely achieve: Compared to other MT-eval annotation data at the sentence level, like DA, MQM data shows more detailed human evaluation. However, it is also hard to largely achieve (even for the DA dataset, it only covers 10+ languages and hundreds k samples through years of work done by WMT).]

• We agree with the reviewer’s description of the trade-off between information density and annotation expense with MQM vs. DA. However, even though MQM data may be expensive to collect, we believe that there is immense potential even in just effectively utilizing existing data, as well as data that will be collected anyway for the evaluation of MT systems.

2. [Feasibility aside, if we only focus on the generality of this technique, this method is hard to generalize to other domains, like QA, as it is hard to say that span annotation also applies to QA data collection.]

• We respectfully disagree. In fact, Wu et al 2023 [1] show in a human study on long-form QA that annotators spend a comparable amount of time providing fine-grained feedback as they do providing preference feedback. Given the similar amounts of effort, QA is likely a promising domain to consider annotating fine-grained feedback for. Other such settings with promise could include reducing toxicity or hallucinations, as annotating a sequence as toxic or “with hallucination” requires identifying the instance of toxicity or hallucination as a precursor.

3. [The baseline is not strong: 1) The baseline model lagged behind the average performance of WMT submission quite a lot. 2) In Table 3, the SFT setting improves results a lot. This gain from SFT is weird if their base model is strong. / Could you show some results of Table-3 when using a stronger model? E.g., apply TWA on M2M100 or NLLB.]

• We agree that to achieve state-of-the-art performance, we would want to invest more effort in building a stronger pretrained base model. In our case, we used only the WMT’23 training data in our experiments to tightly control all the data seen by the model. Even so, the fact that TWA is able to improve performance on top of the gains from SFT demonstrates the promise of the proposed method in utilizing annotation information, not just exploiting a different translation quality between model and data.
• We were unfortunately unable to rerun the experiments from Table 3 with a different model, but we were able to run TWA using MQM annotations of the model’s own translations, in order to simulate the setting where the base model is of comparable quality to the data used in TWA (new section 6.5). We find in this setting that TWA significantly improves performance, from 4.203/0.429 Metric-X/COMET to 3.710/0.456 Metric-X/COMET. This suggests that TWA can offer performance improvements based on annotation information alone, even when there is no difference in translation quality between the model and finetuning data.

4. [Since DPO and SFT are concepts from the LLM community, it would be beneficial to show results on LLM-based MT. (I don't believe it's essential.)]

• Thanks for the suggestion. We were not able to run these experiments at this time, but we agree with the reviewer that it would be useful future work!

(continued in part 2)

Thank you for your review. Addressing your concerns:

1. [Novelty. The main novelty of this work is utilizing additional annotations to improve translation systems, which is not surprising. Otherwise, the proposed unlikelihood training is straightforward.]

• In terms of novelty:
  • This work is the first to propose utilizing MQM data, a readily available source of data in the MT community, to directly finetune MT models. The setting of training on offline annotated data is also a relatively underexplored area.
  • The proposed method is the first to combine the specific concepts of unlikelihood on errors, span-level error loss, and ignoring off-trajectory tokens to utilize offline, annotated data.

2. [Getting annotations is costly. The authors propose to utilize existing annotations, which is scarce. Although in the limited data setting, the proposed method is better than DPO, it’s likely that DPO is still much better in terms of annotation cost.]

• It’s worth noting that in this setup, annotating preference pairs directly would be expensive as we find it to be important to consider every possible pair, i.e., (10 choose 2) = 45 pairs for every source. This is in contrast to annotating each of the ~10 translations per source separately. One could alternatively annotate the translations at a sequence level rather than a finer-grained level and then construct preference pairs programmatically. However, in the case of assigning MQM scores, a sequence-level MQM score is the sum of the scores of the error spans, making the fine-grained annotation of TWA comparable to the sequence-level annotation needed to construct data for DPO.

3. [Relatively weak experimentation. The authors only evaluated two translation directions in one dataset, which may be below standard practice of translation papers.]

• Respectfully, we disagree that this is below standard practice; for instance, multiple translation papers accepted at ICLR last year experimented with two language pairs, e.g., [1][2][3]. We do, however, agree with the reviewer that more language pairs would be better and are working to add an additional language pair (results forthcoming).

4. [How efficient is TWA training compared with SFT?]

• TWA training is as efficient as SFT, utilizing only one model in memory (in contrast to DPO, which requires two models) and needing just a single forward pass to compute the loss.

Thanks again for your review. Please let us know if you have any additional questions or concerns. If not, we would be grateful if you would reconsider your score.

Thanks for the response! A few quick points:

1. The proposed method is the first to combine the specific concepts.

I agree with the authors that the combination may be novel, but I also think the concepts individually aren't very novel. I am happy to leave the decision to the chairs.

2. annotating preference pairs directly would be expensive as we find it to be important to consider every possible pair

I may not understand the authors' point fully. I assumed if it is important to consider every possible pair, that would also apply to annotating error spans. All in all, I appreciate the discussion, which perhaps can be checked with concrete experiments.

Overall, none of my concerns are really addressed, and I maintain my initial assessment.

Thank you for your thorough review. Addressing each comment under weaknesses point-by-point:

1. [Training Data Overlap: MetricX-23 is fine-tuned on MQM WMT’20-’21, and TWA is also trained on this dataset. This overlap suggests that evaluation might leak into training, disqualifying MetricX-23 as an evaluation metric in this setup.]

• As mentioned in section 5.4, we intentionally use MetricX-23, which is fine-tuned on MQM WMT’20-’21, alongside COMET-20, which is not, in order to test the benefits of the proposed approach under an evaluation that is sensitive to the specific information found in MQM data as well as one that is not. This guards against over-indexing to either evaluation.

2. [Motivation for Task: The task is not well-motivated. Obtaining fine-grained annotations is costly, and it’s unclear why methods are needed to utilize this type of supervision. Although this is discussed in the third paragraph of the conclusion, it comes too late (it is better to be addressed in the Introduction), and the motivation largely pertains to other tasks that might benefit from TWA techniques. This raises the question: why focus on machine translation instead of these other tasks?]

• The reason we focus on machine translation is because it is an application where large amounts of fine-grained information is already readily available. Positive results in this setting can motivate the collection of fine-grained results in other settings, which need not be more expensive than sequence-level annotations in many situations. We have incorporated additional discussion in the introduction as suggested; thank you for the suggestion.

3. [Choice of Offline Learning: It’s not well-explained why offline learning is favored over RL-based models. Efficiency might be one reason, which could benefit from further discussion and experimental analysis.]

• As mentioned in the related work, we do not directly benchmark against RL-based methods as they 1. are more memory intensive, requiring at least one additional model in memory to output rewards, 2. are much more difficult to optimize than direct finetuning methods, and 3. do not take advantage of the (already available) offline examples themselves, besides learning a reward model to predict their annotations.

4. [Design Choice Clarity: The design choice mentioned in footnote 1 on page 4 lacks adequate explanation.]

• Thanks for the feedback. We have updated it to: “Under the MQM rating system, some major errors are given a score of -25 (namely those categorized as non-translations), but we use a weight of -5 for these errors as well.” The primary reason was for simplicity.

5. [Evaluation Choices: The choices of evaluation metrics and experimental designs are not well-justified.]

• See above on our motivation for including Metric-X and why we do not compare against RL methods.

6. [Statistical Analysis in Section 6.1: The statistical test mentioned in Section 6.1 lacks detail. It’s unclear what test is used or how it’s conducted. More clarity here would improve the reader's understanding, especially when there is only one instance of each model.]

• We’ve updated the section describing the test in section 6.1. In short, each model is associated with a distribution over source-translation scores (Metric-X or COMET), and we run a permutation test between all pairs to see if results are statistically significant under the null that the scores for each system come from the same distribution. Then, we turn these pairwise significance results into a global ranking via a greedy algorithm that creates a new cluster when a new system is significantly worse than any of the prior systems.

(continued in part 2)

Dear Authors,

I hope this message finds you well.

Given the valuable discussions we've had, I am happy to increase my score. Unfortunately, due to my timezone, I am unable to wait until the last minute for your final responses to the remaining points. I would be happy to leave the interpretation of any further responses to the chairs.

Thank you for your commitment to the discussions

Thank you for your review and for appreciating the novelty and extensibility of this method! We’ve addressed your questions and suggestions below:

Questions:

1. [What was the motivation to include a DPO baseline?]

• We included DPO as a baseline due to its increasing popularity as a post-training method. DPO represents a baseline that translates MQM information into pairwise comparisons via sequence-level information only, in contrast to TWA which utilizes the fine-grained MQM annotations directly.

2. [Clarification: in the SFT baseline, does the finetuning of the base model involve training with MT triples from the MQM data (without annotations)?]

• Correct, SFT ignores the annotation information and only trains on the source-target pairs.

3. [Were there any discussions about evaluation on span-based MT metrics like XCOMET (https://arxiv.org/abs/2310.10482) or GEMBA MQM (https://arxiv.org/abs/2310.13988)?]

• We did not consider span-based MT metrics but agree with the author that including these more widely in MT evaluations would be useful!

Suggestions:

1. [Please include a few more qualitative examples in the Appendix.]

• Great suggestion, we’ve added to the appendix some examples from the Zh->En experiments between TWA, SFT, and DPO.

2. [Please release the code/path to corresponding data after the process.]

• Thanks for the suggestion; we are unfortunately not allowed to release the code, but the MQM data is publicly available here, and the pretraining and evaluation data can be found at the respective WMT webpage, e.g. https://www.statmt.org/wmt20/translation-task.html.

3. [While there is still no consensus about the quality of translations produced by LLMs, it would be useful to add a comment about the extension of this work to LLMs in the discussion section.]

• Thanks for the suggestion. We’ve done so in the updated draft.

4. [To get an idea of the effectiveness of this work with contemporary works, it may be useful to report the performance of a few MT models submitted to the WMT'23 shared tasks (where the outputs are already available)]

• As we started with a base model pretrained only on the WMT’23 training data, we do not expect to beat the state-of-the-art submissions to the WMT'23 shared tasks without additional efforts to utilize other data, etc. However, we are excited about future work to incorporate TWA into efforts to achieve state-of-the-art MT models.

We also agree that evaluating on low-resource languages would be really impactful future work, as would considering extensions with synthetic annotations. In preliminary results with a synthetic annotator model (as described Appendix C) we saw positive results, but more thorough investigations would be useful. Thanks again for all the great questions and suggestions!