Quality Control at Your Fingertips: Quality-Aware Translation Models

1. We are afraid we are hitting a terminology problem here. We showed in the paper that by querying the model with "high quality" prompts we were able to improve translation quality and/or reduce the size of the candidate list for MBR decoding. Note that these two are the main takeaways from the paper, we are not claiming to have developed an improved evaluation for MT quality or anything along these lines. In the response to reviewer SaRa, we provided evidence that our system was even able to perform QE scoring to some degree (which, again, is not the main purpose of the system). We chose the term "quality-aware" as for us it made intuitive sense and we think it provides a valid explanation for the improvement in translation quality. If that is a friction point, we can happily change the name to something less problematic.

2. We took the data from the WMT23 metrics tasks (https://wmt-metrics-task.github.io/) and for each (source, MT output) pair we force decoded with our systems until the end of the MT output. Then we let our system generate the quality label and used this as a QE estimation (to be fair, we restricted the output of our system only to the possible quality lables, in order to force it to act as a QE metric instead of extending the translation). We then evaluated with the official tools of the evaluation, the human scores used for reference are MQM-based, not DA. But we want to stress this point again: the main goal of the paper is the increase in translation quality and the decrease of MBR computational cost, not QE.

3. We probably are hitting the same problem about terminology here. But referring to the approaches you mention definitely makes sense in the scope of the paper. Our approach definitely is much more lightweight than these, and thus limited in its effect.

4. The main forum for MT metrics development and evaluation is the WMT conference, with its yearly metrics task. The findings paper of last year could not be more explicit: "Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust" (https://aclanthology.org/2022.wmt-1.2/). The MT community certainly is shifting away from BLEU, the more general ML community should follow as well.

We thank the reviewer for their constructive comments and answer their questions below.

1. Would you mind elaborating on where specifically you see potential to improve the paper? We agree that the sole prediction of quality labels across all quality levels is not perfect. We also refer to the response for reviewer #2 for a short discussion about correlation with human judgement. Nevertheless, we note that QE is not the goal of this paper since we are mainly concerned with the fact that good hypotheses get good scores, and we can take advantage of that in order to choose a sample with a high quality score. Furthermore, we are reordering the the samples in the highest bucket based on the log probability of the respective score token, which results in better performance w.r.t. reference based metrics, which is our goal of this paper. We will make this objective clearer in the camera ready version.
2. The general idea of our approach is that we supply the quality labels at training time, the system thus learns from them. For more details please refer to the “Method” section in our submitted paper. Regarding our experiments, we evaluated our proposed methods on 2 language pairs as well as provide several metrics, including human evaluations, which show that particularly quality prompting performs superior to baseline methods (Tabel 1 & 2, Figure 1). May we ask you, what in particular you find to be “not convincing enough”?
3. Since BLEU is a lexical metric, it has been shown to be less reliable in judging translations compared to neural metrics or even human evaluations. Since the scientific community is in the process of transitioning away from BLEU, we did not include it in our paper. Nevertheless, we understand that BLEU has been around for a long time and this is why we will include it in the appendix in the camera ready version for the sake of completeness.

• This weakness is straightforward and I believe this paper is not well enough to be accepted.

We thank the reviewer for their constructive comments and answer their questions below.

1. We recommend quality prompting. Although, we did not explicitly test our method on large scale multilingual tasks, the same methods could in principle be used. In particular our experiments in the appendix in table 3 indicate that using the proposed approach for finetuning LLMs on small amounts of data works well. Since LLMs are known to be proficient in multilingual tasks, we are confident that our approach performs similarly as on the tasks mentioned in this paper. We leave it for future work to explore the applicability of the proposed approach in multilingual as well as low-resource settings.

2. This is a very important point, which we encountered during our experiments. The distribution of quality scores is in fact heavily skewed to the left. We tackled this problem by applying a equal mass binning scheme (Eq. 3 in the paper), which led to a balanced training set w.r.t. quality scores. We will include the frequency distribution plot in the camera ready version.

3. We investigated the behaviour of the model regarding possible cases where the model does not produce a quality score. However, we found that the model is able to learn to predict a quality score as early as after 500 steps. We found only a few corner cases where the model does not predict a quality score when we use epsilon sampling and sample 1000 sentences. In this case we assign a score of 0 (lowest bin) to a sample without a quality score.

We thank the reviewer for their constructive comments and answer their questions below.

1. We tested our approach on the WMT23 QE estimation task by force decoding the given source and target sentences and letting the model generate the corresponding QE label. We obtain a Pearson system-level correlation of 0.93. Comparative data points: best QE metric 0.98, CometKiwi 0.95, BLEU 0.92, chrF 0.87 (note that the last two metrics are reference-based). Thus, although it is not the best performing QE system, our approach does learn to estimate the quality of the translations.
2. We agree that the sole prediction of quality labels across all quality levels is not perfect. However, this is not the goal of this paper since we are mainly concerned with the fact that good hypotheses get good scores, and we can take advantage of that in order to choose a sample with a high quality score. Furthermore, we are reordering the the samples in the highest bucket based on the log probability of the respective score token, which results in better performance w.r.t. reference based metrics, which is our goal of this paper. We will make this objective clearer in the camera ready version.
3. This is a great suggestion and we will include examples of translations with different labels in the paper.
4. We refer the reviewer to the following paper for a discussion about the interpretation of neural metrics: https://aclanthology.org/2023.mtsummit-research.16.pdf. When available, human evaluation is the main metric that we use for interpreting the results, and the results on MQM are favorable for out method. As with the last point, we will add examples to the paper. As are representative, we can add some examples here where the (German) output of the quality prompting system is much more natural than the baseline System:

Baseline: Bei Bestellung ist es zur angegebenen Lieferzeit.

QA prompting: Bei Bestellung wird die Lieferzeit angegeben.

—

Baseline: Deshalb bitten wir Sie, diese Vorlaufzeiten bitte zu erlauben.

QA prompting: Deshalb bitten wir Sie, diese Vorlaufzeiten einzuplanen.

With respect to the BLEURT-QE metric we are happy to report that the paper has been accepted for publication in the WMT conference, we refer the reviewer to that paper for a detailed explanation of the metric and analysis of the results.