You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation

Thank you for your highly positive review! Indeed, the primary aim of our paper is to spark debate and prompt researchers to reexamine how they evaluate their systems in the future. We are delighted that you are so receptive to this goal!

Regarding your concerns about the empirical evaluation, we generally agree that there are several ways in which it could be improved, including the ideas you mention. Of course, having as tight an approximation to the true curve as possible would be of immense practical value. However, at present, this requires significant resources, and as such, we leave this for future work.

Thank you for your nice review and especially for your detailed critique. We address your concerns point-by-point below.

Doubts about empirical results fitting the hypothesis. Could we still be far from the curve?

This is a fair concern, and we struggled with the tradeoff (pun intended) between clearly illustrating our theory and the tightness of the presented empirical results.

The main issue is that producing a tighter estimate of the curve is extremely challenging computationally: we would need to optimise a language model using the expectation of the Lagrange dual from Eq (5) for each accuracy-naturalness tradeoff point. Indeed, this method is how Blau and Michaeli obtained their corresponding tradeoff curves for image restoration in their original paper.

Instead, we opted for the more feasible alternative, which is to maximise the one-shot Lagrange dual in Eq (5) over a large number of candidate translations for each source sentence. However, as we note at the end of Section 5.1 and explain in detail in Appendix F, this approach presents an overly optimistic estimate of the curve, making the feasible region appear larger than it actually is.

It is difficult to know how far we are from the curve; we could be far. However, we believe the true curve is probably much closer to the plotted systems than the approximate curve, based on two observations: 1) the state-of-the-art translation systems are already really good and 2) as we also mention as a concrete case study in the paper, in recent years, we consistently saw in WMT MT task evaluations that the winning submission according to automatic metrics (BLEU first, then neural metrics such as AutoRank from last year) are not the winners in human evaluations. We believe that our theory of the tradeoff explains this phenomenon.

"In Figure 3, … I don't observe any systematic "correlation far away from" and "anti-correlation close to” ... I doubt it can be reliably inferred statistically..."

We also agree that it is probably very hard to estimate the "transition point" where the two quantities start to anti-correlate. Our argument only demonstrates that a transition point must always exist. However, we believe the existence result is already a valuable contribution to the literature.

Doubts about the theoretic assumptions

Our theory is based on and reflects current practices in machine translation: the source language is typically derived from a monolingual corpus, which professional translators then translate into the target language. Indeed, there has been a recent explicit effort in the MT community to avoid back-translations or translations in both directions [1, 2].

Philosophically, the issue we had is that defining a joint distribution of perfectly natural source-target language pairs seems unsound/troublesome. Practically, the issue is that parallel distributions are "direction-dependent:" Translating an English corpus into German will inevitably yield a different "joint distribution" compared to translating German into English.

However, our theory sidesteps this issue, and the setup only has one crucial feature: we do not necessarily assume that the target marginal $P_y$ represents the distribution of "perfectly natural texts." Instead, it is up to the practitioner to identify an appropriate monolingual reference distribution $R_y$. This assumption is a key differentiating factor between Blau & Michaeli's work and ours.

Worries about MQM score conversion and interpretation

We are pleased that you have identified these concerns; we had considered them as well but did not have the space in the submission to discuss them. We address your two comments below.

"fluency score doesn't depend on the source, even though annotators have access to the source sentence"

Of course, as current MQM evaluations are conducted, the fluency scores could depend on the source text. While this is undoubtedly an issue that needs to be addressed in future evaluations, we do not think that it should have too significant an effect.

"expected fluency of a natural sentence in the target language is perfect"

This is the more concerning of the two issues. Indeed, this assumption is equivalent to assuming that every critic assigns a higher average score to any unnatural text. While this might sound intuitively true at first, we don't think that it should hold in practice. However, we also don't think that it should cause too significant a difference in practice.

Why did we use monolingual NewsCrawl?

We used it simply to illustrate our point, and we wish to make no claims as to whether it represents the full spectrum of natural-sounding German.

References

1. Toral, A. et al. (2018). Attaining the unattainable? Reassessing claims of human parity in neural machine translation. arXiv.
2. Läubli, S et al. (2020). A set of recommendations for assessing human–machine parity in language translation. JAIR

Relevance to COLM crowd

Naturally, our work is motivated by and primarily targeted at the machine translation audience. Nonetheless, we believe our work fits well with the conference's topics. Concretely, point (3) in the list of topics of interest for COLM calls for submissions that are:

• All about evaluation: benchmarks, simulation environments, scalable oversight, evaluation protocols and metrics, human and/or machine evaluation

In addition to the possible future directions we outlined earlier in our response, our work should be of interest to the broader COLM community and could serve as a basis for exploring interesting future directions. We foresee two particularly relevant future directions:

• For the case of text summarisation, the accuracy-naturalness tradeoff could be extended along a third axis: brevity. Then, this could be put in correspondence with the rate-distortion-perception tradeoff [1].
• Our new statistical distance could be used to detect unnatural / LLM-generated texts, an area that is already gaining significant popularity.

Miscellaneous

l. 178, "By definition ... perfectly accurate": do you mean "optimally" accurate, given Q*? There is no guarantee that the translation system is capable of perfect translation.

Yes, it should be optimally accurate. We will update this in our manuscript.

l. 212, "convex in the first slot": assuming this means D(Q,P) is convex in Q?

Correct!

l. 216: : "convexity of A(N):" should arguably be concavity.

Yes, nice catch! Thank you.

l. 255: I assume this is convex in Q as per assumptions -- can you clarify it?

Indeed. We will provide a corresponding proof of this fact in Appendix E.

l. 263: Why is the perfect score zero, is that an assumption or convenience choice with no loss of generality?

It's a convenience assumption. Indeed, you can see that this assumption is equivalent to assuming that every critic assigns a higher average score to any unnatural text. While this might sound intuitively true at first, we don't think that it should hold in practice. However, we also don't think that it should cause too significant a difference in practice. We made this assumption to simplify the presentation.

l. 355-356: Isn't this suggestion of optimizing non-reference metrics (perplexity) essentially how neural models (NMT & LLM) work anyway?

Yes! Indeed, this is also why we used Gemma for the evaluation in Section 5!

References

• [1] Blau, Y., & Michaeli, T. (2019, May). Rethinking lossy compression: The rate-distortion-perception tradeoff. In International Conference on Machine Learning (pp. 675-685). PMLR.

Thank you for your nice review and the many engaging questions. We are delighted that you found our theoretical work interesting and insightful. We address your comments and concerns below.

"...practical evaluation results (Fig. 2 and 3) do not appear to fit the theory particularly well. Authors argue that the tradeoff curve only becomes relevant for better-performing models. Fair point, however in all examples, everything seems quite far from the curve…"

This is a fair comment given the way we presented the results. However, please note that obtaining a precise estimate for the tradeoff curve is quite challenging and resource-intensive; therefore, we resorted to an approximation, as indicated in Section 5.1. It is difficult to know precisely how crude our approximation is; however, in Appendix F, we explain that the specific method we used is overly optimistic and makes the feasible region appear larger than it is. In this sense, the scatterplot corresponding to the evaluated WMT systems and the approximated A-N curve are not fully comparable.

To get a tighter estimate for the curve, we would need to train a separate translation system for each point (or at least fine-tune a pre-trained model); this is how Blau and Michaeli obtained the curves in their work. While we believe this would be a valuable contribution, we leave it for future work.

We believe the true curve is probably much closer to the plotted systems than the approximate curve, based on two observations: 1) the state-of-the-art translation systems are already really good, and 2) as we also mention as a concrete case study in the paper, in recent years, we consistently saw in WMT MT task evaluations that the winning submission according to automatic metrics (BLEU first, then neural metrics such as AutoRank from last year) are not the winners in human evaluations. We believe that our theory of the tradeoff explains this phenomenon.

Indeed, we think the greatest value of Figures 2 and 3 is that they show the tradeoff between accuracy and naturalness of current systems and that it is worth concerning ourselves with evaluating along both axes.

"and it looks like essentially optimizing to the upper-right corner is the way to go."

We don't agree with this conclusion. Indeed, the main point we hope to convey in the paper is that there is no unique "way to go.” The approximated A-N curves already show that there isn't a unique optimum that we can achieve by optimizing to the upper-right corner, but rather we need to consider a Pareto frontier of solutions.

We can obtain different, Pareto-efficient (i.e., undominated) solutions by training a translation system using the average of the Lagrange dual objective in Eq. (5) with various settings for the slack variable $\beta$.

"The practical impact for the MT community is unclear. If this tradeoff changed the way data compression is evaluated and done, there does not seem to be a clear way to achieve that in MT. Evaluating along multiple dimensions has been tried before (again as rightly pointed out here), and the community has mostly move to one-dimensional metrics that are believed (or hand-waved) to optimize both accuracy and naturalness. It is unclear how this paper would change the state of things. "

We believe that you are slightly conflating evaluation and optimization. We do not argue against optimizing neural (or any other) metrics. Instead, the point of our paper is to alert practitioners to the fact that there does not exist a single metric that can be optimized to attain optimal accuracy and perfect naturalness simultaneously in a practically meaningful way. This theoretical result directly contradicts a fairly widely held, yet formally unjustified, intuition in the community.

As such, we hope that the paper will have a similar effect on the MT community that Blau and Michaeli's work did on the data compression community: researchers will stop looking for a "golden metric" to optimize and will have a more informed view of the inevitable accuracy-naturalness tradeoff of their systems.

Furthermore, unlike in the data compression community, there has been much less research on statistical distances for evaluating translation systems. There have been some attempts, such as the Frechet embedding distance, the counterpart of the widely popular Frechet inception distance (FID) in image quality evaluation, but it is nowhere near as popular or even well-known. Thus, we hope our work will spark a new line of research into text naturalness evaluation techniques.

Of course, we also hope that with our clarified view of accuracy and naturalness, which formalizes and illuminates all previous informal attempts, such as the adequacy-fluency tradeoff, the community will return to evaluating systems along two axes.

Thank you for your nice review and your constructive critique! We address your concerns below.

Worry about neural metrics being true distortion metrics

This is a fair concern, and we certainly worried about things along the same lines. As you mention, neural metrics are not "adversarially robust" in that they can be tricked with universal translations. However, neural metrics certainly perform well in the "average case." Thankfully, the average case is what ultimately matters for our theory since we define accuracy in terms of the expected distortion. Indeed, if neural metrics weren't good in the average case, then they wouldn't have much use.

Nonetheless, getting better neural metrics that are more adversarially robust is an important future direction for MT research and such robust metrics in turn be more reliable to be used in our evaluation framework too.

Handwaving about what neural metrics measure

While this is, in fact, not an issue for the theory we develop, it is an excellent point that also arose during our discussions with colleagues while writing the paper. Indeed, in the community there is an unwritten feeling that neural metrics place a heavier weight on how well a particular translated sentence sounds than on the accuracy of the translation. As a side note, in image compression, there also exist "perceptual distortion metrics" that aim to capture differences that matter to the human visual system.

However, importantly, so long as there is an element of comparison with a reference, and assuming that our assumptions are satisfied, neural metrics are still reference metrics, and the theory applies.

In other words, we are not advocating against optimizing systems using neural metrics. We are merely pointing out that if we optimize a system for a neural metric versus a "pure accuracy metric" such as BLEU or chrF, we will end up on a different part of the accuracy-naturalness curve since BLEU prioritizes accuracy over a neural metric that also measures naturalness.

Choice of the term reference metric

We agree that it is confusing, and we have given careful consideration to our choice of terminology. We ended up choosing this terminology for two reasons: 1) our use of the term aligns with its use in the information theory / learned data compression literature, which is the original inspiration of the work; 2) as we point out in the paper, QE metrics do use the source sentence as a reference. One could argue that QE metrics should be called "target reference-free" metrics.

"Can you comment on the interplay between naturalness and topics or domains?…"

This is a great question, and it really pushes the boundaries of the theory. We briefly discuss this on lines 219-228, but let us be more explicit here. We specifically believe that the appropriate behaviour lies on a spectrum, with the following two approaches as its extremes:

• We produce a full explanation
• We either mirror-translate the term or leave it as is.

The appropriate approach depends on the context. For example, when interacting with a user who is more familiar with internet slang, we might wish to translate the English word "doomscrolling" as "doomscrolling." However, for someone less familiar, we might want to produce a translation that explains the meaning of doomscrolling.

Note that currently, there is no method available to assess this, and it is a critical direction for future work.

Regarding sentence-level naturalness

That is a fair point; we just meant to emphasize that in our theory, naturalness is a property of distributions and not of individual segments. In other words, what makes sense to talk about is whether a "system" sounds natural, but it doesn't make sense to say that individual segments are natural or not from our theory's perspective. Of course, one can still debate sentence-level naturalness outside of our theory. We will clarify this in the camera-ready version of our paper.

Cite prior work

Thank you for the references! We will cite them in the updated version of the manuscript!