Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering

“Conceptually unclear if using QA to evaluate translation makes as much sense as using QA to evaluate other things such as factuality (which prior work did)”

We appreciate the reviewer’s feedback and the opportunity to clarify. While QA-based evaluation has been used in other tasks such as summarization, we argue that factuality—especially at the document level—is also crucial for machine translation (MT), and remains underexplored in the MT evaluation literature: existing metrics such as BLEU and COMET are often limited in their ability to capture these phenomena.

However, to clarify, our questions are not restricted to simple factual or entity-centric forms. As shown in Figure 9, only about 30% are concept-based; the rest span procedural, judgmental, and consequence-based phenomena. Many questions, such as the one in Figure 6 (bottom-left), require reasoning across sentences, demonstrating that the QA signal is not purely local:

“How is the cousin related to Grandma’s litte sister”? requires understanding My father, in fact, had no idea how to do business, but he had gone to seek refuge with a distant cousin. **My grandmother had a younger sister** who had gone to Beijing in her youth, and had gotten married and had children there. **This cousin was her eldest son**, and had always been an outcast in the family.

TREQA is designed to surface translation errors regardless of granularity—be they lexical or discourse-level—and we demonstrate in Table 1 that it performs competitively with state-of-the-art MT metrics. Moreover, Section 5 shows that TREQA questions align well with human-identified errors, further validating their diagnostic value.

Finally, we agree that combining extrinsic QA-based signals with intrinsic metrics (e.g., COMET) may offer complementary strengths, particularly for nuanced paragraph-level evaluation. We will clarify this motivation and its connection to factuality evaluation in other NLP tasks in the final version.

On TREQA’s performance: We agree that TREQA is not always the top-performing metric across all datasets. However, it's important to contextualize these results. Even the strongest reference-based metrics, such as COMET and XCOMET—both of which are trained on large amounts of human judgments—or GEMBA-MQM, which is explicitly designed to target error spans, perform in a similar range. TREQA, despite not being supervised on human judgments and operating in a more general QA-based framework, achieves competitive performance, which we believe is a notable strength. Regarding the Literary dataset specifically, we acknowledge that the performance margin above random (50%) may not appear large. However, this dataset is particularly challenging due to its domain complexity and the small size (180 examples), which inherently limits statistical power. We agree that conducting statistical significance tests would help clarify the robustness of the observed differences, and we consider this an important direction for future analysis.

Finally, it should be noted that the goal of our paper is not to propose a metric to replace existing ones, but rather to develop a metric which captures in a more natural and interpretable way phenomena which existing metrics do not capture well -- we believe our metric can be used in tandem with other metrics such as COMET, and therefore we do not aim to always outperform existing metrics (specially metrics which are specifically trained to correlate with human judgments)."

On Variance Reduction: Please note that including candidates reduces variance in 4/6 cases (across model sizes and the two LPs considered). Having said that, we agree that the claim could be better justified -- we will make it more specific to the results in the header paragraph. Thanks for pointing that out.

“why choosing literature as test? Wouldn't a genre with more objectively measurable success/fail outcomes (...) be better suited?”:

Thank you for raising this important point. In an ideal setting, a truly extrinsic evaluation would require taking actions based on the translations and evaluating quality of the translations based on the success of those actions [1, 2]. However practically this is very challenging and is difficult to scale. Our work takes a more pragmatic yet informative approach by using QA as a proxy for downstream comprehension [3]. This allows us to assess whether readers of the translation extract the same key information and inferences as readers of the original, without requiring complex task-based setups.

We also believe that the literary domain remains a relevant and challenging benchmark for several reasons as these texts demand deep semantic understanding, multi-sentence reasoning and subtle discourse enforced meaning nuances that traditional metrics would fail to capture. While literature is open to interpretation, many questions generated by TREQA focus on elements such as character relationships, event sequences, and causal links, which have stable, answerable properties across translations. Furthermore, by grounding the questions on the candidates, we ensure that the question are relevant in surfacing translation errors which is the main goal of TREQA.

We will add this discussion in the introduction to strengthen the motivation for our choices for this work.

[1] Church, Kenneth W., and Eduard H. Hovy. "Good applications for crummy machine translation." Machine Translation 8 (1993): 239-258. [2] Doyon, Jennifer, Kathryn B. Taylor, and John S. White. "Task-based evaluation for machine translation." Proceedings of Machine Translation Summit VII. 1999. [3] Tomita, Masaru, et al. "Evaluation of MT Systems by TOEFL." Proceedings of the Fifth Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages. 1993.

”document-level metrics for MT seem relevant but missing”:

Please note that all the metrics reported in Table 1 are operationalized at the paragraph level, which aligns with common practice in recent machine translation evaluation research.. While some metrics have been proposed as document-level, they often operate in a context-aware fashion—typically by processing texts at the sentence or segment level and incorporating limited surrounding context (e.g., via context windows) as discussed in Section 6. Such methods often rely on tools like BleuAlign to split paragraphs into aligned segments or make assumptions about the linear one-to-one sentence-level ordering in translation, potentially losing crucial cross-sentence information. In our work, we intentionally avoid such segmentation in order to preserve the full paragraph structure and better reflect real-world paragraph-level translation quality.

However, if there are other document-level metrics that we may have overlooked, we kindly invite the reviewer to share relevant citations. We would be happy to include a discussion or comparison in the final version of the paper.

“The validation of the proposed framework ultimately still seems tied to some sort of human preference at the sentence-level…”

For practical purposes, we validate our framework by showing that 1) TREQA’s rankings of candidate translations align with human judgments on paragraph-level texts (not sentence-level), achieving accuracy scores comparable or better than state-of-the art metrics and 2) the QA pairs created by TREQA target translation-specific errors, as evidenced by their alignment with expert-labeled MQM annotations. However, we agree that a more human-centric evaluation, such as asking annotators to assess the quality of the generated questions or answer correctness, would strengthen our result, but leave that to future work.

“The performance of the framework seems to depend on several factors…"

Yes, the final performance of the metric depends on several design choices, including the input to the QAG model, the QAG model itself, the QA system, and the answer correctness metric. However, the primary contribution of our work is not a fixed configuration optimized for a specific language pair, but rather the introduction of a flexible framework. TREQA is designed to be modular, allowing practitioners to adapt each component to suit their specific needs. We note that we will release the code to reproduce the results reported in the paper. Furthermore, we will also add actionable insights (best practical configuration) and recommendation to the discussion, so that practitioners have a starting point to tailor TREQA to their specific use case, including scenarios involving target languages other than English.

We thank the reviewer for the thoughtful review.

On limited evaluation: We used the reported datasets because, to the best of our knowledge, they are the only available sources containing genuine paragraph-level texts and expert human annotations. Although WMT datasets are labeled as supporting paragraph-level evaluation, only 30% of the examples are “truly” paragraphs. Moreover, WMT evaluations are conducted independently for each translation, whereas both Par3Human and Literary datasets support contrastive evaluation, which better highlights subtle differences in translation quality—an increasingly important concern as MT systems improve [1]. However, we acknowledge the relevance of WMT and consider extending our framework to future WMT datasets, especially those that feature genuine document-level content (e.g., WMT2025). We also clarify that system-level evaluation was beyond the scope of this work as current metrics and previous QA-based approaches already showed high correlations [2]. Our focus was on segment-level quality assessment, as it allows more granular insights into MT performance.

[1] Song, Yixiao, et al. "Enhancing human evaluation in machine translation with comparative judgment." arXiv preprint arXiv:2502.17797 (2025). [2] Krubińsk, Mateusz, et al. “MTEQA at WMT21 Metrics Shared Task”, WMT21

On language direction: Yes, our current experiments focus solely on the xx→en direction (L134-138) as, among other reasons, LLMs have shown superior performance in English compared to other languages (see footnote 5 in the paper). However, TREQA does leverage cross-lingual context in a reference-free setup, which at least partially mitigates concerns about language bias. Specifically, both question generation and answering are informed by the source input and candidate translation, allowing the model to reason across languages. We view our work as a first step and proof of concept. The core framework of using QA for MT evaluation is language-agnostic in principle, and future work could incorporate multilingual LLMs or language-specific QA models to extend the methodology to other directions.

On questions being extractive: As stated in L148–149, our decision to use extractive QA was driven by the desire to ground the QA process strictly in the given texts. This ensures a tighter alignment between question/answer pairs and the input content. While extractive, the generated questions still span diverse linguistic phenomena, as shown in Figure 9 in the appendix. We agree that moving beyond extractive QA is an interesting direction for future work; however, a key challenge is the lack of established evaluation metrics for measuring answer correctness, which is beyond the scope of this study. Our preliminary experiments (footnote 3) using an LLM as a judge for answer correctness showed a tendency to overpredict high scores, highlighting the difficulty of reliable evaluation in this setting.