Evaluating LLMs on Chinese Idiom Translation

Thank you for your valuable feedback and high assessment of our work. We appreciate your helpful suggestions.

In the introduction, there is a missing closing parenthesis in "... (see Table 1"

Thank you for catching this. We will fix this in the final version.

Fonts in some figures (e.g., figure 2, and many other in the annex) are too small and not easily readable in printed version

We will increase the font size in these figures to improve the readability.

IDIOMEVAL is sometimes introduced as a taxonomy (e.g., the abstract) and other times as a framework (e.g., the introduction). Authors need to be consistent with this. My understanding it that it is a framework that, among other things, defines a taxonomy.

You are right. IDIOMEVAL is best described as a framework that includes a taxonomy for evaluating idiom translation. We will revise the wording throughout the paper to ensure consistent use of the terminology.

Why do you think that the performance increases as the model scales up except at 72B? (no need to prove the answer, but just to formulate an hypothesis from your expert view)

Thank you for raising this point. We further investigate the cause of this case. In terms of training loss (cross-entropy), we found that when fine-tuned on thoughts, the 72B model plateaus around 0.2 to 0.3, while smaller models (32B, 14B, 7B) converge more smoothly to a lower loss (~0.02). In contrast, training the 72B model on answers only, especially short answers, results in lower loss and better performance.

In terms of model behavior, we find that the 72B model always outputs the short answer format (“yes” / “no”) even when fine-tuned on longer formats such as long answers or thought-based responses. This differs from all the smaller size models (1.5B to 32B), which follow the training output format. While we do not fully understand the root cause of this behavior, it may explain why training the 72B model on the short answer produces the best performance.

We will include this explanation in the final version.

Please let us know if you have any remaining questions or concerns.

Thank you for your valuable feedback on our work. We appreciate the opportunity to address the points you raised.

In the proposed models, native Chinese speakers fluent in English as annotators are required to obtain the dataset for instruction-tuning. Therefore, the annotators and cost are serious problems.

Yes, annotating the translations is costly, at around $4K USD, which is expected for high-quality, fine-grained annotations. To avoid repeated annotation efforts in future work, we will release our datasets and models publicly so that others can directly build on our work to improve idiom translation performance and develop idiom-aware evaluation metrics. For example, our fine-tuned model can be used for self-training or back-translation to facilitate transfer to other languages.

In detection of idiom translation errors of Table 6, it is not clear whether there are statistical significance or not in F1-Macro between models.

Thank you for the suggestion. We conduct one-sided paired bootstrap significance tests (10,000 resamples) to compare each metric’s macro F₁ score against that of the best-performing model, fine-tuned Qwen2.5-72B. We report the 95% confidence interval of the macro F₁ and p-value. The p-value reflects the proportion of bootstrap samples in which Qwen2.5-72B achieved a higher macro F₁ than the compared model. An asterisk (*) indicates statistical significance at the 0.05 level (p < 0.05). To save space, we show only the top-performing metrics from each method category. This significance analysis will be added to Table 6 in the final version.

The reference-based or reference-free metrics are formulated as a binary classification task using ROC curve. However, it may be a little unfair because the metrics except BLEU based on n-gram matching is not learned to perform a binary classification task.

We agree that traditional translation metrics such as COMET and MetricX are not trained for binary classification. That’s exactly why we apply the ROC curve to determine a threshold on the training set, allowing us to convert their raw scores into binary decisions for fair comparison with prompting and fine-tuned methods on the test set.

In addition, Table 3 displays the correlation between their raw metric scores and human judgements. Together with the binary classification results, these provide a comprehensive picture of how traditional metrics perform in detecting good and bad idiom translations.

This paper needs to evaluate idiom translation of other languages, not only Chinese.

We agree that applying our taxonomy to other languages is exciting. But considering the substantial annotation effort ($4K USD) and author’s expertise in Chinese, we decide to focus on Chinese idioms, which have the uniqueness of proverbs, allegorical sayings, and aphorisms originated from ancient literature and is overlooked in prior work [Line 26-37]. Given budget constraints, we leave the extension to other languages to future work.

Please let us know if you have any remaining questions or concerns.

Thank you for your valuable feedback on our work. We appreciate the opportunity to address the points you raised.

There is not much novelty in the work; the paper mostly just introduces a dataset and the results of an evaluation.

While our paper follows a common format for evaluation work, we believe the novelty of our work lies in the focus and contributions. We introduce a new taxonomy for Chinese idiom translation, provide a high-quality annotated dataset across four domains and nine systems, and conduct the first systematic analysis of both translation models and metrics on this task. In addition, we go beyond the dataset and analysis by training models that outperform existing metrics in detecting idiom translation errors.

Limited language coverage, focusing only on Chinese idioms.

We agree that applying our taxonomy to other languages is exciting. But considering the substantial annotation effort ($4K USD), authors’ expertise in Chinese, and the unique nature of CHinese idioms—which often include proverbs, allegorical sayings, and aphorisms originating from ancient literature and differ from those from prior work studying English, Japanese and European languages [Line 26-37]—we decide to focus on Chinese idioms. Given budget constraints, we leave the extension to other languages to future work.

I have some doubts about the improved methods for evaluation; based on the training data description it looks like the same idioms were present in both training and evaluation, which casts doubt on the positive result.

This is a great point. In fact, only one idiom out of 100 idioms in the test set appears in the training set. This minimal overlap is due to the data split: test set translations are based on 2024 data, while the train/val sets are based on 2023 data [see Line 92–95], leading to different sampled idioms. The train/val split is also done at the idiom level to prevent overlap. We will make this clearer in the final version.

To show the impact of the one overlapping idiom, we present a side-by-side comparison of the original Macro F₁ in Table 6 and updated Macro F₁ after removing the one overlapping idiom from the test set below. To save space, we report only the top-performing metrics from each method category. The differences are very minor across all metrics, and the fine-tuned models show even smaller variation than some non-fine-tuned baselines.

The taxonomy is documented in different places throughout the paper and the number of categories varies. At first you mention 7 different failure modes and Table 1 indeed contains 7 error categories. However, then you mention 9 categories and 13 subcategories. Which version is correct?

Both are correct, let us explain. Table 1 displays 8 categories: 1 for No Error and 7 for different error types. As noted in the Table 1 caption, we also have a More than One category for translations containing multiple errors, bringing the total to 9 categories. Three of these categories have subcategories: No Error has 2, Partial Translation has 3, and No Translation has 2. The remaining 6 categories are stand-alone categories. Counting both the subcategories and stand-alone categories, we have 13 subcategories in total. Please see Appendix A for the full list. We will make this clearer in the final version.

I would suggest using "minor", "major" and "critical" for severity. For annotator confidence, something like "low", "medium", "high" might work better as well.

Thank you for your suggestion. We will use these terms to improve clarity in the final version.

Please let us know if you have any remaining questions or concerns.

Thank you for your valuable feedback on our work. We appreciate the opportunity to address the points you raised.

In Figure 4, the performance drop at the 72B scale for both “short thought with answer” and “long thought with answer” prompts is noted, yet the paper does not sufficiently explore or explain the underlying causes of this decline.

Thank you for raising this point. We further investigate the cause of this case. In terms of training loss (cross-entropy), we found that when fine-tuned on thoughts, the 72B model plateaus around 0.2 to 0.3, while smaller models (32B, 14B, 7B) converge more smoothly to a lower loss (~0.02). In contrast, training the 72B model on answers only, especially short answers, results in lower loss and better performance.

In terms of model behavior, we find that the 72B model always outputs the short answer format (“yes” / “no”) even when fine-tuned on longer formats such as long answers or thought-based responses. This differs from all the smaller size models (1.5B to 32B), which follow the training output format. While we do not fully understand the root cause of this behavior, it may explain why training the 72B model on the short answer produces the best performance.

We will include this explanation in the final version.

As mentioned above, the “Results” section would benefit from a more focused analysis. Consider narrowing the scope of the key arguments to allow for deeper discussion. One suggestion is to move the CoT experiment details to the appendix, thereby freeing up space to expand on the qualitative evaluation—perhaps by including more illustrative examples of idiom translation errors and model behavior.

We agree that including more detailed analysis in the main paper would be beneficial, though we were limited by space constraints. The current version includes example translation errors in Appendix A, and we will expand this with additional examples from each model. If accepted, we will also move some detailed analysis in Appendix D, such as common error types per model and error distribution by frequency, into the main content using the extra page.

Please let us know if you have any remaining questions or concerns.