LLM-as-a-Judge & Reward Model: What They Can and Cannot Do

• The title is too broad: The title of the paper is too broad relative to the specific content covered. While the study centers on the effectiveness of LLMs in judging new languages, a more precise title would better encapsulate the paper’s focus and scope.
• The conclusion is too generalized: The conclusions drawn in the paper are overly generalized, as the authors only evaluate Korean in addition to English. To make the conclusions stronger, it would be good to include evaluation of additional languages to really conclude the circumstances under which automated evaluators are effective in new languages and when not.

The primary concern with this paper is its attempt to address too many questions simultaneously and some of the findings are not particularly surprising. Specifically, it seeks to explore whether LLM judges and reward models can transfer their evaluation capabilities from English to other languages, assess how well these models can identify factual errors, and evaluate their performance on challenging prompts. Due to the breadth of these inquiries, the paper lacks a deep understanding of each individual question.

• Regarding the question of transferability, the claim that LLM judges and reward models can "transfer their skills to other languages" is difficult to substantiate, as the study only tests this on Korean. Additionally, the authors do not provide a detailed examination of the training data for these models.

• Concerning the issue of factuality, the findings are not particularly surprising, given that LLMs are known to hallucinate frequently. If a model inherently holds incorrect beliefs about certain factual knowledge, it is unlikely to assess factuality accurately. Moreover, there are existing works that address factuality error detection through search-augmented evaluations, such as [1][2].

• As for the performance on challenging prompts, it is unsurprising that reward models struggle with problems in domains like biology, physics, and chemistry, as they are not specifically trained for these areas.

[1] FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios

[2] Long-form factuality in large language models

• Lack details about the quality control process: The authors discuss quality analysis on Section 3.2, but did not mention the exact inter-rater reliability (e.g., fleiss’s kappa), which will be essential to gague the quality of the dataset. Also, there lacking discussion for the process where challenge subset is constructed: “which we select one correct and one incorrect response. These are then translated into English” – is the responses being randomly selected? How is translation process completed? “the authors manually review translations for accuracy” - what is the inter-rater agreement here (e.g. does the authors almost always agree on a subset of data)?

• Limited insights from the paper: the authors list findings & observations, yet there's little actionable insights that would allow us to build a better system in the future, rather than "it doesn't work." For example, about the correlation between “English evaluation proficiency and Korean evaluation performance.” (Figure 4). This is rather unsurprising – It is the same task (i.e., judging) with a different language. Rather the correlation between judging performance and KMMLU compares the performance of different tasks, which it is expected to have lower correlation. Furthermore, these results are based on a single language pair, so it is unclear if this trend will generalize across languages (would be interesting if so). I would suggest to further analyzing the results to back up concrete claims that would be helpful for developing better systems, and bring these up in the introduction.

• Limited language pairs & sample size: While the study provides valuable insights about Korean-English transfer, it only focuses on this language pair. Even within Korean, with a limited sample size (90 instructions as mentioned in L135), it is unclear whether the claim that “English evaluation proficiency is a stronger predictor of Korean evaluation performance than Korean language proficiency itself” would hold across different languages. Including more languages (at least one more), especially those from different language families, would help demonstrate the generalizability of the findings.

1. The evidence of weaknesses in proprietary and fine tuned LLM transferability is insufficient, and more detailed analysis and exploration of more dimensions are needed.
2. Section 6 is relevant to LLM's inquiry about linguistic competence. But is it clear to the authors whether there is multilingual data for the training data of the language model in Table 3. This will affect most of the experimental analysis in this paper.
3. This work should introduce more language types for performance analysis.
4. In Table 9 (Hard) llama3.1 70B significantly outperforms other models (e.g., 405B) in Korean, but underperforms in English instead. The authors do not seem to have made an in-depth analysis, which is regrettable.

Limited interesting findings: There are not too many findings to take away. As a NLPer, I did not get anything to learn.

Limited novelty for ICLR audiences: this seems a incremental extension for MT-bench, it might better go ARR.

on the diversity of reward models: See more reward models on

• https://arxiv.org/abs/2410.12784 JudgeBench: A Benchmark for Evaluating LLM-based Judges
• https://arxiv.org/pdf/2403.13787 Evaluating Reward Models for Language Modeling