Learning LLM-as-a-Judge for Preference Alignment

Thank you for your valuable comments. I have a quick question: Could you please clarify the references [1-5] you mentioned? While I have a general understanding of the main issues you addressed, this information will assist us in preparing our response and refining our paper.

Thank you for your kind comment and appreciations of our strengths. In the following, we summarize your comment and address the main issues raised.

The novelty of Con-J.

We acknowledge that SFT+DPO is a popular practice while the novelty of Con-J does not lie in this training target. Con-J is a self-bootstrap framework for training a generative judge model based on preference data (framework presented in Figure 2). Unlike traditional methods for training preference judgment models using instruction tunning data, this approach scales better as it is easier to get human's binary preference rather than high-quality verbal feedback.

We also show that Con-J improves its rationale ability with preference learning as a generative judge, potentially facilitating AI transparency and allowing human-AI collaboration in scenarios such as the construction of preference dataset.

The setting of SFT+DPO.

The general setting of adding an SFT loss term is for stabilizing DPO training by preventing the decrease of the log probability of chosen response (i.e., positive judgment in our paper) [1]. We adopt the SFT+DPO setting as it is a popular practice with both theoretical and empirical evidence. As elaborated in Section A.2, we observe that Con-J shows comparable performance with Con-J w/o SFT in Creation and Code, while outperforms Con-J w/o SFT in Math. Further analysis shows that when trained without SFT, the chosen rewards are relatively small in the Creation and Code datasets, with values of -0.73 and -0.46, respectively, while in the Math dataset, they decrease significantly to -2.37. This implies the benefit brought about by adding SFT term in training on Math dataset.

Here we select the weight of the SFT component according to the reward curve of the chosen response (i.e., positive judgment in our paper). We observed the changes in the chosen reward during training and selected the smallest alpha that would not cause significant changes in chosen reward. This selection results in a relatively small weight than existing work. In previous research of DPO training, the prompts, and chosen responses had diverse formats, whereas in our training, the chosen responses follow a fixed json-style template with a key "rationale" and a key "better_answer". So if the coefficient for the SFT component is too large, it can easily cause the chosen reward to increase rapidly, which could potentially damage the model's performance.

We have revised the relevant discussion in Appendix A.2 and Appendix A.1, but we do not put it in the main text because this training objective is an engineering effort rather than our main contribution. We also added your recommended references and revised the relevant description in Section 3.

[1] Dubey A, Jauhri A, Pandey A, et al. The llama 3 herd of models[J]. arXiv preprint arXiv:2407.21783, 2024.

Human evaluation of the rationales being sampled.

We add a human evaluation with rationales from Con-J and Con-J untrained when given the same set of 100 randomly sampled data samples (refer to Section 4.5 and Appendix A.3). Experimental results demonstrated that Con-J has a higher correctness score for rationales compared to Con-J untrained under human evaluation. The performance gap between Con-J and Con-J untrained under human annotations is even greater than that by GPT-4o annotation.

The control of judgement length.

Thanks for the nice idea.

We controlled the output length of Con-J with a fixed length of 512. In our experiments, we observed that when the initial model needs to produce a lengthy judgment, it might get cut off, leading to an "unknown judgment". Rendering such judgment as negative judgment is actually beneficial, as when failing to do so, we empirically found that the Con-J's outputs will become increasingly lengthy during preference learning.

how can shortened or partial/biased judgements' affect on preference learning be mitigated in light of the inconsistency of rationale vs binary decisions?

The model continuously outputs rationale and preference prediction as verbal text, which means the sampled data is coherent. Hence, the model stops generating the rationale only when it thinks a preference prediction should be made.

We admit the judgment filtering process by treating lengthy judgments as unknown and negative may influence the performance on tasks that might need longer outputs to achieve the COT effect. However, our empirical findings show that rationales improve Con-J through a regularization effect during training, but the impact of rationales as a COT effect on Con-J is as significant during inference. This is particularly because Con-J serves as a judge, and if the answer parts already include a COT, the judge may not require it as much. To address your concerns, we have revised the discussions in Appendix A.1 to elaborate on how we control the output length of Con-J and its potential impact.

Thanks for your kind comments. In the following, we address the main issues raised.

I’m not entirely convinced with the analyses of Consistency and Correctness in Sections 4.3 and 4.5. I think it’s helpful if the paper first defines what “consistency” and “correctness” measure. Suggestion to the authors: define and motivate the aspects of consistency and correctness in the beginning of the paper.

We add the definitions of consistency and correctness at the beginning of the paper. Briefly, correctness refers to whether the rationale provides an accurate analysis supporting the preference prediction, while consistency refers to whether the rationale expresses the same preference as Con-J's final preference prediction.

In addition, it will be helpful to show that the GPT-4o scorer is reliable.

We have added an additional human evaluation to collect human annotations of consistency and correctness (see Appendix B.3). We observe that the Krippendorff's $\alpha$ values between the GPT-4o annotators and human annotators are 0.4427 for correctness and 0.6495 for consistency, indicating moderate and substantial agreement, respectively. This suggests a consistent trend between human annotators and GPT-4o, although some disagreements remain.

The prompt template in Table 9 speaks about the emotional tone, which I think measures a different aspect than consistency. Perhaps that’s also the reason why the results in Figure 3b are surprising (I would expect the consistency of the rationales would also improve) and aren’t consistent with the cited literature on chain-of-thought (CoT). The scoring for Correctness is also confusing: the rubric for 3 can be confused for 2 and 4. I wonder what the findings would be if the rubric is more distinct.

Thanks for pointing out the issue in our prompt template, we acknowledge that the design of prompt can affect the performance. Therefore, we have refined the prompt template in Table 9 with detailed definitions of correctness and consistency as well as descriptions of different consistency levels instead of using "emotional tone".

With the new prompts, similar experimental results are observed. There still exists minor difference, such as a general increase in correctness and a drop in consistency for all checkpoints with the new templates. Although these minor differences still exist, the overall experimental observations remain similar. Specifically, correctness improves while consistency slightly declines during preference learning of Con-J.

We have added a qualitative analysis to investigate the drop in consistency during preference learning and illustrate several solutions to mitigate this issue (see Section A.3.2 and Section A.4). The definition of consistency is whether the rationale and the final preference prediction have the same attitude/preference towards either answer. We observe that when inconsistency happens, the performance of Con-J also drops, which indicates that Con-J refuses to express its attitude towards either answer when it lacks the ability to make judgments. This indicates that the low consistency may not be bad, but potentially indicates that the model is not convincing in its judgment on certain cases and calls for more human interference. In future work, a more effective data construction process may address this problem by designing more appropriate expected rationales for these situations, such as having the model output "I find it difficult to judge, but I can provide some analysis ..."".

It’s also important to describe what correctness means for a subjective task such as Creation.

We have currently conducted experiments only in Math dataset because it allows for an objective analysis on the quality of rationales in the context of math problem-solving. In contrast, the Creation dataset is more subjective. How to evaluate rationales in a potentially subjective scenario is left for future work.

Can you clarify (or point to a URL) how you were able to obtain a test split for Ultrafeedback?

We split Ultrafeedback according to https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized. We have attached the link to our paper.

Line 259-261. The “Code” vertical domain is not mentioned.

We have mentioned the Code dataset at Section 4.1. We have polished the paragraph for a clearer description of the dataset.

Some LaTeX citation commands were used incorrectly

We have gone through the paper and carefully revised the usage of LaTeX citations.

Thank you for your kind comment and appreciation of our strengths. In the following, we address the main issues raised (some of your questions are re-arranged in order to facilitate a progressive reading of our responses).

How exactly does Con-J without rationale differ from the scalar model (Figure 4)?

Con-J without rationale is trained and evaluated with the prompt template presented in Table 7. This indicates that Con-J is prompted to output its preference prediction directly without generating any rationales.

Con-J still differs from the scalar model in that it still leverages the generation capability of the original LLM, while the scalar model replaces the LM head with a value head. Therefore, Con-J w/o rationale lacks the impact of the regularization effect on generating the rationales during training, while SM lacks the regularization effect due to the initialization of the value head. This difference results in a variation in their performance.

The empirical results (Table 10, Figure 4(b)(d)) seem to indicate that the main contribution to Con-J's good results are not the rationales, but (presumably) the regularization in eq. 9 instead (It's unclear to me what exact parts Con-J without rational removes...). Experimentally, what happens when this regularization is ablated? Furthermore, the claims in the abstract/introduction seem to suggest that rationales themselves are the reason why Con-J succeeds. While experimentally it's demonstrated that the rationales do improve robustness to bias, it seems misleading to claim this when the main results actually suggest that the rationales do not improve judgment accuracy at all. It may be good to do a deeper analysis of what is actually inducing this improvement.

To investigate this problem, we conducted an analysis on cases in the general test set where Con-J outperforms Con-J w/o rationale under the verbosity bias with $\gamma=1.0$. Interestingly, amongst these cases, 79.2% of the chosen answers are shorter than the rejected ones. In contrast, in cases where Con-J underperforms Con-J w/o rationale, only 31.6% of the chosen a are shorter than the rejected ones.

We further present the table below showing the accuracy of Con-J and Con-J w/o rationale in cases where the chosen answer is shorter/longer than the rejected one. While the two settings have close overall performance, they behave differently under the two scenarios. Con-J w/o rationale performs well when the correct answer is longer than the rejected one but notably worse when the rejected answer is longer - a manifestation of verbosity bias. A similar trend is observed for Con-J under the two cases, but with a far less remarkable gap, indicating a strong bias counteracting effect of the generated rationale. This finding suggests that even when there is no difference in the average performance of Con-J and Con-J w/o rationale, the influence of verbosity bias on them could be different. We have revised our paper by adding this observation.

What happens if you do have humans (or even other LMs) check the rationales and throw out inconsistent ones? (As far as I understand, the filtering step simply checks the correctness of the final answer, but not the rationale itself.) Is this interpretability actually useful for constructing better / more useful reward models? Will it be better to align downstream models with rationales?

To address this issue, we have made efforts in two aspects.

First, we have conducted human evaluations on the correctness and consistency of these rationales and provided some qualitative analysis (detailed in Section 4.3 and Section A.3). The human evaluation has provided some valuable insights. For example, when the rationale highlights some issues in the answers, humans can quickly verify whether they truly exist and assess the judgment accuracy; conversely, when the rationale refuses to explicitly express its attitude towards either answer, it is likely that Con-J lacks sufficient capability to make a correct judgment. This facilitates the construction of preference data for RLHF training via human-machine collaboration, which makes the annotation process more transparent. Similar approach has been applied to industrial pipelines such as [1].

Second, we conducted an experiment to investigate whether rationales with controlled quality can enhance Con-J's ability (see Section 4.3 and Appendix A.4). Generally, we observe that Con-J trained with rationales filtered by their correctness and consistency scores is also better at making preference judgments.

Thanks for your kind comment and appreciation of our approach and experiments on open-source datasets. In the following, we address the main issues raised.

The approach feels like it would add a lot of overhead in alignment training - RLHF is already extremely computationally intensive and waiting for models to generate rationales before providing the judgement feels like it would not be a feasible approach to use for cutting-edge model alignment.

The practice of RLHF actually involves two important steps: constructing preference data, and training the policy model based on its basis. We discuss the feasibility of Con-J in both steps below.

First, the primary goal of Con-J is to address the challenge of constructing and scaling preference data as a machine annotator. In industry, the construction of preference data typically combines both human annotations, which have a higher alignment with human value, and machine annotations which could be more scalable. While a scalar model does not provide explanations on its annotations, Con-J makes it happen by providing machine-generated rationales on human preferences, contributing to a more transparent annotation process. A similar approach has been applied to industrial pipelines such as annotating code data [1].

Second, we admit that Con-J is less effective than the scalar reward model in RLHF training. However, the majority of time is spent on the policy model's inference process rather than on the reward model during RLHF. To address the efficiency issue, Con-J can be combined with some mechanism that could determine whether generating the rationales preceding the preference prediction is necessary. Since the RLHF training process is beyond the scope of this paper, we leave the combination of Con-J and the RLHF process as future work.

[1] McAleese N, Pokorny R M, Uribe J F C, et al. Llm critics help catch llm bugs[J]. arXiv preprint arXiv:2407.00215, 2024.

I would like to see some qualitative analysis of the rationales - there seems to be minimal information on how good they are and what they might be useful for. Has there been any human evaluation of the rationales when inconsistencies have been detected?

We performed a human evaluation to investigate the correctness and consistency of the rationales. We recruited six annotators from college students to evaluate 100 randomly sampled rationales generated by Con-J in terms of their correctness and consistency with the preference prediction.

As a result, the average consistency value is above 2.85, indicating a high consistency level. We further conducted a qualitative analysis on the very few cases with consistency scores of 2 or 1. We find that in these cases, Con-J often chooses to refrain from clearly expressing its preference, e.g., refuses to indicate which answer to refer to: 'The answer provided a clear calculation process and gave an accurate answer, with more standard mathematical expressions and a clear logical order.', and presents the weakness of both answers: "Answer 1 did not accurately understand the original mathematical problem; Answer 2 overly complicated the solution form, neglecting a more straightforward calculation method."

Furthermore, we found that the consistency of human-annotated cases has a moderate agreement with GPT-4's annotations (Cohen's Kappa=0.47). Therefore, we extended this analysis to examine cases in the full test set that GPT-4 annotated as having low consistency. We found that in cases with low consistency, the accuracy of the preference prediction is 59.9%, which is significantly lower than on the whole test set (85.0%). This suggests that low consistency generally occurs when Con-J lacks sufficient judgment ability, resulting in the generated rationale lacking an explicit attitude towards either answer and supporting the final preference prediction.

We have revised our paper and presented the detailed design of the human evaluation and the qualitative analysis regarding the rationale (including examples) in Section 4.5 and Appendix A.3.