Direct Judgement Preference Optimization

However, from my perspective, these two works need to be carefully compared and analyzed in this paper. Although the authors can compare the Themis with proposed method (CriticGPT is a closed-sourced model), I do not recommend doing so because I believe there may be numerous variables that could affect the experimental results, such as the size of the dataset and the optimization techniques (Themis re-designs the DPO loss).

In terms of experimental comparison against Themis, we believe there are two perspectives in experimental comparison: controlled-setting comparison and practitioner comparison. The first perspective is analysis conducted to determine which training recipe, for fixed preference dataset and teacher model, leads to the best downstream judgment performance. We consider this an exciting line of open research that will inform how the next generation of judge models are trained. However, prior judge models (i.e., all evaluated baselines in our paper) are not trained on a standardized preference mix. Recent efforts have produced high-performant preference datasets, such as the Skywork Reward Preference 80K dataset [2] for the pairwise evaluation setting. Such datasets may serve as a foundation for future research in controlled analysis. However, these datasets were released well after our model development was complete, and as such, we leave a standardized analysis open as future work.

The second perspective is one that we (and other judge papers) implicitly take in our benchmarking efforts. Practitioners who simply want to run a model “out-of-the-box” to get evaluation results or use judge outputs for downstream alignment may heavily prioritize judge performance on standardized benchmarks over training details or dataset mix and size. In order to characterize model performance for this perspective, we run Themis on our single rating evaluation tasks and report the performance below. We agree with the reviewer that different variables almost assuredly impact performance; See above discussion about future lines of analysis for judge training. As such, our stated results are meant to inform those who prioritize “out-of-the-box” performance.

Overall, Themis-8B outperforms many competitive judges in single rating tasks, matching the performance of much larger models like Prometheus-2-8x7B and performing comparably to our 8B model in a subset of benchmarks.

[2] Skywork-Reward-Preference dataset: https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.2

Beyond above comments, this paper, Themis, and CriticGPT heavily rely on human-annotated judgment labels to automatically construct preference datasets. Therefore, the scalability of these methods is limited. I hope to receive some detailed discussion and analysis from the authors on this aspect.

In terms of limitations, we agree with the reviewer that the dependence on human labels limits scalability in the field of auto-evaluation in general, including our work and others mentioned. There are many exciting lines of future work that we wish to explore to overcome the limitations of human labels, especially as the pace of LLM development accelerates. Concurrent work (Self-taught-evaluator [3]) provides an initial exploration into self-taught evaluators that eschew the need for human annotations. However, the empirical performance of the pairwise-specific Self-taught-evaluator still lags our multi-faceted models in pairwise comparisons, indicating a gap in purely synthetic training approaches. Below, we sketch out an approach to re-purpose existing judge models to create the training samples for the next generation of auto-evaluators.

An interesting line of future work in auto-evaluation could be to utilize all judge models as a “crowd” to conduct crowdsourcing. That is, to construct new pairwise preference datasets, we evaluate responses with existing judge model checkpoints (whether pairwise or rating), then aggregate responses. This approach diversifies the source of automated labels appealing to the wisdom of a model crowd, rather than relying on a singular model like GPT-4o to generate labels. Because each crowd member is a model, new exciting approaches in aggregating responses are enabled. Rather than simple majority vote, we can weight each model response by confidence heuristics, like model uncertainty (measured via logits), model size (e.g., give more weight to a 70B model vs. an 8B model), or even model performance on benchmarks (e.g., perhaps we trust a model that achieves 90 on RewardBench more than a model that achieves 75). This entire process can be viewed as a crowd-based self-improvement process, with previous generations of judge models serving as previous “iterations” in self-improvement.

The added bonus is that every new judge model checkpoint released is another member of the crowd, encouraging researchers and practitioners to collectively work towards building gold-standard auto-evaluators. We hope that our models can help overcome current limitations of training judge models. Towards this end, we are planning on releasing our model checkpoints for research purposes at a later date, once we have obtained proper institutional approval, which is currently pending.

[3] Self-Taught Evaluators, https://arxiv.org/abs/2408.02666

Similar to models in the Prometheus family, the current model's input includes an additional evaluation protocol except for the user query and the evaluated response. Therefore, during real-world evaluation setting, additional craft of evaluation criteria is required, as demonstrated in lines 328-329. Although Section 5.4 demonstrates that the impact of the crafted evaluation protocol on evaluation is limited, both this paper and Prometheus-like models require the input of an evaluation protocol (no matter the human-annotated or automatically generated by GPT-4). Therefore, when performing evaluation in real-world scenarios, this severely limits the scalability of the method because it may require additional supervision of humans and advanced models (From my perspective).

We thank the reviewer for bringing up an interesting point. In our view, applying judge models for automatic evaluation requires humans to first define what criteria to use in the evaluation, and to embed such criteria in the prompt to the judge model. We define this characterization of criteria as the evaluation protocol, and is universal in evaluating model responses, regardless of if judge models or human evaluators are used.

We emphasize that unlike Prometheus, evaluation rubrics or reference answers are not required for our model. However, if rubrics or reference answers are available, they can be incorporated in evaluation. In our setting, the evaluation protocol can simply be natural language explanation of what to emphasize (e.g., “focus on how well the output satisfies the user’s instructions”) or a specific manner to conduct evaluation (e.g., PRePair-style prompting of enumerating pros and cons independently before producing a pairwise comparison response). We have updated our paper to explicitly clarify how such protocols differ from Prometheus in lines 142 - 145.

In light of this, we consider the fact that our judge model exhibits relatively stable performance across evaluation protocols in Sec. 5.4 as a strong indicator of the flexibility of our judge model to accommodate evaluation criteria, whether that be task-specific or a fixed evaluation strategy (e.g., PRePair).

Using preference learning for optimization sounds impressive, but some prior work have already utilized preference learning to enhance the evaluation/critique capabilities of LLMs, such as CriticGPT [1] and Themis [2].

Similarly, Themis also utilizes DPO to enhance the evaluation capabilities of language models. Themis also automatically construct the preference data by checking if the model's judgment scores are consistent to human-annotated judgment scores. In terms of motivation and implementation details, there is not much difference between this paper and Themis. Of course, the authors introduce three different preference examples to enhance the capabilities of LLMs.

We thank the reviewer for bringing these two papers to our attention. We have updated the introduction to mention these papers and the related work section with a longer discussion, focused largely on Themis.

The core difference between Themis and our work is that Themis, as the reviewer mentions, makes single-rating specific changes to the DPO loss. In particular, they introduce a margin term into the DPO loss that is based on the difference in value of the ratings between the positive sample and negative sample, modeling the probability of preferring response A or B as

$P(A \succ B) = \sigma\left(r(x, y_A) - r(x, y_B) - \alpha|R(y_A) - R(y_B)|\right).$

Above, $r$ is the parameterized reward model, $\sigma$ is the sigmoid function, and $R(y_i)$ denotes the absolute rating score contained in response $y_i$. As a concrete example, if $y_i$ = “The response is incomplete, but brings up true points. Score: 3”, then $R(y_i) = 3$.

This modification of the DPO loss is specifically designed for learning from single rating evaluations: Because the outputs of judgements are numerical scores, the difference naturally captures some measure of difficulty which can be utilized as a margin. However, it is less clear how to utilize such a DPO loss in training a judge capable of other common types of evaluation settings without numerical score outputs, such as the ones we consider in this paper (pairwise, classification). An additional minor point that arises with this modified DPO loss is the introduction of a new hyperparameter, $\alpha$, which must be set in addition to the typical DPO $\beta$ during training. The effects of $\alpha$ are not characterized by the Themis authors, so we have no insight into the sensitivity of performance with respect to $\alpha$.

In contrast, we propose three different forms of pairwise preference data, without making modifications to the DPO loss, as the reviewer has noted. This design was intentional: Focusing on new ways to construct the pairwise preference data for DPO allowed us to (1) train a judge capable of all three evaluation tasks simultaneously and (2) add new judgment capabilities to the judge simply by adding new types of pairwise preference data and re-training, rather than modifying loss terms. However, we recognize the similarities, and have updated our paper introduction (lines 60-63) and related work section (lines 515 - 530) accordingly.

Thank you very much for the author's response, and I greatly appreciate the additional experiments conducted by the author to address my questions. However, I have a minor concern regarding the experimental results. The proposed 8B model seems to show significantly higher correlation with GPT-4 compared to Themis-8B, but the correlation with human evaluation shows little difference or is slightly worse. Could you provide some extended explanation regarding this issue?

We are happy to provide additional explanations. We begin by noting that our models and Themis are both trained on a mix of human-annotated and model-annotated single rating data. However, Themis only used GPT-4 data that closely matched human annotations. From the Cross-Validation paragraph in Section 3.2 of the Themis paper:

We prioritized samples where the two evaluation ratings were close, which reflected the high consistency between evaluations from humans and GPT-4 and also indicated the potential strong reliability of the samples.

Therefore, it is likely that Themis discarded many samples labeled by GPT-4 that exhibited disagreement with human annotations, skewing the distribution of GPT-4 annotated training data to be close to human annotations. This choice likely explains why Themis exhibits strong human correlation but relatively weaker GPT-4 correlation: Samples that would allow Themis to potentially bridge the gap in correlation with GPT-4 are not used in training.

The fact that our 8B model roughly matches the performance of Themis-8B in terms of human-annotation agreement (+0.01 in BiGGen Bench, -0.02 in FLASK relative to Themis) demonstrates that the additional cross-validation step of removing GPT-4 samples that do not agree with human annotations is not necessary: Judges can generalize their decision making to both types of annotators. This generalization to correlate with GPT-4 annotations may be desirable, as we describe next.

Here, we provide our rationale for including GPT-4 annotated benchmarks in our evaluation suite. While agreement with human annotations is the ultimate goal of automatic evaluation, measuring agreement with GPT-4 annotations is also a scenario that has practical importance. Consider a practitioner that currently uses GPT-4 as a judge model, but is looking to move to a smaller, self-hosted judge model for practical reasons (cost, inference time, ability to continuously fine-tune, removing reliance on an external API, removing dependence on an opaque model-backend which may change with short notice, etc). If the practitioner’s current system works sufficiently well with GPT-4 as a judge, they may want a drop-in replacement that behaves like GPT-4 in its assessments. As a result, the practitioner may prioritize higher correlation with GPT-4 outputs. In this scenario, measurement with human annotators is implicit in that the practitioner is happy with using GPT-4 as a judge, implying that GPT-4’s ratings correlate well with human ratings.

We believe the above rationale is an important point to emphasize, and will update Appendix A (Evaluation dataset overview) in our paper with a discussion of this importance.

Lastly, training details are important in this discussion. As we’ve highlighted, Themis-8B is (1) trained exclusively single-rating tasks and (2) trained using single-rating specific modifications to the DPO training loss. This training approach makes Themis a very competitive single-rating model, especially compared to other 8B models. Our models are less specialized in that they are trained for other tasks and do not make single-rating specific modifications to the training loss. Despite less specialization, our models are competitive with Themis, with the 8B model performing roughly on par with Themis on tasks (human agreement on single-rating tasks) that Themis was trained to excel at.

If this discussion and our earlier answers have addressed your concerns, we would greatly appreciate it if you would consider raising your score. We are also happy to answer any additional questions!

Broadly a comparison to evaluation systems with multiple models is not expected as such, but still it will be nice to compare and check how does your method works compared to methods like reconcile.

Thank you for the suggestion! We have conducted an additional experiment where we compare our model against ChatEval [1], a multi-model system for making pairwise comparisons. ChatEval is similar to ReConcile, where it uses multiple powerful language models to conduct back-and-forth debate before arriving at a final answer. However, unlike ReConcile, which focuses on improving reasoning capabilities, ChatEval is specifically designed for making pairwise comparison judgements of other language model outputs. As such, we run ChatEval on our 7 pairwise comparison benchmarks.

Specifically, we adopt the default setup of ChatEval, where we have 2 agents, both GPT-4, which assume different roles: a “General-public agent” and a “Critic agent”. These agents debate sequentially for four rounds. We keep the default evaluation prompts provided by ChatEval. At the end of the debate, both agents output a final score for each output. We average the scores for both agents, and set the winner as the higher-scoring response. If the final score is tied, we break ties arbitrarily by arbitrarily choosing a winner. Because of the high inference time and API cost of this evaluation, we only run each pairwise comparison benchmark once, so we are unable to report consistency values.

Despite being powered by a much larger model (GPT-4) and relying on multiple turns of reasoning, ChatEval still underperforms even our 8B model in aggregate, trailing by 8.42 points in terms of aggregate performance. ChatEval struggles in key domains such as safety (85.07 on HHH vs. 94.12 for our 8B model) and bias (66.25 on EvalBiasBench vs. 85.00 for our 8B model), highlighting both the impressive capabilities of our judge models and the shortcomings of prompting more capable, general-purpose LLMs. Our results are consistent with results reported in systematic investigations into judging protocols, where multi-agent, multi-round setups often lag other simpler prompting-based approaches; see the performance of ``multi-role-round{1,2}‘’ systems in Table 9 in [2].

[1] ChatEval: Towards better LLM-based evaluators through multi-agent debate, https://arxiv.org/abs/2308.07201

[2] REIFE: Re-evaluating Instruction-Following Evaluation, https://arxiv.org/pdf/2410.07069

It will be interesting to see how easy vs hard negatives can help. Given the current way of generating negatives, it is unclear if they were hard enough and if a mix of easier and hard negatives could help improve further.

Thank you for the interesting question! This is something that we explored at an earlier stage in our model development, and have updated our paper to include this result in Appendix C.6. In particular, we trained two 8B judge models with different teacher model setups. Judge (a) used preference data with both positive and negative pairs sampled from the same teacher model (Llama-3.1-70B). Judge (b) used preference data where positive samples are from Llama-3.1-70B and negative samples are from Llama-3.1-8B. Therefore, one can consider Judge (a) to be trained on “harder” negative samples, whereas Judge (b) is trained on “easier” negative samples.

We report evaluation results for these two models in the above table and have added an additional ablation section in Appendix C.6 with additional discussion. Note that these are earlier checkpoints, and as such, results for Judge (a) differ from our final reported numbers. In particular, training with a weaker teacher model for negative samples resulted in drops in performance across the board, with the most notable drop coming from pairwise comparison consistency, highlighting that harder preference pairs play a critical role in mitigating certain judge biases.

We recognize that with a fixed teacher model, i.e., the Judge (a) setting above, samples may still vary in difficulty. We consider methods for identifying hard negatives in generating pairwise preference data for judges an exciting avenue of future research.

Choice of 70%:15%:15% for DCoT, DStd and DDed sounds somewhat arbitrary. Can you provide further reasoning behind this?

Thank you for highlighting this important aspect! The choice of data distribution reflects our focus on enhancing the reasoning capabilities and interpretability of our LLM judge. Since reasoning is a crucial component for accurate and transparent evaluations, we prioritized D_CoT to ensure the model effectively learns to generate the Chain-of-Thought critique before making the final judgment. At the same time, we maintained a balance with direct supervision ($D_{Std}$) and response understanding ($D_{Ded}$), as both have been shown to significantly contribute to overall performance improvements.

citep vs citet should be used properly. E.g. line 41 should have "GPT-4 (OpenAI, 2023)" and not "GPT-4 OpenAI (2023)". Notations M_t vs M_{teacher} or M'_{teacher} do not match between Section 3 and Fig. 2. Same for student model.

Thank you for pointing out these issues. We have reviewed the paper to make sure citep and citet are used properly. We have also standardized the notation for the teacher/student model in Section 3 to match those in Fig. 2.

Novelty of data curation: As the main contribution of this paper is data curation, it’s better to clarify the difference between this work and existing studies on data curation for LLM evaluation. The key differences are still unclear even after reading the description in Lines 136-150. From Table 1, existing studies have explored the three evaluation tasks (Vu et al., 2024), and have used both SFT and DPO (Want et al, 2024c). This paper seems incremental to combine what has been done in existing studies at a larger scale of training data.

The technical novelty is weak, compared to the significance of releasing a good evaluation model to the community. However, I don’t find the statement of releasing the model checkpoints and the curated data from the paper. It’d be better to clarify this explicitly.

We plan on releasing model checkpoints, once we have obtained proper institutional approval (currently pending). We have added a footnote to Page 1 that clearly states this, as well as a link to an anonymized version of our evaluation code. We apologize for this oversight! Here, we’ll briefly discuss the novelty of our work in relation to prior work:

Our work is different from prior work in that we are, to our knowledge, the first work to explore utilizing DPO to train an explanation-generating judge LLM for multiple evaluation settings. Our contributions are (1) utilizing DPO for training a multi-faceted judge model capable of 3 different evaluation tasks, (2) an exploration of different DPO tasks to enhance the judging and reasoning abilities of judge models, and (3) eventual release of model checkpoints for research purposes, once we have obtained proper institutional approval, which is currently pending. We have re-written portions of the introduction to clarify; please see lines 73-75 and lines 82-87.

In terms of novelty, Self-taught-evaluators (Wang et al, 2024c) is concurrent work, for which the DPO training recipe was released after our model development was completed. In fact, the paper from Wang et al, 2024c only mentions iterative SFT. However, an update to their code-base on September 26 revised the training process to include DPO. Despite how close this update was to the ICLR submission deadline (October 1), we immediately evaluated this model to ensure fair comparison. Because this update was made so close to the deadline, ICLR guidelines are explicit in this characterization as concurrent work: ‘’We consider papers contemporaneous if they are published within the last four months.” See bottom of this link. The caption for Table 1 has noted the ambiguity in the training method of this model.

In terms of technical differences, Self-taught-evaluators use iterative DPO (according to the Github link), requiring multiple rounds of training, to train a judge model that only performs pairwise comparison evaluation. For preference data, they only use pairs of {correct critique + judgment, incorrect critique + judgment}. In other words, Self-taught-evaluators only train their model with preference pairs that include critiques. Our work, in contrast, additionally includes pairs with only standard judgments and a reverse response deduction task (Training Tasks 2 and 3 in Figure 1). The fact that our judge, which is not tailored to pairwise evaluation, outperforms this pairwise-specific model demonstrates (1) the effectiveness (and relative simplicity) of our training recipe, specifically the importance of our additional training tasks.

Presentation: It’s hard to interpret the results and the evaluation with other baseline models (see details in the question section below). The tables and figures are far away from their corresponding text descriptions, making it hard to read them at the same time.

Thank you for your feedback on formatting. We have updated the paper so that tables and figures are closer to their reference points. We will continue to experiment with paper layout to ensure our final version is as readable as possible.

In Line 314, “All baselines are evaluated only on the tasks they are trained for”. Can you report which tasks these baselines are trained for?

Some existing judge models are only trained for certain evaluation settings, like pairwise comparison evaluation only. We have reported different judge model capabilities for the baselines we evaluate in the second column of Table 1. We have updated the paper in Section 4.3 with an explicit reference to Table 1 for tasks, and provide an example of what we mean by “All baselines are evaluated only on the tasks they are trained for” in lines 285 - 288.

In Line 315, what do you mean by evaluating models trained for single ratings on classification? How do you use a model trained for single ratings to evaluate classification tasks?

For all reported numbers, we run publicly available checkpoints in our evaluation setup. To evaluate single rating judges on classification tasks, we prompt the judge model to perform the (binary) classification task directly. That is, the model is prompted to output the natural language text “Yes” or “No”, which is then compared to the ground-truth label. Because a vast majority of judge models are not trained to do classification evaluation, we evaluate single rating judges on classification tasks via the aforementioned prompting approach. We emphasize that we are not using these judge models as logit-based classifier models. Rather, we prompt the model to output a natural language decision. We have made this explicit in our updated paper in lines 289 - 291.

We choose to evaluate single rating judges in this setting because of the pointwise nature of both classification and single rating tasks: they evaluate responses in isolation, without any other answer to compare against, unlike the pairwise comparison evaluation setting. Judge models that are trained exclusively on the pairwise comparison evaluation setting may not generalize to the single rating or classification settings, as we show below

How can you get the results for the tasks that these baselines are not trained for in Table 1 & 2? Do you get the baseline models’ results from their papers or run the baseline models by yourself? Do you test your models on those benchmarks without further fine-tuning your models on the benchmark data at all?

We run all baselines and our judge models ourselves under a fixed evaluation setup. We have attached a link to an anonymized version of the evaluation code in the footnote on Page 1 and have added a note to the text (line 265) to emphasize that we run each baseline model ourselves.

We do not conduct additional finetuning, as judge models are typically released as models ready to use “out-of-the-box”. We ensure that we do not finetune our own judges on benchmark data; i.e., we ensure no test set data contamination. For each benchmark, we conducted a comprehensive evaluation of any data sources used to create such benchmarks, and created our training set with these data sources in mind. In particular, we take caution to only use train or dev sets, and check that such samples do not appear in benchmarks.

We note that with the exception of evaluating models capable of single-rating judgements on classification tasks, we do NOT have results for tasks that baselines are not trained on. Here, we demonstrate that the choice to evaluate models only on training tasks is warranted. We run evaluation for single rating for the Skywork-Critic-8B and 70B models, two extremely competitive pairwise models. These models are trained exclusively on pairwise tasks, and as such, it is not fair to compare such models against models trained to do single rating. We emphasize that these results are for illustrative purposes only.

We prompt the Skywork critic models using a minimally modified version of the Skywork prompt template to perform single rating evaluation. We preserve the original formatting of the pairwise prompt provided by Skywork, minimally changing the instructions to describe a new task (single rating) and to ask the judge to output its decision as [[an integer between 1 and 5]]. Using this prompt template, both the 8B model is only capable of producing outputs “[[A]]” or “[[B]]”, indicating that training has caused such a model to overfit to the pairwise comparison output format. The 70B model is more capable of producing integer outputs, but such outputs correlated extremely poorly with human ratings (Pearson correlation of effectively 0.00 for Skywork-Critic-70B vs. 0.76 for our 70B model).

In Table 5, what are the 6 types of biases? And How do you measure these biases? I understand the position bias can be measured by the consistency of choosing responses even if the order is swapped. But what about the other biases?

We adopt the biases identified by OffsetBias in their paper [1], and have updated our paper to include a description of the biases and how to interpret such results in Appendix A, starting on line 807.

Table 5 presents results on how immune judge models are to such biases. Each category contains pairwise comparisons, where one response is correct and the other response is incorrect, but in a manner that may “trick” the judge model based on the relevant bias. In other words, if the judge model is susceptible to a particular bias, then it will pick the incorrect response. To measure how immune judges are to these biases, we can simply measure the accuracy of a judge in responding to these pairwise comparisons, where the more unbiased a judge is, the more correct responses they will pick.

To summarize, the six biases that were identified by OffsetBias are as follows:

• Length bias: judges prefer longer responses, even if the response does not follow the user’s instructions.
• Concreteness bias: judges prefer responses that are more concrete, such as citing precise percentages or figures, even if they are wrong or irrelevant
• Empty reference bias: Sometimes the input instruction provided by a user is incomplete (OffsetBias authors provide an example of a user requesting a summary of an article, but forgetting to provide an article). Weaker models are susceptible to hallucinating responses based on imagined input content, whereas strong models ask for clarification. Judges tend to prefer hallucinated model responses rather than responses that ask for clarification.
• Content continuation bias: judges prefer responses that continue generating related content to user requests, rather than those that faithfully execute user instructions.
• Nested instruction bias: If the user instruction includes an input (e.g., an article) that includes an instruction, then the judge may evaluate responses based on how well they satisfy the nested response rather than the original user instruction.
• Familiar knowledge bias: Judge models may prefer responses that contain common information (e.g., idiomatic sayings or common facts) rather than responses that precisely follow the user’s instructions.

The authors of OffsetBias arrived at these six bias categories by analyzing failure modes of judge models by generating many judge responses. Then, the authors perform filtering and manual editing to arrive at samples that fit each of the bias categories.

[1] OffsetBias: Leveraging Debiased Data for Tuning Evaluators, https://arxiv.org/abs/2407.06551

Line 464 claims that “we plot the average performance across all three evaluation tasks when removing each training task”, but Fig 4 may miss the SFT+DPO w/o CoT critique.

Our objective is to train a generative judge model that produces both judgements AND critiques. As such, removing the training task that includes CoT critique would mean our judge is not trained on preference samples with critiques. Because such critiques can be extremely useful in improving downstream model performance (see the experiment presented in Section 5.5, with results in Figure 6), we choose not to omit this task during our ablations.

Dear reviewer,

This is a gentle reminder that the discussion period ends on Wednesday, November 26. We are checking in to see if you have any questions or concerns about our response or our revised paper, which was re-written to address reviewer questions and includes new experiments exploring critique quality (Appendix C.7) and training sample difficulty (Appendix C.6).

If you believe our rebuttal and updates address your concerns, it would be greatly appreciated if you would consider increasing your score. We are also more than happy to provide additional clarification and answer any follow-up questions.

Thank you!

The organization of this paper could be further improved, for example, structurize some long paragraphs.

Thank you for the feedback regarding the organization of the paper. We have taken the following steps to improve this aspect of the paper. Please see the updated paper for changes; all major additions or changes to the text have been written in blue font.

• We have broken up longer paragraphs (for example, related work in Sec. 6) and have edited sections for conciseness.
• We have re-arranged the order of figures and tables such that tables appear closer to relevant text.
• We have re-written portions of the introduction to clarify our contributions and other relevant work. For example, see lines 73-75 and 82-87.

The paper could be strengthened by providing a more detailed explanation of the design philosophy, including the rationale for including each component, their individual functions and relationships, and how they relate to the background presented in Section 2. An ablation study or discussion of the impact of removing individual components would also be valuable.

Thank you for your insightful feedback regarding the design philosophy of our work! Below, we highlight our philosophy for utilizing paired preference data in training our judge model, describe the specific reason for including each of the three DPO training tasks, and recap the ablation study presented in the main body. For each point, we provide references to the main text where such explanations appear. If the reviewer has specific questions about design choices, we are happy to further clarify during the discussion period!

1. The use of both positive and negative training examples: The core of our idea is to overcome the limitations of SFT, which only trains the judge model with positive examples, by training with paired preference data. We discuss the limitations of SFT in Section 2 (Lines 146-152) before presenting our approach of utilizing DPO to train our model with three types of preference data in Section 3.

2. Three types of preference data: The rationale of each type of preference data is summarized in the first paragraph of Section 3. Specifically, we explain that (1) the CoT critique is for boosting reasoning while providing interpretability, (2) the standard judgment is for direct supervision and (3) response deduction is for enhancing reasoning. Each following subsection elaborates these rationales before introducing the corresponding method. For example, Section 3.2 explains why the training signal from CoT critique alone is insufficient and how the standard judgement can address this gap. We have also included Figure 3 to visually support our claim.

3. Ablation Study: Indeed we have conducted the ablation study in Section 5.3, where we systematically remove one certain component from each baseline to validate our design choices, as acknowledged by Reviewers xfuc and azFf. The results shown in Figure 4 demonstrate that our complete method yields the best overall performance, underscoring the importance of each component.

If the reviewer has further questions, we are happy to clarify!

Dear reviewer,

This is a gentle reminder that the discussion period ends on Wednesday, November 26. We are checking in to see if you have any questions or concerns about our response or our revised paper, which was re-written to address reviewer questions and includes new experiments exploring critique quality (Appendix C.7) and training sample difficulty (Appendix C.6).

If you believe our rebuttal and updates address your concerns, it would be greatly appreciated if you would consider increasing your score. We are also more than happy to provide additional clarification and answer any follow-up questions.

Thank you!