Thank you for your review!

it is unclear this manual decomposition of plan/execute is most optimal for models

We added additional experiments to showcase the effectiveness of EvalPlanner (planning+execution) for smaller models. In particular, we experimented with Llama-3.1-8B-Instruct and obtained up to 14 points of absolute improvements -- see our answer to Reviewer 1bZh on “Does this method rely heavily on a strong seed model”.

it is never ablated what happens if you simply CoT->verdict and use online preference optimization to optimize

Note that we already have such experiments in the paper. First, one of our baselines, Self-Taught Evaluators (Wang et al., 2024) uses similar data to EvalPlanner and performs preference optimization of simpler CoTs without extensive planning. EvalPlanner outperforms it on RewardBench by up to 4 absolute points (90.0 -> 93.9; Table 2), and by 9-10 points on the other benchmarks (Tables 3, 4, 5) . Second, in Table 7, we also ablate the effectiveness of EvalPlanner’s unconstrained plans with other kinds of plans, wherein we show that our plans outperform other baselines like “list of evaluation criteria”.

isn't it ok to simply reverse the order of the candidates and edit the final answer accordingly

Note that the CoTs (plan+execution) change based on the order of the responses. So, only editing the final answer will make it inconsistent with the corresponding CoT and hence, it is important to perform preference optimization on the data (CoT+final verdict) obtained by reversing the response order.

How is your LLM decoded during evaluation?

As noted in Line 307, we perform greedy decoding when inference.

If you had temperature > 0, what are the standard deviation statistics of the results shown?

We sampled 8 generations with a temperature of 0.8 and top_p of 0.95. Our results on RewardBench are 93.4 with a standard deviation of 0.3. These results are comparable to what we report in our paper, thus showing the effectiveness of EvalPlanner under different decoding hyperparameters.

Did you try using techniques like self-consistency to further boost your results?

We tried it with 8 samples (using the same temperature and top_p as above) and obtained a score of 93.8. So, we did not observe any further improvement in results. This is expected given the fact that the answer space is limited to only two choices (A or B) and the small standard deviation in our results, as noted in the previous answer.

Missing a limitation section

We will add this in the next version.

Thank you for your positive comments about the novelty and the extensive experiments of our paper.

it remains unclear whether the proposed method would also be effective on LLMs from other families or smaller-sized models

EvalPlanner, does in fact, work well even with smaller sized models. To show this, we conducted additional experiments with Llama-3.1-8B-Instruct – see our answer to Reviewer 1bZh on “Does this method rely heavily on a strong seed model”.

Compared to other methods, your inference cost is likely to increase—by how much exactly?

EvalPlanner, on average, generates 1K tokens during inference. Note that the goal of EvalPlanner is to indeed increase test-time compute for evaluation. Complex evaluation is a reasoning problem and in line with recent literature on reasoning, we show that evaluation also benefits from expending more test-time compute.

Thank you for your review and appreciating our work! We are also glad to hear that you’re willing to adjust your scores. We respond to your comments below.

Evalplanner’s applicability to Best-of-N settings

Upon your suggestion, we conducted some experiments and obtained promising results. Please refer to our response to Reviewer bPFz on “Assess whether the proposed judge can be effectively applied to rejection sampling”

Does this method rely heavily on a strong seed model

No, EvalPlanner works equally well, even with 8B models. We happened to experiment with a stronger seed model to maximize performance. As shown below, EvalPlanner w/ Llama-3.1-8B-Instruct improves the seed model by a large 14 points (69.5 → 83.0), almost matching the performance of the much larger Llama-3.1-70B-Instruct and Claude-3.5-Sonnet. Evalplanner does not make any model-specific assumptions and hence should be expected to work with any model for scaling up test-time compute for evaluation.

What would the performance be if only planning or reasoning were used independently

Note that we do such ablations in the paper. First, one of our baselines –- Self-Taught Evaluators (Wang et al., 2024) — trains a judge that generates CoTs without a planning component. Next, in Table 7, we compare EvalPlanner’s plans to other kinds of constrained plans. Having an explicit step-by-step plan allows the model to better reason through it, leading to much better performance. Since the reasoning component always has to be there to produce a verdict, our paper contains ablations of (1) no plan, and (2) other kinds of plans. We’ll clarify this more in a future version.

The performance improvement with the second iteration is promising. What about further iterations?

We did not try a third iteration because of the computational overhead associated with online DPO. This requires obtaining newer/harder prompts and preference pairs, generating outputs from the previous iteration of model, preparing data, and performing preference optimization. That said, with the right kind of data, we believe further iterations should lead to more improvements. We hope that future work can explore it in more detail.

The integration of EvalPlanner with a wider range of CoT-based methods is underexplored

We already explored different forms of CoTs in the paper in Table 7 by varying the type of plan that the judge generates.

The paper lacks deeper experimental analysis, such as the impact of the choice of seed model

Refer to our experiments with a Llama 8B model.

a more granular ablation on decoupling planning and reasoning

We already have such ablations in our paper. First, one of our baselines, Self-Taught Evaluators (Wang et al., 2024) uses similar data to EvalPlanner and performs preference optimization of simpler CoTs without planning. So, it should be seen as a baseline with no planning. EvalPlanner outperforms it on RewardBench by up to 4 absolute points (90.0 --> 93.9; Table 2), and by 9-10 points on the other benchmarks (Tables 3, 4, 5) . Second, in Table 7, we also ablate the effectiveness of EvalPlanner’s unconstrained plans with other kinds of plans, wherein we show that our plans outperform other baselines like “list of evaluation criteria”.

Can you give the training cost of SFT and DPO?

All our experiments are performed on A100 GPUs. SFT of a Llama 70B model requires 3 nodes (24 A100 GPUs) and DPO requires 8 nodes (64 A100 GPUs).

clarify the broader application scope of this method

Evaluation, as we hypothesize in the paper, is a reasoning problem where the judge should plan for the evaluation recipe and then reason through it to arrive at the verdict. Current literature on reasoning (O1/R1) has shown major improvements by scaling up test-time compute. EvalPlanner should be seen as one of the first SOTA recipes for scaling up test-time compute, specifically for evaluation. Through additional Best-of-N experiments, we have also established the effectiveness of Thinking-LLM-as-a-Judge models like our EvalPlanner in improving policy models. Future work could explore its applicability in RLHF pipelines, where both generation and evaluation are scaled up at test time.

Thank you for your review!

Assess whether the proposed judge can be applied to rejection sampling …

First, note that performing extensive RLHF experiments with Evalplanner is beyond the scope of this work and requires separate studies. That said, upon your suggestion, we conducted additional Best-of-N experiments with Evalplanner, on two hard reasoning benchmarks – GPQA (diamond) and AIME2024, obtaining promising results. The experimental setup is as follows:

• For each test data point, we sample N (= 8/16) responses from Llama-3.1-70B-Instruct.
• Since Evalplanner is a pairwise judge, we then prepare samples of response-pairs, amounting to N*(N-1) pairs (considering both orders) for each data point.
• Then we evaluate all these pairs using EvalPlanner and compute ELO ratings to rank the N responses.
• As baselines, we also report results for pass@1, random@N, and self-consistency@N. As an upper bound, we report pass@N.

Below are the results on GPQA (Diamond), where BoN with EvalPlanner improves pass@1 by up to 5 absolute points.

Next, on AIME2024, EvalPlanner also improves Pass@1 by up to 15 absolute points.

Both these results show the promise of EvalPlanner in improving LLMs on downstream tasks.

Analyze how the model's performance evolves over different stages of iterative training to validate its self-improvement capability.

Note that our paper already contains these results in Tables 2 and 3 (with related discussions in Section 4). We present them again below. Recall that EvalPlanner consists of one iteration of SFT and two iterations of DPO. On RewardBench, the accuracies after these three stages are 86.8, 92.3, and 93.9. SFT doesn’t improve results much, especially on the Chat-hard category which contains subtle differences in response-pairs. After we construct DPO pairs teaching models to recognize these differences, it leads to major improvements in the first iteration, even with a small number of prompts (5K). In the second iteration, we obtain further improvements.

Examine how varying the amount of training data affects performance to determine the model’s data efficiency and scalability.

Once again, these results were already presented in the paper in Table 2, associated with a section dedicated specifically to "EvalPlanner is data-efficient and benefits from iterative thought optimization". We show that with as few as 5K synthetic preference pairs, EvalPlanner is competitive with SOTA reward models on RewardBench, obtaining accuracy of 92.3.

It is just a bit straightforward…

Straightforward methods that work well in practice should be preferred. In terms of novelty, to our knowledge, we are the first ones to design a SOTA method that leverages test-time compute for evaluation (via planning and reasoning).

It appears that the performance scores for "safety" and "reasoning" have been incorrectly placed for some baselines

Thanks for noting this! We’ll fix this in the next version.

how the proposed evaluator can enhance generation quality in other policy models

Refer to the answer to your first question!

The baselines in the paper appear to use different training datasets, making it unclear how fair comparisons are ensured.

• First, one of the baselines – Self-taught Evaluator (Wang et al., 2024) uses training data similar to ours and we outperform them by a significant margin.
• Second, for the other baselines, we did not need to match their training data to show the effectiveness of EvalPlanner because EvalPlanner only relies on synthetic pairs and a much smaller number of them. With more or human-annotated data, we expect EvalPlanner to scale even better. This has generally been shown for thinking models that expend more test-time compute for reasoning.

EvalPlanner with weaker seed models

Refer to our response to Reviewer 1bZh on “Does EvalPlanner rely on a strong seed model”.

llm generated planning and evaluation may still contain systematic bias..

It’s a possibility but compared to scalar RMs, we expect a model like EvalPlanner that generates CoTs to be better and more interpretable when dealing with biases.