Thank you for your comments. We address your points below.

• Different Base Models and Datasets: In our revised paper, we added extensive experiments (i) using a larger 9B model (Figure 13), (ii) on another Fractional math and Math odyssey domains (Figure 14, 15, 16, 17), (iii) BoN-aware fine-tuning in the face of Verifier mismatch (Figure 12), (iv) on another HumanEval coding task (Figure 18), (v) other BoN distillation SFT baselines (Figure 11), as well as (vi) co-scaling studies with Gemma 9B model (Figure 9 and 10), for a more comprehensive analysis of our methods. See Appendix D3 in our updated paper and the results summarized in the above responses.

• Rationale for Equation (7): The term penalizing y' in Equation (7) serves as a regularization factor. While learning from high-quality y' is important, preventing overfitting to the BoN sample is also crucial. This term balances the model's learning from the expert (y) and the BoN sample, ensuring robust generalization. It discourages the model from solely relying on maximizing the verifier score, as it should be “aware” of the way the model will be used in inference – by considering both the expert demonstration and the expected win-rate of the N generated solutions.

• Comparative Analysis with Baselines: We added new SFT experiments with baseline training datasets as suggested: (a) training on the best of N, (b) training on all N samples, and (c) training on N samples weighted by verifier scores, (d) labels selected with majority voting. See Figure 11 in Appendix D3 the updated paper and a summary of these results in the following.

BoN Accuracy

Pass@N

While the aforementioned baselines do improve BoN performance over the Base Gemma 2B model, they are still out-performed by our BoN-RL-V method, indicating the value of utilizing the inference BoN strategy explicitly during training. We believe these additions and clarifications significantly strengthen our paper and address your valuable feedback. We welcome further discussion during the rebuttal stage.

Thank you for your positive feedback and insightful questions. We respond to your concerns as below.

• Single Model and Task: In our revised paper, we added extensive experiments (i) using a larger 9B model (Figure 13), (ii) on another Fractional math and Math Odyssey domains (Figure 14, 15, 16, 17), (iii) BoN-aware fine-tuning in the face of Verifier mismatch (Figure 12), (iv) on another HumanEval coding task (Figure 18), (v) other BoN distillation SFT baselines (Figure 11), as well as (vi) co-scaling studies with Gemma 9B model (Figure 9 and 10), for a more comprehensive analysis of our methods. See Appendix D3 in our updated paper and the results summarized in the above responses. All our experiments still showcase the superiority of our BoN inference-aware finetuning methods and their generalizability to broader problem settings.

• Emphasis on N and T scaling over BoN-aware FT: Understanding the relationship between N and T is merely an initial crucial step to optimize BoN inference. Indeed, the main work and experiments of our paper develop BoN-aware fine-tuning of LLMs and demonstrate their effectiveness. To strengthen our comprehensive evaluation, on top of the existing Gemma 2B BoN-aware FT (SFT and RL) experiments and ablation studies on MATH, we also (i) added additional experiments on Gemma 9B model (Figure 9 and 10), (ii) tested our methods on held-out MATH benchmarks, e.g., Fractional math and Math Odyssey (Figure 14, 15, 16, 17), (iii) extended our work to solve HumanEval coding task (Figure 18), and provided additional BoN distillation SFT baselines (Figure 11) over the ones that we already included in the original paper (RLAIF with learned verifier scores, RLEF with system reward, STaR, SFT). This should further strengthen the evaluation of our BoN-aware fine-tuning methods and outline their competitive advantages.

• Generalizability of Inference-Aware Methods: We have not observed a reduction in the model's generalizability with our inference-aware methods. In fact, as shown in Figure 4a, models trained with BoN-aware fine-tuning, e.g., BoN-RL-V that is trained with N'=32, can also improve performance at N=1 (pass@1), indicating improved generalizability not only on BoN policy but also on the base policy itself. Furthermore, as shown in Figure 14, 15, 16, 17 of the updated paper and the following results, our BoN-aware FT models that are trained on MATH manage to also perform well on other MATH domains, e.g., Fractional MATH. This also indicates the generalizability of our BoN-are FT models on held-out benchmarks.

Gemma 2B Fractional Math: BoN Accuracy

Gemma 9B Fractional Math: BoN Accuracy

• Verifiers and Mismatch: For our experiments, we used pre-trained Gemma 2B and 9B models as the verifier to predict pointwise correctness of responses (with 69% and 76% accuracy respectively). To understand how verifiers mismatch in training vs inference influence the performance of BoN policies, we added experiments comparing the test-time BoN performance of LLMs that were trained to align with the true underlying reward but were using a learned verifier in BoN inference. Details can be found in Figure 12 in the updated paper and a summary of numerical results is shown below, indicating the degree of performance degradation (over BoN-RL-V, a BoN-aware FT model that is both trained and tested with the same verifier).

Verifier reward mismatch experiments: BoN Accuracy

• Figure 1 Calculation and Definitions: The empirical frequency in Figure 1 is calculated as the proportion of problems in the MATH 500 evaluation set for which a particular (N,T) pair achieved the highest accuracy. "Best BoN performance" is defined as the highest accuracy achieved by BoN on a given problem. "Easy" problems refer to those for which BoN achieves high accuracy with small T and N, indicating that extensive exploration is not necessary. Conversely, "difficult" problems require larger T for exploration and often larger N for effective exploitation. This distinction is based on the optimal (N,T) pairs found for each problem within the same MATH benchmark.

Hi Reviewer VBde,

We have carefully considered your valuable feedback and have submitted a detailed response addressing the points raised in your reviews. We believe our response clarifies several aspects of the paper, highlights its contributions, and our additional work (attached above and in the updated paper) addresses your concerns. It would be great if you could take some time to review our responses and let us know your feedback.

Thanks in advance, Authors of Paper 8453

• BoN and Majority Vote: While both BoN and majority vote leverage multiple samples, BoN employs a learned verifier to select the best solution, while majority vote relies on the frequency of a particular output. To understand the similarities of these 2 inference methods, in Table 2 (Appendix D3) of the updated paper, we also added R-squared statistics of the performance betwen BoN and MajorityVoting algorithm, indicating strong correlation on performance across LLMs and these 2 inference algorithms.

R-square statistics of Gemma-2B/9B with Pass@N, BoN, and Majority Voting

Furthermore, BoN, with its learned verifier, should have the better potential to capture more nuanced reasoning patterns compared to the simpler majority vote mechanism. Intuitively, suppose the base model would often produce diverse numerical answers, especially during the early stages of training when the probability of generating the correct answer is low. Then, majority voting may degenerate into random selection amongst mostly incorrect outputs, significantly limiting its ability to identify the correct solution. BoN, on the other hand, has the potential to capture higher quality solutions even with an imperfect verifier (absolute accuracy is less crucial to BoN, as long as the ordering of verifier scores can preserve ranking of the response quality). To support the above claim, our proposed BoN-aware RL fine-tuning manages to boost BoN and pass@N performance over models using majority voting, see Figure 11 in Appendix D3 and the following numerical results (Gemma 2B) for detailed comparisons.

BoN Accuracy

Pass@N

Thank you for your valuable feedback. We address your concerns as below.

• Paper Writing and Visual Figure: We have revised the paper for improved clarity, focusing on a more intuitive explanation of our core ideas. We also added a visual schematic figure (see Figure 2) in the updated main paper and some discussions therein to improve the illustrations of our main idea of inference aware fine-tuning with BoN.

• Co-scaling Behavior Analysis: In addition to the Gemma 2B co-scaling experiments, we also present results for Gemma-9B policy and reward models (see Figure 9 and 10 in the updated paper). Using Gemma-9B improves both Pass@N and BoN significantly compared to Gemma-2B. We observe that the gap between using large temperatures (0.7 or 1.0) and very small temperatures (0.1) also increased. While Gemma-2B showed very strong reward model over-optimization for larger N and temperatures, we see a lesser overoptimization for Gemma-9B models. Similar to Gemma2B co-scaling, for Gemma 9B co-scaling, we analyze the optimal exponent $b^*(T)$ w.r.t different temperatures for the functional form in Eq 5.1 and find that a power law functional form can explain the relationship very accurately, achieving very low extrapolation error for Pass@N and BoN (2.75e-05 and 2.87), suggesting that exponent can be accurately predicted from just temperature. We also inspect how optimal N* scales with T in BoN by fitting a power law function plus a linear term which accurately predicts optimal N for unseen temperatures. Predictions of the fitted model can be used to achieve close to optimal performance, achieving less than 0.001 point drop in BoN performance, suggesting that our predictive model makes accurate predictions that keeps the optimal performance.

• Soundness of Experiments (Figure 4): We understand your concern about potential reward model weakness influencing the results. Our reward model, a separately pre-trained Gemma 2B (9B), has around 69% (76%) prediction accuracy on the MATH evaluation dataset. However, the significant improvement of BoN-RL-V over other baselines (e.g., BoN performance is over 30% for BoN-RL-V versus 22% for base Gemma 2B), including those using a broader range of solutions (e.g., RL-V), suggests that the gains are not solely due to increased sample selection. The inference-aware training itself plays a crucial role in enabling the model to better leverage BoN for performance improvement.

• Extra tasks, e.g., AlpacaEval/Arena-Hard: We appreciate your suggestion for experiments on alignment tasks. While our current focus is on reasoning tasks (math and coding), exploring the applicability of our approach to alignment is an interesting future research direction. In our revised paper, we added extensive experiments (i) using a larger 9B model (Figure 13), (ii) on another Fractional math and Math odyssey domains (Figure 14, 15, 16, 17), (iii) BoN-aware fine-tuning in the face of Verifier mismatch (Figure 12), (iv) on another HumanEval coding task (Figure 18), (v) other BoN distillation SFT baselines (Figure 11), as well as (vi) co-scaling studies with Gemma 9B model (Figure 9 and 10), for a more comprehensive analysis of our methods. See Appendix D3 in our updated paper and the results summarized in our main response above.

Gemma 2B on Fractional MATH

BoN Accuracy

Pass@N Accuracy

Gemma 9B on Fractional MATH

BoN Accuracy

Pass@N Accuracy

SFT Baselines Gemma 2B (requested by reviewer gx5H)**

1. BoN-SFT runs fine-tuning on best-of-N sample (N=16)
2. All-SFT runs fine-tuning on all N samples (N=16)
3. Weighted-SFT runs fine-tuning on a verifier weighted version of all N samples (N=16)
4. Maj-SFT runs fine-tuning on majority voting target of samples

BoN Accuracy

Pass@N

Verifier reward mismatch experiments (requested by reviewer VBde)**

BoN Accuracy

R-square statistics of Gemma-2B/9B with Pass@N, BoN, and Majority Voting (requested by reviewer 4VnU)**

We believe these additions significantly strengthen the paper and address the key concerns raised by the reviewers. We detail our specific responses to each reviewer below.