Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning

We thank you for taking the time to review our paper. We are glad you like our paper. To address your concerns, we have: clarified the writing further in places you expressed concern and answered each of your questions. Please let us know if anything is still unclear or if you have further questions. We are happy to clarify anything! If your concerns are addressed, we would appreciate it if you would be willing to raise your score. We respond to each of your main concerns below:

I partially disagree with the claim that the authors made at the end of the introduction. I think what the authors have shown doesn't mean scaling test-time computing can be preferable to scaling pretraining compute. For both medium and hard questions, using …

We do not view our analysis as the last word on test-time scaling, but rather as merely one of the first steps, in which we demonstrate the efficacy of a set of fairly straightforward techniques from the existing literature. While we believe that our statement is correct and that test-time compute can be preferable to pretraining in certain settings (either with easy/medium questions or a low inference budget), we can add additional qualification about potential drawbacks and nuances. There is significant room for future work to improve upon our methods, as discussed in Appendix A. We also further qualify this statement further in the updated draft.

When training a verifier, is it more fair to include the compute to train the verifier as part of the inference-time compute? Similarly for the fine-tuned revision model.

In our case the compute needed for finetuning is trivial relative to pretraining (e.g. less than one thousandth) and thus is has no meaningful impact on the conclusions of our analysis. However, future work may desire to scale up these techniques to training on orders of magnitude more problems during finetuning (either on the same task, or on multiple tasks as you suggested) than the 12k problems from the MATH training set that we used in this work, in this case finetuning compute may become significant. Our goal is not to demonstrate the strongest possible results with test-time compute, but rather to demonstrate what is possible to start with. We therefore leave questions of scaling finetuning for test-time compute as future work. However, we include a note about this as an avenue for future work in the discussion section of the updated draft.

It is a bit difficult to fathom what "compute optimal" is exactly. How is this obtained or how is it optimized? I understand that strategies are selected based on question difficulties but providing the exact detail would be nice.

Compute-optimal scaling refers to running an inference-time method with the best hyperparameters, specific to a given input. That is, given a compute budget B, the compute optimal strategy prescribes a set of hyperparameters customized to a given question so as to obtain the best performance within this budget B. As an example of implementing compute optimal scaling for allocating compute between sequential revisions and parallel best-of-N for a given question we: 1) compute the performance of both techniques on all difficulty bins using a held-out validation set; 2) given a new test question, estimate it’s difficulty and then deploy the strategy which had the highest performance within that difficulty bin on the validation set. We include this example in Section 3.2 of the updated draft, and in the “compute-optimal scaling” part of Sections 5 and 6, we include additional details of the specific hyperparameters that we optimize over. Please let us know if this is still unclear or if you have further questions. We are happy to clarify anything.

The separation of Verifier and Revision is a bit confusing as both require a PRM. The main distinction I think between sections 5 and 6 is one is using search and another is using revision.

While you are right that both verification and revisions use a PRM, revisions modifies the base distribution (i.e., base LLM as well) whereas search using a PRM uses the base LLM directly as the proposal distribution. Concretely, in Section 5, we fix the base LM as the proposal distribution and focus our experiments around search with a learned verifier. In Section 6, we instead focus on optimizing the proposer by training it to do revisions. While there is also a verifier employed in Section 6, the experiments are not centered around different search techniques. We add additional clarifications about this to the updated draft. Do let us know if this explanation helps to clear things up.

We thank you for taking the time to review our paper and are glad for the positive review. We appreciate that you find our experiments to be comprehensive and our analysis to be insightful. We respond to each of your main concerns below:

Given multiple downstream datasets, we only need to scale up training time compute for once while have to scale up inference overhead every time. In this case, if we match the total test time compute with training time one, each task will be allocated much fewer inference time budget, which may lead to very different conclusions.

When scaling inference time compute, dedicated efforts are needed to improve the verifier performance or models' capability of self-correction, which may also consume a non-trivial proportion of compute. This may also lead to unfair comparison to training time compute scaling.

There are great points! In our case the compute needed for finetuning is trivial relative to pretraining (e.g. less than one thousandth) and thus is has no meaningful impact on the conclusions of our analysis. However, future work may desire to scale up these techniques to training on orders of magnitude more problems during finetuning (either on the same task, or on multiple tasks as you suggest) than the 12k problems from the MATH training set that we used in this work, in this case finetuning compute may become significant. Our goal was not to demonstrate the strongest possible results with a test-time compute, but rather to demonstrate what is possible using a few straightforward ideas from the existing literature. We therefore leave questions of how test-time compute scaling improves with additional finetuning data/tasks as future work. However, we include a note about this as an avenue for future work in the discussion section of the updated draft.

Most presentations are excellent but I think there are still some minor issues. was a bit confused when reading Figure 3 (Left) since there is no legends and the overlap of bars make colors even more complex

Glad that you find most presentations excellent. Good catch regarding Figure 3! We correct this in the updated draft. To answer your question here: green=majority orange=best-of-N blue=beam-search

Also, the main body oof the paper should be self-constrained, so it may not be quite suitable to put the results in Sec 5.3 to Figure 14 which is inconvenient to readers.

We have moved this figure into the main paper in the updated draft.

It seems all PRM results are presented in the way of weighted Best-of-N. Is PRM alone not working? If so, is it fair to compare lookahead search with the other two, given that the value in lookahead search can not be augmented by majority vote?

Best-of-N performs well; it just somewhat underperforms weighted best-of-N. Specifically, best-of-N refers to selecting the best single response from all samples. best-of-N weighted effectively combines best-of-N and majority voting by summing over the scores of all solutions which reach the same final answer and returning the answer with the greatest sum. We include a plot with Best-of-N in appendix G.2 of the updated draft.

Thanks for the response and for the great question! Regarding the majority voting baseline in our results, we do not believe that our PRM outperforms majority vote simply because we used a weak model. This is because we believe this result is consistent with other results in the community:

• Prior results show that verifiers outperforming majority voting is a standard result, including in cases where much larger or stronger models are used. For example, the MathShepard paper [1] shows in Table 1 that the MathShepard verifier beats majority vote for a number of strong base LMs (DeepSeek-67B, LLemma-34B, LLaMA 2-70B). Additionally, the Let’s Verify Step by Step paper [2] shows a PRM verifier significantly outperforming majority vote with a very strong base model in Figure 3 (majority is ~68%).
• In our results, we see a very consistent trend, across three different base LLMs in Figure 3 (right) and Figure 16, in which majority voting performs similarly to ORM/PRM on easy questions, but on medium/hard questions the gap widens significantly with verifiers performing much better. Indeed, for a much more capable model with much higher absolute performance on the MATH benchmark (say 90-95% for example), all problems in the benchmark would effectively become “easy problems” and thus majority voting may be all we need. However, even for such a model we should still be able to construct much harder questions than anything in MATH (e.g. IMO level), which would benefit from a verifier. There should always be “hard problems” (unless all reasoning problems can be solved), just that they may not exist in the MATH dataset.

For these reasons, we have a strong reason to believe that the general findings that we observe on MATH in our setting should hold more generally, even as we move to much more capable LMs. That said, we want to clarify that we are not proposing that PRMs are the best way to use test-time computation -- there are likely better ways of using test-time computation that lead to better results. Our point is just to show that using test-time compute via PRMs can enable us to obtain better performance, though it may not always the best way to do so.

[1]: Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

[2]: Let’s Verify Step by Step

Thank you for the review and for the constructive feedback! We are glad that you liked the paper. To address your concerns, we have: 1) adjusted for the cost of calling the verifier in our analysis; 2) added additional results using other base models for PRM search; and 3) answered each of your questions. Please let us know if our responses and edits to the paper address your concerns and if so, we would be grateful if you are willing to raise your score. We would be happy to discuss further.

Both the PRM and ORM models utilize the PaLM 2-S* base language model, which is costly. However, the computational cost associated with PRM during test time is not clearly defined and calculated.

Thanks for the question! We first note that these changes do not change the main conclusions of our analysis (we discuss why below), but we have now updated the draft to modify our analysis to account for the verifier cost. Using a verifier is twice as expensive as the majority voting baseline, so to account for this, we shift our best-of-N plots accordingly. Additionally, we can update our analysis of exchanging pre training to reflect the cost of calling the verifier. In practice, the effect of this is to half the R values (D_inference/D_pretrain) corresponding to each of the vertical lines in Figure 7. With these adjustments, the majority curves still fall below our best-of-N baselines and our conclusions with regards to how pre-training/test-time compute can be exchanged remain unchanged.

In a specific domain, analyzing the trade-off between pre-training and test-time computation is more meaningful, but the paper primarily focuses on pre-training.

We might be misunderstanding your question, so please point out if this answer does not address your concerns. To our understanding, this question is about potentially analyzing tradeoffs of fine-tuning vs only using test-time computation. It is true that for our analysis in Section 7 we could have fine tuned the larger model as another baseline comparison. However, we do not believe this would represent a meaningful improvement in performance so as to change any of our main findings. Namely, finetuning models using techniques like Rest^EM or STaR on the same number of questions we use for training the verifiers/revision models (e.g. 12K questions from MATH) does not improve accuracy by more than ~5% on MATH [1]. Our test-time compute techniques instead improve performance by up to 30% in some cases. Therefore, we do not believe this would represent a meaningful improvement in performance so as to change any of our main findings.

Whether the observations are consistent across different datasets and models remains unclear…

We have some additional results with training different size verifier models, and we see the same trends for these other models. We include these results in Appendix I of the updated draft.

As for datasets, MATH is a standard dataset used in a number of related works [3,4,5], and it is really the only datasets which is hard enough, but not too hard, to provide a good test bed for analyzing test-time compute at our level of model capability. So we believe that the dataset should not be grounds for rejection. That said, if the reviewer has thoughts on what could be a better dataset to study test-time compute, we could consider incorporating that for the camera ready as well.

What do the different colors in Figure 3 (left) represent? It appears the legend is missing.

Good catch! We correct this in the updated draft. To answer your question here: green=majority orange=best-of-N blue=beam-search

Thank you for your response. Most of my questions have been answered. For the second weakness, yes, I want to ask for fine-tuning instead of pertaining; thank you for answering it. However, I still have concerns about whether those observations can be generalized to datasets from different domains and difficulty levels. Reviewer 19PN hoB2 also mentioned a similar concern...

We respond to this comment first, and we will respond to your other comment shortly.

Thank you for the response! We’re glad that we were able to answer most of your questions. Regarding datasets, unfortunately most datasets traditionally used in the LM literature are not a great fit for studying test-time compute. For example MMLU and triviaQA are more knowledge intensive, rather than reasoning. In fact, according to [1], which conducted an extremely thorough meta-analysis of which tasks benefit from chain of thought, most of these datasets (CSQA, MMLU, etc..) do not benefit from chain of thought. Only pure math tasks saw improvement. This is not to say that we expect test-time/reasoning compute to only benefit on math, it just means that the settings where today’s generation of LMs actually see improvement from reasoning are limited. We expect reasoning will likely help in many settings long term, such as agentic problems. We will consider other coding datasets (e.g., Codecontests) for the final version of the paper. However, according to [2] llama 3-8B-Instruct has a very low oracle pass@100 rate on Codecontests (e.g. 10-15% verses 80-90% on MATH) , so obtaining useful results on this task may have a very high time/computational cost, and thus it would be infeasible to get this running in the rebuttal period.

That said, we believe that despite that fact that our analysis is conducted only on MATH, our difficulty bin analysis does provide some insights about how we should expect different test time compute techniques to perform on tasks of varying difficulties. Namely, we see that on really easy tasks simple techniques like majority voting are likely sufficient. However, on harder tasks, more sophisticated techniques are often beneficial.

[1]: To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning

[2]: Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

Regarding my questions, it seems the authors generally agree that the performance of test-time scaling is highly sensitive to the accuracy of the verifier. I believe it would be valuable to include a more detailed discussion on this point. I agree that the verifier is well trained, and I believe the observations based on a well-trained verifier. That's exactly the reason why I have concerns about the verifier. I'm curious what will happen if we can not get a well-performing verifier or if we can not afford high verifier training costs. Also, as the response 'Indeed the verifier accuracy is a bit lower at the higher difficulty levels.', what happens if it's more unbiased?

Specifically, with different verifiers, the exchange between pretraining and test-time computing may vary significantly. It’s also unclear whether observations such as “beam search is more effective on harder questions and at lower compute budgets” remain consistent under different verifier configurations. Could you provide additional information or clarification on this?

Also, thanks for the effort to add the verifier's inference cost. But should the verifier's training cost also be considered in the pretraining and test-time compute exchanging discussion?

Thank you for the review and for a positive assessment of our paper. We are glad you appreciate the comprehensiveness of our experiments. We respond to each of your main comments and concerns below:

Regarding the “proof-of-concept” nature of our analysis.

As stated in the introduction our findings “[suggest] that with a fairly naive methodology, scaling up test-time computation can already be preferable to pretraining, with only more improvements as test-time strategies mature.” We do not wish to claim that our work is the final word on test-time compute and indeed believe that there are many future directions to expand upon our work, including: alternative designs for difficulty bins, novel approaches to training verifiers, and fundamentally different methods for utilizing test-time computation. We could not study all of these due to computational and infrastructural limitations, but we discuss many of these possible directions in the discussion section (Appendix A). We believe that the specific experimental decisions we made in the paper provide for a strong starting-point for understanding test-time scaling (e.g. we use 5 difficulty bins as it aligns with the 5 bins in the MATH dataset; we study improving the proposer and verifier as these represent two primary axes with which test-time compute can be scaled as discussed in the paper; the specific methods we implement are not novel and instead based on existing approaches in the literature). We also note that there are some specific limitations to our work, such as the cost of estimating question difficulty (see limitations in Section A).

In the specific case of our difficulty-bin analysis, our main contribution is to show the existence of a statistic that can help us predict how to best use test-time compute, since, in principle, no one test-time strategy is going to work on all problems (e.g., it is easy to construct worst case examples where each strategy is bad). Of course, while we use a fairly naive way of computing this statistic, recent work [1], already builds a cheap method to estimate this quantity, showing positive results.

It'll be much clearer and more helpful if the authors can discuss more about the possible "hyper-parameters", compensating the framework described in Eq. 1.

Section 3.2 of the updated draft includes a clear example of how Eq. 1 is used. We also describe the specific hyperparameters considered in the compute optimal results sections of both Section 5 and 6 (the changes are in blue).

My understanding from the paper is that different FLOPs budgets of SFT can also be taken into consideration

The finite nature of math finetuning data (e.g. 12k questions in the MATH training set) means that the total SFT compute in terms of FLOPs contribute negligibly w.r.t pretraining FLOPs in Section 7 (to be precise, less than one-thousandth the compute used for pretraining) and thus will have no meaningful effect on our conclusions. We can spend additional FLOPs on each question by training for additional epochs or generating additional solution traces per questions but scaling in this way will quickly overfit [2] (plus we already do early stopping with all of our models).

On the other hand, future works could consider experimenting with a much larger custom scraped dataset of, say hundreds of thousands of questions (or millions), to better understand the scaling of these different techniques with finetuning compute. It is likely that verifiers or revision models trained on this much larger dataset would perform much better. We leave this exploration to future work.

This is an interesting and insightful paper. I have another question (better called thought) -- what if we scale on both test-time compute and pre-training compute

Thanks for the question. This is a very insightful point, there is certainly an interesting question of what is the optimal ratio of compute to allocate to pretraining versus test-time. Expanding upon our empirical findings, we hypothesize that the optimal model given a total FLOPs budget that accounts for pretraining and test-time compute will be different from the compute-optimal pre-training solution by itself. While we do not answer this question in the paper, it is an interesting question that future work should concretely answer, building on our analysis.

To see intuition as to why the compute-optimal pre-training solution might change, consider over-training: while overtrained models are generally better when considering pretraining + standard inference (e.g. no extra test-time compute) FLOPs, they might be suboptimal when accounting for test-time FLOPs as these models may not produce samples with enough diversity for test-time search or revisions to be effective. We add a discussion of this to the discussion section in the updated draft.

[1]: Learning How Hard to Think: Input-Adaptive Allocation of LM Computation

[2]: RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold