Rational Metareasoning for Large Language Models

Thank you for your comments. We hope to address your concerns and clarify the purpose of our paper in the following response:

---

(E)

Quiet-STaR has indeed improved LLM reasoning performance by modifying pretraining to generate intermediate reasoning chains between tokens. While this is effective for downstream tasks, it increases computational costs by generating reasoning chains at every step, even when unnecessary. In contrast, our approach is specifically designed to enhance efficiency. Additionally, Quiet-STaR operates at the continual pretraining scale, whereas our approach is applied at the fine-tuning scale, making the methodologies distinct.

We appreciate the suggestion to explore combining these methods, but we were unable to do so within the timeline of the rebuttal. It would be interesting to investigate whether our VoC reward could similarly reduce the cost of Quiet-STaR, as we have demonstrated for standard CoT.

---

(F)

Thank you for flagging this! We agree with your observations, but we believe they actually motivate and support our method. We’ll try to clarify what we mean here, and will update the Results section accordingly.

Looking at Figure 3 and Figure 4 in the Appendix, it seems to me the model learned to simply rely on its internalized knowledge

This is correct. One of the key objectives of our approach, Rational Metareasoning, is to train the model to only use reasoning when necessary. When the model already has the knowledge accessible, reasoning is superfluous, and RM-trained models use shorter reasoning chains. This is in line with our claims: RM reduces inference costs while maintaining performance compared to other reasoning methods, which cause the model to generate costly but often useless reasoning tokens.

one conclusion from this could be that ARC and CommonsenseQA simply aren’t great benchmarks to assess complex reasoning capabilities

We agree with this as well! Again, our method is intended to improve efficiency rather than performance. We use these datasets to highlight the fact that standard reasoning methods are insufficiently flexible, causing them to waste tokens when reasoning is not needed.

I also need to point out that the reduction in reasoning lengths is much less impressive on the harder reasoning benchmarks GSM8K and Proofwriter (see Table 5). Thus, my current hypothesis is that the proposed approach works well for problems that are not hard reasoning problems, but is much less effective on harder reasoning problems where we actually need more CoT-like methods.

You are again correct that we see a less significant reduction in token generation for benchmarks which do require reasoning (GSM8K and Proofwriter). However, we also view this as a strength of our method: we still reduce inference costs on these datasets, without compromising performance.

The kind of adaptive use of different amounts of reasoning demonstrated across datasets is actually one of the strengths of Rational Metareasoning. The pattern of “significant reasoning reductions on ARC/CommonsenseQA and small reductions on GSM8K/Proofwriter” demonstrates that Rational Metareasoning is in fact teaching the LLM to spend inference time adaptively, based on what is effective in each task.

We also added an the requested ablation where the model is directly finetuned on the answer for the phi-2 model, without intermediate CoT (Appendix F). It can be seen that while this provides an even bigger efficiency gain, the performance is less consistent.

---

(G)

Thank you for highlighting this relevant benchmark. Although we did not have the opportunity to include this dataset in the training set, we tested the out-of-domain generalization capabilities of the methods on its test set, as detailed in Appendix C. The results show that performance remains comparable, while our method uses significantly fewer reasoning tokens, confirming its efficiency.

---

(H)

Thank you for your thoughtful suggestion. The models used in our study were not instruction-finetuned, so they lacked the ability to effectively summarize their reasoning. While your proposed approach is compelling, it differs from the central objectives of our work. Our primary focus is on enabling the model to autonomously determine when additional reasoning is required and to improve itself through a self-improvement loop. While fine-tuning on reasoning chains generated by bigger models or humans is an interesting idea that we agree should be explored, it does not align with the core objective of our work, which emphasizes use of reinforcement learning without involving external guidance.

---

(I)

Unfortunately, due to limited computational resources, we were not able to test our method on excessively large models.

Thank you for your insightful comments. Regarding the potential limitation that shorter responses may not necessarily be human-friendly, we would like to clarify that we do not treat reasoning chains as explanations for the model's answers intended for humans. Instead, they serve as intermediate computations that assist the model in arriving at its answer. Nevertheless, we recognize the importance of maintaining interpretability. We have updated the discussion in the appendix Appendix E to address your concern.

---

Comments on Clarity

Thank you for catching these errors! We have corrected the algorithm and updated the paper (main text, lines 166–167 and 169–170) to address your valid concerns. Specifically, the optimal policy approximation refers to selecting reasoning chains that maximize the reward. These chains are then used to fine-tune the model, shifting its policy closer to the optimal one.

---

Experiments

1. Instruction-Tuned Model Baseline
   Thank you for this suggestion. While we did not test this baseline prior to submission, we agree it is an interesting approach (albeit applicable only to instruction-tuned models). We conducted an experiment using Llama-3-8B-Instruct. Unfortunately, this method did not effectively adapt response lengths to match task complexity in our setting. However, we acknowledge it as a noteworthy baseline, especially for larger models or with more refined prompt engineering. The paper now includes this baseline and discussion (Appendix D).

2. Batching Technique
   This choice was partially based on the STaR approach, where the authors found that slower initial training benefits performance. In retrospect, we agree that an ablation study to test its impact on our method would be valuable. We have started further experiments to investigate this but may not complete them by the rebuttal deadline.

---

Minor Writing Suggestions

Thank you for these suggestions and corrections—we have updated the paper to address them!

---

Questions

1. Connection Between VoC and the Reward Function
   Upon re-reading the manuscript, we agree that this connection was not described in sufficient detail. First, individual reasoning tokens correspond to intermediate computations $c$, and the reasoning chain $z$ is a sequence of these computations. The actions $a$ represent the space of possible LLM outputs, while the target sequence $y$ is one such sequence ($y \in \mathcal{Y}$). The belief state $b$ corresponds to the LLM’s internal state after generating zero or more reasoning tokens, and the input $x$ determines the LLM’s initial belief state. Therefore, $\pi_\theta(y \mid x)$ (from Eq. 3) is equivalent to $\mathcal{E}[U(a) \mid b]$. However, an important distinction exists: Eq. 1 describes the value of an individual computation/token, whereas Eq. 2 describes the value of a sequence of computations (a reasoning chain). The LLM’s initial “belief” about the value of actions or outputs is captured by $\pi_\theta(y \mid x)$ (Eq. 3), and its final “belief” after reasoning is described by $\pi_\theta(y \mid z, x)$. We hope this explanation clarifies the connection; please let us know if further points are unclear. The paper has also been updated to clarify this connection (main text, lines 109–111, 117–120).

2. Rejection Sampling Threshold and Alternative RL Algorithms
   A fixed threshold (e.g., 0) makes the hyperparameter $\gamma$ highly influential, as heavily weighting costs can yield negative rewards. We chose a dynamic threshold to increase robustness to hyperparameter choices. Regarding alternative algorithms, the VoC reward function can be readily adapted to other algorithms (for example, DPO would entail training on samples with the highest and lowest reward respectively). We felt it was most important to demonstrate generalization across models and benchmarks, so in this work we focused on a single algorithm (expert iteration) because it has been found to be most sample efficient for reasoning [1]. Unfortunately, we did not have time or computational resources to experiment with alternatives during the rebuttal phase. However, we have updated the discussion (main text, lines 459-463) to clarify our motivation.

3. T in Lines 165–169
   The $T$ refers to the number of examples sampled from $\mathcal{D}$.

4. Expert Iteration Steps
   We perform up to 8 expert iteration steps, but performance typically plateaus after 4–5 iterations. Intermediate checkpoints were not preserved due to their size. We plan to rerun Phi-2 training with STaR and MR and will update the paper with these details once available.

References

[1] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning, 2024.

Weakness 1

Thank you for raising this concern! We agree that agentic reasoning (e.g., ReAct [1] or Toolformer [2]) is an important and emerging domain, and our method may indeed be applicable in such settings. However, the task format for tool-based methods differs significantly, requiring tailored datasets for accurate evaluation.

Additionally, measuring the computational cost of agentic behavior is challenging. Different methods produce varying numbers of LLM calls, and external tools introduce variable and often hard-to-quantify costs. For these reasons, we focused on CoT-style reasoning to clearly demonstrate the efficiency gains of our methodology.

That said, we recognize that applying our method to agentic settings is a valuable direction for future work, which we neglected to mention in the original discussion. We have updated the paper (main text, lines 454–458) to include this point.

---

Weakness 2

We agree it was an oversight not to discuss other reasoning approaches in the paper. While we acknowledge that CoT prompting does not always produce the optimal trajectory, more complex methods, such as Tree of Thoughts [3], Step-back Prompting [4], and Self-Refinement [5], may achieve higher performance. However, our approach is designed to improve efficiency, not performance.

CoT is the most basic and concise reasoning approach, making it an appropriate baseline for testing our method. The demonstrated ability of our method to reduce reasoning tokens without compromising performance in this minimal setting suggests that similar efficiency gains could extend to more complex methods, which typically involve significantly more reasoning steps.

We hope this explanation clarifies our choice of CoT. The paper has been updated to reflect this motivation (main text, lines 217–220).

---

References

[1] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models, 2023. URL: https://arxiv.org/abs/2210.03629.

[2] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023. URL: https://arxiv.org/abs/2302.04761.

[3] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models, 2023. URL: https://arxiv.org/abs/2305.10601.

[4] Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. Take a Step Back: Evoking reasoning via abstraction in large language models, 2024. URL: https://arxiv.org/abs/2310.06117.

[5] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative refinement with self-feedback, 2023. URL: https://arxiv.org/abs/2303.17651.

Alternative Model Architectures

Regarding the use of alternative model architectures, such as SSMs, we agree this would be an interesting direction for future research. In this work, however, we prioritized demonstrating generalization across widely used models and benchmarks.

Robustness Testing on Noisy Inputs

For robustness testing on noisy inputs, we hypothesize that performance would align with the baselines. Instead, we focused on testing in an out-of-domain setting, which we considered a higher priority. Unfortunately, due to limited computational resources, we were unable to test our method on excessively large models.

---

Q1. Likelihood of Not Generating the Correct Answer

We do not have specific statistics but estimate it to be similar to the model's performance in the few-shot prompting setting. Rationalization intuitively increases the sample size, offering more options when selecting the argmax of the VoC reward. We conducted a limited ablation experiment on the phi-2 model by removing the rationalization step. This resulted in a reduction of 12 tokens on average in the validation set but had no effect on performance. We have updated the paper to clarify the motivation behind using rationalization (main text, lines 137–139).

---

Q2. Input Tokens in Failed Cases

Yes, the input tokens of failed cases are included in the count of input and output tokens.

---

Q3. Time Complexity Reduction

The time complexity reduction is proportional to the reduction in tokens, as each token requires the same amount of time on the fixed-depth transformer architecture we used. However, this may vary with different architectures.

---

Q4. Alternative Reward Functions

This work focuses on exploring the potential of VoC when applied to LLMs. Investigating alternative reward functions is indeed an interesting direction for future research.

A1. Rationalization and STaR Details

Thank you for suggesting additional details on STaR. We agree that our original presentation was insufficient and have added the relevant details to the paper (main text, lines 147–149).

For rationalization, we followed the approach described in the STaR paper. Intuitively, rationalization increases the sample size, providing more options when selecting the argmax of the VoC reward. For example:

• Without rationalization: If there are 4 samples (1 correct, 3 incorrect), the correct sample might be chosen even if it has a suboptimal VoC due to its length.
• With rationalization: Rationalization can increase the size of the sample set, potentially yielding 3 correct samples and 1 incorrect one. This provides more choices and allows us to select the best trade-off between utility and cost.

However, we did not evaluate the impact of rationalization on the metrics used to assess the models prior to submission. Since receiving this feedback, we conducted a limited experiment on the phi-2 model, ablating the rationalization step. We found that rationalization provided a marginal reduction in length (12 tokens on average in the validation set) without affecting performance. The paper has been updated to include the motivation behind using rationalization (main text, lines 137–139).

---

A2. Cost Function and Gamma Parameter

We found that using a linear cost function resulted in slightly lower performance during early iterations on the validation set. A logarithmic cost function aligned utility and cost more effectively, especially given token-length variance. However, your comment prompted us to conduct further testing, which revealed that the performance difference was negligible.

Gamma, treated as a hyperparameter, was tested with values 1, 0.1, and 0.01 on the phi-2 model. Gamma = 0.1 provided the best balance, aligning cost and reward distributions effectively. Overall, the method is robust to variations in gamma, provided that we filter for correct answers (within 0.5 standard deviation in accuracy and five tokens in length).

We have updated the paper (main text, lines 118–120 and 250–252) to reflect these choices and findings.

---

A3. Instruction-Tuned Model Baseline

Thank you for this suggestion. While we did not test this baseline prior to submission, we agree it is an interesting approach (albeit applicable only to instruction-tuned models). We conducted an experiment using Llama-3-8B-Instruct. Unfortunately, this method did not effectively adapt response lengths to match task complexity in our setting. However, we acknowledge that it is a noteworthy baseline, particularly as it may perform better with larger models, or with extensive prompt engineering. We have updated the paper to include this baseline and a corresponding discussion (Appendix D).

---

A4. Exploring Other Algorithms

We agree that this is an interesting open question! Rational metareasoning can be readily adapted to other algorithms (for instance, DPO could involve training on samples with the highest and lowest rewards, respectively). In this work, we prioritized demonstrating generalization across models and benchmarks. Consequently, we focused on a single algorithm (expert iteration) because it has been found to be the most sample-efficient for reasoning tasks.

Unfortunately, due to time and computational constraints, we were unable to experiment with alternative algorithms during the rebuttal phase. However, we have updated the discussion (main text, lines 459–463) to clarify our rationale for choosing expert iteration and to outline directions for future work.

---

B1. Visualization of Performance-Efficiency Tradeoff

We’re sorry to hear that these figures were not helpful. We plotted accuracy against the logarithm of the weighted sum of input and output tokens (weighted 3:1, as is common in API pricing). We have included the plots in Appendix B of the paper. We hope this provides a clearer visualization of the performance-efficiency tradeoff.

---

B2. Out-of-Domain Few-Shot Evaluation

Thank you for this suggestion! Since this section focuses on testing out-of-domain generalization, we evaluated results using out-of-domain few-shot examples, specifically with 5 shots from the datasets used for training. We found that performance remained comparable, while our method used fewer output tokens. We have updated the paper (Appendix C) to include this information.

---

Q1. Expert Iteration Steps

We perform up to 8 expert iteration steps. However, we observe that performance generally plateaus after iterations 4 or 5. Unfortunately, the intermediate checkpoints and their associated details were not preserved due to their large size. We plan to rerun the training of Phi-2 with STaR and MR and will update the paper with the corresponding training details once they are available.

Figure in Main Paper

Thank you for your feedback. The choice to skew the y-axis was intentional for two reasons: first, we are optimizing for the logarithm of sequence length; second, we found that this representation enhances readability. Regarding including this figure in the main paper, we agree it would be a valuable addition. Being constrained with page limit, we decided to replace the main paper figure with this new one, and moved the old main figure to the appendix.

---

Length Adaptation for Instruction-Tuned Models

Thank you for pointing this out. We realized that we had omitted the table with the length reduction ratio and have now updated the Appendix to include it. As shown, the instruction-tuned version generates longer reasoning chains compared to the other methods, and is is less adaptive to task difficulty (length reduction ratio). While the order of the datasets is the same, we feel that the ratio between datasets and hard/easy questions is more indicative of the adaptive nature of the model.

---

Please let us know if you have any further questions, we remain at your disposal.