Optimizing Test-Time Compute via Meta Reinforcement Finetuning

Thank you for your positive review of the paper! We will add more experimental details to the Appendix – in particular, regarding the experimental setting for both MRT (STaR), MRT (RL), and analysis on R1, and we will also cite the o1 and o3 blog post in the paper. We will address your concerns below, and would appreciate it if you would be willing to raise your score if you find your concerns addressed. We are happy to answer any remaining questions.

In Fig. 2 of the analyses of Deepseek-R1, the authors mix pass@k, majority@p and the episodes together, which makes this plot rather confusing at first glance. Also, I do not know how the authors make the plot for accuracy versus log-tokens. I would assume that different problems would have very different number of tokens per episode/pass. Those details are not clear even after reading the relevant appendices.

It's correct that different problems would have very different numbers of tokens per episode and different numbers of episodes. Therefore, we first split the problems into different groups based on the number of episodes. For all solutions with a certain number of episodes (e.g., 6-10, 26-30), we average the number of tokens per episode and accuracy across those solutions. In other words, to plot the blue line, we fix j, find the average number of tokens up to j episodes, and then average maj@k given j episodes in the thought block across different solutions. Please let us know if this is clear, and we will add this discussion to the paper as well.

The authors suggest that one can either use another LLM or the same LLM as the policy model to estimate the information gain. However, which LLM is used and how it is used to estimate the information gain is not clear from the paper for the experiments conducted.

Thanks for the question! We will use the one extra page in the final version to also include more details in the main paper and definitely add the rest to the appendix.

In regards to your question above, we note that there are multiple ways to estimate the information gain or rewards. In this paper, we use Monte Carlo rollouts with the base model as the estimator. Specifically, to compute the reward of a given prefix, we sample multiple rollouts to complete this prefix with the base model. We then use the success rate among these rollouts to represent the reward of the prefix. The reason we mention that "one can either use another LLM or the same LLM as the policy model to estimate the information gain" is because Monte Carlo rollouts are not the only method to estimate rewards. One can also train a progress reward model to assess how effective a prefix is for a given problem, or use an LLM as a judge to provide the estimation.

It is a little confusing for RL to run for multiple iterations since the authors use an online-RL framework. I guess the authors mean running with additional randomly sampled subset of the dataset. I suggest the authors clarifying this confusion.

Yes, and multiple iterations are also useful to update the reference model to prevent it from being too different / off-policy.

Would it be helpful to steer the model to continue the episodes with MRT? Would it continuously improve the performance as reported in the s1 paper or would it saturate instead?

New results in more open-ended settings:

To extend MRT to more episodes, we ran MRT directly on top of distilled variants of DeepSeek-R1. We refer to this setting as the "open-ended" setting, since the episodes now – much like our analysis in Section 4 – are not constrained to follow a specific format. We defer the training details to the supplement. We evaluated MRT and GRPO on AIME 2024/25, and AMC 2023 datasets (20 samples per problem) from different base models in the supplement. Our models fine-tuned on DeepScaleR-1.5B-Preview achieve state-of-the-art performance for their size: 47.2% success on AIME 2024 and 39.7% on AIME 2025. Across multiple base models, MRT's relative performance improvement is about 2–3x compared to outcome-reward RL (GRPO).

We also measure the cumulative regret metric for both MRT and other baselines (STaR, GRPO, base model). To do so, we first run the R1 analysis on our own models and take the optimal policy $\pi^*$ to be the one that achieves perfect accuracy in one episode. Intuitively, we are computing the red area (denoting cumulative regret) normalized at different cutoff points/token budgets. As shown in the supplement, models trained with MRT have the lowest regret. Moreover, in extrapolation regions where we steer our trained model to think more with "Wait" (similar to S1), the performance of MRT doesn't plateau but continues to improve.

Thank you for your feedback! To address your concerns, we clarify the notion of token efficiency and interpret the results of MRT in comparison with length-penalty and baseline GRPO, add new results running MRT on top of DeepSeek-R1 models to extend beyond three-episode setting, and add numerous visualizations of regret with and without MRT to justify the efficacy of the regret metric. Please let us know if your concerns are addressed, and if so, we would be grateful if you could raise your score.

[New expt.] Results for multiple episodes

To extend MRT to more episodes, we ran MRT directly on top of distilled variants of DeepSeek-R1. This “open-ended setting” removes the constraints on the episodes to follow a specific format, making them freeform similar to our analysis in Section 4. We defer the training details to the supplement.

Results: We evaluated MRT and GRPO on AIME 2024/25, and AMC 2023 datasets from different base models in supplement. Our models fine-tuned on DeepScaleR-1.5B-Preview achieve state-of-the-art performance for their size: 47.2% success on AIME 2024 and 39.7% on AIME 2025. Across multiple base models, MRT's relative performance improvement is about 2–3x compared to outcome-reward RL (GRPO).

Comparisons to length penalty: We also run an additional comparison on top of the DeepScaleR-1.5B model, where we apply an explicit length penalty but fine-tune it with GRPO. In agreement with findings in the submission, we find that incorporating a length penalty results in worse pass@1 accuracy.

In the supplement, we also measure the cumulative regret of MRT, GRPO/STaR, and base models. To do so, we choose $\pi^*$ to be the one that achieves perfect accuracy within one episode. Intuitively, we are computing the red area in the supplement normalized for different token budgets. MRT attains smallest regret, even when extrapolating beyond training budget (similar to s1).

Token efficiency and length penalty

To clarify, token efficiency refers to maximum performance at minimal tokens. The tradeoff of using length penalty is that although it reduces the number of tokens substantially, the performance plateaus beyond a point (e.g., see Figure 10, when tokens > 8000). MRT surpasses it when we allow a larger number of tokens, and the model's performance continues to increase. In addition, note that in the above experiments, MRT even outperforms length penalty in terms of pass@1 performance.

Computation cost of MRT

In supplement, we compute the total FLOPs used by MRT and STaR/GRPO for sampling and training. For NuminaMATH (20,000 problems) with Llama-3.1-8B-Instruct, STaR required 2.62×10²⁰ FLOPs while MRT needed 2.64×10²⁰ FLOPs (1.01× more) while attaining 1.7× fewer inference tokens. Similarly, GRPO used 6.34×10¹⁹ FLOPs versus MRT's 6.86×10¹⁹ FLOPs (1.08× more) but MRT used 1.6× fewer inference tokens. MRT uses <8% more computation while requiring 60% fewer tokens during inference to achieve the same performance.

Understanding of Figure 4

Yes, your understanding is correct. To add to your point, this result is saying that sequential sampling does not realize the full potential of tokens spent, as the naive strategy of parallel sampling (maj@k) outperforms sequential thinking. When in principle, sequential thinking should easily express maj@k with the same number of tokens.

How is $c_k$ generated?

Sorry for the confusion, it should be $c_j$, which consists of the prefix (first j episodes) sampled from the previous checkpoint. $k$ is defined as the number of episodes of the rollout. To avoid confusion, we provide a more detailed explanation of update in the supplement.

Sec 6.2, warmstart dataset construction and other construction schema.

In Figure 5, for each node from 0-13, we compute the information gain by using Definition 5.1, and select the one that maximizes the information gain. And yes, we can construct in other formats as suggested. Motivated by this, we extend the method to an open-ended setting.

Besides, I checked Appendix C, The Omni-MATH subset (40 problems) and AIME (30 problems) are small and may not reflect broader generalization.

As shown in the supplement, we added more results by evaluating Deepseek-R1 on AIME problems from 2015-2024 (293 problems in total). Our findings from the submission still hold, where the performance of off-the-shelf models does not improve as the thinking budget increases, and simple early termination with parallel sampling outperforms, but the model couldn't discover such solution.

Thank you for your feedback! We've added new results measuring regret for MRT and baselines, showing MRT attains smaller regret and improved performance in more general settings. We will also update the paper with more clarifications and the definition of STaR. If your concerns are addressed, we'd be grateful if you could raise your score.

Significance of maj@k results in Figure 2 and measuring regret

Motivation for measuring maj@k. Our goal with maj@k is not to compute regret, but to demonstrate a simple baseline using partial thinking traces that outperforms sequential thinking with more episodes. If sequential thinking worked effectively, it should have outperformed this basic maj@k approach (blue >> green).

Regret measurements: We initially didn't measure regret because, similar to RL, this requires comparison against an optimal policy π* that's unknown beforehand. Performance over more episodes served as our proxy. However, we can consider the optimal comparator π* to be the one that achieves perfect accuracy in one episode. Here, the regret is the area between the blue / green / orange lines in Figure 2 and the horizontal line at y = 1. For each point on the R1 scaling curve, we can plot corresponding regret, normalized by episodes or tokens used. As shown in the supplement, the regret of direct and [maj@k]_j are lower compared to [maj@1]_j on solutions with more episodes, indicating that sequential episodes do not use tokens efficiently compared to majority voting from an earlier episode or the direct model.

Performance gain over baselines from MRT

We want to highlight that the metric is not just pass@1 performance, but also token efficiency. We redrew the plots in the supplement by omitting iter1 and highlighting token efficiency. With linearized evaluation, MRT achieves the same performance as GRPO with 1.6x fewer tokens.

New results in more open-ended settings: We extended MRT to distilled variants of DeepSeek-R1, where episodes aren't constrained to follow a specific format. We evaluated MRT and GRPO on AIME 2024/25, and AMC 2023 datasets (20 samples per problem) from different base models in supplement. Our models fine-tuned on DeepScaleR-1.5B-Preview achieve state-of-the-art performance for their size: 47.2% success on AIME 2024 and 39.7% on AIME 2025. Across multiple base models, MRT's relative performance improvement is about 2–3x compared to outcome-reward RL (GRPO).

The 95% confidence interval for our method fine-tuned from DeepScaleR-1.5B-Preview on AIME2024 is ±0.13%. Given the stable estimation from 20 samples, we prioritized evaluating on more problems and omit the CI for other models.

Computational overhead in MRT training.

As in supplement, we compute the total FLOPs used by MRT and STaR/GRPO for sampling and training. For NuminaMATH (20,000 problems) with Llama-3.1-8B-Instruct, STaR required 2.62×10²⁰ FLOPs while MRT needed 2.64×10²⁰ FLOPs (1.01× more) while attaining 1.7× fewer inference tokens. Similarly, GRPO used 6.34×10¹⁹ FLOPs versus MRT's 6.86×10¹⁹ FLOPs (1.08× more) but MRT used 1.6× fewer inference tokens. MRT uses <8% more computation while achieving same performance with 60% fewer tokens during inference.

Equation 2 clarification

Thanks for pointing this out. It should be $c_{j}$​, which consists of the prefix (first j episodes) sampled from the previous checkpoint $\pi_\text{old}$. We provide a more detailed explanation of updated equation 2 in the supplement.

clarify $\pi_\text{old}$ and $\mu$

Policy $\mu$ can be any LLM (e.g., an "-instruct" model which is told to utilize episodes so far to guess the best answer). For implementation simplicity we use the success rate of Monte-Carlo rollouts on $\pi_\text{old}$ to represent $\mu$. We will add this clarification to the paper.

proof of claim in Line 26

The theoretical argument behind this line is from Section 3 of the TRPO paper, which shows optimizing policy under the state distribution induced by the old policy with a KL-constraint on actions results in monotonic performance improvement.

optimality of augmented reward

As shown in the supplement, the optimal policy for the augmented reward (Right, w/ information gain) will also attain maximal reward under the original reward (Left, w/o information gain). However, not every policy that achieves the highest original reward exhibits maximal information gain. To see this intuitively, note that Equation (1) only guarantees correctness of the outcome, whereas maximal information additionally does so quickly.