Response to Reviewer N2ms

Thank you very much for your detailed and encouraging review. We greatly appreciate your recognition of both the theoretical and empirical contributions of UFT, as well as your thoughtful questions. Below, we address each of your points.

Model scale and generalization to larger LLMs

We fully agree that demonstrating UFT on larger models would further strengthen our claims. However, due to resource constraints (our current experiments already cost approximately $10,000), we were unable to scale up the experiments further.

That said, our current results reveal key trends that may generalize. For instance, on Qwen2.5-1.5B:

We observe that RFT performs well on Countdown and MATH(3,4,5) but fails on Logic, where the model does not get enough knowledge through pretraining. UFT, by contrast, significantly outperforms RFT in this setting. We expect similar benefits on larger models when fine-tuning for extremely complex reasoning tasks or tasks requiring specific domain knowledge.

Use of model-generated hints

UFT is not limited to human-annotated hints. It only requires:

• A reasoning trace. It can either be human annotations or chain-of-thoughts generated by other models.
• A reward signal (e.g., a rule-based reward model).

The “hints” used in our work are not curated step-by-step annotations, but simply prefixes of existing solutions contained in the dataset. For each solution, we will divide it by sentence, and the hint length is the number of sentences. Here is an example from the MATH dataset:

[
"For the piecewise function to be continuous, the cases must \"meet\" at $2$ and $-2$. ",
"For example, $ax+3$ and $x-5$ must be equal when $x=2$. ",
"This implies $a(2)+3=2-5$, which we solve to get $2a=-6 \\Rightarrow a=-3$. ",
"Similarly, $x-5$ and $2x-b$ must be equal when $x=-2$. ",
"Substituting, we get $-2-5=2(-2)-b$, which implies $b=3$. ",
"So $a+b=-3+3=\\boxed{0}$."
]

Therefore, this approach is easily extendable to LLM-generated traces or even imperfect intermediate outputs.

In fact, prior work [3] has shown that LLM-generated explanations often outperform human-written ones for supervision. Recent work s1 [4] also utilized LLM-generated responses for fine-tuning. Hence, we expect UFT to perform even better when paired with high-quality model-generated responses.

Description of the algorithm

Thank you for highlighting this. We will update the main text to briefly explain Algorithm 1 at a high level, describing how hints are sampled and how the hybrid loss is scheduled during training.

Ablation study on cosine annealing and supervised loss

Figure 5 already includes an ablation where RFT is combined with cosine-annealed hints but without the supervised log-likelihood loss. The drop in performance in this setting demonstrates that the supervised loss on hints plays a key role in UFT’s effectiveness.

Ablation on the hint schedule

In Figure 5, we can see that $R^3$ (the hint length is sampled from a uniform distribution throughout the training) performs worse than RFT with cosine-annealed hints. Both of them adopt the standard RFT objective (3.1) without the log-likelihood on hints. This result also justifies the necessity of gradually annealing the hint length.

On the other hand, the specific choice of hint length distribution is not important. To assess the sensitivity of UFT to this choice, we also experimented with a two-point distribution: for any expected hint length $p\cdot L\in [n, n+1)$, we set $l=n$ with probability $n+1-p\cdot L$ and $l=n+1$ with probability $p\cdot L-n$. $p$ is still determined by the cosine-annealing scheduler. Using this scheme, the final accuracy of Qwen2.5-0.5B (Countdown) is:

• Two-point: 48.63%
• Binomial distribution: 50.39%

This suggests that the choice of distribution's impact on overall performance is relatively small.

Clarifications on Theorem 4.3

For any rule-based reward model, the assumption holds because there is always a gap between the reward for the correct answer and the incorrect answer. For instance, Deepseek R1 [1] and DAPO [2].

Regarding “partial rewards,” if this refers to assigning intermediate rewards (e.g., for correct reasoning steps), we currently do not use partial rewards in our setup. All rewards are terminal (binary correctness). However, extending UFT to settings with partial rewards is a promising direction for future work.

Avoiding overreliance on hints

Indeed, overreliance on hints is a real concern, and the cosine annealing strategy is specifically designed to mitigate this. As training progresses, the expected hint length smoothly decays, ensuring that the model learns to reason without external help, aligning training with the final evaluation distribution (no hints at inference time).

Computation time

UFT is faster than RFT per training step. Here is the training time per step averaged across tasks:

This improvement stems from shorter generation lengths during the hint phase. We also report the rollout time per step below, showing clear gains:

While UFT introduces a hybrid loss, this overhead is offset by a more efficient trajectory rollout procedure.

References

[1] Guo D, Yang D, Zhang H, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning[J]. arXiv preprint arXiv:2501.12948, 2025.

[2] Yu Q, Zhang Z, Zhu R, et al. Dapo: An open-source llm reinforcement learning system at scale[J]. arXiv preprint arXiv:2503.14476, 2025.

[3] Ren X, Wu B, Liu L. I learn better if you speak my language: Understanding the superior performance of fine-tuning large language models with LLM-generated responses[J]. arXiv preprint arXiv:2402.11192, 2024.

[4] Muennighoff N, Yang Z, Shi W, et al. s1: Simple test-time scaling[J]. arXiv preprint arXiv:2501.19393, 2025.

[5] Agarwal A, Kakade S M, Lee J D, et al. On the theory of policy gradient methods: Optimality, approximation, and distribution shift[J]. Journal of Machine Learning Research, 2021, 22(98): 1-76.

[6] Jin C, Liu Q, Wang Y, et al. V-Learning--A Simple, Efficient, Decentralized Algorithm for Multiagent RL[J]. arXiv preprint arXiv:2110.14555, 2021.

Response to Reviewer BALi

Thank you for your thoughtful and detailed feedback. We sincerely appreciate your insights, and we address your comments point by point below.

Ablation study on distributions

We chose the binomial distribution for hint length sampling due to its simplicity and desirable properties: (i) it is discrete with support over integers (hint lengths), and (ii) the expected hint length can be precisely controlled, which is crucial for implementing the cosine-annealing schedule.

To assess the sensitivity of UFT to this choice, we also experimented with a two-point distribution: for any expected hint length $p\cdot L\in [n, n+1)$, we set $l=n$ with probability $n+1-p\cdot L$ and $l=n+1$ with probability $p\cdot L-n$. $p$ is still determined by the cosine-annealing scheduler. Using this scheme, the final accuracy of Qwen2.5-0.5B (Countdown) is:

• Two-point: 48.63%
• Binomial distribution: 50.39%

This suggests that the choice of distribution's impact on overall performance is relatively small.

KL-Divergence estimation in (3.2)

As we discussed in Line 180-186, $KL(\pi^{\*}|| \pi)$ (red term) is equivalent to the log-likelihood on the hint (the blue term in Eq 3.3) by definition of the KL-divergence.

Specifically, $KL(\pi^{\*}|| \pi)=\sum_a \pi^{\*}(a|s)\log \frac{\pi^{\*}(a|s)}{\pi(a|s)}=\sum_a \pi^{\*}(a|s)\log \pi^{\*}(a|s) + \sum_a \pi^{\*}(a|s)\log \frac{1}{\pi(a|s)}$.

Since only $\sum_a \pi^{\*}(a|s)\log \frac{1}{\pi(a|s)}$ is related to our policy $\pi$, minimizing the KL is equivalent to minimizing $\sum_a \pi^{\*}(a|s)\log \frac{1}{\pi(a|s)}$. Lastly, the log-likelihood on the hint, $\log \frac{1}{\pi(a|s)}$ with $a\sim \pi^{\*}(\cdot|s)$, is an unbiased estimator of $\sum_a \pi^{\*}(a|s)\log \frac{1}{\pi(a|s)}$.

Performance gain

The objective function of UFT is equivalent to that of RFT (GRPO) when $\sum KL(\pi^{\*}||\pi)$ is removed (c.f. Equation (3.1) and (3.2)). The only difference after removing $\sum KL(\pi^{\*}||\pi)$ is that UFT starts reasoning from the hint, while RFT starts from scratch.

In Figure 5, we did an ablation study on using hint but with/without the modified objective function (the additional log-likelihood on hint, equivalent to the KL as stated above). It shows that the additional log-likelihood contributes to the final performance gain and helps the model to learn new knowledge.

Implication of theory.

Modeling LLM text generation as a uniform tree (each node representing the generation of the next token) is a standard and widely used abstraction in prior work [1][2].

For LLM text generation, if we take each branch as the next token, then the whole text generation forms a path in the tree. Since the token table is fixed for LLMs, the branching factor $B$ is the same for all nodes. In other words, the search tree is uniform.

Finally, our theoretical results serve to justify that integrating the supervised signal during RL helps improve sample complexity, as noted by Reviewer N2ms. While our analysis assumes uniform trees, the results can extend to imbalanced trees with minor adjustments. For instance, the lower bound in Theorem 4.2 can be understood more generally as requiring exploration of at least $\frac{\text{Number of Leaves}}{4}$ to achieve 50% accuracy.

$\beta$ value

We acknowledge the reviewer’s request for a more detailed explanation regarding $\beta$. The formal version of Theorem 4.3, provided in Appendix E, shows that $\beta$ should satisfy: $\beta\leq O(\frac{\Delta}{(H+1)^2\log B})$.

This is a conservative upper bound that holds under worst-case for arbitrary reference policies $\pi^{ref}$. A tighter (and more practical) condition depends on the KL divergence between the and the reference policy, which is $O(\frac{\Delta}{(H+1) KL(\pi||\pi^{ref})})$.

In practice, due to pretraining, $\pi^{ref}$ might be close to $\pi^{\*}$, making it feasible to choose a larger $\beta$. In our experiments, we choose $\beta=0.001$, which satisfies the bound when $\pi^{ref}$ is not far from $\pi^{\*}$.

References

[1] Xie Y, Goyal A, Zheng W, et al. Monte carlo tree search boosts reasoning via iterative preference learning[J]. arXiv preprint arXiv:2405.00451, 2024.

[2] Chen G, Liao M, Li C, et al. Alphamath almost zero: process supervision without process[J]. Advances in Neural Information Processing Systems, 2024, 37: 27689-27724.

Response to Reviewer MekW

Thank you for your comprehensive and thoughtful review. We deeply appreciate your questions and suggestions. Below, we address each of your points in detail.

When to use UFT

UFT’s main advantage lies in its robustness across tasks and model capacities. While SFT, RFT, or SFT-RFT may perform slightly better in specific scenarios, it is often difficult to determine the optimal fine-tuning strategy beforehand. UFT offers a strong, consistent alternative that adapts across regimes.

As demonstrated in the table below (Qwen2.5-1.5B), RFT surpasses SFT-RFT on Countdown and MATH(3,4,5), but fails entirely on Logic. UFT, by contrast, maintains competitive or superior performance across all tasks:

This illustrates that while task complexity and model capacity are important, their interaction is nontrivial. UFT offers a reliable choice when the best training scheme is unknown.

Computational cost comparison

We evaluated the average training step time across models. Surprisingly, UFT is faster than RFT, due to shorter outputs in the early training phase (thanks to hints):

This improvement stems from shorter generation lengths during the hint phase. We also report the rollout time per step below, showing clear gains:

Pure SFT remains the most efficient (about 6-9x faster than RFT/UFT), as it does not involve trajectory sampling or reward computation.

Hint length ablation

We conducted a study on Qwen2.5-0.5B (MATH(3,4,5)), testing dividing the solution into $L=4/5/6$ pieces uniformly and sampling hint length among $\{0,1,\dots, L\}$. We choose MATH(3,4,5) instead of Countdown because the solution length of MATH(3,4,5) is relatively longer.

• Hint length $L=4$: 27.44%
• Hint length $L=5$: 27.15%
• Hint length $L=6$: 25.20%

This suggests that UFT is relatively insensitive to the total number of pieces we divided the solution into.

$\beta$ ablation

We evaluated different $\beta$ values on Qwen2.5-0.5B (Countdown):

• $\beta=0.0005$: 46.09%
• $\beta=0.001$: 50.39%
• $\beta=0.002$: 59.95%

A higher $\beta$ amplifies the impact of the supervised log-likelihood term, helping the model learn more from hints. For all main experiments, we adopt $\beta$ = 0.001, the default in VERL. For larger models, our preliminary results show that varying $\beta$ yields minimal performance changes, likely due to their stronger pretrained priors. Here are the results of Qwen2.5-3B (Countdown).

• $\beta=0.0005$: 77.93%
• $\beta=0.001$: 75.59%
• $\beta=0.002$: 78.22%

Length of hint phase ablation

We tested different durations for the hint phase on Qwen2.5-0.5B (Countdown):

• $T_{hint}=200$: 52.15%
• $T_{hint}=300$: 50.39%
• $T_{hint}=400$: 50.00%

These small variations confirm that UFT is robust to the hint phase length.

Data allocation strategy

We appreciate your insightful suggestion. We agree that placing harder examples in the supervised phase and easier ones in the RL phase may further enhance performance. Moreover, adaptively adjusting hint length based on the hardness of each data point is also promising.

However, the current paper’s goal is to demonstrate the value of integrating supervising signals to reinforcement learning, and we leave adaptive data allocation to future work.

Clarification of theory

• Tree Structure (Lines 82–101): These definitions are necessary to formally define the trajectory sampling and expected return in Section 3.
• Optimal Step Count: Since GRPO is a convergent algorithm, the more steps we train, the higher the final accuracy is. The accurate value of $T$ of achieving $50\%$ accuracy is given in Lemma E.1.
• In Algorithm 2, using reward as the Q-value follows the practice in GRPO [1], where advantage is computed as $\tilde A(s,a)=\frac{r(s,a)-mean({\bf r})}{std({\bf r})}$. Our implementation omits normalization for simplicity, in line with GRPO variants [2].

References

[1] Shao Z, Wang P, Zhu Q, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models[J]. arXiv preprint arXiv:2402.03300, 2024.

[2] Liu Z, Chen C, Li W, et al. Understanding r1-zero-like training: A critical perspective[J]. arXiv preprint arXiv:2503.20783, 2025.

Thank you for your patient and detailed response, which has addressed most of my concerns. The computational cost analysis showing UFT's efficiency advantage over RFT is particularly insightful, and the ablation studies demonstrate the method's robustness across different hyperparameter settings. I look forward to seeing UFT's performance on larger models. In recognition of the authors' thorough clarifications and the value of their contribution, I am raising my score.

Response to Reviewer KErd

We sincerely appreciate your thoughtful review and constructive feedback. Below, we address your primary concern regarding the generalization of UFT to larger models and large-scale experiments.

UFT's generalization to larger models

Even though we cannot afford training a 70B model, we believe UFT will be helpful for 70B models on tasks out of its capacity (unseen tasks during pretraining / extremely hard reasoning problems).

While we are currently unable to run experiments on 70B-scale models due to computational constraints, we believe that UFT will be beneficial even for such models, especially on tasks that exceed their pretraining capabilities. For instance, some tasks involving specific domain knowledge or extremely complex reasoning problems.

The following empirical results may hint at this. For instance, on the logic puzzle, which requires more abstract reasoning, Qwen2.5-1.5B performs poorly with RFT alone (0.00%) but achieves 23.00% with UFT:

In conclusion, although Qwen2.5-1.5B has enough capacity for Countdown and MATH(3,4,5), it is insufficient in solving the logic puzzle and UFT outperforms RFT by a large margin. Therefore, it is likely that for larger models, UFT will be helpful when solving extremely hard tasks.

This stark difference in Logic suggests that even when a model appears to have “enough capacity” (as Qwen2.5-1.5B does for Countdown and MATH), it can still benefit significantly from UFT when the task challenges its reasoning ability. Hence, for much larger models facing extremely complex tasks, we anticipate that UFT will continue to provide meaningful advantages.

Experiments on large models

We fully agree that broader experiments would offer more insight. However, as noted in Appendix B.2, our current experiments already incurred a cost of approximately $10,000, and scaling to 70B models would be prohibitively expensive for us at this stage.

That said, we emphasize that absolute model size is not the sole driver of UFT's benefits. What matters more is the relative complexity between the model's capacity and the task. Our findings show that UFT outperforms RFT in settings relatively hard for the model (such as the logic puzzle for Qwen2.5-1.5B), which is a likely scenario even for large models on particularly difficult or unfamiliar problems.

I definitely appreciate the significant cost of experimentation. The authors have partially addressed my concern with their argument whereby Qwen2.5-1.5B has sufficient capacity for countdown and MATH but fails on Logic when RFT is used. I buy that. However, instead of having to rely on this argument without further experimental evidence, it could be better/easier for the authors to simply limit the scope and positioning of this paper to SLM models. There's no reason to make an argument for larger models if you can't validate it with experiments. You can simply argue that SLMs are important for edge compute cases, and your proposed UFT outperforms SOTA on SLMs. I would +1 the score to Accept given such an argument. But it is much harder to agree to your claim that UFT is more generally applicable (plainly because generalizability to LLMs is not substantiated, regardless of the reason). My score was already positive to begin with, so I'm keeping it the same. I would assign a higher score either if you (a) limited the scope to SLMs or if you (b) provided experimental validation of claims of UFT generality for LLMs. I hope my rationale makes sense. It's good work -- as reflected in my extensive list of strengths. Thank you.