LILO: Learning to Reason at the Frontier of Learnability

We thank the reviewer for their time and valuable feedback that has helped us improve this work. We are encouraged by the positive comments and score. We address the suggestion of an additional ablation below and hope this resolves any of the outstanding concerns.

Tuning N_learnability:

This is a great point, and different models and hyperparameters may have different optimal N_learnability. Sadly, we don’t have the compute resources to complete a full training run for a variety of choices of N_learnabilty. However, a simple approach to finding N_learnability at the start of training could be to sample a batch of questions, and plot how the choice of N_learnability affects the overall learnability of the selected batches. We do this for GSM8K and MATH below:

N_learnability=4 or 8 seem to be a nice balance between high learnability batches and low sampling costs.

We hope this adequately addresses the issue highlighted and invite any more feedback or points you believe are significant.

We thank the reviewer for their time and valuable feedback that has helped us improve this work. We are encouraged by the positive comments around the improvement that prioritising learnability brings to training LLMs to reason with reinforcement learning. We address the weaknesses and questions one-by-one below and hope this resolves any of the outstanding concerns and that the reviewer will consider revising their rating.

Originality:

• There is a rich history of curricula for traditional RL settings, and as you note, Learnability has been proposed before, such as in [16].
• We emphasise that the relevance of [16] on reasoning with LLMs is extremely small - there is no clear method for computing and prioritising learnability, there is no suggestion that this should work beyond the REINFORCE algorithm, and is it considered in traditional RL tasks.
• Beyond [16], it is non-trivial to decide which curricula methods from traditional RL are suited for the LLM reasoning setting:for example Havrilla tried some and these failed. This is true for other areas of RL too, for example value networks, which are popular in deep RL, have been shown to not transfer to the LLM setting.
• We identified that Learnability based curricula are particularly well suited for the LLM setting because it is non-stochastic, and uses sparse, binary rewards. Even once we identified this, there was still work to do to adapt it to the LLM setting. For example, in Appendix E Figure 11 we detail how the frequency with which you recalculate learnable questions, which is typically done every 10-50 timesteps in the traditional setting, is catastrophic in the LLM setting.
• Crucially, this results in LILO being the first to show that a curriculum method can boost overall performance on LLM reasoning tasks. Given the popularity and resource consumption of this setting this is highly impactful and useful to the community.

Naming of learnability:

• We completely agree that “learnability” is a loaded phrase, and ideally a different term, such as "reward variance" would be used. However there is a large body of previous work (including [16,17,18]) that terms this ‘learnability’ and as such, we must use this term. There are proper mechanisms to attempt to change this, such as community meetups, blog posts and position papers, but reviews of individual papers are not a suitable place.
• We emphasise that LILO does NOT prioritise variability in responses or mistakes, but rather variability in rewards. It does not matter whether the model makes the same mistake every time, or a different mistake every time. If all samples are incorrect, it is a mathematical fact that existing policy gradient algorithms will not learn, no matter how many updates.
• If there are some correct and some incorrect examples, and thus some gradient signal, it is an interesting hypothesis to consider that it might be easier to learn to fix 1 mistake than 10 different ones, and we thank the reviewer for this suggestion for future work.

Impact of theorem 3.1:

• We collected over ~10,000 gradient samples from the checkpoints used during training on GSM8K and MATH and found a statistically significant correlation between learnability and $|| \nabla_\theta J(\theta)||^2$, as show in the table below. See the rebuttal to reviewer GyS9 for more information on this.

This table shows the correlation between learnability and $|| \nabla_\theta J(\theta) ||^2$.

• The theorem is the first to explain this phenomena and directly led to our method, which in turn led to significant improvement and acceleration of LLM training methods. Given the popularity of LLM reasoning, this constitutes significant downstream impact of Theorem 3.1.
• Looking at the update magnitudes produced during training, we see that selecting the highest learnability produces 1.46x larger updates than not using learnability. Theorem 3.1 explains this, but it also explains why prioritising learnability produces 1.44x larger updates than simply rejecting any questions that are always answered correctly or incorrectly, and 1.40x larger updates than prioritising the most difficult questions. Existing theory cannot explain this.

“Combining theoretical RL and LLM reasoning is strange”:

• The reviewer is correct to note that it has become commonplace in LLM papers to present methods or heuristics that work, without any attempt to explain or motivate them with theory.
• It is our position that ideally new methods are presented with insightful theory that motivates and explains them. The required theory to explain our method did not previously exist, and thus we presented it alongside the method. We are not opposed to publishing a more general theory paper in future.

Wall clock time: Whilst we did provide wall-time impact in Table 1, we agree that it would be more useful to the reader if speed-ups were reported in wall-clock time alongside speed-ups in training steps. We have therefore amended Table 2 to be the following:

We cannot upload graphs during this rebuttal, but commit to changing the x-axis on training curves, Figure 1, 2, 3 and 4 for future versions.

Learning stagnation:

• LILO does prioritise some trajectories over others, which is precisely why it speeds up training.
• Crucially however, it does not ignore them forever: If the model has no insight into a certain problem, and is mostly wrong when answering it, it will have low learnability, and training on it would have lower impact than training on a higher learnability question. If no questions with higher learnability exist, LILO will still train on this question. Methods that don't use LILO gain no benefit from training on these questions earlier on in the process. The same is true for questions that model consistently gets right.

We kindly request that the reviewer examines our further clarifications and assesses whether they satisfactorily address the concerns previously raised, potentially altering the evaluation of our work. We welcome any additional insights or comments that may be deemed pertinent.

We thank the reviewer for their time and valuable feedback that has helped us improve this work. We are encouraged by the positive comments around the both simplicity and effectiveness of our approach at boosting the performance of RLVR algorithms. We address the weaknesses and questions one-by-one below and hope this resolves any of the outstanding concerns and that the reviewer will consider revising their rating.

Originality:

• You are completely right to note the long history of studying hard sample mining in the traditional RL setting. We were inspired by many of these works especially Prioritised Level Replay [26] to try to produce a simple and effective method for the LLM RLVR domain. (For the avoidance of confusion, we note that our samples are not the hardest samples, they are the ones at the correct level of difficulty, thus providing the largest gradient update)
• Beyond hard sample mining, there is a rich history of many other types of curricula for traditional RL settings, and as you note, Learnability has been proposed before, such as in [16].
• We emphasise that the relevance of [16] on reasoning with LLMs is extremely small - there is no clear method for computing and prioritising learnability, there is no suggestion that this should work beyond the REINFORCE algorithm, and is it considered in traditional RL tasks.
• Beyond [16], it is non-trivial to decide which curricula methods from traditional RL are suited for the LLM reasoning setting:for example Havrilla et al tried some and these failed. This is true for other areas of RL too, for example value networks, which are popular in deep RL, have been shown to not transfer to the LLM setting.
• We identified that Learnability based curricula are particularly well suited for the LLM setting because it is non-stochastic, and uses sparse, binary rewards. Even once we identified this, there was still work to do to adapt it to the LLM setting. For example, in Appendix E Figure 11 we detail how the frequency with which you recalculate learnable questions, which is typically done every 10-50 timesteps in the traditional setting, is catastrophic in the LLM setting.
• Crucially, this results in LILO being the first to show that a curriculum method can boost overall performance on LLM reasoning tasks. Given the popularity and resource consumption of this setting this is highly impactful and useful to the community.

Quality (The validity of Theorem 3.1):

• We greatly appreciate the valuable remarks regarding Theorem 3.1. Prompted by the reviewer’s comments, we have revisited the proof of Theorem 3.1, and realized that the assumptions needed for the proof can be relaxed. Previously we assumed independence between $g_{t, \theta}$ and $A_{t’}$, for any $t$, $t’$. The new, less restrictive, assumptions we need, are as follows:
  • For REINFORCE and RLOO we require that $|| \sum_{t=0}^{T-1} g_{t, \theta} ||^2$ is uncorrelated with $A_t = r(s_T)^2$ in the case of REINFORCE, and uncorrelated with $A_t = ( (r(s_T) - \mathbb{E}{\pi_\theta} [ r(s_T) ])^2$ in the case of RLOO. This is a less restrictive assumption than the independence between $A_t$ and $g_{t’, \theta}$, for any $t$, $t’$, and lack of correlation is much easier to test for than independence.
  • For PPO and VinePPO we now only require that $A_t^2 = (V^{\pi_\theta}(s_{t+1}) - V^{\pi_\theta}(s_t))^2$ and $|| g_{t, \theta’} ||^2$ are uncorrelated for any $t = 0, …, T-1$. This is a less restrictive assumption due to again replacing independence with lack of correlation, and since we no longer need any assumption on the dependence of $A_t$ and $g_{t’, \theta}$, at different times $t \neq t’$.
• We in particular appreciate the excellent comment that $|| g_{t, \theta} ||^2$ or $|| \sum_{t = 0}^{T-1} g_{t, \theta} ||^2$ could depend on learnability. We completely agree that those terms could depend on the underlying question of the MDP, and thus on the learnability of that question. We have thus also tested the correlation of those terms with learnability, in addition to all the revised assumptions made in the theorem. Below are all the details and results regarding those statistical tests:
• For REINFORCE and RLOO:
  • We randomly sampled questions, answers and gradient locations from the training process, drawing ~10,000 datapoints for each of the MATH and GSM8K experiments.
  • We found a very small correlation between a) $|| g_{t, \theta} ||^2$ and $A_t$ b) $||g_{t, \theta}||^2$ and $t$, and c) $|| \sum_{t=0}^{T-1} g_{t, \theta} ||^2$ and learnability. Testing end to end, we found that these small negative correlations are dwarfed by the increase of learnability itself. As explained by our theorem, $|| \nabla_\theta J(\theta) ||^2$ has a strong statistically significant positive correlation with learnability. These are shown in the tables below.
• For PPO, VinePPO: As mentioned above, we have relaxed the assumptions needed for our proof, and now only require that:
  • $||g_{t, \theta} ||^2$ and $(V^{\pi_\theta}(s_{t+1}) - V^{\pi_\theta}(s_t))^2$ are uncorrelated, for any $t = 0., …, T-1$. We found no statistically significant correlation between those terms, when testing this.
  • $\mathbb{E}$ ||g_{t, \theta} ||^2 $$ does not depend on $t$. We found no statistically significant correlation between $|| g_{t, \theta} ||^2$ and $t$ when testing this.
  • $||g_{t, \theta} ||^2$ does not depend on learnability. We found no statistically significant correlation between $||g_{t, \theta} ||^2$ and learnability when testing this.
  • Testing end to end, we found that as explained by the final result of our theorem, $|| \nabla_\theta J( \theta) ||^2$ has a strong statistically significant correlation with learnability.
• To summarize, we have thoroughly tested empirically the validity of the assumptions in Theorem 3.1, and conclude that those are reasonable assumptions to make. We thus believe that Theorem 3.1 provides a well justified quantitative explanation of how LILO works under the hood.

This table shows the correlation between learnability and $|| \nabla_\theta J(\theta) ||^2$.

We invite the reviewer to consider our supplementary explanations and assess if they resolve the raised concerns, possibly influencing their appraisal of our work. Should there be any other aspects or feedback you find relevant, please do not hesitate to share.

We thank the reviewer for their time and valuable feedback that has helped us improve this work. We are encouraged by the positive comments around the effectiveness of our method for boosting performance of algorithms that train LLMs to reason with RL. We address the weaknesses and questions one-by-one below and hope this resolves any of the outstanding concerns and that the reviewer will consider revising their rating.

The naming of “learnability”:

• You’re completely correct that training signal variance, algorithmic utility (or even just reward variance) are great names for this quantity. However, there is a body of previous work [16,17,18] that terms this ‘learnability’ and as such, it makes sense to use this term.

"The model isn’t learning robust reasoning - it is just getting better at solving specific problems:"

• This is indeed very much a concern with LLM reasoning, RL or even just machine learning in general. It doesn’t feel like a particularly fair critique of this paper specifically, which does not claim to make improvements in this area.
• Methods for improving the generalisation of LLM reasoning are definitely needed (we also suggest this in Section 7), and we thank the author for acknowledging this.
• This is a method to speed up training, and boost the overall test accuracy achieved. Given the popularity and resource consumption of LLM reasoning, this is an equally worthwhile and impactful direction orthogonal to improving generalisation.
• As with the term “learnability”, one can take issue with established phrases such as “learning to reason”. Given the prevalence of “learning to reason” RLVR papers, it is reasonable to use this term to avoid needlessly adding new language to this topic, but we also recognise its limitations.

What happens if GSM8K has incorrect Q,A pairs?

• As with any most machine learning methods, if the data is wrong or low quality, the model training would likely suffer.
• There is an argument that LILO could help to reduce the negative effects of mislabelled data. For example, if the answer is wrong and the model consistently answers correctly but is deemed incorrect, the learnability of this sample would be 0, and we would ignore training on it. This is in contrast with not using LILO which would continually prioritise training on this question.

Theorem 3.1’s closing of a “vital gap”:

• There is a rich history of curricula for traditional RL settings, and as you note, Learnability has been proposed before, such as in [16].
• We emphasise that the relevance of [16] on reasoning with LLMs is extremely small - there is no clear method for computing and prioritising learnability, there is no suggestion that this should work beyond the REINFORCE algorithm, and is it considered in traditional RL tasks.
• Beyond [16], it is non-trivial to decide which curricula methods from traditional RL are suited for the LLM reasoning setting:for example Havrilla et al tried some and these failed. This is true for other areas of RL too, for example value networks, which are popular in deep RL, have been shown to not transfer to the LLM setting.
• This paper therefore provides the theory to explain why learnability based curricula are particularly well suited for the LLM setting. Previous work does not provide any theory as to why learnability would be useful for modern RL algorithms such as RLOO, GRPO, VinePPO and PPO.
• Looking at the update magnitudes produced during training, we see that selecting the highest learnability produces 1.46x larger updates than not using learnability. Theorem 3.1 explains this, but it also explains why prioritising learnability produces 1.44x larger updates than simply rejecting any questions that are always answered correctly or incorrectly, and 1.40x larger updates than prioritising the most difficult questions. Existing theory cannot explain this.
• Crucially, this results in LILO being the first to show that a curriculum method can boost overall performance on LLM reasoning tasks. Given the popularity and resource consumption of this setting this is highly impactful and useful to the community.

Assumptions within Theorem 3.1:

• We refer the reviewer to our rebuttal to reviewer GyS9, where we comprehensively address this with statistical testing of our assumptions and the theory.

LILO + GRPO:

• Thank you for pointing this out! We definitely should have made this clearer. We use the unnormalised version of GRPO that has shown to outperform normalised GRPO for reasoning tasks [cite DrGRPO and Replicating R1]. As we point out in Section 3, using the unnormalised GRPO does benefit from prioritising learnability.
• We wanted to ensure that the experimental setups we were using already had optimised baselines, and therefore that LILO’s improvement could not be brought about e.g by tuning hyperparameters. We therefore used the VinePPO library which provides optimised baselines of PPO and VinePPO for GSM8K and MATH, and the Oat library which provides optimised baselines of GRPO on ORZ57K.

Error analysis and rejection sampling:

• In Figure 5 we visualise how the number of reasoning steps chosen by LILO changes throughout training. This is strong evidence that rejection sampling would not have been sufficient - questions with a higher number of reasoning steps only become learnable later on during training.
• We agree that further visualisation here of the exact questions that progressed from hard to easy throughout training would be useful to the reader, and we commit to adding these in the camera-ready version.

Errata:

• Thank you for your diligence in picking up on the mistake on line 121, we have fixed this.
• Thank you for the suggestion of rewording to use “up to 3x reduction”. This is indeed a fairer reflection of Table 2.

We respectfully ask the reviewer to evaluate our added details: we believe we comprehensively address any concerns, and that as such it would be reasonable to modify the assessment of our contribution. We are open to any additional feedback or relevant points you might wish to discuss.

Dear Reviewer xAAo,

Thanks so much for the continued discussion, we have considered your response and prepared a reply to each point below (note we had to spread this over 3 comments)

On the terms ‘learnability’ and "learning to reason"

• We completely agree that “learning to reason” should be reserved for the discussion of learning correct and generalisable cognitive processes. Sadly, it is our experience that this isn’t reflected in much of the recent literature. It seems to us that “Learning to reason” has become a popular term for improving LLMs' ability at “reasoning style” questions (maths, science, logic etc), often using reinforcement learning. In additional to the seminal OpenAI O1 release [1], here are 4 papers from 2025 that all have “learning to reason” in the title [2,3,4,5], and 6 papers that all refer to “learning” and “reasoning” [6,7,8,9,10,11]. All of these papers claim that the model has “learnt to reason” via evaluation on holdout problems, and as far as we can tell do not obviously do any deep analysis on the type or generalisability of the cognitive processes used. If one is happy to use “learning to reason” to mean improving test accuracy, it seems reasonable to use “learnability” to mean questions that when trained on yield a higher test accuracy.

• On the other hand, you are correct that there is an alternative strand of work that pushes back against this, with papers such as [12,13,14,15,16] that analyse whether the “reasoning” learnt using RL on LLMs is actually a robust, generalisable, cognitive process. Since our work is directly aimed at those practising RL on LLMs for maths style tasks, we chose to go with the former “learning to reason” terminology, but we appreciate that it can be misleading for those more familiar with the latter referenced work. We’ve suggested how we might perhaps find a compromise between these two styles below, but are open to your suggestions.

• You’re right to point out that previous work on learnability has been only in “traditional RL tasks”. We actually have a similar gripe to the reviewer when ‘learnability’ is used in traditional settings. When teaching an agent to solve mazes: is selecting learnable training levels ones where it might learn general principles for solving mazes (ie the right hand rule), or are these just levels that haven’t yet been memorised? However, learnability has become the standard term for reward variance in traditional settings. Since Theorem 3.1 applies to any sparse MDP, whether in the traditional RL or LLM setting, we chose to stick with the term, instead of redefining it, but we can appreciate how for some readers this may come with added connotations. We're happy to add an addendum/clarifying note to this effect, if you see fit (see below for suggestions).

• This leaves us in a sticky spot - concept drift like this is annoying, especially when some (often closely related) areas of ML overload the same term. We’ve done our best to balance concept drift with situating work within existing literature and terminology which is also important for clarity, for paper visibility, and for giving due credit to the lineage of ideas.

• Honestly, we would greatly appreciate the reviewer's guidance on how best to resolve this - would they be happy with some reconciling / defining terms early on in the paper, such as:

  • By learning to reason, we specifically mean here “learning to solve unseen maths problems through training on a set of existing problems”. By “learnable problems”, we simply mean problems with high variance in rewards for which the model will likely obtain high gradient updates. `

  Also perhaps in limitations, we could be more explicit:

  • Our method helps improve the overall accuracy achieved on the unseen test set, and does so in fewer training steps. Our method does not directly aim to solve the broader goal of ensuring the model is learning correct and generalisable cognitive processes.

  We hope this would suffice, but if not, we could consider changing the name of the paper. It would be a shame for an issue of semantics to interfere with a paper whose broader contribution we really believe in. Please do let us do know your opinions on how best to resolve this dilemma.

[1] OpenAI (2024) Learning to reason with language models. Available at: https://openai.com/index/learning-to-reason-with-llms/ (Accessed: 8 August 2025)

[2] Chen, M., Li, T., Sun, H., Zhou, Y., Zhu, C., Wang, H., Pan, J.Z., Zhang, W., Chen, H., Yang, F. and Zhou, Z., 2025. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503.19470.

[3] Zhao, X., Kang, Z., Feng, A., Levine, S. and Song, D., 2025. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590.

[4] Yan, J., Li, Y., Hu, Z., Wang, Z., Cui, G., Qu, X., Cheng, Y. and Zhang, Y., 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945.

[5] Zhang, J., Huang, J., Yao, H., Liu, S., Zhang, X., Lu, S. and Tao, D., 2025. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937.

[6] Dang, Q.A. and Ngo, C., 2025. Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't. arXiv preprint arXiv:2503.16219.

[7] Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C. and Tang, C., 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599.

[8] Havrilla, A., Du, Y., Raparthy, S.C., Nalmpantis, C., Dwivedi-Yu, J., Zhuravinskyi, M., Hambro, E., Sukhbaatar, S. and Raileanu, R., 2024. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.04642.

[9] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X. and Zhang, X., 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

[10] Wang, Y., Yang, Q., Zeng, Z., Ren, L., Liu, L., Peng, B., Cheng, H., He, X., Wang, K., Gao, J. and Chen, W., 2025. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571.

[11] Rastogi, A., Jiang, A.Q., Lo, A., Berrada, G., Lample, G., Rute, J., Barmentlo, J., Yadav, K., Khandelwal, K., Chandu, K.R. and Blier, L., 2025. Magistral. arXiv preprint arXiv:2506.10910.

[12] Hazra, R., Venturato, G., Martires, P.Z.D. and De Raedt, L., 2025. Have Large Language Models Learned to Reason? A Characterization via 3-SAT Phase Transition. arXiv preprint arXiv:2504.03930.

[13] Li, Z.Z., Zhang, D., Zhang, M.L., Zhang, J., Liu, Z., Yao, Y., Xu, H., Zheng, J., Wang, P.J., Chen, X. and Zhang, Y., 2025. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419.

[14] Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S. and Farajtabar, M., 2025. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506.06941.

[15] Zhang, Y., Wang, H., Feng, S., Tan, Z., Han, X., He, T. and Tsvetkov, Y., 2024. Can LLM Graph Reasoning Generalize beyond Pattern Memorization?. arXiv preprint arXiv:2406.15992.

[16] Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J. and Cambria, E., 2025. Are large language models really good logical reasoners? a comprehensive evaluation and beyond. IEEE Transactions on Knowledge and Data Engineering.