Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges

Thank you for your thoughtful review and appreciating the novelty, clarity, and thoroughness of our approach and experiments. We address your specific comments and suggestions below:

Computational Cost

Length generalization is interesting for two reasons: First, because we don't have training data with long sequences; and second, because training on long sequences is very costly. ... While I believe that addressing only the first issue is important, the authors should discuss this limitation explicitly. Additionally, I think that discussing the overall computational cost of the method (and ideally compare it to other methods for length generalization) would be helpful."

We appreciate your highlighting this point. While our manuscript briefly mentions the inherent differences of our method from other length generalization approaches (lines 82, 623-626), we agree that explicitly mentioning computational costs is crucial. Indeed, our iterative training method does require higher computational resources compared to single-round training strategies, as it involves multiple iterations of model training. We will clearly discuss this computational trade-off and compare it with other methods explicitly in the revised manuscript.

Comparison with Positional Encoding Method

I think the authors should directly compare their results, at least in some of the experiments, to other length generalization methods on the same tasks.

We acknowledge your suggestion to explicitly compare our approach with other positional encoding (PE) methods. We want to emphasize that our framework is fundamentally architecture-agnostic and can be combined seamlessly with various positional encodings, including both NoPE and RoPE. Specifically, we used NoPE in experiments involving small models trained from scratch due to its better length generalization properties compared to absolute position encoding (APE) or RoPE. Conversely, for our pretrained model experiments, we employed pretrained LLaMA models utilizing RoPE. Hence, the choice of positional encoding is orthogonal to our core contribution. We will clarify and elaborate this point.

Clarification on Majority Voting Filtering

In the majortiy voting based filtering, the authors use an ensemble of models, all trained from scratch with different seeds and datasets. This seems much more expensive than sampling with temperature. Does temperature sampling simply not work, requiring this more complex method? I think that discussing the computational cost of this ensemble method would be useful, and also maybe comment on how it could be improved (e.g., how large should k be for the method to work)?

We did not consider temperature scaling because our tasks are structured to have a specific input-output format. Thus, temperature sampling did not yield sufficient output diversity. Specifically, we observed very high confidence (over 0.999) in top-1 outputs using beam search (for reverse addition task), indicating minimal diversity even with temperature sampling. However, we anticipate temperature scaling could be more effective for pretrained models that can generate more varied outputs.

Regarding computational cost, we conducted ablation studies for majority voting in Appendix Section B.2.4, exploring data cost and performance trade-offs. Given computational costs scale linearly with the number of ensemble models, we will add explicit commentary on this trade-off, as well as provide new results on how accuracy and performance vary with ensemble size (k) and consensus thresholds in our revision.

Q1. Generalization beyond 100 digits

In the addition experiments, the authors claim that they achieve generalization to over 100 digits, but from the plots it seems that performance could keep improving. How far does the generalization actually reach? Did the authors run experiments beyond this range?

Our experiments show that self-improvement can continue indefinitely as long as data accuracy remains high. We halted our experiments around 100-digit sequences due to memory constraints, but since performance remained stable, we anticipate continued successful generalization beyond this length.

Q2. Relative Filtering

The description of relative length filtering is unclear. How is used? Are the lengths filtered based on a fixed constant, or relative to the maximal length L?

We apologize for the unclear description. Currently, relative length filtering uses fixed thresholds (e.g., threshold value of 2 for forward addition and 10 for multiplication). We will clarify this clearly in our revised manuscript. Following the suggestion from Reviewer vrfN, we will also add sensitivity analyses for filtering thresholds. Employing thresholds relative to maximal sequence length L would be a valid (and insightful) approach as well, although it is not used in our experiments.

We appreciate your recognition of our extensive experimental validation and clear presentation. We respond to your questions inline below.

W1 Limitations to Synthetic Setting

It would be better if the authors can discuss more on how their findings can potentially help facilitate real-world tasks.

Please refer to our general response below in our response to Reviewer MReD (General response to overlapping concerns).

W2 Filtering

The method in this paper is limited to synthetic settings with several implicit assumptions: ... (2) the existence of (unsupervised) filtering rules that enable near-perfect filtering, without which the accuracy drops obviously after few iterations.

It is not surprising to see reliable filtering is critical for the success of [.] (note: this comment appears to be cut off)

We agree that filtering is central to the framework. One interesting result is that simple majority voting over models trained with different seeds is surprisingly effective. This unsupervised, task-agnostic approach allows for easy application to a variety of tasks, including more complex ones.

Difficulty schedule

The method in this paper is limited to synthetic settings with several implicit assumptions: (1) single task with fine-grained controllable difficulties ...

I am particularly interested in how much difficulty increased in each round of iteration for each task. As mentioned in the Introduction, controlling the weak-to-strong curriculum is crucial, as models require a structured difficulty schedule to avoid catastrophic failure. But the authors only discuss this by directly putting the final setups without explaining the rationales in the Appendix.

Thank you for pointing out the need to clarify how we selected difficulty increments. Our primary consideration was to ensure data quality sufficient for reliable training in subsequent rounds, thus preventing error avalanches (Section 8). We recognize that our rationale behind difficulty increments could be clearer, and we will explicitly connect the choice of difficulty scaling with the avoidance of error accumulation. Additionally, we will include an experiment demonstrating the consequences of overly aggressive difficulty scaling, further clarifying the importance of difficulty schedule. Our existing "accelerated self-improvement" experiments (Section 7.2, Section B.4, Figures 12 & 23) also offer insights into the flexibility of our difficulty schedules, demonstrating the allowable range of difficulty increases with self-improvement rounds.

Connection to Length Scaling of Reasoning Models

Results in this paper seem to relate to the inference length scaling of reasoning models[1].It would be interesting to discuss the connection between length scaling during RL and the findings in this paper. ([1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning)

Thank you for pointing out the potential connection between our results and the inference length scaling observed in reasoning models like DeepSeek-R1. Indeed, both lines of work emphasize iterative and incremental learning paradigms. We add a short discussion of this connection and its implications in our revised manuscript.

Thank you for your careful review and for recognizing the thoroughness of our experimental design.

W1 Limitations to simple tasks

This paper is very much limited by the simplicity of the task[...] length of the input corresponds nicely to the difficulty of the problem. However, in many (and more realistic) cases, such a correspondence is not that straightforward.

We agree that for most of the tasks we considered, longer length implies higher difficulty. But for tasks like maze solving, a larger number of nodes do not correspond strictly to harder instances; it is possible to have a smaller maze that is harder due to a higher branching factor. It would indeed be interesting to consider tasks whose difficulty scales constantly or inversely with length. Moreover, we absolutely agree that estimating the difficulty of real-world tasks is an important future direction, and we discuss this point more in the Limitations section of our paper.

For more on real-world applicability, please refer to our general response below.

W2 Data filtering

Without the data filtering procedure, the self-improvement framework will lead to much worse result. It is unclear how to design good data filtering procedure on more complex tasks.

Indeed data filtering is crucial to the success of our self-improvement framework. One key finding from our experiments is the effectiveness of simple majority vote filtering. This strategy is notable because it does not rely on task-specific heuristics, making it potentially applicable to more complex scenarios. Moreover, filtering based on majority voting to ensure self-consistency is a widely-used approach, as we mention in our paper (L265).

W3 Implications for Complex Settings

Therefore, the implications of the findings presented in this paper to more complex settings is not clear. Besides stating this as a limitation, the authors should at least discuss how the results on simple tasks would be useful for more complex tasks.

We acknowledge your valid concern regarding the implications of our findings for more complex, real-world settings. As we discuss explicitly in our Limitations Section, defining and quantifying task difficulty remains an open and significant challenge in practical applications. However, our results provide valuable insights into systematically scaling task difficulty, an essential step for enabling transformers to tackle more complex real-world problems effectively.

Moreover, our experiments show that models exhibit robustness to imperfect difficulty scheduling, especially when starting from pretrained models. This robustness improves further with increased training rounds. Additionally, we find that decomposing complex problems into intermediate steps enhances the model's ability to generalize. Notably, pretrained models are particularly effective at leveraging these decompositions, highlighting a promising avenue for applying our method to complex tasks.

---

General response to overlapping concerns:

Real-world Applicability and Generalization Beyond Synthetic Tasks.

We acknowledge this valid concern. We selected synthetic tasks for their well-defined difficulty and clear evaluation metrics, which allow for controlled and rigorous analysis.

Additionally, the scope of our chosen tasks aligns with common practice in the length generalization literature. Many influential works (Li et al., 2023;Ruoss et al., 2023;Zhou et al., 2023;McLeish et al., 2024;Kazemnejad et al., 2024;Sabbaghi et al., 2024;Cho et al., 2024;Zhou et al., 2024;Fan et al., 2024) have focused on the problem of generalizing beyond lengths seen during training using synthetic tasks, gaining recognition for providing critical insights into model limitations and capabilities. In fact, this line of research dates back to early works such as Neural GPUs(Kaiser and Sutskever, 2016) and Universal Transformers (Dehghani et al., 2019). We cite these studies to legitimize our methodology, demonstrating that insights derived from synthetic tasks are widely considered meaningful and impactful.

Compared to many prior works, our work considers a broader range of synthetic tasks with varying difficulty, encompassing arithmetic operations, string manipulations, and maze-solving problems. These tasks allow us to stress-test self-improvement in diverse settings and build insights applicable to more complex domains. Although out of the scope of this paper, we plan to extend self-improvement to long-context natural language reasoning benchmarks.

Overall, we believe our paper presents an important step forward. We demonstrate improvements to the existing length generalization literature by providing a method that can continue the trend of generalization indefinitely, on a wider range of tasks. We supplement the work with robust empirical analyses. Our work establishes foundational insights on how self-generated data and iterative self-improvement can extend the capabilities of transformer models.

We thank the reviewer for their detailed feedback and for acknowledging that the iterative framework is a compelling strategy to push the boundaries of the generalization capabilities of Transformer models, and recognizing our extensive ablations and analyses. Please find our responses to each of the concerns you have raised below.

W1,W2, Q1 Real world applications

W1 The experiments focus solely on synthetic tasks. More real-world applications would be valuable to assess practical utility. I would expect the authors try their method to adapt a model trained on shorter text sequences to process longer inputs

W2 The method's performance on tasks with more complex or less structured data remains to be explored.

Q1 How do you envision scaling this self-improvement framework to tasks beyond synthetic benchmarks? Could you provide examples of potential real-world applications or domains where you expect similar performance gains?

Please refer to our General Response below in our response to Reviewer MReD (General response to overlapping concerns) regarding the limitations on synthetic tasks.

W3-1 Necessity of self-improvement vs. positional encodings

Why do we need self-improvement while we can use positional encoding and not requires training to generalize to longer sequence?

While approaches like RoPE indeed improve length generalization, they are complementary rather than replacements. Our self-improvement method is architecture-agnostic, requires no positional encoding modification, and naturally scales across diverse applications without architecture changes. This generality positions our method advantageously, especially when considering broad applicability to existing large pretrained models, avoiding retraining costs or architectural modifications. For example, for our pretrained model experiments, we employed pretrained LLaMA models utilizing RoPE. Hence, the choice of positional encoding is orthogonal to our core contribution. We will clarify and elaborate this point.

W3-2. On RoPE scaling for long-context

The authors did mention that 'While effective in controlled setups, these approaches are often incompatible with large language models (LLMs) in practice, as they introduce task-specific modifications that are difficult to scale across diverse applications.' but RoPE, for example is applied on Deepseek and it is not 'task-specific'."

RoPE itself is only a positional encoding choice, but we agree that RoPE scaling, used in DeepSeek, is proposed as a way to generalize to longer contexts without retraining the model. However, existing works on long context model evaluation [https://openreview.net/pdf?id=293V3bJbmE] shows that RoPE still struggles to achieve reliable performance on OOD input lengths. Furthermore, applying RoPE scaling on our synthetic tasks did not help with length generalization in our experiments. We will add our result on using RoPE scaling in the Appendix.

Q2. Sensitivity of Filtering Thresholds

Have you conducted experiments to assess how sensitive the method is to the choice of filtering thresholds in both length filtering and majority voting? What guidelines can you offer for tuning these parameters in practice?

We agree this is important and will add further clarification in the paper. Based on our current observations, the impact of length filtering is task-dependent. In tasks like multiplication where errors tend to drop intermediate steps, the threshold is less sensitive. In contrast, tasks like forward addition(where incorrect outputs are typically just 1–2 digits short), stricter filtering is needed to avoid error propagation. Regarding majority voting, we plan to expand our ablations to show how the number of models affects filtering effectiveness. As indicated in prior work (e.g., https://arxiv.org/pdf/2306.15400), even a small number of highly accurate training examples can suffice for strong length generalization. Therefore, filtering for high-precision data by strong consensus among multiple models - even at the cost of sample quantity - can be beneficial. We thank the reviewer for suggesting this analysis and will include it in our revision.