SkyLadder: Better and Faster Pretraining via Context Window Scheduling

Thanks for your valuable feedback and for recognizing the value of our controlled experiments. Here are our responses to your questions:

Changing the context length is not a new idea.

We fully acknowledge prior works that employ the idea of growing the context length. However, as discussed in Section 2, these approaches were introduced primarily for efficiency reasons and lacked rigorous and public evaluation of their impact on final model performance. To our knowledge, no prior work has conducted a controlled, large-scale study across multiple model sizes, datasets, and masking strategies to quantify:

1. How pretraining context length affects downstream task performance under a fixed token budget.
2. How different scheduling strategies compare in terms of performance.

For (1), our controlled experiments challenge the common assumption that longer context windows are better if compute permits. We show that naïve long-context pretraining does not consistently improve performance. This overturns the default “longer is better” expectation and highlights context length as a critical but underexplored dimension in pretraining design.

For (2), while existing models often extend context in multiple stages, there has been no public and systematic analysis of how the schedule pace affects outcomes. In Table 7 (illustrated in Figure 11), we show that different schedules, despite reaching the same final 32K context, can lead to significantly different performance. This shows that a systematic investigation of the schedule is necessary and deserves more open and public research.

---

It might be better to use a more appropriate method such as 2-stage training, i.e., short-context pretraining + long-context adaptation

We agree that current models often adopt a two-stage procedure: an extended period of training at a fixed mid-range length (e.g. 4K or 8K), followed by long-context continual pretraining on a much smaller token budget with gradual expansion. We therefore include a Cont. Pretrain baseline following this setup and compare it with SkyLadder.

We experimented with pretraining 1B models with 32K context. The SkyLadder schedule starts from 32 at the beginning and gradually expands towards 32K. In contrast, the Cont. Pretrain setup spends 95% of training tokens at 4K length, before expansion to 32K with the rest 5% tokens. The performances are as follows:

While two-stage training improves over the constant baseline, SkyLadder achieves the highest performance on all tasks. This demonstrates that scheduling should be planned over the entire pretraining process, not only during the adaptation stage. This controlled comparison shows that SkyLadder provides genuine gains over the commonly used heuristic approaches.

---

Here are our responses to the “Questions” Section:

Given that RoPE, and especially NoPE can actually be extended to unseen lengths, have you ever tested the performance of these models on long-context tasks?

We evaluated whether short-context models can match long-context models when extended to 8K using RoPE adjustments or NoPE.

First, we take RoPE-based models pretrained with 512–4K contexts and apply three common extension methods (Linear, Dynamic NTK, and YaRN) to scale the context window to 8K. The table below reports evaluation perplexity (PPL) at 8K. For reference, a model pretrained directly at 8K achieves a PPL of 8.25.

Observation: YaRN provides the best results, but only the 4K→8K model approaches the 8K-pretrained baseline. Models trained on very short contexts (512 or 1K) perform poorly when extrapolated, showing that post-hoc scaling cannot fully substitute long-context pretraining.

Next, we evaluate these extended models on the MDQA-30 document task. For reference, a model pretrained directly on 8K achieves a score of 17.7.

Observation: Even with the strongest extrapolation techniques, short-context models underperform the 8K-pretrained model on MDQA-30. The performance gap widens as the pretraining context length decreases.

For NoPE models, we encountered training stability issues and were able to do small-scale experiments (120M parameters, with 15B tokens). Due to the limited model size, we only evaluated perplexity (PPL) on held-out validation text, with $L_e$ indicating the evaluation context window.

Observation: PPL increases sharply whenever the evaluation length exceeds the pretraining length. While NoPE models show some extrapolation ability, they still fail to match the performance of models trained natively with longer contexts.

Overall, both RoPE-based extensions and NoPE provide only limited long-context capability. Short-context models cannot fully substitute dedicated long-context pretraining, especially on complex long-document tasks. This highlights the necessity of long-context training. Our paper thus investigates the challenges in long-context pretraining and proposes SkyLadder to improve both long and short performance.

We hope that our response has addressed your concerns, and we would greatly appreciate it if you could consider increasing the score.

Thanks for your helpful feedback. We appreciate that you found our work clear and valuable. Here are our responses to your questions:

The core idea is not particularly surprising.

Our contribution is the first large-scale, controlled study across multiple model sizes, datasets, and masking strategies, demonstrating that shorter windows consistently outperform long ones under a fixed token budget. These findings challenge the current trend of pretraining with longer context windows (Figure 1).

Previous works may apply short-to-long pretraining for efficiency reasons without showing a clearer performance gain, as discussed in Section 2. We show that proper context scheduling can improve performance on both short and long tasks, in addition to training speedup. We verified this effect across model sizes and pretraining corpora. This reveals that the schedule context length is a critical yet underexplored dimension of pretraining design.

---

More analysis on why short context pretraining performs better.

Short-context pretraining provides more stable learning dynamics overall. We identify three key observations below:

First, as increasing the context window mostly affects the attention layer, we choose it as the focus of our study. In Section 4.6 Figure 7, we plot the dynamics of attention sink and entropy during the pretraining: we observe that SkyLadder delays the emergence of attention sink. This is consistent with the findings of concurrent work [1] that short-context pretraining results in a lower attention sink. Overall, short-context models have more active attention heads that do not sink to the first token, leading to more capacity in processing the context.

Second, we monitor the max attention logits $S_{\text{max}} = \max_{i,j} q_i \cdot k_j$ across the training steps, following a recent study of K2 [2]. A large attention logit suggests that a head is malfunctional and leads to numerical instability. We pretrained 120M models with different context lengths (each row), and get $S_{\text{max}}$ as training progresses (each column represents a checkpoint pretrained with X tokens).

We observe that the 16K model shows growing $S_{\text{max}}$, and still increases steadily at the end of training. In contrast, the 1K model exhibits smaller $S_{\text{max}}$ overall, and the value quickly saturates after 5B tokens. Thus, short-context models give smaller max attention logits.

Finally, we directly monitor the training dynamics by investigating the training loss and gradient behavior. We compute four stability metrics for the first $N = 30K$ steps of pretraining, where $L_t$ is the training loss and $G_t$ is the gradient norm before clipping:

• Loss Volatility: measures local fluctuations of loss over a sliding window ($w=10$), computed as $\frac{1}{N} \sum_{t=1}^{N} \text{Std}(L_{t-w+1}, \dots, L_t)$. Lower values indicate more stable training.
• Loss Smoothness: the average loss change between consecutive steps, $\frac{1}{N-1} \sum_{t=2}^{N} |L_t - L_{t-1}|$. Smaller values mean smoother convergence.
• Mean Loss Ratio: measures temporary increases in loss relative to the best loss so far, $\frac{1}{N-1} \sum_{t=2}^{N} \frac{L_t}{\min(L_1, \dots, L_{t-1})}$, where smaller values indicate fewer loss spikes.
• Average Gradient Norm: $\frac{1}{N} \sum_{t=1}^{N} \min(G_t, 1)$, where larger values indicate more aggressive gradient updates.

We find that longer-context models exhibit higher loss volatility, less smooth loss curves, more frequent upward loss spikes, and larger gradient norms, indicating less stable optimization dynamics during training. This suggests that short-context pretraining is inherently more stable, potentially providing more reliable gradient updates and facilitating better convergence.

Due to the rebuttal policy, we are unable to present figures to visualize the dynamics. We will plot these interesting results in the updated version of our paper.

---

Why does the proposed SkyLadder method fail to improve performance on certain reading comprehension tasks, as shown in Table 2?

We note that in the reading comprehension setup, Random+SkyLadder performs the best across all tasks. In contrast, for the IntraDoc setup, adding SkyLadder is overall comparable to the baseline. For tasks improved by SkyLadder, the difference is large. For tasks that with a performance drop, the decrease is very small (<0.8%). Our evaluation setup mainly follows [3], where a performance fluctuation is also observed at around 1-2%.

Several factors may explain this fluctuation:

• Few-shot prompting noise: We prepend 2-4 demonstrations with passages and questions to each test sample. The context thus includes extra noisy information that the model needs to handle, causing performance variations.
• Open-ended generation: These QA tasks require the model to output free-form text, which naturally gives more variations than multiple-choice questions.
• Strict exact-match metric: The generated text is evaluated by an exact match to the answer. Small differences in phrasing or format may be marked incorrect.

All these factors may contribute to performance fluctuation. Even though IntraDoc+SkyLadder does not seem to improve reading comprehension significantly, it still improves training efficiency and standard benchmark scores.

Overall, we hope that we have addressed your concerns by carrying out the above analysis. We plan to include these results in the revised version of the paper, and would appreciate it if you could consider updating your score.

---

References

[1] Barbero et al. Why do LLMs attend to the first token?. Arxiv 2504.02732

[2] Kimi Team. Kimi K2: Open Agentic Intelligence. Technical Report 2025.

[3] Zhao et al. Analysing The Impact of Sequence Composition on Language Model Pre-Training. ACL 2024

Thanks for your valuable feedback. We are delighted that you find our work interesting. Here are our responses to each question:

1. More tokens for Figure 1.

We conducted additional experiments by training 1B models from scratch on 200B Fineweb-Pro tokens, with a context length of 1K, 4K, and 16K. Results on downstream tasks are shown below:

The trend remains consistent: shorter context windows lead to better overall performance under a fixed token budget. We plan to extend training to even larger budgets (e.g., 500B tokens) in future work.

The following table represents the length characteristics of these datasets:

These benchmarks are commonly reported when a base model is released. While these benchmarks are relatively short, they are standard for assessing a base model’s knowledge and reasoning abilities: we do not want a long-context model to hallucinate simply because the user provides a short query.

---

2. Generalization to larger models.

We also hope to validate SkyLadder on larger models like 7B. However, such experiments require more tokens and compute, which is infeasible during the rebuttal period. Our current results already demonstrate generalizability across multiple model sizes, context lengths, datasets, and tasks. As shown in Table 5, the relative gains of SkyLadder increase with model size (from +1.1% for 120M to +3.5% for 3B), indicating that the benefits are likely to be even larger as model size grows.

---

3. Extend Table 1 and 2 to more corpora.

We reported the performance of pretraining on Fineweb-Pro in Table 4. We report the performance of these models on standard benchmarks below:

Models with SkyLadder added show steady performance gains on these commonly used benchmarks, even when trained on a different corpus.

For benchmarks listed in Table 2, the scores are as follows:

SkyLadder achieves better or comparable performance on these benchmarks as well, demonstrating its generalizability.

---

4. More long-context benchmarks.

We evaluate our largest 3B model on additional long-context tasks (Tables 12 and 13 in Appendix A.7.3). Key results are summarized below:

First, we follow a related work [1] to evaluate models on TOEFL and QuALITY. SkyLadder achieves better performance on these benchmarks.

Many-shot ICL: We follow [2] and [3] to test on text classification. Numbers in brackets are the number of labels for each task.

SkyLadder shows significant gain for tasks with many labels. Both models achieve comparable performance on binary tasks, possibly because the performance is almost saturated above 90%.

Retrieval-Augmented Generation: We follow [4] for evaluating long-context RAG performance on multiple QA datasets. We fill the entire 8K context for this setup.

SkyLadder outperforms the baseline by a large margin across all QA datasets in the RAG task, which is an important application scenario for long context models.

Overall, as shown by these additional results, SkyLadder can be better than or comparable with the baseline on multiple long context evaluations while achieving significant performance gains on standard benchmarks.

---

5. Random + SkyLadder outperforms IntraDoc + SkyLadder

We thank the reviewer for spotting this intriguing phenomenon. First, we note that both SkyLadder and IntraDoc create shorter contexts; thus, the gain of applying SkyLadder to IntraDoc is smaller compared to Random. Towards the end of training, Random+SkyLadder (denoted as RS) is trained on overall longer contexts compared to IntraDoc+SkyLadder (IS): the RS model is able to see contexts beyond the current document. Thus, RS is exposed to more diverse and noisy multi-document contexts. Such exposure likely teaches the model to filter out irrelevant content and focus on key information. Thus, it shows better performance on the benchmarks, especially the longer ones with noisy contexts.

---

6. Information about dataset lengths

As we are unable to attach figures in the rebuttal, we have to present the statistics as a table. We first show the distribution of the pretraining corpora we used:

Both distributions are right-skewed, which means long documents are rare. Overall, the CC corpus has longer documents, as Fineweb-Pro has been carefully cleaned and processed.

As for the evaluation datasets, besides the ones we reported earlier in #1 in this response, the statistics for other benchmarks are as follows:

We will put more dataset information in the revised version of the paper.

We hope the above response addresses your concerns, and we would appreciate it if you could consider raising your score.

---

References

[1] Pouransari et al. Faster LLM Training with Variable Sequence Length Curriculum. NeurIPS 2024.

[2] Zhao et al. Analysing the Impact of Sequence Composition on Language Model Pre-training. ACL 2024.

[3] Shi et al. In-context Pretraining: Language Modeling Beyond Document Boundaries. ICLR 2023.

[4] Yen et al. HELMET: How to Evaluate Long-context Language Models Effectively and Thoroughly. ICLR 2025.

Thanks for your valuable feedback. We appreciate that you find our study insightful. Here are our responses to your concerns in the Weaknesses section:

The paper provides solid experimental evidence for the effectiveness of context window scheduling, but the overall innovation is limited.

Thanks for acknowledging the rigor of our study. We want to clarify that the main contribution of our paper is to provide a new insight that context window scheduling can boost the downstream task performance, which is "novel" compared to previous works. As discussed in the Related Work section, previous works that explore related ideas are mostly designed to accelerate training, without demonstrating consistent performance gains. Our work is the first to conduct a large-scale, controlled study (up to 3B models, 32K context) showing that context window scheduling not only improves training efficiency but also significantly boosts downstream task performance.

Furthermore, unlike approaches that select documents of certain lengths (which risk introducing domain biases), SkyLadder dynamically adjusts the effective context window through attention masking, mitigating the distribution shift issues in long-context training. This combination of scale, performance improvements, and bias mitigation constitutes an important contribution beyond prior works.

---

Scaling to larger models.

We are also curious about further scaling and acknowledge this as a limitation. We are unable to finish the pretraining of 7B models during the rebuttal period, as training a larger model naturally requires more tokens beyond our compute capability. However, we believe our findings have provided strong evidence of scalability:

• Positive scaling trend: In Table 5, As shown in Table 5, the performance gain from SkyLadder grows larger as model size increases (from 120M to 3B parameters), suggesting its potential in larger models.
• Consistent gain across different settings: We demonstrate consistent improvements across different corpora, masking strategies, evaluation tasks, and model architectures. This shows that SkyLadder is not specific to a single configuration and should generalize to larger models

These results support the scalability of our method to larger models, which we leave as future work.

---

The lack of comparative analysis with other advanced models, such as LLaMA 3.2 3B, Mistral 3B, and Qwen 2.5.

We appreciate the reviewer's suggestion to include comparisons with advanced models. However, a direct comparison is not meaningful due to fundamental differences in training setups:

• First, our paper looks at pretraining a model from scratch. These public LLMs are trained on orders of magnitude more data (>10T tokens), and their superior performance is largely attributed to the large data scale.
• The pretraining data for these models is not publicly available, making it impossible to control for differences in data quality or preprocessing.
• Our goal is not to release a new state-of-the-art model, but to isolate and study the effect of context scheduling under a fixed token budget, where we can rigorously control for all other variables.

Our 3B model exactly follows the Llama3.2 3B architecture (Table 10), and we train two models on exactly the same tokens to verify the effectiveness. This is a strictly controlled setup that rules out confounding factors such as improved data cleaning.

For other architectures, we note that Mistral has no 3B model officially. Therefore, we additionally pretrain Qwen 0.5B models from scratch on 100B tokens to show the generalizability of our approach. The table below shows the PPL on the held-out validation set.

SkyLadder improves the validation PPL across all evaluation lengths.

On common benchmarks, SkyLadder also performs better overall. These results confirm that our method is applicable to Qwen models and generalizes across architectures. We have also verified its effectiveness on hybrid-attention models such as Gemma 3 (Table 17; Appendix Line 742).

---

Here are our responses to the specific questions in the “Questions” section.

What training scale was used in the "Continual Pretraining" experiment (Tab. 7)? Was an evenly spaced expansion (e.g., 4K → 8K → 16K → 32K) used for context window sizes?

Table 7 shows results for 1B models pretrained on 100B high-quality Fineweb-Pro tokens, with a final target length of 32K. We did not employ gradual expansion for the Cont. Pretrain setup in Table 7. However, we agree that it is interesting to examine the evenly spaced expansion as suggested, and we conducted additional experiments. The results are shown below:

Although this gradual continual pretraining setup outperforms the baseline, SkyLadder still achieves the best performance across all tasks. This result indicates that the schedule should be applied from the beginning of pretraining, rather than only during the continual pretraining stage.

---

Does SkyLadder affect the model's ability to generalize to unseen sequence lengths?

Thanks for this interesting question. Our answer is that it improves the generalization ability to unseen lengths, as shown by the following observations.

First, we directly evaluate pretrained checkpoints trained with an 8K window on sequences longer than 8K, following the protocol in [1]. The table below shows the perplexity (PPL) on the GovReport dataset for different evaluation lengths (each column) up to 32K:

Interestingly, the SkyLadder model, with gradual expansion, achieves lower perplexity beyond the training length. In contrast, the PPL of the Random baseline quickly explodes.

Second, we also implemented long-context continual pretraining, where both models are extended from 8K to 32K using 2B additional tokens from [2]. We evaluate the models on the RULER benchmark.

In this case, the SkyLadder model outperforms the baseline as well. This confirms that gradual context expansion leads to a stronger model with better generalization to unseen lengths.

We hope our responses have addressed your concerns and clarified the contributions of our work. We would greatly appreciate it if you could consider raising your score.

---

References

[1] Peng et al. YaRN: Efficient Context Window Extension of Large Language Models. ICLR 2024.

[2] Fu et al. Data Engineering for Scaling Language Models to 128K Context. ICML 2024.