NExtLong: Toward Effective Long-Context Training without Long Documents

We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.

---

Q1: "Regarding Table 1, it seems that NExtLong is using extra negatives when pre-training on the two datasets compared to other baselines, meaning that it may use more training examples, which makes the comparison not fair."

A1: Thanks for your question. We appreciate the opportunity to clarify this point.

We would like to clarify that NExtLong does NOT use any additional training example (lines 251-256). The negatives utilized by NExtLong are also drawn from the two datasets (lines 208-212). Specifically, we construct a FAISS index from these datasets, and NExtLong retrieves hard negatives from the FAISS index to synthesize long-context data. The FAISS index used by the baseline methods is also built from the same datasets; the only difference between those methods and ours is how they rearrange documents into long-context data.

Importantly, the total number of training examples and the experimental settings are identical across all methods, ensuring a fair comparison. This is explicitly stated in the following sections of our paper:

• Lines 208-212: "Various methods, including NExtLong and baseline approaches, are employed to synthesize target-length samples concatenated from these short documents."
• Lines 251-256: "...The same training configuration is applied to all methods for a fair comparison."
• Lines 263-272 provide further details on how baseline methods synthesize long-context data.

We apologize for any confusion and hope this clarification addresses your concern. Please let us know if any further details would be helpful.

We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.

---

Q1: "I would like to suggest to include the above work in the related section. It is the first work try to use interleaved chunk data to improve long-context performance."

A1: We sincerely appreciate your valuable suggestion. We acknowledge the significance of the mentioned work as the first to explore the use of interleaved chunk data for improving long-context performance. We will incorporate this important reference in the revised version to ensure a more comprehensive discussion in the related work section. Thank you for bringing this to our attention.

Q2: "Limited Novelty: The novelty is limited given the existence of previous works [1] and [2]."

A2: We would like to emphasize that our contribution is not using interleaved chunks. As we explicitly state in lines 55–57: "An intuitive approach to building documents with long-range dependencies is to insert additional text between dependent segments.[1]" Additionally, we discuss the limitations of those methods in lines 62–65, noting that they do not fully capture the real-world challenge of extracting long-range dependencies amid extensive distracting information.

We replace hard negatives with random chunks to implement a baseline (RandomD) similar to [1][2], and Figure 7 shows that it underperforms NExtLong. The results indicate that our method better reflects real-world scenarios where models must extract relevant information despite interference, thereby fostering more robust long-range dependency learning (As mentioned in lines 83-88).

Q3: "Compatibility with Modern Techniques: The proposed data preparation method is not compatible with the commonly used intra-document attention approach for long-context training."

A3: We would like to clarify that our method is indeed compatible with intra-document attention. In fact, Llama-3-8B-NExtLong-512K-Base is trained using the intra-document attention approach and achieves strong results (Table 2). Specifically, we construct 512K and 64K datasets (lines 326–329) and apply intra-document attention to the 64K subsets, enabling their concatenation into 512K context length. This strategy aligns with ProLong[4] (lines 318–320), which uses a 64K dataset to train 512K models. Our results in Table 2 demonstrate that NExtLong performs better under this setup.

Q4: "For Table 1, I suggest that the authors provide the results averaged over 32K, 64K, and 128K in the appendix, following the same evaluation protocol as in ProLong."

A4: Thank you for the suggestion. We will incorporate the results averaged over 32K, 64K, and 128K in the appendix. We report the averaged results in the table below, and NExtLong maintains a performance advantage across sequence lengths beyond 16K, further supporting the effectiveness of our approach.

Q5: "In Table 2, the results are averaged over 8K, 16K, 32K, 64K, and 128K. Does Table 5 follow the same procedure? I suggest that the authors provide more detailed explanations of the experimental setup."

A5: Yes, Table 5 follows the same reporting procedure. Specifically, the "Head" strategy in Table 5 corresponds to the NExtLong results in Table 1.

We sincerely apologize for any confusion caused by the insufficient explanation of the experimental setup. We will include more details in the revised version. Please let us know if any further information would be helpful.

---

[1] Tian, J., Zheng, D., Cheng, Y., Wang, R., Zhang, C., & Zhang, D. (2024). Untie the knots: An efficient data augmentation strategy for long-context pre-training in language models. *arXiv preprint arXiv:2409.04774.

[2] Zhao, L., Wei, T., Zeng, L., Cheng, C., Yang, L., Cheng, P., ... & Zhou, Y. (2024). Longskywork: A training recipe for efficiently extending context length in large language models. *arXiv preprint arXiv:2406.00605.

[3] Wang, H., Liu, Q., Du, C., Zhu, T., Du, C., Kawaguchi, K., & Pang, T. (2024). When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training. *arXiv preprint arXiv:2411.13476.

[4] Gao, T., Wettig, A., Yen, H., & Chen, D. (2024). How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660.

We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.

---

Q1: Dependency on FAISS retrieval quality

A1: The quality of FAISS retrieval depends significantly on the effectiveness of the embedding model. As embedding models advance, retrieval performance is expected to improve accordingly. Given ongoing advancements in this area [1][2], we anticipate these advancements will further enhance the performance of NExtLong and plan to explore the impact of more advanced embedding models in future work.

---

[1] Choi, Chanyeol, et al. Linq-Embed-Mistral Technical Report. arXiv preprint arXiv:2412.03223 (2024).

[2] Wang, Liang, et al. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 (2024).

Q2: "computational cost for indexing"

A2: The computational cost of indexing is only incurred during the generation of synthetic training data and does not impact the inference phase of the model. Since this is a one-time cost, the resulting performance improvements provide lasting benefits for downstream tasks. Given these long-term advantages, the computational expenditure is justified and contributes to overall performance.

Q3: "not comparing against models using realistic long documents."

A3: We would like to clarify that some of the models compared in Table 2 are indeed trained with realistic long documents. For example, ProLong [1] uses realistic long documents (as mentioned in lines 323-325) and extensively explores how to utilize them effectively.

To ensure a fair comparison, Table 2 directly compares NExtLong with ProLong. It shows that Llama-3-8B-NExtLong-512K-Base outperforms Llama-3-8B-ProLong-512K-Base, further validating the effectiveness of NExtLong (lines 319-322).

---

[1] Gao, Tianyu, et al. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660 (2024).

Q4: "Although I understand it is not the main point of this paper, but I am curious about if the proposed method can be applied to long documents to further extend the effective context length of the model? ..."

A4: Based on the hypothesis that shorter original documents pose more significant challenges for a fixed target length and given that training a long-context model is resource-intensive, we prioritized creating a more challenging experimental setting within our limited training resources. Our current experiments deliberately use relatively short documents, creating an effective 64x increase in sequence length compared to the original documents (as mentioned in lines 651–652).

Despite this challenging setting, NExtLong still achieved strong performance, leading us to believe that when the original documents are longer, they can be merged into even longer sequences while maintaining promising results. We are currently collecting longer documents and will systematically investigate the impact of documents exceeding 8K tokens in future work.

Thank you for your insightful question!

We sincerely appreciate your response! Your detailed and insightful feedback plays a crucial role in improving our article. The following text further clarifies some questions.

---

Q1: "It would be better to explain why improving the long-context ability of LLMs can decrease the performance on short text benchmarks."

A1: Previous long-context extension approaches [1] often degrade short-context performance because they rely exclusively on non-synthetic long documents, which are scarce across most domains (as noted in lines 35–37). This view also aligns with [2], which states that "a data mixture that keeps the domain mixture ratio the same as the pretraining mixture...this is the primary reason our solution improves long context tasks while maintaining short context performance."

Quest [3] demonstrates that using a long-text dataset synthesized from short texts with good domain diversity does not decline short-text performance. Following Quest, we also use short-text data sources with sufficient diversity, such as Cosmopedia V2, which covers over 34,000 topics. As a result, Table 5 demonstrates that NExtLong does not degrade short-text performance on average.

Thanks for your valuable suggestion! We will elaborate on this point in the revised version.

Q2: "Compared to prior methods, the novelty of this approach feels somewhat limited. It would be helpful to more clearly highlight which aspects of the previous methods made them suboptimal."

A2: As mentioned in lines 51-53: "those methods typically concatenate short documents based on random or similarity-based rankings, lacking a clear mechanism for capturing long-range dependencies." Previous studies (e.g., Quest [3]) synthesize long-context data by concatenating similar documents sequentially, allowing models to perform the next token prediction without relying on preceding documents. That concatenation strategy weakens the learning of long-range dependencies, and Table 1 shows that those methods achieve only marginal improvements compared to the Standard approach.

While approaches like [4][5] increase dependency length using interleaved chunk data, they do not capture the real-world challenge of extracting long-range dependencies amid extensive distracting information. To verify this, we replace hard negatives with random chunks to implement a baseline (RandomD) similar to [4][5], and Figure 7 shows that it underperforms NExtLong.

In contrast, as mentioned in lines 83-88: "By inserting these distractors between originally dependent meta-chunks, NExtLong not only increases the distance between dependent chunks—effectively transforming the dependencies into long-range ones—but also introduces distracting noise." This design better reflects real-world scenarios where models must extract relevant information despite interference, thereby fostering more robust long-range dependency learning.

We will provide a more detailed comparative analysis in the revised version. Thanks for your constructive suggestion!

---

[1] Chen Y, Qian S, Tang H, et al. Longlora: Efficient fine-tuning of long-context large language models. The Twelfth International Conference on Learning Representations (ICLR'24).

[2] Fu Y, Panda R, Niu X, et al. Data engineering for scaling language models to 128k context. In Proceedings of the 41st International Conference on Machine Learning (ICML'24).

[3] Gao C, Wu X, Fu Q, et al. Quest: Query-centric data synthesis approach for long-context scaling of large language model. The Thirteenth International Conference on Learning Representations (ICLR'25).

[4] Tian, Junfeng, et al. Untie the knots: An efficient data augmentation strategy for long-context pre-training in language models. arXiv preprint arXiv:2409.04774 (2024).

[5] Zhao, L., Wei, T., et al. Longskywork: A training recipe for efficiently extending context length in large language models. arXiv preprint arXiv:2406.00605 (2024).