Scaling Long Context Training Data by Long-Distance Referrals

Dear reviewer kWvS,

We sincerely appreciate your valuable and insightful comments. We found them extremely helpful for improving our manuscript. We will strive to address each comment in detail, one by one below.

---

W1 Format Errors

We thank the reviewer for pointing out all the format errors. In the revision, we fixed all the 1) missing space after “LongPack”; 2) missing brackets of the citation

W2 Messy Experiment Results

We thank the reviewer for pointing out that the experiment results are scattered. We upload a revision of the paper with the following changes:

• Section 3 discusses the definition and findings of referral and its distance. It first formally defines referral and referral distance, then explains how we measure long referral density. After that, it elaborated the finding that long referrals improves the language model’s long context understanding. The corresponding experiment setup and result, as well as the figures, are moved in this section (Now Figure 1,2,3). This deduplicates previous Section 4.2, and avoids referring figures and data at a page far away.

• Section 4 discusses LongPack’s approach to construct the data, which consists of Section 3.2 and 4.3 in the previous version.

• Section 5 now only includes experiments of data from LongPack. This includes referral distance analysis and model quality when trained with LongPack’s data.

W3 & Q1 Lack of baselines

We trained models by constructing data chunks of Book (PG-19) or long dialogue (ShareGPT). Both are trained and tested on a 128K context window. This follows the practice of previous works (the Book is for YaRN-128K [1] and the long dialogue is for LongChat [2]). We reported the score of both as:

The two new pipelines are both better than Upsample on the long context benchmark for 10%. However, our method still significantly outperforms all these two popular long-context data pipelines on the overall score (17%). Besides, our method is better at each subtask except for a score close to Dialogues at “multi-hop tracing” and “aggregation”.

We added the new baselines and discussion into our revision (Section 5.4).

Regarding ProLong, we first point out that it is orthogonal to our method, as it designs metrics to distillate data to improve its quality, while LongPack aims to scale up long documents. Since ProLong does not release a publicly available dataset or model, we compare the 9B-model trained with LongPack (1B tokens) with the Prolong-7B (20B tokens) reported in their paper. We use lost-in-the-middle repository [3] and LongBench [4] to run the experiment:

The result shows that LongPack, even with only 5% tokens, performs slightly better than Prolong except for two tasks (GovReport and RepoBench-P). We reason the second as that we didn’t upsample code data in our dataset.

We also highlight that previous work ([5]) shows that training with domain shifts will lead to a non-trivial drop of the model’s quality on other domains, measured by the validation loss of different categories. Hence, Books, Long Dialogue, and Prolong (mainly from code and books), might all suffer from such a shift of data corpus. On the other hand, LongPack is based on the crawl of the whole internet data, which takes most of a pretraining data corpus, and thus does not hurt the model quality.

[1] Yarn: Efficient context window extension of large language models, Yushi Bai et al. [2] How Long Can Context Length of Open-Source {LLM}s truly Promise?, Dacheng Li et al. [3] https://github.com/nelson-liu/lost-in-the-middle [4] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding, Yushi Bai et al. [5] Data Engineering for Scaling Language Models to 128K Context, Yao Fu et al.

Dear reviewer kWvS,

We reported the experiment result of ProLong-reproduced dataset v.s. LongPack below. Both models are trained on top of GLM-4-9B with 1 billion tokens, with the same hyper-parameter reported in the paper:

---

Result of RULER benchmark:

Result of lost-in-the-middle and LongBench:

The results indicate that LongPack outperforms ProLong in almost all benchmark tests.

Dear reviewer qBkR,

We sincerely appreciate your valuable and insightful comments. We found them extremely helpful for improving our manuscript. We will strive to address each comment in detail, one by one below.

---

W1, W3 Format Errors

We thank the reviewer for pointing out all the format errors. In the revision, we fixed all the 1) missing space after “LongPack”; 2) missing brackets of the citation

W4 Using pseudocode to describe the data pipeline

We add the pseudo-code in the appendix of our revision (Appendix.D), and paste a copy below.

REQUIRE Root document r and its url u, HTML database indexed by URL H, Cleaned Web Page database indexed by URL P
    html := look_up(H, u)
    references := regex_substr_all(html, REGEX_REF)
    retrieve_target := []
    FOR (r in references)
        url := regex_substr(r, REGE_URL)
        key := regex_substr(r, REGEX_KEY)
        retrieve_target = retrieved + [(url, key)]
    retrieve_target = dedup_url_merge_key(retrieve_target)

    packed := “”
    FOR ((url, keys) in retrieve_target)
        content := look_up(P, url)
        packed = packed + keys + ":\n" + content

    packed = packed + "root:\n" + r
OUTPUT packed

W5 insufficient experiment content

We add more content in the experiment in two ways:

On one hand, we move some analysis of each category of the RULER benchmark from the appendix to the main text, because we noticed that some tasks are more relevant to the document length, while some prefer more fine-grained documents.

On the other hand, we add more baselines.

We trained models by constructing data chunks of Book (PG-19) or long dialogue (ShareGPT). Both are trained and tested on a 128K context window. This follows the practice of previous works (the Book is for YaRN-128K [1] and the long dialogue is for LongChat [2]). We reported the score of both as:

The two new pipelines are both better than Upsample on the long context benchmark for 10%. However, our method still significantly outperforms all these two popular long-context data pipelines on the overall score (17%). Besides, our method is better at each subtask except for a score close to Dialogues at “multi-hop tracing” and “aggregation”.

We added the new baselines and discussion into our revision (Section 5.4).

Q1 motivation of base model selection

We explain this from two sides: 1) regarding the model size, we choose a 7~10B model due to the computation budget. Currently, our setup exactly fits in a 8x80GB H100 node. Training a model larger than that with a 128K sequence length will need multiple nodes, and raises unnecessary engineering effort on the system side; 2) regarding the model architecture, we chose GLM-4 because it has state-of-the-art performance. The pretrained base model’s ability limits the score after continuous training with long sequence data, and we want to ablate this issue as much as possible. As mentioned in Table 1, row “Ideal”, GLM-4 has almost 100% correctness on the first three categories. As a reference, the score of Llama-2 on a 4K sequence length is: “79.0, 99.9, 87.0, 82.4, 69.5, 35.6”, which is much worse than GLM-4. We believe our result can be migrated to other model architectures, such as the Llama family, because GLM-4 also uses Gated-MLP, multi-query attention, and Rotary positional embedding.

W4 & Q8 More baselines

We trained models by constructing data chunks of Book (PG-19) or long dialogue (ShareGPT). Both are trained and tested on a 128K context window. This follows the practice of previous works (the Book is for YaRN-128K [1] and the long dialogue is for LongChat [2]). We reported the score of both as:

The two new pipelines are both better than Upsample on the long context benchmark for 10%. However, our method still significantly outperforms all these two popular long-context data pipelines on the overall score (17%). Besides, our method is better at each subtask except for a score close to Dialogues at “multi-hop tracing” and “aggregation”.

We also highlight that previous work ([4]) shows that training with domain shifts will lead to a non-trivial drop of the model’s quality on other domains, measured by the validation loss of different categories. On the other hand, LongPack is based on the crawl of the whole internet data, which takes most of a pretraining data corpus, and thus does not hurt the model quality.

We added the new baselines and discussion into our revision (section 5.4).

W5 & Q6 MISC points

• LongPackis highly scalable —> LongPack is highly scalable ... (similar errors occur also in other places)
• The citation format in Section 4 Experiment is not in standard
• The caption of Table 2 is … making me confused about the numbers in the Table 2

We thank the reviewer for the valuable suggestions. In the revision, we fixed all the 1) missing space after “LongPack”; 2) missing brackets of the citation; 3) missing metric (“validation loss”) of the “general performance” in Table 2’s caption.

d0’s meaning is not explained

We thank the reviewer for pointing this out. $d_0$ is the threshold of the referral distance, when counting long distance referrals. We added this detail at the definition of referral distance in Section 3.1 (Line 156)

How is “near-natural” property of the generated long documents been evaluated?

We use “near natural” to describe that LongPack generated documents have a referral distance similar to that of natural long documents. We added the explanation of this form in Section 3.2 (Line 306)

Dear reviewer i48T,

We sincerely appreciate your valuable and insightful comments. We found them extremely helpful for improving our manuscript. We will strive to address each comment in detail, one by one below.

---

W1 Paper organization

We upload a revision of the paper with the following changes:

• Section 3 discusses the definition and findings of referral and its distance. It first formally defines referral and referral distance, then explains how we measure long referral density. After that, it elaborated the finding that long referrals improves the language model’s long context understanding. The corresponding experiment setup and result, as well as the figures, are moved in this section (Now Figure 1,2,3). This deduplicates previous Section 4.2, and avoids referring figures and data at a page far away.

• Section 4 discusses LongPack’s approach to construct the data, which consists of Section 3.2 and 4.3 in the previous version.

• Section 5 now only includes experiments of data from LongPack. This includes referral distance analysis and model quality when trained with LongPack’s data.

We’d hope this could make the experiment result more clear to the conclusion it serves, and are looking forward to any further suggestions.

W2 & Q 1-3 Missing detail of the paper

Thank you for pointing out the missing detail. We added the missing detail one by one in the revision, and also place them below:

How are the scores of dataset examples calculated in Figure 1(a)?

This score is from the RULER benchmark. We added this explanation in both figure caption (Line 241) and the experiment setup (Line 206)

How the length of the document is calculated, by token, by word or by sentence?

The distance is counted at the sentence level. We highlighted this detail at the definition of referral distance in Section 3.1 (Line 182)

How are the order of the retrieved contents decided?

The retrieved pages are prepended simply by the order when the corresponding hyperlink appears in the root document. We added this detail in Section 4 (Line 351)

W3 & Q4 & Q5 More analysis of the experiment result

We thank the reviewer for the thorough investigation of the experiment result, and add interpretation of the result as follows:

Why the #referral in the referral distance bucket of 512-2048 from docs with length (16k, 32k) in the packed documents significant drop compared to the refinedweb dataset?

We reason this as that 512-2048 sentences are at a large scale, especially when comparing to the document length (16-32K tokens).

As shown in Figure 6(a) and (b), if we only count neighboring referrals instead of all pairwise referrals, both RefinedWeb and LongPack have a very low density for the (16-32K document, 512-2048 sentences) case.

By definition, the pairwise density is approximately quadratic to that of neighboring density, which magnifies the bias.

It is said that in Figure 5 (b), the performance of (16k-64k) group is close to (15k-32k) group. This statement is not accurate.

We notice that for many task categories (single-pos retrieval, multi-pos retrieval, QA), the performance is directly relevant to the referral distances (Figure 2).

On the other hand, all document length groups perform roughly the same on aggregation.

In multi-hop tracing, (4-8K) and (16-32K) perform better than other groups. We hypothesize this as the fact that the task scatters a chain of information into the context, and the model is required to trace the whole chain. This requires the model to focus on every 128K / N context window, where N is the number of vertices on the chain.

In this way, training with long context could lead to a “lost-in-the-middle” [1], while models trained on medium length documents can fit the distance of two neighboring points on the chain. As evidence, in Figure 3(b) of the updated paper, “segmented” and (16-64K) uses the same document, and is much worse than (16-64K) in all other tests. However, it is better in the “multi-hop tracing”. This could be reasoned as that in “segmented”, each data point contains 4K-token segments not relevant to each other, and is thus specialized in understanding the 4K context window.

[1] Lost in the Middle: How Language Models Use Long Contexts, Nelson F. Liu et al.

Dear Reviewer i48T,

Kindly remind you to look at our rebuttals. We have carefully addressed your concerns in detail, and are looking forward to hearing from you about the feedback or any further question.

Best regards, Authors

Dear reviewer i48T,

We are writing to draw your attention to the rebuttal we submitted for our paper, as the discussion period is approaching its conclusion.

It holds great significance for us if you could review our rebuttal to ensure a fair and balanced evaluation. We believe that our responses have addressed the points in your review.

Best, Authors

Dear reviewer 7FCd,

We sincerely appreciate your valuable and insightful comments. We found them extremely helpful for improving our manuscript. We will strive to address each comment in detail, one by one below.

---

W1 related work

The idea of using links between documents during language model pre-training has been explored ...

We acknowledge that LinkBERT shares some similar ideas at a high level. However, LongPack still differs significantly in the following novelty and contribution:

• The scope of the problem is different. LinkBERT studied improving the performance of masked language modeling on specific tasks (multi-hop reasoning and few-shot QA), while LongPack focuses on scaling up the context length of pretrained auto-regressive large language models.
• LongPack identified and studied long-distance referral density as the key to improve the quality of long-context training dataset, which is one of our unique contributions and the key motivation behind using document reference, while LinkBERT only intuitively uses hyper-links and other references.
• LinkBERT embeds the document relation into a new loss function (named “document relation prediction” in the paper), while LongPack packs documents by references, and considers the packed document as a single new document during the training (w/o changing the next token prediction loss)

W2 & Q1 Statistical significance of experiments

We thank the reviewer for pointing out the importance of a statistical analysis of the result. We’d like to clarify the statistical significance in two aspects: evaluation and training.

On evaluation, the benchmark we use, RULER, samples 500 times for each test and counts the average score. Hence we believe our results are statistically significant at that stage.

On training, due to the huge training cost (each experiment requires 200 H100 hours), it is not financially possible to launch enough runs for statistical analysis. We additionally train 3 more times with LongPack data for different random seed, and reported the variance of RULER score as below:

Note the std error of the overall score is only 0.14, yet our data outperforms 18 points against the Upsample baseline. We expect this indeed shows a statistical significance of our method.

We plan to run 3 times with different random seeds and add the std error of all end-to-end experiments in the next revision.

Focus on web-based documents

We acknowledge that LongPack currently focuses only on web pages. As discussed in Section 2, web pages dominate the pretraining data corpus (e.g. 87% of Llama pretrain data are from web pages, and 4.8% from GitHub, which is also available on the internet. [1]). As a result, web pages not only provide abundant data resources, but represent the majority of the pretrain dataset and avoid a domain shift in continuous training.

Besides, while we only use web pages, our method can be applied to any domain with reference between documents. We have discussed this in the last section of the paper (Line 514-517 of the revision). For example, academic papers have citations, and code files explicit show the dependency by “import” or “#include” sentences. We leave the experiment using the combination of all these references as a future work.

[1] https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T

Dear Reviewer 7FCd,

We are writing to kindly remind you to look at our rebuttals. We have carefully addressed your concerns in detail, and are looking forward to hearing from you about the feedback or any further question.

Best regards, Authors

Dear reviewer 7FCd,

We are writing to draw your attention to the rebuttal we submitted for our paper, as the discussion period is approaching its conclusion.

It holds great significance for us if you could review our rebuttal to ensure a fair and balanced evaluation. We believe that our responses have addressed the points in your review.

Best, Authors