Make Your LLM Fully Utilize the Context

Thanks for your valuable feedback on our work. We hope the following response could further address your concerns.

W1: From the viewpoint of Scaling Laws, comparing Mistral-7B and its further trained model FILM-7B is unfair.

R1: We present further experimental results to compare IN2 training and normal instruction tuning (IT) under the same training data size. These results demonstrate that IN2 training can effectively alleviate the lost-in-the-middle problem while the normal instruction tuning cannot. In the following table, both Mistral+IN2 and Mistral+IT have the same training data size (20% of our full data size, ~300K examples). Furthermore, the PDF file in our general response illustrates the performance curves of Mistral+IT. Compared with Mistral+IN2, the gains from normal instruction tuning are marginal and unstable.

	Doc		Code		DB		All
Model	Avg	Gap↓	Avg	Gap↓	Avg	Gap↓	Avg	Gap↓
Mistral	74.2	32.1	20.3	59.5	47.5	77.0	47.3	56.2
Mistral+IT	69.0	25.9	30.2	76.5	53.4	54.4	50.9	52.3
Mistral+IN2 (ours)	82.9	11.5	74.5	27.7	83.5	31.6	80.3	23.6

Q1: How well does FILM-7B perform on few-shot setting?

A1: We conduct further evaluations on the few-shot learning tasks in LongBench, including TREC (few-shot classification), TriviaQA (few-shot QA) and SAMSum (few-shot summarization). Although FILM-7B does not outperform GPT-4-Turbo, it shows that IN2 training can significantly close the gap with proprietary models on few-shot settings. These results demonstrate that even though the surface forms differ across various long-context scenarios, we believe that the underlying capability to comprehend long-context information is transferable.

Model	TREC	TriviaQA	SAMSum	Avg.
GPT4-Turbo	77.0	91.7	39.7	69.5
Mistral	71.0	84.5	35.8	63.8
FILM (ours)	76.0	90.0	39.5	68.5

Thanks for your encouraging feedback on our work. We hope the following response could further address your concerns.

W1: Since the data is synthesized and the chunk is randomly injected into the Doc, would the model memorize such synthetic data but lose the generalizability on other long-context tasks beyond such scenarios?

R1: Our evaluation results on real-world long-context tasks (Table 2) could address this concern. For instance, the long contexts in NarrativeQA are entire books or movie scripts, and the long contexts in Qasper are full academic papers. These natural long contexts are quite different from our synthetic data, and the improvements on these tasks demonstrate the generalizability beyond the synthetic pattern.

We also further evaluate on the following few-shot learning tasks which also have different surface forms with our synthetic data. These tasks are taken from LongBench, including TREC (few-shot classification), TriviaQA (few-shot QA) and SAMSum (few-shot summarization). The improvements on these tasks further demonstrate the generalizability on general long-context tasks.

Model	TREC	TriviaQA	SAMSum	Avg.
GPT4-Turbo	77.0	91.7	39.7	69.5
Mistral	71.0	84.5	35.8	63.8
FILM (ours)	76.0	90.0	39.5	68.5

W2: The authors may add more details about their construction of the training data.

R2: We will add further details about the process of constructing our training data, including the selection strategy on filling segments $[r_j]$ and the data quality analysis before generating data on a large scale. Moreover, Appendix D contains details about the considerations and implementation for our segmentation process.

• Selection strategy on filling segments $[r_j]$. We randomly sample $[r_j]$ from a large candidate pool where $s_i$ and also $[s_i]$ are excluded. During sampling $[r_j]$, we restrict the distribution of len($[r_j]$) in order to manage the distribution of the final context length.
• Data quality analysis. We manually accessed 40 synthetic examples before generating data on a large scale. Some of examples are contained in our Appendix H. We find that GPT-4 is highly capable of generating QA pairs that are specifically related to the information provided in the given contexts.

Q1: Will the training data be released to the community?

A1: Our data will be released upon completion of the internal data review process.

Q2: A related paper about multi-hop QA style Needly-in-the-Haystack benchmark.

A2: Are you referring to the RULER benchmark [1]? This benchmark purports to access "multi-hop tracing and aggregation". Our Appendix F includes the evaluation process and results on the RULER benchmark. It shows that our FILM-7B is the first ~7B model that achieves 32K effective context length.

Models	Claimed Length	Effective Length	4K	8K	16K	32K
Llama2-7B	4K		85.6	-	-	-
LongChat-7B	32K	<4K	84.7	79.9	70.8	59.3
Together-7B	32K	4K	$\underline{88.2}$	81.1	69.4	63.0
LWM-7B	1M	<4K	82.3	78.4	73.7	69.1
ChatGLM3-6B	128K	4K	$\underline{87.8}$	83.4	78.6	69.9
Mistral-7B	32K	16K	$\underline{93.6}$	$\underline{91.2}$	$\underline{87.2}$	75.4
FILM-7B	32K	32K	$\underline{92.8}$	$\underline{88.2}$	$\underline{88.1}$	$\underline{86.9}$

[1] RULER: What's the Real Context Size of Your Long-Context Language Models?

Thanks for your detailed review and constructive suggestions. We hope the following response and our additional experiments have addressed your concerns.

W1 and Q4: Require experimental comparison with normal instruction tuning.

R1: We present further experimental results to demonstrate that normal instruction tuning (IT) cannot alleviate the lost-in-the-middle problem. We conduct a comparison between the effectiveness of IN2 training and normal instruction tuning, with the results shown in the following table. Specifically, both Mistral+IN2 and Mistral+IT have the same training data size (20% of our full data size, ~300K examples) and do not incorporate adjustment of RoPE. The training data for Mistral+IT are sampled from a large-scale instruction-following dataset OpenOrca [1]. Compared with Mistral+IN2, the gains from normal instruction tuning are marginal and unstable.

	Doc		Code		DB		All
Model	Avg	Gap↓	Avg	Gap↓	Avg	Gap↓	Avg	Gap↓
Mistral	74.2	32.1	20.3	59.5	47.5	77.0	47.3	56.2
Mistral+IT	69.0	25.9	30.2	76.5	53.4	54.4	50.9	52.3
Mistral+IN2 (ours)	82.9	11.5	74.5	27.7	83.5	31.6	80.3	23.6

Furthermore, the PDF file in our general response illustrates the performance curves of Mistral+IT. These comparisons demonstrate that IN2 training can effectively alleviate the lost-in-the-middle problem while the normal instruction tuning cannot.

W2 and Q3: Require further explanations for “higher information intensity” and the insights behind adjusting RoPE.

R2: The term "higher information intensity" is not a quantitative measure but a conceptual phrase we utilize to characterize our IN2 training. In our QA-style training data, the essential information for different questions are spread evenly across all positions, instead of being concentrated only at the beginning and the end (which would indicate high sparsity).

The rationale behind our adjustment of RoPE is derived from practical experiences with context-extension training. Studies [2,3] show that when the context window was extended from 4K to 32K during training, the RoPE base $\theta$ was increased from 10,000/50,000 to 1,000,000. There is a high-level similarity between context-extension training and our IN2 training: both aim to enhance the model’s capability to perceive information from a wider range of input positions compared to the previous training stages. Consequently, we carry out experiments to investigate whether the experience from context-extension training could also benefit our IN2 training.

W3: Require further clarification on notations and sampling process.

R3: We will revise our Section 2 to make it clearer for readers.

• About $[r_j]$ and $s_i$/$[s_i]$: It is satisfied that $[r_j]$ will not contain $s_i$/$[s_i]$. Moreover, we make a restriction that 2 <= len($[s_i]$) <= 16. We set an upper bound here considering the time cost of calling GPT-4 for an excessively long input sequence.
• About the sampling process for length balance: There is a typo in line 108: the “reject sampling on $[r_j]$” should be “restricted sampling on $[r_j]$”. During sampling $[r_j]$, we restrict the distribution of len($[r_j]$) in order to manage the distribution of the final context length. Thanks for pointing out this error!

Q1: Why synthesize QA data instead of using an existing QA dataset?

A1: There are mainly two reasons.

• First, the use of synthetic data allows our methodology to be scalable in terms of accessing larger training data sizes. Moreover, it offers flexibility in training a long-context model in a specific domain where there may be a scarcity of manually annotated QA pairs. Leveraging synthetic training data has recently become a prevalent data construction strategy for training LLMs [4,5].
• Second, as we have taken many existing QA datasets in our evaluations (including both long-context and short-context datasets), by utilizing synthetic data exclusively, we can circumvent potential data contamination issues between training and evaluation.

Q2: Would IN2 training be beneficial to extend the context length?

A2: That's an insightful question. We think the answer is Yes, and we have some preliminary results and observations: applying IN2 training on a 32K context window and subsequently extending it to 64K with YaRN (without fine-tuning) still yields improvements (as illustrated in Figure 8 of our Appendix). These results may not be as robust as within the 32K context, but we believe that incorporating fine-tuning on the 64K context window could further enhance the performance.

[1] https://huggingface.co/datasets/polinaeterna/OpenOrca.

[2] InternLM2 Technical Report

[3] Qwen2 Technical Report

[4] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

[5] The Llama 3 Herd of Models

Thanks for your insightful questions about our work. The following are our responses to these questions.

Q1: What are the reasons for using synthetic data rather than existing natural long context datasets during fine-tuning?

A1: There are mainly two reasons.

• First, many existing natural long context datasets are often purposed for evaluation rather than training, including datasets such as L-EVAL [1], LongBench [2], and SCOLLS [3]. Therefore, by utilizing synthetic data exclusively, we can circumvent potential data contamination issues between training and evaluation.
• Second, the use of synthetic data allows our methodology to be scalable in terms of accessing larger training data sizes. Moreover, it offers flexibility in training a long-context model in a specific domain where there may be a scarcity of manually annotated QA pairs. Leveraging synthetic training data has recently become a prevalent data construction strategy for training LLMs [4,5].

Q2: What is the context we use to mix 128-token segments with the gold answer?

A2: When selecting the filler segments $[r_j]$ to create a lengthy context, we want that $[r_j]$ bears no connection to the QA pair. This way, to respond to the given question, the model will be trained to carefully find out and utilize the related $s_i$ (or $[s_i]$) from a large amount of redundant information. For implementation, we randomly sample $[r_j]$ from a large candidate pool where $s_i$ (or $[s_i]$) are excluded. We have manually inspected 40 synthetic cases to verify that this random selection generally meets our expectations of unrelatedness.

We did not use the similarity-based selection strategy due to the following concerns:

• Similar segments could have supporting information for the QA pairs, creating a shortcut during training. Given that $[r_j]$ could also provide useful information to answer the question, the model may just use the $[r_j]$ at the beginning and end of the context, without thoroughly searching for the $s_i$ in the middle.
• Similar segments might contain information that could lead to ambiguity in answering the question (for instance, different individuals with the same name). This ambiguity could potentially affect our training.

It is a valuable research question to explore how to properly incorporate the similarity-based selection with our IN2 training. Mitigating the above issues requires further designs and ablations, such as carefully controlling the mixture proportions, designing and adjusting the rule-based/embedding-based filters, and also incorporating human check if necessary. Given the high training cost, we consider the use and ablation of similarity-based selection for future work.

Q3: Require further analysis on data quality and fine-tuning approach.

A3:

• For the data quality, a manual assessment of 40 synthetic examples was conducted prior to large-scale data generation (20 for local information QA and 20 for multi-hop QA). Some of these examples are included in Appendix H. We find that GPT-4 is highly capable of generating QA pairs that are specifically related to the information provided in the given contexts. To generate multi-hop QA pairs, a simple prompting strategy was employed that utilizes ** to emphasize the need for "using at least two segments" to generate the QA pair (refer to Appendix I). Using this prompt, all 20 sampled multi-hop QA pairs have successfully incorporated information from at least two segments.

• For the fine-tuning approach, we take the standard instruction-tuning approach for fine-tuning, which involves updating the model parameters only based on the loss on the output side. We do not include the input-side loss on the synthesized long contexts because we are concerned that it will dominate the training process and interfere with natural language modeling. For hyper-parameters, we initially tried four learning rates (1e-5, 5e-6, 1e-6 and 5e-7) in the first 100 training steps. We chose the 1e-6 learning rate as it exhibited the best loss trend in the first 100 training steps. Other hyper-parameters in Section 2.2 follow the general settings for post-training.

Q4: Why fine-tuning on QA-style data can enhance performance in other long-context tasks such as summarization?

A4: Although the surface forms differ across various tasks, we believe that the underlying capability to comprehend long-context information is transferable. Our further evaluation results on few-shot learning tasks can also demonstrate this transferability. These tasks include TREC (few-shot classification), TriviaQA (few-shot QA) and SAMSum (few-shot summarization). Although the surface form of few-shot learning is quite different from our zero-shot training data, these results further support that an improved capability to perceive long-context information can be generalized beyond surface forms.

Model	TREC	TriviaQA	SAMSum	Avg.
GPT4-Turbo	77.0	91.7	39.7	69.5
Mistral	71.0	84.5	35.8	63.8
FILM (ours)	76.0	90.0	39.5	68.5

Q5: What are bold in Table 2?

A5: The bold numbers are SOTA-level results among ~7B open-source models. Specifically, for each task, we bold the highest result and the results within a margin to the highest one (0.5 for summarization tasks and 2.0 for others).

[1] L-Eval: Instituting Standardized Evaluation for Long Context Language Models

[2] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

[3] SCROLLS: Standardized CompaRison Over Long Language Sequences

[4] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

[5] The Llama 3 Herd of Models