Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

We sincerely appreciate Reviewer LVQT for supporting our work and providing constructive suggestions. To address Reviewer LVQT’s concerns, we provide point-wise responses below.

[Q1: No Lost-in-the-Middle Effect] Thank you for the insightful question. We have observed similar findings to those in [1], particularly with LLaMA-2-7B-Chat, which does not exhibit the lost-in-the-middle phenomenon but rather shows performance loss at the beginning. The results are reported in Table R1. Additionally, after applying Ms-PoE, consistent improvements are achieved by an average of 1.5 accuracy. This further demonstrates the effectiveness of our approach for enhancing the context utilization of LLM inference. We’ve added the results in the updated draft.

[1] ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models.

Table R1. Comparison results of LLaMA-2-7B-Chat with and without Ms-PoE on the MDQA tasks.

[Q2: More Benchmark Results] Thanks for the suggestions. We conducted additional experiments on all 13 tasks from the LongBench benchmark for further evaluation, and the results are presented in Table R2. Our findings show that Ms-PoE consistently demonstrated effectiveness across all 13 tasks, with improvements reaching up to 6.67 and an average of 2.2. It is important to note that we fixed $R_{min}=1.2$ and $R_{max}=1.8$ for all tasks without tuning on the test set.

Table R2. Comparison results of LLaMA-2-7B-Chat with and without Ms-PoE on the LongBench benchmark. We use the same scaling ratios without further turning.

[Q3: Determination of $R_{min}$ and $R_{max}$] Thanks for the question, we determine the $R_{min}$ and $R_{max}$ via an ablation study on the MDQA task. Specifically, we randomly selected 500 samples from MDQA tasks as the validation set and examined the effect of different scaling ratios, the results are reported in Table 4 where $R_{min}=1.2$ and $R_{max}=1.8$ demonstrates superior performance. We then apply the same ratios to other downstream tasks without further adjustment.

[Q5: Welch’s t-test of Table 3] Good suggestion. We conducted a further statistical analysis of the results presented in Table R3. As illustrated in Table R3, the baseline method shows a significant difference in performance when critical documents are positioned in the middle versus the beginning of the inputs, as well as between the middle and the end. However, the difference between the beginning and the end is not significant (p-value > 0.05). In contrast, when applying our Ms-PoE method, there is no significant difference in performance between the beginning, middle, and end positions, and it also achieves better average performance. These findings further confirm the effectiveness of our approach, and we have included these results in the updated manuscript.

Table R3. Welch’s t-test of the results when different ordering metrics are applied. The Null hypothesis states that there is no difference between the means of two results.

We appreciate Reviewer xsrg for acknowledging our method is “efficient”, and the observation of attention heads is “insightful”. To address Reviewer xsrg’s concerns, we provide pointwise responses in the following.

[Q1: Limited Improvements] We respectfully disagree that our performance improvement is limited compared to “Self-Extend.” As demonstrated in Table 2, “Self-Extend” achieves similar performance to direct positional interpolation, or “Single-Scale Positional Encoding,” with gains of 1.20 and 29.76 on the MDQA and KV Retrieval tasks, respectively. In contrast, our Multi-Scale Positional Encoding achieves significantly higher gains of 3.92 and 43.72 on the same tasks, respectively, clearly surpassing the improvements achieved by “Self-Extend.”

[Q2: Ablation Studies of Context Lengths] That’s a good suggestion. We conducted additional evaluations of our method across varying input lengths by changing the number of documents from 3 to 15. In these experiments, the key document is consistently positioned in the middle of the input to assess the LLMs' ability to capture middle context information. As depicted in Table R1, when the input length is short, our approach shows negligible improvement compared to the Baseline, with a 0.2 accuracy gain. However, as we increase the number of documents, thereby lengthening the input, significant improvements are achieved, with gains up to 6.4 in accuracy.

Table R1: Results of the effectiveness of our approach across different input lengths. The relevant key documents are located in the middle of the inputs and experiments are conducted with Vicuna-7B

[Q3: Results on LongBench-EN Benchmark] Thanks for the question. We conduct additional experiments of all 13 tasks from the LongBench-EN benchmark. Results are reported in Table R3. We can observe that Ms-PoE achieves consistent performance improvement without any finetuning.

Table R3. Comparison results of LLaMA-2-7B-Chat with and without Ms-PoE on the LongBench benchmark. We use the same scaling ratios without further turning.

[Q4: Determination of $R_{min}$ and $R_{max}$] Thanks for the question, we determine the $R_{min}$ and $R_{max}$ via an ablation study on the MDQA task. Specifically, we randomly selected 500 samples from MDQA tasks as the validation set and examined the effect of different scaling ratios, the results are reported in Table 4. We then apply the same ratios to other downstream tasks without further adjustment.

We thank Reviewer qA6u for acknowledging our work as “interesting and intriguing”. We provide pointwise responses in the following.

[Q1: Limited Improvements] We respectfully disagree with the claim that our improvements are limited. Our method offers a plug-and-play solution to enhance current open-source LLMs without any fine-tuning. And the enhancements are consistent across multiple benchmarks, including a 3.8-point average improvement on ZeroSCROLLS (Table 1), a 3.92-point increase in Multi-Document QA tasks, and a 43.72-point gain in KV Retrieval tasks (Table 2). Additionally, as shown in Table R1 in the uploaded PDF, our evaluations on the LongBench benchmark further demonstrate a consistent improvement of up to 6.67 points. Therefore, we believe our improvements are both consistent and significant.

[Q2: Studies of Scaling Ratio Allocation] Thank you for the insightful comments. In our experiments, we chose the heuristic linear (arithmetic sequence) rescaling ratio due to its simplicity and input-independent nature. We can also rescale each attention head using other strategies, such as directly via the position-aware score ($S_P$) while it introduces additional computational costs. Since when implementing a scaling ratio based on the position-aware score, each input would have distinct rescaling ratio values. This would necessitate recalculating the $\theta$ value in RoPE on the fly, leading to additional computational overhead. Additionally, we can also allow different heads to share the same scaling ratio.

To further address Reviewer qA6u’s concerns, we explored multiple scaling ratio allocation strategies and reported the results in Table R2. For the exponential and cosine strategies, we aimed to examine whether non-linear allocation strategies perform well. For the stepwise solution, we allowed different attention heads to share the same scaling ratio, and the position-aware score provided evidence for rescaling directly based on its raw scores. We observed that the position-aware strategy achieves similar results to the linear strategy but requires extra computational overhead. Similarly, allowing different attention heads to share the same scaling ratio yields comparable results. Therefore, based on the superior performance and computational efficiency, we chose the linear assignment method.

Details: We compare several strategies in our experiments, including (i) Exponential: $r_i = 1.8 - (1.8-1.2) \cdot 0.9^{i}$, where $r_i$ is the i-th scaling ratio and 0.9 is the multiplication factor for scaling ratio decay. (ii) Cosine: $r_i = 1.8 - (1.8-1.2)cos(\frac{i}{n_h-1}\phi)$, where $n_h$ is the number of attention heads. (iii) Stepwise: $r_i = 1.2 + \frac{1.8-1.2}{3} round (\frac{i*4}{n_h})$, (iv) Position-Aware score: $r = 1.2 + \frac{(1.8-1.2)\cdot(S_P-min(S_P))}{max(S_P)-min(S_P)}$, as well as the Linear strategy in our current implementation.

Table R2. Results of different scaling ratio strategies on the MDQA task with Vicuna-7B.

[Q3: Visualization of Attention Pattern before and after Rescaling] Thank you for the suggestion. We visualized the attention patterns of different attention heads and reported the variation of $S_P$ before and after rescaling. The results are demonstrated in Figures R1 in the uploaded PDF. We found that the rescaling step effectively enhances the context utilization of LLMs, making some non-"position-aware" heads focus on critical information. Quantitatively, $S_P$ consistently increases with an average improvement from 1.79% to 1.93%.

[Q4: Determination of $R_{min}$ and $R_{max}$] Thanks for pointing it out. We’d like to clarify that we didn’t search the best hyperparameter of $R_{min}$ and $R_{max}$ on the test set. Instead, we randomly selected 500 samples from MDQA tasks as the validation set and did the ablation studies of different scaling ratios on this validation set. We found that selecting a ratio between 1.2 and 1.8 significantly boosts the performance. Then. we apply the same ratios to all other tasks without further adjustment.

[Q4: Scaling Ratio Affects the Favored Zone] Thanks. We report the raw accuracy results corresponding to Figure 3 in Table R3 where we position the key documents at the beginning, middle, or end of the sequences. From Table R3, we can observe that “changing the scaling ratio also affects the favored zone of LLMs. With a small scaling ratio (e.g., 0.5), LLMs tend to focus more on the most recent part of the context, while with a large ratio (e.g., 2.5), LLMs favor the beginning part”. We have included this table in the revised version for a clearer understanding of the content discussed in Lines 212-215.

Table R3: Accuracy results for the MDQA task when key documents are positioned at various locations within the sequences. Different rescaling ratios are applied, with all attention heads sharing the same rescaling ratio.