Thank you for your insightful comments!

Q1: Training from Scratch

We initially performed from-scratch pretraining on smaller models and found that the properties of the positional vectors were largely similar to continually-trained models, but the models trained from scratch had inferior performance. To address your question, we pretrained TinyLlama from scratch with the same configuration, but using different positional encodings and attention patterns. Due to time and resource constraints, we only trained TL-NoPE-new for 50B tokens, while the other configurations were trained for only 5B tokens.

• Formation of positional vectors for TL-NoPE: For the TL-NoPE-new trained from scratch, the formation of its positional vectors is similar to the continually trained one: the initial tokens exhibit distinct positional information after the first layer (as shown in Figure 1 in the PDF). As the table below shows, removing the positional information of the initial tokens significantly harms the performance, indicating that the flow of this positional information facilitates the formation of subsequent tokens' positional information.

• Formation of positional vector for TL-NoPE-Window-new: For window attention with NoPE, the positional information flow from initial tokens to subsequent tokens also occurs, and the distinct positional vectors gradually propagate across both windows and layers. The distinct positional vectors of the initial layers are shown in the table below.

• Effect of positional vectors on attention: we subsequently remove different components from the query and keys (shown in Figure 2 in PDF), and observe that after removing the positional vectors or positional basis, the long-term decay and attention sink all disappear.
• Positional vectors change beyond the context window: as shown in the following table, with inputs exceeding the context window (2K), only models with consistent positional vectors exhibit length extrapolation capacities. In addition, when we employ attention scaling on TL-NoPE, its PPL score on 8K length is 14.05, slightly larger than 11.75 in the 2K length of the original model. The similarity between the original and extended models is 0.97, indicating the interpolation of positional vectors.

• Effect of OOD Positional Vectors: similarly, the attention sinks and logits of positional vectors of TL-NoPE-new also change after exceeding the context window.

• Proposed Methods: Finally, we test our proposed methods with the from-scratch trained TL-NoPE-new and TL-NoPE-window-new, and observe similar phenomenons. Both positional vector replacement and attention window extension can achieve context window extension. However, due to the properties of these models, the hyper-parameters and the original performance are different.

Q2: Absence of Needle in the Haystack

We initially explored the "needle in a haystack" task. However, due to the poor performance of TinyLlama, the models we continued to pre-train exhibited diminished retrieval capabilities. In our tests, even with an input text length of 200 tokens, the models failed to generate accurate responses, regardless of the positional embeddings and attention patterns employed. Given the difficulty of the "needle in a haystack" task, we did not include the outputs in our paper.

Q3: How to Decide Values of Alpha and Lambda

For the alpha value in positional vector replacement, we conducted experiments in Appendix E according to the effective interpolation ratio and PPL. We tested different combinations and identified the optimal interpolation ratio, replacement layer, and interpolation times (alpha). For lambda in the attention window extension, we similarly used a PPL-based search to determine the scaling factor. We will further point out how to decide these hyper-parameters and add citations in the final version of our paper.

Thank you for your response and valuable suggestion!

To assess the long-context utilization capability, we have adopted your recommendation and evaluated the LLMs using ICL. Specifically, we randomly sampled consecutive substrings from the RedPajama dataset, with lengths of 20, 50, and 200 tokens, respectively. These substrings were then repeated to achieve varying lengths, and we evaluated the accuracy of the models' ability to repeat the first tokens of substrings correctly. The results are presented in the following table.

After exceeding the context window, the performance of models deteriorates significantly, whereas our methods maintain a degree of capability to utilize longer contexts effectively. Specifically, for TL-Window-NoPE, as the substring length increases, the model exhibits slower degradation of performances. For TL-NoPE, with much longer substrings (for example, 200 tokens), the model even demonstrates superior performance at 8K tokens compared to 2K tokens. These enhancements underscore that our models can leverage longer contexts for enhanced performance.

Thank you for your helpful comments! We will revise Figure 4 in the final version.

Thank you for your insightful comments!

W1: Unnecessaries of Section 4

In Section 4, the proposed methods are significant evidence for our analysis of the relationship between positional vectors and the context window. Our experiments substantiate our previous viewpoints. For instance, interpolating positional vectors can extend the context window to some extent. Although it has poorer performance than Dynamic NTK, it provides a new approach to solving the problem of context window extension and can be applied to models without RoPE. In the final version with one additional page, we will adopt your suggestions by supplementing more experiments and condensing the content of this section to some extent.

W2: Failure of Window Attentions in Needle in the Haystack

Why can window attention hardly solve the needle in the haystack problem?

• Some work has reported that models with full attention can utilize "retrieval heads" to directly retrieve critical information from the needle [1], while window attention cannot directly attend to these tokens.

• Models with window attention acquire knowledge only through the indirect transmission of information between windows. However, previous work has highlighted that there is severe attenuation in the information passing between windows, making it difficult to leverage semantic information from distant tokens [2].

• Positional information tends to be a kind of global information at a given position, ensuring the model can produce coherent output, and is orthogonal to the semantic information of the input.

To validate this, we first compare the hidden states of tokens when using the full input sequence versus using only the window-sized input at a time. We test TL-RoPE-Window and Mistral-7B-v0.1 with input lengths of 16K and 32K, respectively. We find that the similarity of the last layer's hidden states under both settings was above 0.98, suggesting that information from tokens outside the window has little impact.

Additionally, we test the scenario where the "needle" is placed at the beginning but is modified into random tokens from the vocabulary. The hidden states of the outputs remain largely unchanged, further supporting our hypothesis.

[1] Retrieval Head Mechanistically Explains Long-Context Factuality

[2] Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis

W3: Absence of Some Mainstream LLMs

Considering resource constraints, we only continuously pre-trained TinyLlama with modified positional encoding under identical settings for a fair comparison. The mainstream models are all based on RoPE, and continuing pre-training is prohibitively expensive; hence, we only examined the positional vectors of the original Llama3-8B, Yi-9B, and Qwen1.5-7B.

• Formation of positional vectors: After the first layer, the initial positions have significantly different positional vectors compared to other tokens, as shown in Figure 1 of the PDF. Furthermore, we remove different components from values at different positions and observe changes in the positional vectors and PPL, as shown in the table below. The findings are consistent with the original paper, indicating that the initial tokens play a critical role in shaping the positional vectors and influencing the model’s PPL. When the positional information of the initial tokens is lost, the model loses its ability to produce coherent output. Moreover, Llama-3-8B and Yi-9B are more sensitive to semantic information than smaller models. | | | original | w/o value || w/o positional vector | | w/o positional basis | | w/o semantic basis | | |---|---|---|---|---|---|---|---|---|---|---| | position | - | - | 0-4 | 32-256 | 0-4 | 32-256 | 0-4 | 32-256 | 0-4 | 32-256 | | Llama-3 | simi | 1 | 0.75 | 1.0 | 0.75 | 0.96 | 0.21 | 1.0 | 0.93 | 0.87 | | Llama-3 | ppl | 6.74 | 16.27 | 6.63 | 17.20 | 8.4 | >1000 | 6.60 | 17.6 | 15.18 | | Yi-9B | simi |1| 0.98 | 1.0 | 0.92 | 1.0 | 0.54 | 1 | 0.91 | 1.0 | | Yi-9B | ppl | 6.51 | 8.03 | 6.56 | 37.92 | 6.62 | >1000 | 6.52 | 42.27 | 7.08 | | Qwen1.5-7B | simi | 1 | 0.98 | 1.0 | 0.98 | 1.0 | 0.74 | 1.0 | 1.0 | 0.99 | | Qwen1.5-7B | ppl | 7.97| 9.51| 8.03 | 9.51 | 8.04 | 217.13 | 7.98 | 8.09 | 8.68 |

• Impact of Positional Vectors on Attention: We examine the changes in attention after removing the positional vectors, as illustrated in Figure 2 of the PDF. We observe that attention sinks and long-term decay are eliminated after removing the positional vectors or basis.

• Effect of Positional Vectors Beyond the Context Window: As shown in the following table, after exceeding the context window, models without an extended context window experience a sharp change in positional vectors and PPL, similar to TL-RoPE. By using dynamic NTK, we achieve interpolation of the positional vectors, maintaining high similarity to the original positional vectors. | model | context window(W) | PPL(W) | PPL(2W) | Simi(2W) | PPL(NTK, 2W) | Simi(NTK, 2W) | |---|---|---|---|---|---|---| | Llama-3-8B | 8192 | 6.74 | >1000 | 0.30 |7.84 | 0.96 | | Yi-9B | 4096 | 6.51 | 102.58 | 0.24 | 6.70 | 0.99 |

• Effect of OOD Positional Vectors: After exceeding the context window, the properties of high attention scores on the first token (attention sinks) in Llama-3 and Yi-9B rapidly disappear, and the logits of the positional vectors differ from those within the window, as shown in the table below. However, the change in logits for Yi is not obvious, although it has formed the same pattern as shown in Figure 5 (right) of the paper. | model | context window (W) | property | 0W | W1.5W | 1.5W~2W | |---|---|---|---|---|---| | Llama-3-8B | 8192 | attention sink | 0.47 | 0.1 | 0.005 | | Llama-3-8B | 8192 | logits similarity | 1 | 0.9 | 0.88 | | Yi-9B | 4096 | attention sink | 0.68 | 0.34 | 0.06 | | Yi-9B | 4096 | logits similarity | 1 | 0.98 | 0.97 |