Why Does the Effective Context Length of LLMs Fall Short?

Thank you for a detailed review and valuable suggestions. In response to your questions, our answers are as follows.

Q1: Where to find our implementation of STRING.

A1: The link in our manuscript is a placeholder, and it will be replaced in our camera-ready version. The pytorch code of STRING can be found in the submitted supplementary materials. We have also created an anonymous GitHub page detailing our implementation to ensure transparency and reproducibility.

Q2: Show the variability in the results.

A2: All the experiments run in our paper use greedy search following previous work, and there is no training procedure involved. To show the error bar, we use a top-p sampling strategy with top-p set to 0.7.

We run the model with top-p decoding 3 times denoted as sampling-1, sampling-2, sampling-3 and the results are shown in this table.

Q3: Whether the same findings apply to medium- and large-sized LLMs

A3: We agree that conducting experiments with larger models at scales of 7B or 70B parameters could enhance the robustness of our results. However, a 1.3B parameter model is the largest size that our current resources can support for pretraining from scratch. In fact, most models pretrained with open-source corpus from the community are usually at the scale of GPT-2 or TinyLlama. Section 3 is also named as a probing experiment to avoid any overclaims. The analysis of position frequency in large language models in should be conducted during the pretraining phase and we are unable to use models from big companies. Despite utilizing many acceleration strategies following the popular project TinyLlama, we still require one month to train each model with 16 NVIDIA A100 GPUs .

For large-sized models, we demonstrate the importance of position frequency by results in Section 4: by abandoning the use of infrequent positions for gathering distant information, large-sized models like Llama3.1 70B and Qwen2 72B can be greatly improved. This also indicates that the long-context ability of large-sized models is also hindered by infrequent positions.

Q4: More insights about STRING and potential limitations of STRING.

A4: The core idea of STRING is to use well-trained positions to gather distant information while maintaining the closest distance to neighboring tokens. In the original ROPE, distant information is gathered by the most infrequent positions, causing the model to overemphasize local dependency rather than long-range dependency. For example, in long-context QA, each token needs to focus on its neighboring tokens to ensure the fluency, so tokens with a distance of 1 are always more important than tokens with a distance of 1K. However, tokens at a distance of 1K are not necessarily more important than those at a distance of 100K. It is possible that the tokens at a distance of 100K are precisely the ones that can be used to infer the answer. Compared to ROPE, STRING is more conducive to gathering such information.

A potential bias of STRING is introducing non-consecutive position indices. Taking the position matrix of Llama3.1 as an example,

the (42K-1)-th diagonal is filled with 42K-1 but the 42K-th diagonal is filled with 128. However, we have conducted extensive experiments to verify STRING on 22 synthetic and real-world sub-tasks from RULER and InfiniteBench, and we have not observed any tasks where STRING fails to outperform ROPE. This may suggest that the benefits brought by STRING for current LLMs greatly outweigh the potential bias it may cause.

Dear Reviewer QJGa,

Thank you sincerely for taking the time and effort to review our submission.

Below, we address the points you raised:

• The GitHub link currently included in our manuscript serves as a temporary placeholder and will be updated in the final camera-ready version of our paper.
• We have added error bars to the tables in our experimental section to better illustrate the variability of our results.
• We have expanded the discussion on STRING, providing deeper insights and outlining its potential limitations.
• Our current computational capacity is limited to 16 NVIDIA A100 GPUs and it took us about 2 months to pretrain these models from scratch. Larger models developed by big companies, such as Llama2 7B, require significantly more resources (e.g., Llama2 7B demands about 3.3 million GPU hours). However, the effect caused by infrequent positions on large-sized models can be demonstrated by the improvements from STRING.

As the public discussion phase comes to an end, please let us know if there are any additional questions or concerns we can address to assist with your review. We are more than willing to provide any further information or clarification you might need.

Thank you once again for your valuable feedback.

We are grateful for your valuable review and thoughtful suggestions, and we try to answer your questions as follows.

Q1: Observation (1) in Section 3 (line 216) is overstated.

A1: We double-checked our paper and found no claims about training efficiency. The original claim in line 216 is 'Larger training context window consumes fewer tokens to achieve the same effective context length' and we did not make claims like 'Larger training context window achieves the same effective context length with better training efficiency'. The number of consumed tokens is closely related to the position frequency studied in this work. We have added the following sentence in our updated manuscript (line 221): 'But it does not mean larger context training window can save more computation' for better clarity.

Q2: STRING is not a foundational solution.

A2: We appreciate the concern that STRING may not be classified as a foundational or final solution. However, the primary objective of our work is to highlight a critical issue that contributes to the limited long-context capabilities of current large language models (LLMs).

We introduce STRING with two main goals:

• STRING is designed as a plugin that can be seamlessly integrated into existing models, delivering significant improvements in their long-context handling abilities.
• By showcasing STRING's effectiveness, we aim to illustrate the substantial potential in addressing the identified problem.

We hope that the community can make significant strides in enhancing the long-context processing capabilities of LLMs inspired by our analysis and the design of STRING.

Q3: Extrapolation experiments

A3: Thank you for your suggestions and we agree that the extrapolation results provide a valuable dimension to our method. Positions exceed the training length can be mapped to the trained interval with STRING.

We verify the extrapolation version of STRING with language modeling and NIAH (4-needle). The test length is set to 1.0x training length to 2.5x training length. STRING achieves a better performance than popular extrapolation methods: NTK, YaRN and ReRoPE. The detailed results are shown in the following tables.

PG19 Language modeling

Needle-in-a-haystack

Thank you for the clarifications and for your efforts on the extrapolation experiments. My concerns are resolved, and I believe STRING will provide valuable insights for future improvements on position embeddings.

Thank you for a detailed review and valuable suggestions!

Q1: Automatically setting S and W.

A1: Thank you for the suggestion! In our ablation study (see Figure 7), we find that the performance variation remains minimal when W ranges from 32 to 512. Automatically setting W may not significantly impact the results. However, for S, its value is influenced by the number of well-trained positions in the model. In our experiments, we set S to $\frac{1}{3}$ of the training length for all tasks and models to facilitate reproducibility and follow-up by the community. Automatically setting S could potentially yield better results, such as through a grid search on synthetic data. We plan to explore this in future work.

Q2: Ignoring distant positions during training.

A2: In our training setting, we use Llama2 4K and the latest Llama3.1 128K as the base model. For Llama2 4K, we use conversations from ShareGPT and set the training length to 4K. For Llama3.1 128K, we train with a popular long-context SFT dataset, ChatQA-2[1] and set the training length to 128K. We train the models for one epoch.

The table below shows the results on RULER, where -FT indicates the finetuned version of STRING.

The table below shows the results on RULER, where STRING-FT indicates the finetuned version of STRING. For Llama2, our test length is set to 4K, and for Llama3.1, it was 128K. The results show that continual training can lead to more performance gain. On Llama2 4K, the improvements are significant, but on Llama3.1, the benefits are less pronounced, possibly due to data quality issues.

In addition to simplicity and flexibility, our primary reason for not including the training settings in the initial version is to exclude the impact from data. The training-free improvements are clearly attributed to the exclusion of infrequent positions rather than the use of additional training data.

[1] ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities

Q3: Potential side-effects caused by the re-positioning on short-context benchmarks

A3 Part-I: STRING will produce the same results as RoPE on short-context benchmark.

Considering the Pseudocode Code for Llama3.1 128K in our implementation:

l = input_length 

L = 128K # training length
S = 42K # L // 3, shifted size

diag_attn = flash_attn(q, k, v, sliding_window=S)
if l <= S:    # for short context, l<= S is always True
    return diag_attn
else:
    do shifted self-attention
    merge sliding window attention and self-attention
    return merged_attn

When the input length $l < S$, the sliding window attention calculates attention within the neighboring $S$ tokens. In the short-context setting, we always have $S \gg l$, and the sliding window attention calculates full attention for the short sequence without shifting position indices. Results on the long-context coding dataset code.debug which requires the model to adhere to syntax are shown in Table 3 of our manuscript. STRING is able to yield more than a 3-point score improvement on Llama3.1 8B/70B.

A3 Part-II: additional discussion.

Since short-context input does not involve infrequent positions, there is also no motivation to re-order positions in this scenario. However, we agree that studying the side effects of re-positioning on short-text tasks is a very interesting idea.

We update the code so that the shifted size S is now determined by the input length rather than the training length.

l = input_length # can be a small value

S = l//3  #  shifted size determined by input length
N = 2*l//3

diag_attn = flash_attn(q, k, v, sliding_window=S) # l // 3 
q = apply_shifted_pos(q) # apply shifted position ids for q
shifted_attn = flash_attn(q[-N], K[:N], V[:N]) # shifted-self-attention (the bottom-left triangle)

output = merge(diag_attn, shifted_attn)

If we still keep W = 128, it will result in the following position matrix:

When the input length $l=384$, we have $\frac{l}{3}=128$, which has the same effect as RoPE. Whether $\frac{l}{3}<128$ or $\frac{l}{3}>128$, re-positioning occurs. We study the side-effect of the re-positioning with Llama2 and Llama3.1 on HumanEval and GSM8K.

GSM8K

HumanEval

Overall when $\frac{l}{3}<128$, the performance decline is most noticeable and as the sequence length increases, the negative effect of re-positioning becomes smaller. Only in the GSM8K with 4-shot and 16-shot settings, re-positioning shows a slight advantage. Compared with math problems, in coding tasks, using re-positioning often leads to worse outcomes as it may disrupt code syntax as you mentioned.

Q4: Latency overheads.

A4: We show the inference speed and GPU memory usage of STRING and RoPE in Appendix Figure 9. The throughput and GPU memory consumption are negligible compared to RoPE. The current implementation of STRING is based on calling Flash-Attention twice, and there is room for optimization from the kernel.

Truncated comment: Thank you again for your kind words and interesting insights. We are pleased to hear that you enjoyed reading our paper. It seems that Weakness 1 was truncated in OpenReview. Could you please repeat it in a later comment?

We appreciate your thorough review as well as insightful comments, and try to answer your questions as follows.

Q1: Detailed explanation of how we protect local relationships.

A1: To illustrate this, consider the relative matrix of Llama3.1 128K,

where the value on the 42K-th diagonal is set to W instead of 42K (using ROPE) to achieve a shifting effect. This means each token treats the token 42K positions away as if it's at a distance of W. In an extreme case, if W=0, the current token would treat the token 42K positions away as itself. As we know, LLMs rely heavily on neighboring tokens to maintain fluency. If W is too small, the model can not correctly focus on these nearby tokens. By setting W=128, we ensure that relative positions 0-127 are unique and assigned to neighboring tokens. This maintains fluency while enhancing long-context performance.

Q2: The impact of altering the relative positions between neighboring tokens.

A2: As explained in A1, when W is set to a small value, it disrupts the relative positions of neighboring tokens. We investigate this effect by testing three versions of Llama with different training lengths, and we vary W from 0 to 128. The table below shows the results on the NIAH task:

In this table, we observe that the nearest 8 tokens are the most crucial for fluent generation. When W > 8, there is no significant degradation. Setting W to a larger value, such as 128, usually results in more stable outcomes.

Dear Reviewer SwNh,

We sincerely appreciate the time and effort you have devoted to providing thoughtful and constructive feedback on our work.

Here is a summary of our response:

• A detailed explanation of how the local value W=128 works and why positions 0~127 are only assigned to neighboring tokens.
• When distant tokens are assigned relative positions less than 4, the model will collapse.
• STRING produces the same results as RoPE on short-context input because only the sliding window attention function is invoked.
• Side effects caused by re-positioning on math and coding tasks.

As the public discussion phase is nearing its conclusion, we wanted to kindly follow up to see if there are any additional questions, concerns, or points that we could clarify or address to further assist with your review process. We are more than happy to provide any additional information or details you might need.

Thank you once again for your valuable comments.