E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

We are deeply grateful for the reviewer’s valuable comments, which helps enhance the quality of our work. We have crafted detailed responses for each suggestion and incorporated them into the revised manuscript. In our response, comments from the reviewer are highlighted in italics, with our responses directly following. For clarity, quotations from the adjusted manuscript are presented in markdown quotation mode. The corresponding modifications in the paper are highlighted in blue. Unless otherwise stated in our response, all pages, equations, sections, and bibliographical references refer to those in the revised paper.

W1 - The authors seem to overclaim that "E2LLM solves all the challenges of the impossible triangle" since the performance on long-context tasks of E2LLM still falls far behind "continual pertaining on longer texts + long-context SFT" paradigm [1, 2] (though there is no comparison between E2LLM and this approach in the paper), which is the mainstream and SoTA approach now and widely used in most advanced LLMs such as Llama3.1 and GLM4. This makes the real-world applicability of E2LLM doubtful. A comparison between E2LLM and "continual pretraining + SFT" approach should be presented to support the claim.

We would like to argue that most long-context extension methods aim to enhance the context window with minimal to no fine-tuning, as explicitly stated in studies like Position Interpolation (Chen et al. (2023a)) and YaRN (Peng et al., 2023). This is consistent with E2LLM’s goal of compatibility with pretrained models. As discussed in Section 1 (Lines 041-044),

Achieving this compatibility is crucial for effectively leveraging the pretrained knowledge contained in these models, allowing for **parameter and sample efficiency without necessitating extensive additional training with large datasets.

This rationale underpins our decision not to focus on continue-pretraining (CPT) plus supervivsed fine tuning (SFT) in the first place, as it deviates from the original motivation behind long-context extension methods. Our core aim is to demonstrate that, given an equivalent training dataset, E2LLM—being tuned with fewer parameters—can outperform baseline methods that necessitate more extensive tuning. This assertion is validated in Section 4 (see Tables 1-2 and Figure 3), where E2LLM effectively harnesses the knowledge embedded in the pretrained text encoder and LLM decoder.

On the other hand, we acknowledge that CPT plus SFT is a prevalent practice to achieve practical long-context LLMs, as seen in the Llama and Qwen series. However, due to resource constraints and time limitations, we could not conduct CPT plus SFT using the E2LLM framework on a substantial amount of high-quality long-context data. Instead, we provide a discussion in Appendix I.1 (Lines 1303-1323 on Page 25) regarding how CPT can be conducted within the E2LLM framework as follows:

While E2LLM demonstrates comparable or superior performance compared to various baseline methods when utilizing the same amount of fine-tuning data, it has become increasingly common to engage in continue-pretraining (CPT) alongside supervised fine-tuning (SFT). This approach typically involves leveraging substantial quantities of high-quality, long-context data. By doing so, systems can achieve a long-text language model (LLM) that exhibits versatility across diverse long-text tasks. This practice has been effectively illustrated by recent models in the Llama and Qwen series (Dubey et al., 2024; Yang et al., 2024; Hui et al., 2024).

In this context, we will explore the methodology for conducting CPT within the E2LLM framework. Given that E2LLM functions akin to an encoder-decoder architecture, it is logical to adopt pretraining tasks prevalent in other established encoder-decoder frameworks, such as T5 (Raffel et al., 2019), BART (Lewis et al., 2020), and GLM (Du et al., 2022). However, considering the unique characteristics of the E2LLM model, we specifically recommend the use of prefix language modeling (PLM) (Wang et al., 2022) as the pretraining task. This choice ensures that the CPT process aligns seamlessly with the subsequent SFT process.

Concretely, suppose that the chunk size of the text encoder is $C$ and the length of the decoder is $L$. In this setup, we can create random pretraining sequences with lengths of $\ell C + L - \ell$, where $1 \leq \ell \leq L-1$ represents the number of chunks. We then can partition the prefix segments of length $\ell C$ into $\ell$ individual chunks, which are then fed into the text encoder. The task for E2LLM during this CPT phase is to predict the remaining segments of length $L - \ell$. This structured approach not only enhances the model's ability to comprehend and generate lengthy texts but also sets a solid foundation for effective fine-tuning on targeted applications thereafter.

Q2 - Can E2LLM effectively harness the longer context windows? In other words, does the PPL/performance of E2LLM continue decreasing/increasing when the context window becomes longer?

Thank you for your valuable comment.

In response, we have investigated the sensitivity of E2LLM's performance to variations in context length for the QMsum and NarrativeQA datasets in Appendix C (Page 18, Lines 956-960) as follows:

Next, we investigate the sensitivity of the models' performance to variations in context length. To do this, we categorize samples from the QMsum and NarrativeQA datasets into five groups based on their context lengths and then evaluate the perplexity (PPL) of the answers within each group. Our findings are summarized in Table 6.

Table 6-1: Performance as a function of context length. The best results are in bold, the second are underlined, and the third are wavy underlined.

Table 6-2: Performance as a function of context length. The best results are in bold, the second are underlined, and the third are wavy underlined.

The results presented in the table indicate that E2LLM demonstrates a strong resilience to variations in context length for both summarization (QMsum) and question-answering (NarrativeQA) tasks, typically achieving the best results among all models. This robustness can be attributed to the "understanding" task incorporated during the training of E2LLM (see Section 3.2). By reconstructing different parts of the context, E2LLM effectively comprehends the information, regardless of its length.

Q1 - What is the maximum compression rate of E2LLM such that the context can be recovered losslessly?

First, we would like to clarify that the reconstruction or the "understanding" task is an auxiliary task in E2LLM. Our main focus is on the "reasoning" task such as document summarization and QA. To make this distinction clear, we have elaborated on it in Section 3.2 (Lines 264-269, Page 5):

It is important to note that the “understanding” task serves as an auxiliary task, while our primary focus remains on the “reasoning” task. We determine the final checkpoints exclusively based on the validation loss associated with the “reasoning” task. In this context, we do not anticipate that E2LLM can achieve lossless compression of the context. However, we believe that the LLM decoder is capable of retaining or comprehending essential information from the context. The LLM operates as a "suggestion feature" for input methods, leveraging hints to generate meaningful responses. In this case, the chunk tokens provided by the text encoder serve as these essential hints.**

On the other hand, we acknowledge that it is interesting to check whether E2LLM can stably recover the context. To this end, we conduct an experiment using QMsum training data with chunk sizes of 128, 512, 1024, and 2048. We evaluated the Precision, Recall, and F1-score of the reconstructed sequences across various lengths (100, 500, 1000, 2000, 3000 tokens) in relation to the ground truth. The results of this evaluation are summarized in the following Table.

Our findings indicate that Precision consistently increases with the length of the reconstructed sequence, while Recall gradually declines, irrespective of the chunk size. The F1 score demonstrates an initial increase with longer sequence lengths, followed by a decline. This trend can be attributed to the fact that the reconstruction task is generally oriented toward reconstructing sequences of approximately 3000 characters (or 500-600 tokens) during training. During inference, if the length of the true sequence is shorter than 500 tokens, the generated sequence tends to be longer than the ground truth. Conversely, when the true sequence exceeds 600 tokens, the generated output tends to be shorter than the ground truth. As a result, the generated sequences often introduce additional tokens when the true sequence is shorter than 500 tokens, leading to lower Precision but relatively higher Recall. In contrast, when the true sequence is longer than 600 tokens, the model struggles to capture all tokens from the ground truth, resulting in higher Precision but lower Recall.

Moreover, we can find that optimal results are achieved with a chunk size of 1024, which aligns with the observations in our ablation study regarding chunk size (Appendix H.2, Page 24-25, Lines 1288-1297).

W2 - The figure of Needle-in-a-Haystack is deceptive since the color is all green (the score less than 5 should be marked in red). Though outperforms the baseline methods, E2LLM still struggles in recalling the needle with only 30% success rate， while current long-context LLMs can achieve nearly 100% performance.

In the original version, we use a green-toned color bar to represent accuracy levels, with varying shades indicating different accuracy values. Following your suggestion, we have updated the color bar to a green-to-red gradient, which enhances the intuitiveness of the visualization. The revised figure can be found in Figure 3 on Page 9.

Regarding the accuracy on the Needle In A Haystack (NIAH) task, we believe it is influenced by several factors, such as whether large-scale continual pretraining has been conducted and the parameter size of the model. Both factors directly impact the performance of LLMs on NIAH tasks. In our experiments, neither our method nor the baselines undergo large-scale continual pretraining. Indeed, we only used 13K samples to train all models. Furthermore, we use the relatively small and limited LLaMA2-7B model as the backbone, which inherently restricts the accuracy achievable, making 100% unattainable in this context.

To clarify this point, we have mentioned the number of samples we used to train all models in Appendix D (Lines 1006-1012) as:

Note that we use all training samples from QMSum, GovReport, Quality, and NarrativeQA, but randomly select 3,000 samples from TriviaQA, as the sample size of this dataset is much larger than that of others, but the average sequence length is the shortest. The total number of training samples is around 13,000.

Notably, the performances of YaRN, LongLoRA, CEPE, RAG, and LongLLMLingua also exhibit insensitivity to context length. On the other hand, LLoCO's performance declines slowly with increasing context length. Finally, streamingLLM and the original Llama2-7B demonstrate sensitivity to context length; streamingLLM loses more information in the middle of the context as length increases due to its specific $\Lambda$-shaped attention mask, while Llama2-7B struggles to handle long contexts altogether, as its maximum length has not been extended.

Dear Reviewer uKEN,

We would like to thank you once again for your constructive feedback on our paper. We have made significant efforts to address the concerns you raised and have revised the manuscript accordingly. Given that the discussion period is nearing its end, we kindly ask if you could review the updated version and let us know if there are any remaining issues or points that require further clarification.

We are more than happy to address any additional questions or concerns you might have.

Thank you for your time and consideration.

Q3 - In line 265, it would be easier to understand if "sequence length" could be replaced by "context window" for the input capacity of the model.

Thank you for your valuable feedback. Following your suggestion, we have replaced "sequence length" and "input length" with "context window" to enhance readability.

W4 - The paper misses some related works. For example, for position embedding extension methods, there have been better methods [1] or improvements to YaRN [2]. For methods similar to CEPE, the concurrent work FocusLLM [3] has a very similar design to E2LLM.

Thank you for pointing this out!

We have cited and discussed CLEX and FocusLLM in our paper in Section 2 (Lines 124-126 and Lines 178-183) as:

Another approach, CLEX (Chen et al., 2024a), replaces manual design with learned scaling factors through neural differential equations, effectively overcoming the limitations inherent in traditional positional extrapolation techniques.

Additionally, we note that, concurrently with our work, FocusLLM (Li et al., 2024b) has also adopted a strategy of chunking long contexts and summarizing each chunk using the hidden states of the local context from all layers of an LLM. These hidden states are concatenated to serve as the key-value cache for the same LLM, providing answers to user queries. From the perspective of E2LLM, FocusLLM essentially employs an LLM as a text encoder, which influences both training and inference efficiency.

Furthermore, we have conducted a comparison of CLEX with E2LLM and other baseline methods. Due to time constraints, we focused our evaluation on the QMsum dataset, following the settings outlined in Section 4.1. The results are summarized in the table below. While CLEX achieves a lower perplexity (PPL), its G-mean performance is inferior to that of E2LLM.

Table R1: Performance of CLEX on QMSum.

Q1 - In line 396, you mentioned "E2LLM addresses these challenges...", which includes the challenge that StreamingLLM requires training. However, E2LLM also requires training to align the encoder, adaptor, and decoder. Please explain this claim.

E2LLM is designed to navigate the complexities of the "impossible triangle," which encompasses high performance, low complexity, and compatibility with pretrained models. While it is true that E2LLM necessitates some level of training to align the encoder, adaptor, and decoder, it is important to emphasize that this training is efficient.

E2LLM is compatible with pretrained text encoders and LLM decoders, allowing us to achieve parameter and sample efficiency without the need for extensive training on large datasets (as noted in Lines 42-43 on Page 1). This efficiency is further supported by the results presented in Table 1 on Page 7, which demonstrate that E2LLM typically achieves superior performance with a reduced number of trainable parameters when trained on the same amount of data.

In contrast, while StreamingLLM does not require training, its performance is generally inferior to that of other methods. Thus, it does not effectively address the challenges posed by the "impossible triangle."

Q2 - In line 404, you mentioned that YaRN faces the challenge that "attention mechanisms can become dispersed in exceedingly long contexts". However, YaRN proposes the dynamic scaling method for the attention scores to alleviate this problem.

We would like to clarify the distinctions between the issues addressed by YaRN and the concept of attention dispersion.

The dynamic scaling method only alleviates abrupt performance degradation when the context length exceeds what was seen during training, and the primary focus of Yarn is to ensure that the position embeddings of nearby tokens remain distinguishable. This approach helps maintain the model's sensitivity to local context.

Conversely, attention dispersion refers to the phenomenon where, as the number of input tokens increases, attention scores for each query become widely dispersed across a larger set of keys. This can result in overlapping keys with different semantic meanings, complicating the model's ability to concentrate on relevant information.

It is important to note that YaRN does not directly address the problem of attention dispersion. Simply distinguishing position embeddings of tokens is not sufficient to resolve this issue, as position embeddings alone do not fully determine the attention scores.

A more straightforward approach for mitigating attention distraction is to reduce the number of input tokens, as pointed out in [R1]. Indeed, E2LLM effectively decreases the number of input tokens through soft prompt compression.

[R1] Deng et al, FltLM: An Intergrated Long-Context Large Language Model for Effective Context Filtering and Understanding, 2024.

We greatly value the reviewer’s constructive feedback, which has been instrumental in refining both the presentation and the substance of our paper. In response, we have carefully addressed each comment and provided a detailed explanation to ensure all concerns are thoroughly resolved. In our response, comments from the reviewer are highlighted in italics, with our responses directly following. For clarity, quotations from the adjusted manuscript are presented in markdown quotation mode. The corresponding modifications in the paper are highlighted in blue. Unless otherwise stated in our response, all pages, equations, sections, and bibliographical references refer to those in the revised paper.

W1 - The authors would like to introduce E2LLM as a general framework that could work with any LLMs, which is intended to show the method's flexibility. However, when compared to similar methods like CEPE, both the encoder and the LoRA rank used are different. This both makes the "Trainable Parameters" in Table 1 incomparable and makes me doubt whether the improvement comes from the more powerful GTE-Large-en encoder or the proposed method.

First, we would like to emphasize the difference in the number of trainable parameters as illustrated in Table 1. CEPE has a significantly higher number of trainable parameters (1.31B) compared to E2LLM (16M). Despite this disparity, E2LLM demonstrates superior performance, highlighting its effectiveness even with fewer parameters. We believe this comparison is valid and underscores the strength of E2LLM.

Second, while the GTE-Large-en encoder is indeed a powerful component of E2LLM, one of the key advantages of our framework lies in its ability to fully leverage existing pretrained text encoders and LLM decoders. As indicated in Section 1 (Lines 041-043), a primary goal of E2LLM is to maintain compatibility with pretrained models (T3).

Achieving this compatibility is crucial for effectively leveraging the pretrained knowledge contained in these models, allowing for parameter and sample efficiency without necessitating extensive additional training with large datasets.

In contrast, CEPE is constrained by its requirement for the encoder to align with the decoder's tokenizer, which limits its ability to directly benefit from pretrained models and necessitates extensive training of the encoder from scratch.

Furthermore, we would like to highlight the performance comparison with another baseline, RAG, which employs the same chunking strategy, the same GTE encoder, and the same LLM decoder as E2LLM. Despite these similarities, our proposed method consistently outperforms RAG, further demonstrating the unique strengths of E2LLM.

In summary, we attribute the success of E2LLM to its flexibility in seamlessly integrating powerful pretrained text encoders and LLM decoders, rather than relying exclusively on LLM decoders.

W2 - It is unsure how the authors trained E2LLM and the baselines on the tasks specified in Appendix C. How were the hyperparameters for each method decided?

Thank you for pointing this out. The parameter settings for E2LLM and the baselines are detailed in Appendix B on Page 15-16, i.e., Overview of Baseline Methods. However, we realize that we have omitted the implementation details of E2LLM's adapter.

We have now included the following description in Appendix B (Page 16-17, Lines 861-863, 874-875):

W3 - The analysis of E2LLM's complexity has flaws. As per Figure 2, the input contains a span (prompt + query) that is directly put into the decoder. The complexity needs to separate the computation of the "context" part and the "prompt" part.

Thanks for pointing this out! We appreciate your observation regarding the contribution of the prompt to the complexity calculation.

In response, we have clarified our discussion in Section 3.2 (Page 6 Lines 279-280) as:

Let us denote the original context length (excluding the prompt or instruction) as $L$ and the chunk size in E2LLM as $C$.

Specifically, we have replaced "input length" by "context length (excluding the prompt or instruction)".

Regarding the Adapter, its structure is designed as a two-layer Multilayer Perceptron (MLP). The first layer's input and output neuron numbers correspond to the embedding dimensions of the encoder and decoder, respectively, with GELU used as the activation function. The second layer maintains equal input and output dimensions, aligning with the decoder's embedding size.

Dear Reviewer JbEJ,

We would like to thank you once again for your constructive feedback on our paper. We have made significant efforts to address the concerns you raised and have revised the manuscript accordingly. Given that the discussion period is nearing its end, we kindly ask if you could review the updated version and let us know if there are any remaining issues or points that require further clarification.

We are more than happy to address any additional questions or concerns you might have.

Thank you for your time and consideration.

Thank you for your feedback. We would like to further clarify the number of trainable parameters in CEPE and the encoder it uses in the experiment.

Regarding CEPE, we would like to argue that the composition of its trainable parameters, which includes the encoder that needs to align with the decoder’s tokenizer, as well as the additional randomly initialized cross-attention layers in each layer of the decoder’s transformer, results in its large number of parameters. Therefore, the number of CEPE's trainable parameters is fixed, and is not related to the LoRA rank, which represent its inherent limitation on parameter efficiency.

Moreover, as we previously mentioned, due to the need to ensure consistency in vocabulary between the encoder and the decoder, CEPE pre-trains a masked language model (MLM) on the RedPajama dataset. Therefore, it cannot directly apply the pre-trained GTE-Large-en as the encoder, which is another compatibility limitation of CEPE.

In summary, the differences in the encoder used and the number of trainable parameters in CEPE arise from its inherent design. In fact, this highlights the advantages of E2LLM in terms of parameter efficiency and compatibility.

We hope our clarifications address your concerns. Should you have any further questions, please do not hesitate to ask, and we would be happy to provide further explanations.

W5 - Can we do the eval on RULER?

Thank you for your insightful suggestions. In response, we have incorporated experimental results on the RULER benchmark, as presented in Table 8. Furthermore, we have included a detailed discussion of the model's performance on the RULER benchmark to provide a more comprehensive analysis. As shown in Appendix F, Line 1076-1109 on Page 20-21.

Table 8: Performance on RULER Benchmark. The best results are in bold, the second are underlined, and the third are wavy underlined.

We do not consider the retrieval tasks here, as they can be considered variants of the Needle-in-a-Haystack test. For the VT task, we set the number of variable name-binding chains and the number of times binding variable names in each chain to be 1 and 4, respectively. For the CWE and FWE tasks, we set the frequency of ten common words to be 30, uncommon words to be 3, and alpha as 2.0. Finally, for the QA task, we use two single-hop short-context QA datasets SQuAD and HotPotQA. For models that requires training, we reuse the checkpoints trained in Section 4.2.

The results are listed in Table 8. Given the diversity of tasks presented in RULER, we can clearly identify the strengths and weaknesses of each baseline method. Although Yarn and LongLoRA perform relatively well in the QA task, they struggle significantly with the CWE and FWE tasks. This is likely due to an attention distraction problem, which hampers their ability to focus on specific common or frequent words. Additionally, both methods encounter out-of-memory issues when the context length exceeds or equals 64K; for reference, we utilized an A100 GPU with 80GB of memory for inference. This suggests that the space complexity of YaRN and LongLoRA is too high for scenarios with limited resources. On the other hand, LongLLMLingua excels in the FWE task but underperforms in the others. The soft compression methods, E2LLM and LLoCO, manage to strike a balance between performance on the aggregation (CWE and FWE) and QA tasks, yielding comparable results. E2LLM tends to favor QA tasks, while LLoCO is better suited for aggregation tasks. It is worth noting that E2LLM can take advantage of increasingly sophisticated text encoders that are continuously being open-sourced, as demonstrated in our ablation studies; meanwhile, the encoder used by LLoCO is fixed to AutoCompressor. Lastly, we observe that all methods perform poorly on the VT task, which demands a nuanced comprehension of the long context, presenting a challenge that may be too great for the current models.

W4 - With the encoder and adapter, how does the inference speed compared to the baseline?

Thank you for pointing that out. In fact, we conduct experiments related to inference efficiency, as discussed in Section 4.4 (Lines 507-511 on Page 10) and Appendix G on Page 21-22.

As displayed in Figures 4b and 4c, E2LLM stands out with impressive results, demonstrating the lowest runtime and memory usage, especially for lengthy sequences at 73K. This efficiency is primarily due to its relatively high compression ratio of approximately 100 times, which dramatically reduces the number of chunk tokens processed by the LLM decoder to a size much smaller than the original number of text tokens.

W2 - The chunk size is pretty heuristic and with different tasks, the performance varies with chunk size. Could we make it learnable as well?

According to our ablation study on chunk size in Table 11 and Appendix H.2 (Page 24, Lines 1290-1297), we find that

the differences in performance metrics across different chunk sizes are relatively small for all datasets used in this study, indicating that the alignment process in E2LLM can effectively mitigate the impact of chunk size on performance. Nonetheless, selecting an optimal chunk size can still provide a slight performance boost.

On the other hand, we acknowledge that making the chunk size learnable is interesting and is beneficial to the performance of E2LLM. To this end, we provided a discussion on learnable chunk size in Appendix I.2 (Page 25, Lines 1326-1343) as:

While the performance of E2LLM is not highly sensitive to chunk size, selecting the optimal size can enhance its effectiveness, as illustrated in Table 11. This raises an intriguing question: Can we determine a chunk size that further boosts the performance of E2LLM? We believe there are two primary approaches to achieving this goal.

First, we can apply techniques commonly used to optimize hyperparameters in neural networks or during neural architecture search (NAS) to the chunk size learning process. Approaches such as Bayesian optimization, reinforcement learning, and meta-learning can be adapted to optimize both the chunk size and E2LLM model parameters simultaneously.

Second, chunk size can also be optimized independently of the E2LLM model parameters. One promising strategy is to explore chunking based on the semantic relationships between tokens, such as through meta-chunking (Zhao et al., 2024). However, a significant challenge with existing chunking methods is that the tokens within a chunk only capture information from that specific chunk, resulting in a loss of contextual information from nearby chunks. To address this issue, we can consider the "late" chunking method (Gunther et al., 2024}. This approach first embeds all tokens of a long text using a long-context embedding model and then applies chunking by mean pooling the token embeddings within each chunk. While this method provides chunk embeddings that encapsulate full contextual information, it comes with the drawback of increased complexity, as the computational demands of text encoders scale quadratically with the length of the input text.

W3 - Maybe another concern is the real time application, how do we plan for that given this framework.

Thanks for pointing this out! We notice that CPT (continue-pretraining) followed by SFT (supervised fine-tuning) is a prevalent practice to achieve practical long-context LLMs, as seen in the Llama and Qwen series. As such, we provide a discussion in Appendix I.1 (Lines 1303-1323 on Page 25) regarding how CPT can be conducted within the E2LLM framework as follows:

While E2LLM demonstrates comparable or superior performance compared to various baseline methods when utilizing the same amount of fine-tuning data, it has become increasingly common to engage in continue-pretraining (CPT) alongside supervised fine-tuning (SFT). This approach typically involves leveraging substantial quantities of high-quality, long-context data. By doing so, systems can achieve a long-text language model (LLM) that exhibits versatility across diverse long-text tasks. This practice has been effectively illustrated by recent models in the Llama and Qwen series (Dubey et al., 2024; Yang et al., 2024; Hui et al., 2024).

In this context, we will explore the methodology for conducting CPT within the E2LLM framework. Given that E2LLM functions akin to an encoder-decoder architecture, it is logical to adopt pretraining tasks prevalent in other established encoder-decoder frameworks, such as T5 (Raffel et al., 2019), BART (Lewis et al., 2020), and GLM (Du et al., 2022). However, considering the unique characteristics of the E2LLM model, we specifically recommend the use of prefix language modeling (PLM) (Wang et al., 2022) as the pretraining task. This choice ensures that the CPT process aligns seamlessly with the subsequent SFT process.

Concretely, suppose that the chunk size of the text encoder is $C$ and the length of the decoder is $L$. In this setup, we can create random pretraining sequences with lengths of $\ell C + L - \ell$, where $1 \leq \ell \leq L-1$ represents the number of chunks. We then can partition the prefix segments of length $\ell C$ into $\ell$ individual chunks, which are then fed into the text encoder. The task for E2LLM during this CPT phase is to predict the remaining segments of length $L - \ell$. This structured approach not only enhances the model's ability to comprehend and generate lengthy texts but also sets a solid foundation for effective fine-tuning on targeted applications thereafter.

We sincerely appreciate the insightful feedback provided by the reviewer. Incorporating their comments has greatly enhanced the clarity and depth of our paper, thereby improving its overall quality. To address each point comprehensively, we have prepared a detailed response for every comment raised. In our response, comments from the reviewer are highlighted in italics, with our responses directly following. For clarity, quotations from the adjusted manuscript are presented in markdown quotation mode. The corresponding modifications in the paper are highlighted in blue. Unless otherwise stated in our response, all pages, equations, sections, and bibliographical references refer to those in the revised paper._

W1 - The encoder-decoder framework is reasonable but I am concerned that what if the LLM is a large scale one like llama 70B. will the train be feasible. Seems author just conductor experiments on small scales.

Thank you for your valuable feedback. E2LLM can indeed scale to larger language models, such as Llama2-70B. We have attempted to use Llama2-70B as the decoder and evaluated its performance on QMSum. The results are presented in Appendix H.1 (Lines 1232-1259 on Page 23), as follows:

We adopt Llama2-70B as the decoder to further validate the feasibility of E2LLM on larger-scale language models (denoted as E2LLM-70B). During training, we apply 4-bit quantization using QLoRA's Parameter-Efficient-Finetuning (PEFT) method. We conduct training and evaluating on QMSum, assessing its performance using the R1, R2, RL, G-mean, and PPL metrics as well as their relative improvement over those corresponding to the 7B model. The results are shown in Table 10.

Table 10: Performance on E2LLM with larger-scale model.

As shown in the table, the performance of E2LLM significantly improves when using Llama2-70B, particularly in terms of Rouge-L and PPL. It is important to note that Rouge-1 and Rouge-2 evaluate unigram and bigram overlaps, respectively, measuring the match between the generated text and reference text at the word and phrase levels. In contrast, Rouge-L evaluates the similarity of the generated and reference texts based on the longest common subsequence (LCS), which measures structural similarity at the sentence level. This indicates that by leveraging a larger model, E2LLM is able to better capture the overall sentence structure and word order. Additionally, the reduction in PPL further demonstrates the model’s ability to generate more coherent and reasonable content.

Dear Reviewer t6Dh,

We would like to thank you once again for your constructive feedback on our paper. We have made significant efforts to address the concerns you raised and have revised the manuscript accordingly. Given that the discussion period is nearing its end, we kindly ask if you could review the updated version and let us know if there are any remaining issues or points that require further clarification.

We are more than happy to address any additional questions or concerns you might have.

Thank you for your time and consideration.

Q1 - In Figure 4, why do the inference time and memory usage increase for StreamingLLM? It keeps a constant size of starting and recent KV caches, so it should stay constant past a certain length.

Indeed, StreamingLLM does maintain a fixed-size KV cache for both starting and recent tokens, which allows it to handle streaming inputs and outputs effectively. However, as indicated in the official implementation, for any given context, StreamingLLM must initially load the entire KV cache associated with that context. During the subsequent generation process, it utilizes Sink Attention to preserve the KV caches for both the starting and recent tokens. Consequently, in long-context scenarios, the memory usage and inference time for StreamingLLM still exhibit quadratic growth.

We have addressed this point in Appendix G (Lines 1142-1147, Page 22) when discussing the results in Figure 4.

Q2 - Why is E2LLM better than YaRN and LLoCO on NarrativeQA in Table 1 but not in Table 2?

Thanks for pointing this out! The difference primarily arises from the experimental settings of Table 1 and Table 2. Specifically, in Table 1, each dataset is trained and tested independently. As mentioned in Section 4.1 (Page 7, Lines 341-342),

We employ the validation sets of each dataset for testing and split the training sets into training and validation subsets using a 95:5 ratio.

In contrast, Table 2 involves training on a mixture of all five datasets. As stated in Section 4.2 (Page 8, Lines 421-422),

The methods that require training utilize data collected from all five tasks discussed in the previous subsection.

Furthermore, Appendix E (Page 20, Lines 1071-1072) clarifies:

For methods that require training, the training data utilized are identical to those employed during the Needle in a Haystack experiment.

We hypothesize that interactions between datasets can have varying effects, maybe some positive and others negative. This explains why E2LLM demonstrates better performance in the single-task experiments of Table 1 but falls behind some baselines in the multi-task setting of Table 2.

Q3 - In Sec 4.1, the datasets listed are QMSum, GovReport, HotpotQA, NarrativeQA, and TriviaQA, but the corresponding Table 4 lists Quality instead of HotpotQA for the statistics, and Table 1 shows Quality instead of HotpotQA.

Thank you for bringing this to our attention. We acknowledge that there was an error in Section 4.1 and Table 1, where It should indeed be "Quality" rather than "HotpotQA". The introductions of the datasets in Appendix C, as well as the notations in Tables 4 and 5, are accurate. We have made the necessary corrections in the revised manuscript.

Q4 - How important are the two training tasks in Sec 3.2? It's not clear how they are used during training and how they affect the performance and the training efficiency, as reconstructing the inputs from the encoder introduces additional costs.

We have conducted an ablation study regarding the weight of the "understanding" task in Appendix H.2 (Page 24, Lines 1267-1275). Moreover, we have made it clear in Section 3.2 (Page 5, Lines 264-266) that

It is important to note that the “understanding” task serves as an auxiliary task, while our primary focus remains on the “reasoning” task. We determine the final checkpoints exclusively based on the validation loss associated with the “reasoning” task.

Regarding the impact of the "understanding" task on training efficiency, we acknowledge that it does introduce some additional costs, as the training speed is influenced by the length of the context that must be reconstructed. Nevertheless, we contend that this task is essential. In the absence of continual pretraining, a self-supervised task like reconstruction or "understanding" is vital for maintaining the model’s ability to retain context. Without it, we observe a substantial degradation in performance. This phenomenon is illustrated in Figure 6 on Page 23, where we see a marked decline in performance as the weight of the "understanding" task approaches zero (equivalent to the logarithm of its weight approaching negative infinity).

W2 - A key difference between previous methods and E2LLM is that they are typically studied in the pre-training/continual pre-training settings where they are first trained on a corpus like RedPajama (such as in the case of YaRN, LongLoRA, CEPE, and others), and then evaluated either zero-shot or with ICL on downstream tasks. It would be important to see how E2LLM would perform in such settings in addition to the supervised fine-tuning setting.

We would like to argue that most long-context extension methods aim to enhance the context window with minimal to no fine-tuning, as explicitly stated in studies like Position Interpolation (Chen et al. (2023a)) and YaRN (Peng et al., 2023). This is consistent with E2LLM’s goal of compatibility with pretrained models. As discussed in Section 1 (Lines 041-044),

Achieving this compatibility is crucial for effectively leveraging the pretrained knowledge contained in these models, allowing for parameter and sample efficiency without necessitating extensive additional training with large datasets.

This rationale underpins our decision not to focus on continue-pretraining (CPT) plus supervivsed fine tuning (SFT) in the first place, as it deviates from the original motivation behind long-context extension methods. Our core aim is to demonstrate that, given an equivalent training dataset, E2LLM—being tuned with fewer parameters—can outperform baseline methods that necessitate more extensive tuning. This assertion is validated in Section 4 (see Tables 1-2 and Figure 3), where E2LLM effectively harnesses the knowledge embedded in the pretrained text encoder and LLM decoder.

On the other hand, we acknowledge that CPT plus SFT is a prevalent practice to achieve practical long-context LLMs, as seen in the Llama and Qwen series. However, due to resource constraints and time limitations, we could not conduct CPT plus SFT using the E2LLM framework on a substantial amount of high-quality long-context data. Instead, we provide a discussion in Appendix I (Lines 1303-1323 on Page 25) regarding how CPT can be conducted within the E2LLM framework as follows:

While E2LLM demonstrates comparable or superior performance compared to various baseline methods when utilizing the same amount of fine-tuning data, it has become increasingly common to engage in continue-pretraining (CPT) alongside supervised fine-tuning (SFT). This approach typically involves leveraging substantial quantities of high-quality, long-context data. By doing so, systems can achieve a long-text language model (LLM) that exhibits versatility across diverse long-text tasks. This practice has been effectively illustrated by recent models in the Llama and Qwen series (Dubey et al., 2024; Yang et al., 2024; Hui et al., 2024).

In this context, we will explore the methodology for conducting CPT within the E2LLM framework. Given that E2LLM functions akin to an encoder-decoder architecture, it is logical to adopt pretraining tasks prevalent in other established encoder-decoder frameworks, such as T5 (Raffel et al., 2019), BART (Lewis et al., 2020), and GLM (Du et al., 2022). However, considering the unique characteristics of the E2LLM model, we specifically recommend the use of prefix language modeling (PLM) (Wang et al., 2022) as the pretraining task. This choice ensures that the CPT process aligns seamlessly with the subsequent SFT process.

Concretely, suppose that the chunk size of the text encoder is $C$ and the length of the decoder is $L$. In this setup, we can create random pretraining sequences with lengths of $\ell C + L - \ell$, where $1 \leq \ell \leq L-1$ represents the number of chunks. We then can partition the prefix segments of length $\ell C$ into $\ell$ individual chunks, which are then fed into the text encoder. The task for E2LLM during this CPT phase is to predict the remaining segments of length $L - \ell$. This structured approach not only enhances the model's ability to comprehend and generate lengthy texts but also sets a solid foundation for effective fine-tuning on targeted applications thereafter.

We deeply value the thoughtful feedback received. Addressing the comments from the reviewer has significantly improved both the clarity and the depth of our paper, elevating its overall quality. To ensure a thorough response, we have prepared a detailed reply for each point noted by the reviewer. In our response, comments from the reviewer are highlighted in italics, with our responses directly following. For clarity, quotations from the adjusted manuscript are presented in markdown quotation mode. The corresponding modifications in the paper are highlighted in blue. Unless otherwise stated in our response, all pages, equations, sections, and bibliographical references refer to those in the revised paper.

W1 - About the baseline methods, Appendix B states that Llama 2 7B is “the backbone for the other methods”, but E2LLM uses Llama 2 7B Chat as the decoder. It doesn’t seem fair to compare methods that use different backbone models. What would the performance of E2LLM be if Llama 2 7B were used as the backbone model instead? Or if the other methods use the chat model as the backbone?

Thank you for your feedback. For the other baselines, we use the same backbone as E2LLM, namely LLaMA2-7B-Chat. As described in Appendix B (Page 15, Lines 796-797)，

LLaMA2-7B: This refers to LLaMA2-7B-Chat without additional training or fine-tuning, serving as the backbone for the other methods.

We realize that inconsistent naming may have caused some confusion. To enhance the readability of the paper, we have standardized the naming throughout the manuscript, replacing all occurrences of "LLaMA2-7B-Chat" with "LLaMA2-7B" in relevant sections.

Dear Reviewer 9zKM,

We would like to thank you once again for your constructive feedback on our paper. We have made significant efforts to address the concerns you raised and have revised the manuscript accordingly. Given that the discussion period is nearing its end, we kindly ask if you could review the updated version and let us know if there are any remaining issues or points that require further clarification.

We are more than happy to address any additional questions or concerns you might have.

Thank you for your time and consideration.

W5 - Model architecture details are missing. Maybe this can be benefit from adding a clear model architecture figure and justifying the design choice. Figure 1 demonstrates the idea at very high level and paragraph descriptions are not very clear in 3.1.

Thanks for pointing this out! We would like to clarify that we provided the model architecture in Figure 2 on Page 2.

We also identified an error in the previous version of Section 3.1; it should reference "Figure 2" as the illustration of the E2LLM framework architecture instead of "Figure 1." This mistake has been rectified in the revised manuscript.

Furthermore, we have enhanced Section 3.1 with a more comprehensive description of the model architecture as:

For long input contexts, E2LLM first performs chunking. Each resulting chunk is then processed by the encoder, which captures its semantics. The adapter facilitates the mapping of the encoder's outputs into the LLM decoder’s embedding space, allowing the decoder to interpret these representations effectively. Ultimately, the decoder utilizes these embeddings as substitutes for the original context and executes two fine-tuning tasks—“understanding” and “reasoning”—to train the entire framework.

Q1 - What's the adapter is chosen and why?

As mentioned in Section 3.1 (Page 5, Lines 236-237) and Appendix B (Page 16-17, Lines 862-863, 874-876),

Regarding the Adapter, its structure is designed as a two-layer MLP. The first layer's input and output neuron numbers correspond to the embedding dimensions of the encoder and decoder, respectively, with GELU used as the activation function. The second layer maintains equal input and output dimensions, aligning with the decoder's embedding size.

Our decision is informed by inspiration drawn from vision-language models (VLM), as discussed in Section 3.3 (Page 6). In the literature, we identified three primary adapter designs: the linear layer employed in MiniGPT-4 (Zhu et al., 2024), the MLP used in LLaVA (Liu et al., 2024), and the cross-attention layer with learnable queries found in Qwen-VL (Bai et al., 2023), which was later replaced by an MLP in Qwen2-VL ([R1]).

We opted for a simple yet effective adapter design. Our ablation study, outlined in Appendix H.2 (Lines 1284-1287), reveals that a single linear layer falls short when compared to the two-layer MLP; this is largely because a linear transformation may inadequately map the output of the text encoder to the embedding space of the LLM decoder. Additionally, we have identified significant limitations with cross-attention layers. Specifically, these adapters tend to convert an arbitrary number of input chunks into a fixed number of learnable queries, regardless of the context length. This approach seems impractical in our scenario as it fails to account for the variability in context length.

[R1] Peng Wang et al., Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. arXiv 2024.

W3- I think the Needle task is not well presented. I understand that authors wanna focus on the accuracy, but the heat map might give us a clear image on where the model fails, and the trend per position more clearly w.r.t needle depth. Is it possible to have a similar analysis?

We would like to clarify that we have included a heat map in Figure 3 on Page 9 and have discussed its implications in Section 4.2.

Following your suggestion, we have further plotted the average score as a function of the depth percent in Figure 5 and discussed the results in Appendix D (Lines 1014-1019) as:

In addition to the results presented in Section 4.2, we further illustrated the average score over the context length as a function of depth percentage in Figure 5. It is evident that the performances of Llama2-7B, YaRN, LongLoRA, LongLLMLingua, and E2LLM are largely insensitive to the depth at which the needle is inserted. In contrast, LLoCO achieved the best results when the needle was positioned close to the answer, as discussed at the end of Section 4.2. Furthermore, E2LLM typically delivers the best performance across all depths.

W4 & Q2 - I am not so sure that I understand the compression process. The major compression is depending on the chunk size in this paper, but it is not well studies how this chunk size will influence the results, or how will influence the results, semantically for example. I would encourage authors break down the Negative LL per token position to demonstrate the effect of chunk size, particularly at boundaries.

We have indeed examined the impact of chunk size on the performance of E2LLM, as detailed in Table 10 on Page 24 and further discussed in Appendix H.2 (Lines 1290-1298) as:

Results in Table 11 show that the differences in performance metrics across different chunk sizes are relatively small for all datasets used in this study, indicating that the alignment process in E2LLM can effectively mitigate the impact of chunk size on performance. Nonetheless, selecting an optimal chunk size can still provide a slight performance boost. While smaller chunks might reduce compression and better preserve inputs, they may hinder context capture in longer sentences or paragraphs, making it difficult for the encoder to grasp semantics, which affects downstream tasks. Conversely, larger chunk sizes increase diversity and noise, complicating semantic capture and leading to decreased performance, especially in tasks like DocumentQA where relevant sentences may be overlooked.

Additionally, we would like to clarify that we are unable to compute the negative log-likelihood per token position for the input context in E2LLM. This is because the loss in E2LLM is calculated solely based on the answers generated by the LLM decoder, rather than on the context provided to the text encoder. The context serves merely as a prefix in the E2LLM framework.

We appreciate your feedback and hope that our explanations address your concerns regarding the chunk size and its effects on model performance.

W2 - Also adding some recent benchmarks, for example Ruler[1], and HELMET[2] to have real long context evaluation thoroughly.

Thank you for your insightful suggestions. In response, we have incorporated experimental results on the RULER benchmark, as presented in Table 8. Furthermore, we have included a detailed discussion of the model's performance on the RULER benchmark to provide a more comprehensive analysis. As shown in Appendix F, Line 1076-1109 on Page 20-21.

Table 8: Performance on RULER Benchmark. The best results are in bold, the second are underlined, and the third are wavy underlined.

We do not consider the retrieval tasks here, as they can be considered variants of the Needle-in-a-Haystack test. For the VT task, we set the number of variable name-binding chains and the number of times binding variable names in each chain to be 1 and 4, respectively. For the CWE and FWE tasks, we set the frequency of ten common words to be 30, uncommon words to be 3, and alpha as 2.0. Finally, for the QA task, we use two single-hop short-context QA datasets SQuAD and HotPotQA. For models that requires training, we reuse the checkpoints trained in Section 4.2.

The results are listed in Table 8. Given the diversity of tasks presented in RULER, we can clearly identify the strengths and weaknesses of each baseline method. Although Yarn and LongLoRA perform relatively well in the QA task, they struggle significantly with the CWE and FWE tasks. This is likely due to an attention distraction problem, which hampers their ability to focus on specific common or frequent words. Additionally, both methods encounter out-of-memory issues when the context length exceeds or equals 64K; for reference, we utilized an A100 GPU with 80GB of memory for inference. This suggests that the space complexity of YaRN and LongLoRA is too high for scenarios with limited resources. On the other hand, LongLLMLingua excels in the FWE task but underperforms in the others. The soft compression methods, E2LLM and LLoCO, manage to strike a balance between performance on the aggregation (CWE and FWE) and QA tasks, yielding comparable results. E2LLM tends to favor QA tasks, while LLoCO is better suited for aggregation tasks. It is worth noting that E2LLM can take advantage of increasingly sophisticated text encoders that are continuously being open-sourced, as demonstrated in our ablation studies; meanwhile, the encoder used by LLoCO is fixed to AutoCompressor. Lastly, we observe that all methods perform poorly on the VT task, which demands a nuanced comprehension of the long context, presenting a challenge that may be too great for the current models.

Table 6-2: Performance as a function of context length. The best results are in bold, the second are underlined, and the third are wavy underlined.

The results presented in the table indicate that E2LLM demonstrates a strong resilience to variations in context length for both summarization (QMSum) and question-answering (NarrativeQA) tasks, typically achieving the best results among all models. This robustness can be attributed to the "understanding" task incorporated during the training of E2LLM (see Section 3.2). By reconstructing different parts of the context, E2LLM effectively comprehends the information, regardless of its length.

Notably, the performances of YaRN, LongLoRA, CEPE, RAG, and LongLLMLingua also exhibit insensitivity to context length. On the other hand, LLoCO's performance declines slowly with increasing context length. Finally, streamingLLM and the original Llama2-7B demonstrate sensitivity to context length; streamingLLM loses more information in the middle of the context as length increases due to its specific $\Lambda$-shaped attention mask, while Llama2-7B struggles to handle long contexts altogether, as its maximum length has not been extended.

We're grateful for your insightful comments! We believe that addressing the reviewer’s comments has resulted in improving the clarity and presentation of the paper’s contributions and has brought the paper to a higher standard. A comprehensive reply has been prepared to address every point raised by the reviewer. The reviewer's comments are presented in italics, followed by our response. Quotations from the revised paper are included in markdown quotation mode. The corresponding modifications in the paper are highlighted in blue. Unless otherwise stated in our response, all pages, equations, sections, and bibliographical references refer to those in the revised paper.

W1 - I would encourage authors to break down the task accuracy by length to show the real benefit of their method.

Thank you for your valuable suggestions. Following your advice, we conduct an evaluation on the QMSum and NarrativeQA datasets based on context length to gain deeper insights. The results are presented in Appendix C on Page 18 as follows:

Next, we investigate the sensitivity of the models' performance to variations in context length. To do this, we categorize samples from the QMsum and NarrativeQA datasets into five groups based on their context lengths and then evaluate the perplexity (PPL) of the answers within each group. Our findings are summarized in Table 6.

Table 6-1: Performance as a function of context length. The best results are in bold, the second are underlined, and the third are wavy underlined.