E$^2$-RAG: Towards Editable Efficient RAG by Editing Compressed KV Caches

Thank you for your insightful feedback! We appreciate your suggestion regarding experiments with larger-scale models.

We agree that evaluating E²-RAG on models in the 14B, 32B, or even larger ranges would indeed offer valuable insights into its scalability and further strengthen the generalizability and practical relevance of our findings. The decision to focus on the 3B and 8B model sizes in the current study was primarily guided by the available computational resources. We consider extending our experiments to these larger models in future work.

Thank you for taking the time to provide insightful feedback. As stated in the Reasons To Reject, the two concerns in your comments are 1) lack of model generalization, and 2) lack of deeper theoretical grounding and discussion. Our point-wise responses are as follow:

1. "Limited Model Generalization"

We agree that testing on a broader range of model architectures would further strengthen our findings. Our current experiments were constrained by available computational resources, but we see this as an important direction for future validation.

2. "Superficial Discussion on Conflicts"

You are correct that while we identify and discuss knowledge conflicts arising from DELETE and UPDATE operations (e.g., in Section 6.3), the current work primarily observes their effects and hypothesizes on the editor's behavior (e.g., $\Delta V$ acting as a "mask" or "replacer"). A deeper theoretical grounding and more sophisticated resolution mechanisms are indeed significant areas that we leave for future work, as our current findings highlight the potential for further advancements in this direction.

Thanks for your detailed reviews. Since your feedback is diverse, here please let us to address each point you have made.

For your Reasons To Reject:

1. "Concerns about the method."

We understand your concern regarding the method's efficiency and the editor module's training. But the editor's training is an upfront investment and the models are readily available for direct download from open-source platforms like Huggingface. Moreover, for scenarios with frequent updates (e.g., news, finance, or Wikipedia with 0.6 million daily edits), the speed-up of our method is substantial and quickly amortizes the initial training cost. Re-reasoning or re-encoding the entire context for every small edit is far more computationally expensive in the long run.

2. "The approach is flawed."

• "Why is a similar operation not required for the 'value'?"

This is because the positional embedding is not applied to the 'value' in attention[1], there is no need to remove the positional embedding from 'value'.

In Rotary Position Embedding (RoPE), positional information is exclusively applied to the "query" and "key" vectors within the attention mechanism. RoPE encodes the position information into the "query" and "key" using a rotation matrix, rather than directly adding position information to the "value". Specifically, the encoding for "query" and "key" is as follows:

$q_m = R_{\Theta, m} W_q x_m$

$k_n = R_{\Theta, n} W_k x_n$

where $R_{\Theta, m}$ is the rotation matrix, $ W_q $and$ W_k $are the linear transformation matrices for "query" and "key", and$ x_m $and$ x_n $are the input hidden vectors for position$m$and$n$.

When computing the attention weights, the inner product of "query" and "key" naturally incorporates position information. Specifically, the inner product formula is:

$q_m^T k_n = (R_{\Theta, m} W_q x_m)^T (R_{\Theta, n} W_k x_n)$

By expanding this formula, we can see that position information is encoded into the inner product through the rotation matrices $R_{\Theta, m}$ and $R_{\Theta, n}$. This means that position information has already been passed to the attention weights through the interaction of "query" and "key", and there is no need to add position information to the "value".

• "The Editor Module appears to simply perform fine-tuning on newer data with reduced memory embedding"

We understand your perspective on the editor module. While it's true that our editor module performs fine-tuning on editing data with LoRA and editing embeddings, our innovation isn't centered on this. Instead, the core novelty of E²-RAG lies in the editor's specific architectural design and its unique objective: to directly generate offset KV caches $(\Delta K, \Delta V)$. These offset caches are then used to precisely modify existing, compressed KV caches based on our defined editing operations (INSERT, DELETE, and UPDATE). This constitutes a targeted mechanism for performing edits directly at the compressed representation level, which distinguishes it from general model fine-tuning that typically operates on raw text or aims for broader behavioral adaptations.

• "Would the introduction of additional key-value cache entries not negatively impact the inference speed?"

We appreciate the your concern. However, the impact of adding a small number of KV cache entries from the editor module is negligible. Specifically, even though additional KV entries slightly increase attention computation, the number of "editing tokens" (e.g., 4 in our setup) is tiny compared to the hundreds or thousands of tokens typically processed during generation.

To quantify this, we model the relative speed as inversely proportional to the total number of tokens in the KV cache:

$\text{Relative Speed}= \frac{s_0}{s}$

where $s_0 = 1$ is the speed when there are no KV caches. We summarize the theoretical speed impact of introducing 4 editing tokens in the table below:

As shown, adding 4 tokens has a negligible impact on speed, especially when the sequence length is large (e.g., hundreds of tokens). Moreover, E²-RAG operates on compressed representations, and the primary efficiency gain comes from pre-computed KV caches, even with minor edits. This leads to up to 3× faster generation compared to standard RAG systems.

[1] Roformer: Enhanced Transformer With Rotray Position Embedding

3. "There are concerns regarding the quality of the dataset."

We understand this point. The quality of our datasets is ensured because they are constructed by systematically modifying established QA benchmarks, leveraging their inherent quality for our specific editing tasks. For instance, in the INSERT operation, we reused existing questions from the original document QA, with new answers generated by Qwen2.5-7B based on the inserted knowledge. For DELETE, when necessary knowledge was removed, Qwen2.5-7B generated appropriate refusal-to-answer responses. Similarly, for UPDATE, the same language model was used to create conflicting new knowledge along with the corresponding questions and answers.

4. "The proposed method requires model training. When comparing editing efficiency with the baseline, was the training time taken into account?"

We understand your concern. However, the editor's training is an upfront investment. Users can directly download the models from the openscore website (Huggingface). So we believe the training time for the editor module is a one-off, amortized cost and is not included in this per-edit operational efficiency metric, which is standard practice for evaluating editing/update mechanisms.

5. "The methods section of this paper is not clearly written."

• $W_E$ is indeed the token embedding matrix of the LLM, which transforms one-hot input text tokens $\mathbb{I}_X$ into dense embeddings $E$
• $Q_{append}$ is the query of the question related to the inserted knowledge in attention, so it can perform dot product in attention.
• $O$ represents the output of the attention when processing deleted knowledge. It describes how the attention scores $(\tilde{A}, \Delta A)$ applied to the original values $\tilde{V}$ and the offset values $\Delta V$ combine. In the DELETE operation, $\Delta V$(part of the offset KV cache from the editor) acts as a "mask" to suppress targeted segments of knowledge needed to delete.
• These sections describe the specific objectives and mechanisms of our editor module for each defined operation (INSERT, DELETE, UPDATE). They are not merely general analyses but detail how the editor, using components like $\Delta K$, $\Delta V$, and the attention mechanism, achieves the intended edit on the KV caches. These are core to explaining how E²-RAG performs editable efficient RAG.

For your Questions To Authors:

1. "In line 225, the authors characterize Golden Edit as representing an upper-bound performance. However, the results reported in Table 1 indicate that ..."

We appreciate you pointing out this interesting observation. As we have mentioned in paper, this phenomenon can be attributed to two main factors: firstly, our E²-RAG editor's specialized training for optimally integrating new knowledge directly within compressed KV caches, and secondly, the inherent advantages of operating within this simplified, compressed knowledge domain. Specifically, the E²-RAG editor is explicitly trained to generate offset KVs $(\Delta K, \Delta V)$ that optimally integrate new information. As noted in lines 248-250 of our paper, "the KV cache for new knowledge is added after the original KV caches, bringing it closer to the query tokens, which causes the model to direct more attention toward the new knowledge". Moreover, as stated in lines 253-256, compressing context into KV caches "extracts and simplifies information, omitting less important details, which enhances the model's retrieval ability" compared to standard RAG systems that might be vulnerable to irrelevant information in a longer, albeit golden, text. This principle of the effectiveness of compressed representations is also supported by prior work[2].

2. "I find the distinction between the "old" and "new" settings in Table 1 somewhat unclear. Could the authors clarify what specifically differentiates these two settings, including any changes in evaluation protocol, data distribution, or editing targets?"

We acknowledge the need for clarity; however, we have provided this distinction directly in the caption of Table 1, "We use “Old” and “New” to indicate whether the knowledge involved in the question is added through the INSERT operation or is inherent to the old chunk." And we also have provided examples in Appendix D.

[2] xrag: Extreme context compression for retrieval-augmented generation with one token

We have carefully considered your comments and detailed replies to address your concerns. We are eager to know if these responses adequately resolve the issues.

Please feel free to reach out if you have additional questions. If you believe there is potential for improving the original score, we are fully willing to make any further efforts necessary.

Best regards,

Authors

Thank you for your appreciation and detailed feedback. After carefully reading your comments, we have summarized your main concerns as: 1. Lack of analysis and discussion on impact of outdated entries related to a certain query while n >> k. 2. The need for experiments on more models, given its comparatively lower effectiveness on the Qwen 2.5 7B model. We are happy to address each of these points.

1. The analysis and discussion on the impact of outdated entries related to a certain query when n>>k

We acknowledge the importance of investigating this scenario. To explore how models behave when facing in-contextual conflicts arising from outdated information, an experiment could be designed as follows: One might select approximately 100 questions from the test set of Squad for the three operation and duplicate the associated information chunk five times. Subsequently, the ratio of edited (updated) chunks could be varied systematically from 0.2 (1 out of 5) to 1.0 (all 5 out of 5). This would allow for an examination of model performance under varying degrees of information conflict. We provide the results for Llama-3.2 3B and Llama-3.1 8B in the table below:

Firstly, all the models and operations demonstrate a similar trend where performance slightly drops when the ratio of edited chunks is reduced from 1.0 to 0.2. This is a reasonable result because when the number of edited chunks decreases, the model can be influenced by the unedited chunks, which may confuse the model and lead to incorrect answers. However, to our surprise, our method also shows a certain level of robustness in this scenario. Especially in the UPDATE operation, even when only 20% of the chunks are updated, the model can still notice the change and answer correctly. For instance, the performance of the 3B model remains at 0.75 when the ratio of edited chunks is only 0.2, compared to 0.82 when the ratio is 1.0.

We acknowledge that updating by selecting the top-k relevant outdated entries is indeed a naive method, as it can lead to the issue of incomplete updates, as you mentioned. However, since our work is the first to explore how to efficiently keep the RAG database up-to-date, we believe there will be future follow-up work to better address this challenge.

2. "It is less useful on the Qwen 2.5 7B model reported in Appendix H."

We recognize this observation. It is important to note that our computing resources are extremely limited. The Qwen 2.5 7B model, as reported, was fine-tuned for only 8,000 steps. This is a substantially lower number of training iterations compared to those applied to the Llama series models presented in our work, which likely contributes to the observed differences in performance.

We provide the training loss curve in this anonymous link.

If you have any further questions, please feel free to let us know.