Let the Code LLM Edit Itself When You Edit the Code

Thanks for your careful review and suggestions. We respond to each of your concerns as below.

Regarding Weakness 1: Trivial Algorithm

The problem we address, i.e., efficiency in real-time code editing, is of substantial practical importance and remains largely underexplored in existing literature. Our method, although simple in design, is the first to effectively tackle this challenge. The simplicity of our approach enhances, rather than diminishes, its impact, highlighting its elegance and practicality in addressing a complex issue. By introducing a simple yet effective solution for real-time code editing, we believe our work makes a significant contribution to the field and opens new avenues for future research aimed at improving LLM efficiency in dynamic environments.

Regarding Weakness 2 and Question 1: More Experiments

In addition to evaluating PIE on DeepSeek-Coder, we have reported the performance of CodeLlama in Tables 5, 6, and 7 in Appendix A.2, which further validates the effectiveness of PIE. Following your suggestions, we also conduct experiments on Yi-Coder-9B (We choose it because it has the same architecture as Llama, making it convenient for code adaption). The results are shown in Table R.1 below:

Table R.1: Performance comparisons of models on insertion experiments of Repobench-C-8k Python

Besides, we additionally conduct experiments on code generation tasks (as also suggested by Reviewer Bo49), using HumanEval [1] and its C++ version from HumanEval-X [2]. We randomly choose three consecutive lines in the prompt as the user inserted code and report Pass@1 for different methods. It's important to note that the context (prompt) in HumanEval consists of only a few lines (with an average of 15 lines), making each edit semantically significant. The results are shown in Table R.2 below:

Table R.2: Performance of DeepSeek-Coder 6.7B on HumanEval and HumanEval-X

Table R.1 and Table R.2 further validate the effectiveness of PIE on various models (DeepSeek-Coder, Codellama, Yi-Coder) and datasets (Repobench-C, Humaneval, HumanEval-X)

[1] Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint arXiv:2107.03374, 2021.

[2] Zheng Q, Xia X, Zou X, et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x[C]//Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023: 5673-5684.

Dear reviewer,

Thank you for your constructive review and comments! We have provided point-to-point responses to your review. Considering the author-reviewer discussion is ending very soon, could you kindly take a look at our responses and let us know your feedback? We are happy to discuss more if you still have any concerns. Thank you again and look forward to your feedback!

Upper Bound on the Discrepancy between PIE and Full Recomputation

We derive an upper bound on the difference between the outputs of the Positional Integrity Encoding (PIE) method and full recomputation at a given position $i$.

Objective
Let:

• $\mathbf{o} _i^{\text{PIE}}$ denote the output at position $i$ using PIE.
• $\mathbf{o} _i^{*}$ denote the output at position $i$ using full recomputation.
• $\Delta \mathbf{o} _i = \mathbf{o} _i^{\text{PIE}} - \mathbf{o} _i^{*}$ is the discrepancy we aim to bound.

Our goal is to find an upper bound for $\| \Delta \mathbf{o} _i \|$.

1. Adjustment of Key Vectors in PIE

For positions $j'$ after the edit (where $j' \in [j+1, n]$), PIE adjusts the key vectors as:

$\mathbf{k} _{j'}^{\text{edit}} = R _{\Delta j} \mathbf{k} _{j'}$

where:

• $\Delta j = (i + m - j)$ is the positional shift due to the edit,
• $R _{\Delta j}$ is the rotary positional encoding matrix corresponding to shift $\Delta j$,
• $\mathbf{k} _{j'}$ are the original key vectors before adjustment.

2. Value Vectors

PIE leaves the value vectors unchanged:

$\mathbf{v} _{j'}^{\text{edit}} = \mathbf{v} _{j'}$

In contrast, full recomputation updates both keys and values:

where $\mathbf{h} _{j'}^{*}$ are the updated hidden states after full recomputation.

3. Discrepancy in Attention Scores

The attention score between a query at position $i$ and a key at position $j'$ differs between PIE and full recomputation:

• PIE Attention Score:

  $\alpha\_{i, j'}^{\text{PIE}} = \text{softmax}\left( (R\_i \mathbf{q}\_i)^\top \mathbf{k}\_{j'}^{\text{edit}} \right)$

• Full Recomputation Attention Score:

  $\alpha\_{i, j'}^{*} = \text{softmax}\left( (R\_i \mathbf{q}\_i^{*})^\top \mathbf{k}\_{j'}^{*} \right)$

4. Discrepancy in Output Representations

The difference in the attention output at position $i$ is:

$\Delta \mathbf{o} _i = \mathbf{o} _i^{\text{PIE}} - \mathbf{o} _i^{*} = \sum _{j} \left( \alpha _{i, j}^{\text{PIE}} \mathbf{v} _{j} - \alpha _{i, j}^{*} \mathbf{v} _{j}^{*} \right)$

We can rewrite this as:

$\Delta \mathbf{o} _i = \sum _{j} \left( (\alpha _{i,j}^{\text{PIE}} - \alpha _{i,j}^{*}) \mathbf{v} _j + \alpha _{i,j}^{*} (\mathbf{v} _j - \mathbf{v} _j^{*}) \right)$

Take the norm and apply triangle inequality:

$\| \Delta \mathbf{o} _i \| \leq \sum _{j} \left( \left| \alpha _{i,j}^{\text{PIE}} - \alpha _{i,j}^{*} \right| \cdot \| \mathbf{v} _j \| + \alpha _{i,j}^{*} \cdot \| \mathbf{v} _j - \mathbf{v} _j^{*} \| \right)$

5. Computing Expected Norms

Assume $\mathbf{v} _j \sim \mathcal{N}(\boldsymbol{\mu},\mathbf{\Sigma})$ and $\mathbf{v} _j \sim \mathcal{N}(\boldsymbol{\mu},\mathbf{\Sigma})$.

a. Norm of $\mathbf{v} _j$:

The expected squared norm:

$E\left[ \| \mathbf{v} _j \|^2 \right] = \| \boldsymbol{\mu} \|^2 + \operatorname{Tr}(\mathbf{\Sigma})$

b. Norm of $\mathbf{v} _j - \mathbf{v} _j^{*}$:

Since $\mathbf{v} _j$ and $\mathbf{v} _j^{*}$ are Gaussian, the difference is also Gaussian:

$\mathbf{v} _j - \mathbf{v} _j^{*} \sim \mathcal{N}(\boldsymbol{\delta}_{\mu}, \mathbf{\Sigma} + \mathbf{\Sigma}^{*})$

where $\boldsymbol{\delta} _{\mu} = \boldsymbol{\mu} - \boldsymbol{\mu}^{*}$.

The expected squared norm:

$E\left[ \| \mathbf{v} _j - \mathbf{v} _j^{*} \|^2 \right] = \| \boldsymbol{\delta} _{\mu} \|^2 + \operatorname{Tr}(\mathbf{\Sigma} + \mathbf{\Sigma}^{*})$

6. Bounding the Discrepancy

a. Bounding the First Term:

$\left| \alpha _{i,j}^{\text{PIE}} - \alpha _{i,j}^{*} \right| \cdot \| \mathbf{v} _j \| \leq \left| \alpha _{i,j}^{\text{PIE}} - \alpha _{i,j}^{*} \right| \cdot M _v$

where:

$M _v = \sqrt{ \| \boldsymbol{\mu} \|^2 + \operatorname{Tr}( \mathbf{\Sigma} ) }$

b. Bounding the Second Term:

$\alpha _{i,j}^{*} \cdot \| \mathbf{v} _j - \mathbf{v} _j^{*} \| \leq \alpha _{i,j}^{*} \cdot M _{\delta v}$

where:

$M _{\delta v} = \sqrt{ \| \boldsymbol{\delta} _{\mu} \|^2 + \operatorname{Tr}( \mathbf{\Sigma} + \mathbf{\Sigma}^{*} ) }$

7. Total Discrepancy Upper Bound

Combine the bounds:

$\| \Delta \mathbf{o} _i \| \leq M _v \sum _j \left| \alpha _{i,j}^{\text{PIE}} - \alpha _{i,j}^{*} \right| + M _{\delta v} \sum _j \alpha _{i,j}^{*}$

Since $\sum _j \alpha _{i,j}^{*} = 1$:

$\| \Delta \mathbf{o} _i \| \leq M_v \cdot D _{\text{TV}}(\alpha^{\text{PIE}}, \alpha^{*}) + M _{\delta v}$

where $D _{\text{TV}}(\alpha^{\text{PIE}}, \alpha^{*})$ is the total variation distance between the attention distributions:

$D _{\text{TV}}(\alpha^{\text{PIE}}, \alpha^{*}) = \frac{1}{2} \sum _j \left| \alpha _{i,j}^{\text{PIE}} - \alpha _{i,j}^{*} \right|$

8. Final Upper Bound Expression

Thus, the discrepancy is bounded by:

$\| \Delta \mathbf{o} _i \| \leq 2 M _v \cdot D _{\text{TV}}(\alpha^{\text{PIE}}, \alpha^{*}) + M _{\delta v}$

9. Interpretation of the Terms

• $M _v$: Represents the magnitude of the value vectors $\mathbf{v} _j$. It depends on the mean and variance of the Gaussian distribution of $\mathbf{v} _j$.

• $D _{\text{TV}}(\alpha^{\text{PIE}}, \alpha^{*})$: Measures the difference between the attention distributions of PIE and full recomputation. A smaller $D _{\text{TV}}$ implies more similar attention weights.

• $M _{\delta v}$: Represents the magnitude of the difference between the value vectors $\mathbf{v} _j$ and $\mathbf{v} _j^{*}$. It depends on how much the means and variances change due to the edit.

10. Assumptions for Small Discrepancy

• Small Changes in Attention Weights:

  If the attention distributions are similar, $D _{\text{TV}}(\alpha^{\text{PIE}}, \alpha^{*})$ is small.

• Small Changes in Value Vectors:

  If the edits do not significantly change the means and variances, $M _{\delta v}$ is small.

11. Conclusion

Under the assumption that:

• The attention distributions between PIE and full recomputation are similar.
• The value vectors before and after the edit are similar (small $\boldsymbol{\delta} _{\mu}$ and small changes in $\mathbf{\Sigma}$).

Then:

$\| \Delta \mathbf{o} _i \| \leq \epsilon _{\alpha} M _v + \epsilon _v$

where:

• $\epsilon _{\alpha} = 2 D _{\text{TV}}(\alpha^{\text{PIE}}, \alpha^{*})$ is small.
• $\epsilon _v = M _{\delta v}$ is small.

Therefore, the discrepancy between PIE and full recomputation is bounded and can be made small under reasonable assumptions about the edits and model behavior.

Thank you once again for taking the time to review our paper. We would be happy to provide any additional details or clarifications if you have further questions.

Thank you for your support of our paper! We respond to each of your concerns as below.

Regarding Weaknesses 1 and Question 1: Evaluation on Code Generation tasks.
While our current work focuses on real-time editing scenarios, we agree that evaluating PIE on code generation tasks can further validate its effectiveness. We add experiments on code generation tasks, using HumanEval [1] and its C++ version from HumanEval-X [2]. We randomly choose 3 consecutive lines in the prompt as the user inserted code and report Pass@1 for different methods. The results are shown in Table R.1 below:

Table R.1: Performance of DeepSeek-Coder 6.7B on HumanEval and HumanEval-X

Table R.1 shows that PIE significantly outperforms baseline methods, closely approaching the performance of full recomputation. It's important to note that the context (prompt) in HumanEval contains only a few lines (an average of 15 lines), further highlighting PIE's effectiveness.

Regarding Weaknesses 2 and Question 2: The chosen evaluation metric.

We would like to clarify that in code completion tasks, it is standard to use EM (Exact Match) and ES (Edit Similarity) as evaluation metrics, as adopted by previous works such as CodeXGLUE [3], CrossCodeEval [4], and RepoBench [5]. Additionally, many test samples are not executable, making it challenging to evaluate Pass@1. However, we have reported Pass@1 for code generation tasks in Table R.1 above, which validates the effectiveness of PIE.

Regarding Weaknesses 3 and Question 3: Evaluation for other models.
In addition to evaluating PIE on DeepSeek-Coder, we have reported the performance of CodeLlama in Tables 5, 6, and 7 in Appendix A.2, which further validates the effectiveness of PIE. Following your suggestions, we also conduct experiments on Yi-Coder-9B (We choose it because it has the same architecture as Llama, making it convenient for code adaption). The results are shown in Table R.2 below:

Table R.2: Performance comparisons of models on insertion experiments of Repobench-C-8k Python

Table R.1 and Table R.2 further validate the effectiveness of PIE on various models (DeepSeek-Coder, Codellama, Yi-Coder) and datasets (Repobench-C, Humaneval, HumanEval-X).

We appreciate your thorough review. We have carefully replied to your questions and will enhance our paper based on your suggestions. We are happy to go into more detail regarding any further questions.

[1] Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint arXiv:2107.03374, 2021.

[2] Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x[C]//Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023: 5673-5684.

[3] Lu S, Guo D, Ren S, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation[J]. arXiv preprint arXiv:2102.04664, 2021.

[4] Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion[J]. Advances in Neural Information Processing Systems, 2024, 36.

[5] Repobench: Benchmarking repository-level code auto-completion systems[J]. arXiv preprint arXiv:2306.03091, 2023.

Thanks for your careful review and suggestions. We respond to each of your concerns as below.

Regarding Weakness 1 and Question 1: Limited Evaluations of models and datasets.

In addition to evaluating PIE on DeepSeek-Coder, we have reported the performance of CodeLlama in Tables 5, 6, and 7 in Appendix A.2, which further validates the effectiveness of PIE. Following your suggestions, we also conduct experiments on Yi-Coder-9B (We choose it because it has the same architecture as Llama, making it convenient for code adaption). The results are shown in Table R.1 below:

Table R.1: Performance comparisons of models on insertion experiments of Repobench-C-8k (Python)

Besides, we additionally conduct experiments on code generation tasks (as also suggested by Reviewer Bo49), using HumanEval [1] and its C++ version from HumanEval-X [2]. We randomly choose three consecutive lines in the prompt as the user inserted code and report Pass@1 for different methods. The results are shown in Table R.2 below:

Table R.2: Performance of DeepSeek-Coder 6.7B on HumanEval and HumanEval-X

Table R.1 and Table R.2 further validate the effectiveness of PIE on various models (DeepSeek-Coder, Codellama, Yi-Coder) and datasets (Repobench-C, Humaneval, HumanEval-X).

[1] Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint arXiv:2107.03374, 2021.

[2] Zheng Q, Xia X, Zou X, et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x[C]//Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023: 5673-5684.

Regarding Weakness 2 and Question 3: Limited Evaluations on longer edits and contextually related edits.

Following your valuable suggestions, we increase the number of edited lines. Additionally, we conduct experiments on contextually related edits. We utilize Edit Distance to identify similar lines to the target line in the context as user edits. For quick feedback, we conduct experiments (see Table R.3 below) on the Cross-File-First setting of RepoBench-C-8k, maintaining the other settings as in the paper.

Table R.3: Performance comparisons of DeepSeek-Coder 6.7B on contextually related insertion experiments

As shown in the Table R.3, PIE still maintains its performance compared to full recomputation on longer edits and contextually related edits.

Regarding Weakness 3: Dependency on RoPE

PIE indeed depends on RoPE. However, given that RoPE is the most widely used positional encoding in LLMs (e.g., LLaMA[3,4,5], PaLM[6], DeepSeek[7], Mistral[8,9], Phi[10,11], Baichuan[12], Qwen[13], Grok[14], Yi[15]), our method is compatible with a broad range of models.

[3] Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models[J]. arXiv preprint arXiv:2302.13971, 2023.

[4] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.

[5] Dubey A, Jauhri A, Pandey A, et al. The llama 3 herd of models[J]. arXiv preprint arXiv:2407.21783, 2024.

[6] Chowdhery A, Narang S, Devlin J, et al. Palm: Scaling language modeling with pathways[J]. Journal of Machine Learning Research, 2023, 24(240): 1-113.

[7] Guo D, Zhu Q, Yang D, et al. DeepSeek-Coder: When the Large Language Model Meets Programming--The Rise of Code Intelligence[J]. arXiv preprint arXiv:2401.14196, 2024.

[8] Jiang A Q, Sablayrolles A, Mensch A, et al. Mistral 7B[J]. arXiv preprint arXiv:2310.06825, 2023.

[9] Jiang A Q, Sablayrolles A, Roux A, et al. Mixtral of experts[J]. arXiv preprint arXiv:2401.04088, 2024.

[10] Gunasekar S, Zhang Y, Aneja J, et al. Textbooks are all you need[J]. arXiv preprint arXiv:2306.11644, 2023.

[11] Li Y, Bubeck S, Eldan R, et al. Textbooks are all you need ii: phi-1.5 technical report[J]. arXiv preprint arXiv:2309.05463, 2023.

[12] Yang A, Xiao B, Wang B, et al. Baichuan 2: Open large-scale language models[J]. arXiv preprint arXiv:2309.10305, 2023.

[13] Bai J, Bai S, Chu Y, et al. Qwen technical report[J]. arXiv preprint arXiv:2309.16609, 2023.

[14] https://github.com/xai-org/grok-1

[15] Young A, Chen B, Li C, et al. Yi: Open foundation models by 01. ai[J]. arXiv preprint arXiv:2403.04652, 2024.

Dear reviewer,

Thank you for your constructive review and comments! We have provided point-to-point responses to your review. Considering the author-reviewer discussion is ending very soon, could you kindly take a look at our responses and let us know your feedback? We are happy to discuss more if you still have any concerns. Thank you again and look forward to your feedback!

Thanks for your careful review and suggestions. We respond to each of your concerns as below.

Regarding Unrealistic Setting for Interactive Editing and Unconvincing Conclusions Due to Limited Evaluation Scope

We opted for a random edit setting due to the lack of real-world datasets that capture real-time user editing with corresponding labels. Since no such datasets exist, we leveraged the current dataset and constructed a random edit scenario. This allows us to ensure comprehensive coverage of diverse editing scenarios without making assumptions or imposing restrictions on user input. In real-world applications, user inputs are highly varied and unpredictable, and random edits, which can occur at any point in the code, provide a more general and robust evaluation of our method across a wide range of potential situations.

Additionally, following your valuable suggestions, we incorporated a reuse baseline and conducted experiments in more diverse editing conditions, specifically on: (1) multiple edits and contextually related edits, and (2) fill-in-the-middle (FIM) generation, as detailed below.

1. Multiple edits and contextually related edits

To simulate multi-edit scenarios, we increase the number of edited lines. We also design a contextually related insertion setting, where we use Edit Distance to identify lines in the context that are similar to the target line, simulating user insertions. We conduct experiments (see Table R.1) on the Cross-File-First setting of RepoBench-C-8k, keeping all other settings the same as in the paper.

Table R.1: Performance of DeepSeek-Coder 6.7B on contextually-related insertion experiments

The reuse baseline has been newly incorporated following your suggestion. The results in Table R.1 demonstrate that PIE is robust in handling longer edits and outperforms the baseline methods, approaching the performance of full recomputation.

To further address your concern about the effectiveness of PIE, we conduct experiments on code generation tasks (as suggested by Reviewer Bo49), using HumanEval [1] and its C++ version from HumanEval-X [2]. We randomly choose three consecutive lines in the prompt as the user inserted code and report Pass@1 for different methods. It's important to note that the context (prompt) in HumanEval consists of only a few lines (with an average of 15 lines), making each edit semantically significant. The results are shown in Table R.2 below:

Table R.2: Performance of DeepSeek-Coder 6.7B on HumanEval and HumanEval-X

The results in Table R.2 demonstrate that PIE significantly outperforms the baseline methods, approaching the performance of full recomputation. Given the importance of these edits in such short contexts, updating the KV cache with PIE becomes essential for maintaining high prediction accuracy, whereas the performance of the reuse baseline drops substantially.

2. Fill-in-the-middle (FIM) generation

Since RepoBench doesn't provide the right context, we use the CrossCodeEval dataset [3] for left-to-right evaluation. We utilize the complete left context and prompt, truncating the right context to a maximum of 40 lines to construct the fill-in-the-middle task. We use Edit Distance to identify five consecutive lines in the left context that are similar to the target line, simulating user insertions. The results are provided in Table R.3 below.

Table R.3: Performance of DeepSeek-Coder 6.7B on Fill-in-the-middle task

Table R.3 shows that PIE still maintains performance in the Fill-in-the-middle task, which is consistent with the left-to-right task.

Regarding Limited Technical Novelty

Our paper's primary contribution is PIE, an simple yet effective approach designed for real-time code editing. While previous research has largely overlooked the simultaneous modification of context and model representations (KV cache) in dynamic code editing scenarios, our work directly addresses this gap. By resolving temporal confusion in real-time editing, PIE offers a novel solution for improved efficiency in dynamic environments. We believe that by resolving temporal confusion in real-time code editing, PIE opens new pathways for improving LLM efficiency in dynamic environments, contributing a novel and impactful solution to the field.

We appreciate your thorough review of our paper. We have carefully replied to your questions and will improve our paper according to your suggestions. We look forward to your re-evaluation of our submission based on our responses, and we are also happy to go into more detail regarding any further questions.

[1] Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint arXiv:2107.03374, 2021.

[2] Zheng Q, Xia X, Zou X, et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x[C]//Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023: 5673-5684.

[3] Ding Y, Wang Z, Ahmad W, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion[J]. Advances in Neural Information Processing Systems, 2024, 36.

Dear reviewer,

Thank you for your constructive review and comments! We have provided point-to-point responses to your review. Considering the author-reviewer discussion is ending very soon, could you kindly take a look at our responses and let us know your feedback? We are happy to discuss more if you still have any concerns. Thank you again and look forward to your feedback!

In real-world applications, user inputs are highly varied and unpredictable, and random edits, which can occur at any point in the code, provide a more general and robust evaluation of our method across a wide range of potential situations.

I respectfully disagree with this claim. Realistic edits are often temporally and contextually correlated, as users tend to modify interdependent parts of the code. For example, after editing a function signature, a user is likely to update related call sites. In such cases, the ability to attend to newly edited code is essential for maintaining prediction accuracy. Random edits, by contrast, fail to capture these dependencies, making them a poor representation of real-world editing scenarios and reducing the relevance of your evaluation.

Additionally, following your valuable suggestions, we incorporated a reuse baseline and conducted experiments in more diverse editing conditions, specifically on: (1) multiple edits and contextually related edits, and (2) fill-in-the-middle (FIM) generation, as detailed below.

Thank you for including the reuse baseline and additional experiments. However, the performance of this new reuse baseline appears to significantly outperform the original baselines used in your paper. This undermines the original baseline's relevance and raises concerns about its suitability as a comparison point. I strongly recommend incorporating the reuse baseline across all new results provided in your replies to other reviewers. The fact that the reuse baseline—essentially a "do nothing" approach—captures most of the gains PIE demonstrated over Conflict Fast Encoding further questions the significance of PIE’s remaining improvements. This raises the possibility that these improvements may stem from benchmarks crafted in ways that favor PIE, rather than inherent advantages of the method itself.

I also don’t feel that my concerns regarding the single-edit limitation have been adequately addressed by the new results that simply increase the number of edited lines. Consider the definition-call site example again: a user might first edit a function definition and then proceed to edit all associated call sites. Under PIE's attention pattern, the transformed context would be able to attend neither the newly edited definition nor any of the newly edited call sites, making it very difficult (if not impossible) to correctly predict the remaining call sites that needed an update. If your evaluation instead increases the size of a single edit site or simply picks more random edit sites, this artificially increases the chance that at least one new call site remains untouched and leaks the correct calling pattern into the context. This, however, reflects an unrealistic evaluation scenario and does not align with how PIE would behave in practical usage. In summary, PIE’s inability to allow context tokens to attend to newly edited tokens seems like a critical limitation. making it highly unlikely for PIE to match the performance of full recomputation in realistic usages. I encourage the authors to address this gap with more concrete examples or additional analyses.

That said, I acknowledge that the inclusion of the reuse baseline and the new experiments have strengthened the overall results. Assuming that the reuse baseline will be incorporated into all reported results, I am raising my score from 1 to 3 to reflect these improvements. However, I remain concerned about the broader applicability of PIE in realistic editing scenarios.

Dear Reviewer qWyU,

Thank you for your further comments and for raising specific concerns about the broader applicability of PIE in realistic editing scenarios. We appreciate the opportunity to address these points in detail.

1. Regarding the reuse baseline compared to PIE

We appreciate your suggestion to incorporate the reuse baseline as a comparison point. While the reuse baseline may capture some performance gains due to the redundancy in code, it performs poorly when user edits involve highly important and semantically significant changes. This limitation is particularly evident in HumanEval (Table R.2), where the context (prompt) consists of only a few lines (with an average of 15 lines), making each edit semantically significant.

To further expand on this, we conduct additional experiments using a more challenging setting, where nine consecutive lines were chosen as the user-inserted code. The results for this setup are shown in Table R.4 below:

Table R.4: Performance of DeepSeek-Coder 6.7B on HumanEval and HumanEval-X (9 lines)

The results in Table R.4 clearly demonstrate that the reuse baseline essentially fails in this task, dramatically underperforming compared to Conflict Fast Encoding and PIE.

2. Regarding the single-edit setting and multi-place edits

You raise the valuable example of a user editing a function definition followed by updates to all associated call sites. To simulate this scenario, we treat each line in the context as a potential site and use Edit Distance to identify multiple sites in the context that are similar to the target line, thereby simulating multi-place insertions. We conduct experiments in this contextually related multi-place editing setting on the Cross-File-First task of RepoBench-C-8k. The results are shown in Table R.5 below:

Table R.5: Performance of DeepSeek-Coder 6.7B on contextually-related multi-place insertion experiments

The results in Table R.5 demonstrate that PIE is robust in handling multi-place edits and outperforms the baseline methods.

We hope these additional experiments and analyses address your concerns and provide further evidence of PIE’s robustness and applicability to real-world editing scenarios. Thank you again for your thoughtful review and suggestions. We are happy to provide further details or conduct additional experiments if you have any further questions.

Thank you for your detailed response and for conducting additional experiments to address my concerns. I appreciate the effort in expanding the evaluation to include the reuse baseline and exploring multi-place edits. However, I have reservations about the realism of the new settings you proposed.

If I understood correctly, the multi-place editing setup appears to simulate edits by removing insertion points from the final version of the code and treating the resulting broken code as the "original code" for the reuse baseline. This does not reflect how code is typically edited iteratively over time or incrementally updated across related call sites. By using broken code as input for the reuse baseline, the comparison feels artificially skewed in favor of PIE and does not fairly evaluate the reuse baseline's performance in a realistic iterative editing workflow.

Real-world editing often involves scenarios where users progressively refine their code, and the ability to attend to new edits (e.g., function definitions and their associated call sites) is critical. This evaluation setup does not adequately capture those dynamics, which remain a key concern about PIE's applicability.

Given these issues, I am unable to update my current score. I still appreciate the additional experiments and analyses, but I encourage further exploration of settings that more accurately reflect real-world iterative editing processes. Thank you again for engaging with my feedback and providing thoughtful responses.

Dear Reviewer qWyU,

Thank you for your continued engagement and for raising concerns about the realism of our multi-place editing setup. We appreciate the opportunity to clarify our methodology and address your points in detail.

1. Clarification of the Multi-Place Editing Setup

You correctly understood our multi-place editing setup: it simulates edits by removing insertion points from the final version of the code and treating the resulting broken code as the "original code".

However, we respectfully disagree with your statement that this comparison is "artificially skewed in favor of PIE" and does not fairly evaluate the reuse baseline’s performance.

Regardless of whether edits are applied progressively or simultaneously, the reuse baseline inherently fails to capture the changes introduced by the edits, as it relies entirely on the original, unmodified context. This limitation is fundamental to the reuse baseline and is not influenced by how edits are simulated or applied. By definition, it cannot adapt to the edited context and will yield the same output regardless of the editing workflow. Therefore, we believe that our comparison remains fair and highlights the limitations of the reuse baseline relative to PIE.

Moreover, we would like to highlight the results in Table R.2 and Table R.4, where the reuse baseline essentially fails, dramatically underperforming compared to both Conflict Fast Encoding and PIE. Unlike the reuse baseline and conflict fast encoding, PIE effectively utilizes the inserted code, enabling it to retain most of the performance and achieve results much closer to full recomputation.

2. Progressive Refinement

You also mentioned that real-world editing often involves progressive refinement, where users iteratively update their code. To address this, we provide the following clarification:

In the insertion experiments, if the edits are applied progressively (i.e., each insertion is made, followed by PIE updates) or simultaneously (i.e., all edits are made at once, and PIE is applied afterward), the results will be identical as long as the insertions are orderly (e.g., from the beginning of the context to the end).

If the insertions are unordered or dispersed, we agree that there may be slight differences between progressive and simultaneous updates. However, we argue that this difference is likely to be small, as the core mechanism of PIE—resolving temporal confusion in the KV cache—remains consistent.

Thank you for your thoughtful feedback. We hope this response clarifies our experimental design and addresses your concerns about PIE's applicability in realistic scenarios. We remain committed to further improving our evaluation methodology and welcome any additional suggestions you may have.

Best regards,

Authors