Coeditor: Leveraging Repo-level Diffs for Code Auto-editing

Thank you for the review and questions. Please see our response below:

Comment: I would be curious to see how CodePlan + Coeditor perform on the PyCommits dataset (especially since the CodePlan paper partially features the results using the Coeditor model), although I realized that CodePlan was published after the ICLR submission deadline. Is it possible to add?

Regarding combining CodePlan with Coeditor for the PyCommits dataset, it's important to note CodePlan and Coditor’s different focuses. Coeditor, along with the PyCommits dataset, focuses on making precise changes within specific code regions. CodePlan focuses on orchestrating large-scope changes across multiple files, based on some clear specification and utilizing a local editing model similar to Coeditor. This difference in their operational scopes means PyCommits would not be an ideal evaluation for CodePlan + Coeditor, as CodePlan’s planning strength is not needed for PyCommits' localized edits.

---

Comment: Another interesting baseline could be BluePencil model focusing on repetitive edits (see "On the fly synthesis of edit suggestions" by Miltner et. al) this is also available through IntelliCode Suggestions VS extension.

BluePencil and Coeditor share a common goal of predicting user edits based on recent user changes. However, BluePencil uses rule-based program synthesis techniques and is limited to suggesting repetitive changes. In contrast, Coeditor has the capability to produce novel changes beyond BluePencil's scope. Furthermore, BluePencil's implementation is specific to C# programs, while Coeditor has been trained exclusively for Python. This difference in target programming language precludes a direct comparison of the two tools.

Thank you for the review and questions. Please see our response below:

Q1: Were the baseline LMs evaluated with a 1K context, same as Coeditor, or with their maximum number of tokens (when possible)?

We let all baseline LMs utilize their full context by including as much surrounding code as possible. The context size of each model is detailed in Table 3.

---

Q2: Was signature information only used in Coeditor? If so, is this intended to be a core part of the contribution? Please provide ablations with other models using this information, and with Coeditor fully trained without this information in any case.

We would first like to clarify that all our ablation results were obtained by retraining Coeditor with the corresponding information removed at training time (as mentioned at the beginning of Section 4.3.)

The signature information (based on usage-analysis) is only used by Coeditor in our evaluation and is part of the contribution of this work. While prior work has explored the use of signatures in tasks like code completion [1] and type inference [2], their value in code editing has not been studied before. In fact, our initial hypothesis was that signatures might be less critical in editing tasks, given that the model has access to the old version of the code, which can aid in inferring signatures from existing usages.

However, our ablation studies (Table 5) uncovered a significant finding: both recent editing history and signature information are crucial for editing performance, and neither alone is sufficient. In particular, the first row of Table 5 shows that not having editing history (while still keeping the signatures and a 16K context size) leads to the largest drop in performance, reducing overall EM from 42.1% to 26.1%. This result counters the statement that “a large component of the performance does not come from the presence of diffs, but from a more banal form of information that related work has already explored”.

To understand why the combination of signatures and recent editing history is so useful for this task, we manually inspected examples where Coeditor was successful and found that the editing history often clarifies the desired types of changes, while the signature information aids in identifying where similar changes should be applied. For example, if the model sees that the user has added a statement obj.some_attribute *= 2 and needs to predict a similar change obj.another_attribute *= 2 , it often requires seeing what other attributes are defined in obj’s class signature in order for the model to predict which other attributes need to be similarly mutated.

[1] Toufique Ahmed et al. "Improving Few-Shot Prompts with Relevant Static Analysis Products." arXiv preprint arXiv:2304.06815 (2023).

[2] Jiayi Wei et al. "TypeT5: Seq2seq Type Inference using Static Analysis." The Eleventh International Conference on Learning Representations. 2023.

---

Q3: How do the baseline models perform in the multi-round setting?

The intent of the multi-round setting is to evaluate how the model performs when iteratively prompted with additional user changes (if the model didn’t predict all the needed changes in the first round). Since the code infilling baselines cannot predict where and how to make edits, we had to evaluate them on a simplified code completion task. This adaptation means that we cannot meaningfully apply the multi-round editing framework to these models.

For example, if the ground truth is

-def sort_by_name(persons):
+def sort_by_name(persons, reversed=False):
     persons.sort(key=lambda p: p.name)
+    if reversed:
+        persons.reverse()
     return persons

We create the corresponding code completion problem below, where we only select the last changed line as the code completion target and apply the remaining code changes to form the new context. This creates a normal completion problem such that the resulting code (after adding the missing line) is the after-commit version of the code above.

def sort_by_name(persons, reversed=False):
    persons.sort(key=lambda p: p.name)
    if reversed:
        <target> persons.reverse() </target>
    return persons

This is similar to the last round in a multi-round editing setting, arguably the easiest part of a multi-round editing loop for the baseline models, which were not pre-trained to handle a partially edited context.

Q: I would still appreciate clarification on why Tab. 5 shows a best-case EM rate of 42.1% while Tab. 3 shows a much higher value of 60.4%.

This is because these two tables show results on different tasks. Tab. 3 shows the EM rate on the PYCOMMITS-ONELINE dataset, which is a code completion task involving only one missing line. Tab. 5 shows the single-round EM rate on PYCOMMITS, which is an editing task requiring identifying and performing multiple missing changes. Because the latter is a fundamentally harder problem, the EM rate is lower.

Comment 1: Regarding the multi-round editing task: I understand that the baseline models are not particularly compatible to this task [...] I think it's not a stretch to come up with an evaluation that uses traditional models in a similar way. In any case, I'll leave this up to you.

The fundamental difficulty of the multi-round editing task is that it requires predicting more than one change, hence if a model misses some changes in the first round, it can potentially benefit from additional changes made by the user. Code completion models like StarCoder are only trained to predict a single missing expression (via the Fill-in-the-Middle objective), so we think there is no meaningful way to evaluate such models on the PYCOMMITS dataset in a way that’s comparable to our model. In any case, we’ll consider this more deeply for future work; thanks for the suggestion!

Comment 2: It is important to acknowledge this in writing, both noting that diff information alone is often insufficient in the aforementioned sections, and either adding more equal comparisons with baselines or noting the absence of signature information in baseline models as a limitation of the evaluation.

We will ensure that both points you highlighted are addressed in our upcoming revisions. The decision to withhold signature information from the code completion models stemmed from their inherent limitations, as out-of-the-box, they typically struggle to utilize signatures effectively due to how they were pre-trained (context is effectively limited to a single document consisting of code rather than signatures). Consequently, it becomes a challenge to effectively prompt these models with signature information while also balancing the limited token budgets.

Thank you for the review and questions. Please see our response below:

Comment: The edit granularity choice and dataset can introduce bias: developers often commit many intermediate commits that get squashed into a single commit before pushing and changes can get overwritten.

This is a good point: in practice, commits in a pull request are often squashed before merging, resulting in training commits that are larger than what users typically encounter at editing time. There are several potential remedies to this issue. For example, at inference time, we could extend our approach to include editing history beyond the last commit, in order to better match the content of a squashed commit. Future work can also explore using LLMs to identify squashed commits and decompose them into smaller, semantically independent units. Notably, however, many squashed commits collected from open source repos can still be relatively small and contain closely related changes. Therefore, training our model on a substantial volume of such commits may already provide a robust training signal that is sufficient for teaching the model to utilize editing histories.

Question: With the move towards an extension, was using more granular edit events considered, what about navigation events or other similar side-channel information? Is the main concern the context size or are there other trade-offs that moved the authors closer to git diff, line-level information?

Our choice of using commit history is mainly motivated by data availability. Acquiring detailed user edit events entails installing specialized tools for data collection, posing scalability and privacy challenges that are beyond the scope of this work. A promising future direction, however, could involve a hybrid approach that combines large-scale pre-training (using the commit-based data proposed by this work) followed by fine-tuning with a smaller dataset of detailed edit events.

GPT-3.5-turbo experiments

Since GPT3.5 is not a code infilling model, we tried different ways to prompt it for this task and report two results below. The first prompting method, called “GPT3.5-plain”, simply put the suffix before the prefix using the format

{suffix code}
—---
{prefix code}

and set the message role to “assistant”. This way, we encourage the model to send a reply that continues generating the remaining prefix code, and we take the first line in the reply as the model suggestion.

The second prompting method we tried, “GPT3.5-instruction”, formulates a one-shot prompt. We send the following instructions to the model using the role “user”:

You are a programming expert tasked to fill in a missing line for a given Python code
snippet. The snippet may have been truncated from both ends, and the missing line is
indicated by a special token `<MISSING LINE>`.
You should output the missing line (along with any leading whitespaces) and
nothing more. For example, if the input is
—
def fib(n):
   if n < 2:
<MISSING LINE>
   else:
       return fib(n-1) + fib(
—
Your output should be "        return 1" (without the quotes) and nothing more.

Now fill in the code snippet below:
—
{prefix code}<MISSING LINE>{suffix code}
—
Your output:

However, both prompting strategies performed worse than text-davinci-003, as indicated in the table below, so we did not include these results in the paper.

---

Q4: Can we use instruction finetuned version of the code generation models for code editing task?

While there is recent work focusing on instruction-based editing, our approach focuses on a markedly different setting that doesn't require user instructions and is more similar to traditional code completion tools. Combining explicit developer instructions with editing history into our model could be an interesting direction for future work.

---

Q5: Instead of a seq2seq model, can we use a decoder-only LM for the editing task?

A decoder-only LM could potentially be adapted for the task presented in this work. However, some specific designs, like how T5’s mask tokens are used to model editing as masked span infilling, would likely require some adaptation for decoder-only models to work well on this task.

---

Q6: Is it possible to evaluate Coeditor on the PIE benchmark [1]?

The focus of this work is predicting probable code edits based on recent programmer changes made elsewhere, whereas the PIE benchmark focuses exclusively on performance-improving program rewrites without conditioning. Therefore, the PyCommits dataset differs fundamentally in its objectives from the PIE dataset.

Thank you for the review and questions. Please see our response below:

Comment: The primary issue of the paper is over claim when it compares with SOTA code infilling models […] The paper didn't explain clearly how the generic infill models are prompted to solve the code editing tasks. Therefore, there is no way we can fairly compare the proposed approach with the baseline models compared in this work.

Our comparison with infill models is detailed in Section 4. We extracted single-line code completion problems from real-world edits and fed the prefix and suffix of each problem to the infill model, assessing whether it can accurately output the correct middle span. We discuss other prompting strategies below under Q3.

Note that the single-round editing evaluation has a structure that is designed to be fair to infilling models as follows: if the ground truth changes in a code unit $u$ contain multiple changed lines, we only select the last changed line as the code completion target and remove it from $u$, and we apply the remaining code changes to $u$. This creates a normal completion problem such that the resulting code (after adding the missing line) is the after-commit version of $u$. For example, if the ground truth is

-def sort_by_name(persons):
+def sort_by_name(persons, reversed=False):
     persons.sort(key=lambda p: p.name)
+    if reversed:
+        persons.reverse()
     return persons

We create the corresponding code completion problem as

def sort_by_name(persons, reversed=False):
    persons.sort(key=lambda p: p.name)
    if reversed:
        <target> persons.reverse() </target>
    return persons

This is similar to the last round in a multi-round editing setting, arguably the easiest part of a multi-round editing loop for the baseline models, which were not pre-trained to handle a partially edited context.

---

Q1: Why Coeditor is trained with a fixed batch size of 1? The Nvidia GPU used to train the model has 48GB memory which should be good to accommodate batch size > 1 with a 220M param model, right?

In our training setup, we utilize about 20% of the available GPU memory, achieving roughly 70% average GPU utilization. While a larger batch size is feasible, the block-sparse attention pattern (which varies for each training example) coupled with the diverse context sizes of different editing problems (ranging from no reference blocks to 15K reference block tokens), limited the benefits of larger batch sizes for our task.

---

Q2: Does the PyCommits dataset composed of Github projects that have permissive licenses?

Yes, as stated in Section 3.4's last paragraph, we constructed the PyCommits dataset from the commit history of 1,650 Python projects with permissive licenses (MIT, Apache, BSD) sourced from GitHub.

---

Q3: How the generic infill models are prompted to solve the code editing tasks? Is few-shot prompting used for the models? For example, [1] used demonstration augmented prompting for Codex model (when finetuning is not possible).

Given that the single-line code completion task is closely related to the original infilling task, we did not employ few-shot prompting for the text-davinci-003 model. Instead, we utilized the maximum available context size (4K tokens) to store as much surrounding code as feasible. The reported performance of text-davinci-003 was obtained by directly calling OpenAI’s infilling API, which expects a prefix and a suffix.

We did experiment with different prompt formats on GPT-3.5-turbo, as described below (in the next comment).