Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions

Thank you for the insightful review and suggestions on our work.

Missing references

We will cite all of these and acknowledge Guo et al. as concurrent work.

Please provide more details about the dataset creation procedure.

The annotators were recruited through academic networks, these included researchers and engineers in formal methods, programming languages, AI, astrophysics, and mathematics, all of whom use Python heavily in their work. They were encouraged to come up with problems covering a range of difficulty to differentiate models that are very strong at code editing tasks from ones that are not. The annotators were provided with the breakdown of questions from the outset, and each was told what topic and change type to focus on. For each problem, the annotator wrote both the lazy and descriptive text, which later required approval by the lead. There was no starter code for the annotators, however, the lead gave annotators example problems to look at before they began. They were taught how to write the questions unambiguously to give the model a fair chance of passing the unit tests. For example, an ambiguous instruction may cause the model to implement a function body correctly but with a name that was not consistent with the unit test. The project lead reviewed all instructions and several model completions to make sure this wasn't the case.

Does this evaluation benchmark include [documentation-related] edits?

This benchmark does not deal with updates in documentation such as comments or README's, since they can not straightforwardly be tested with unit tests.

"GPT-4…" Could you explain how you came to this conclusion?

It is true that GPT-4 is certainly trained on data involving code. We meant is that it's not trained primarily on code, which is the case for models such as StarCoder. We will clarify in the revised version.

It's not clear why Table 4 was not presented like Table 3

In our ablations we were interested in identifying the overall most effective training configuration, thus we displayed the overall pass@1 and ExcessCode scores. We will include the table with lazy and descriptive columns in the revised version.

Table 6 does not include all models in Table 3.

We provided only EditCoder and OpenAI models in Table 6 to compare the best open models to closed models. We agree that showing all models would be insightful and will provide the full table in the main body for the revision.

Thank you for the insightful review and suggestions on our work.

The evaluation metric is limited as there are only pass@k and ExcessCode in metrics. Authors might propose some refined assessment techniques for code editing tasks or add human evaluation.

We note that the outputs of certain models did go through human inspection to ensure that all questions were unambiguous. During this process, we were able to get a qualitative understanding of how our models were performing. We provide some of these completions in Appendix D. We would be happy to include a more comprehensive human evaluation in the camera-ready if needed for acceptance.

Similar contributions with Guo et al.[1]. Authors might consider to compare with this work if possible.

Please note that Guo et al. appeared on ArXiv in April 2024, after the COLM submission deadline. Guo et al. sources problems from popular competitive programming websites, such as LeetCode and CodeForces, thus focusing on data structures and algorithms, while our benchmark encompasses several other topic categories. Additionally, our dataset features unique sets of dual (lazy and descriptive) instructions for each problem, unlike the hardcoded prompts used in their edit categories. We will acknowledge Guo et al. as concurrent work in our revision.

The code languages types are limited since there is only Python language.

We acknowledge that there are inherent limitations to a benchmark of problems written only in Python. However, mechanically translating Python benchmarks to other languages is an active research topic [1, 2, 3]. We identify expanding our benchmark to other languages as an opportunity for future work.

[1] Cassano et al., MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation

[2] Orlanski et al., Measuring The Impact Of Programming Language Distribution

[3] Athiwaratkun et al., Multi-lingual Evaluation of Code Generation Models

Thank you for the insightful review and suggestions on our work.

A potential concern arises from the limited scope of the benchmark, which encompasses only 105 manually crafted instructional code editing problems. This constraint could result in a restricted coverage of domains within the evaluation.

We acknowledge that CanItEdit only contains 105 tasks, but we point out the following:

1. We provide two variants of task specifications per task, effectively creating 210 problems of varying difficulty.
2. Several other notable code generation benchmarks are composed of between 100-200 tasks, such as HumanEval [1] and LeetcodeHardGym [2]
3. Among the 210 problems, CanItEdit covers over 5 broad categories topics, with 3 different expected change types.

We hope that the high quality and diverse range of tasks within our dataset effectively mitigate limitations posed by its modest size, ensuring robust and comprehensive evaluations.

[1] Chen et al., Evaluating Large Language Models Trained on Code

[2] Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning

Thank you for the insightful review and suggestions on our work.

Quantified correlation with the code generation capability.

We agree that code generation and code editing capability are correlated. It's possible that if a model exhibits strong code generation capabilities, it will also show strong performance on our benchmark. In Appendix A.4 we provide a table with results of some of our models on the HumanEvalPack benchmark, which includes a code generation split (Synthetize). We compute the Pearson correlation coefficient between the Synthetize scores and the average between the Lazy and Descriptive scores on CanItEdit. We find a coefficient of 0.661, suggesting a moderate positive correlation. We will add this result to the revised version of the paper.

Lacking of human evaluation. It would be better if the authors could provide the performance of human annotators in the benchmark as a guideline for the (half) open-ended problems.

We would be happy to include this in the camera-ready if needed for acceptance.

if all the "before" segments do not met the requirements, do they still fall into any of the three task categories?

We categorized problems as corrective, perfective, or adaptive based on the changes applied in the ground truth solution. A model may produce an edit that does not conform to the labeled changed type. However, such an edit likely won't pass the test cases, as we have built them to reflect the correct change kind. We will clarify in the revised version.

Can the authors provide clarification on the "relatively small" and "quite low" from "The mean Levenshtein distance between the ‘before’ and ‘after’ code segments is 197.1 characters, indicating that the changes are relatively small. We also analyze the distribution of the commit message lengths, and find that the mean token count is 10.1, which is quite low."?

We believe these numbers are small with respect to the ones found in our benchmark. In our benchmark, the mean Levenshtein distance is 302.1, while the mean token count is 81.7 for descriptive instructions and 35.6 for lazy instructions. We will clarify in the revised version.

Is there any overlap between the two training sets?

We have found 4 overlapping items between EditPackFT and Commits2023FT by finding exact matches between the before, instruction, and after chunks. These overlapping items are filtered out by the MinHash+LSH deduplication process when the two datasets are combined.