NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits

We thank the reviewer for the insightful feedback on our work. We answer the questions asked by the reviewer below:

My concern lies in why the LLM-generated dataset is realistic and representative? Specifically, ..., how could the synthesized dataset to represent such distribution.

Our synthetic data generation, by design, supports multiple correlated edits within a single task. Indeed, our pipeline generates multi-edit examples and we will add concrete examples in the appendix in the revised version. This approach mirrors real-world software development, where changes often span multiple classes and interrelated code segments. By generating data that captures the complexity of correlated changes, we ensure an authentic representation of complex code editing scenarios.

To further validate our dataset's ability to handle multi-file edits, we finetuned larger QwenCoder-2.5 variants and evaluated them on Aider Polyglot, a challenging benchmark with multi-file edit problems. Due to char limits, we are unable to give a concrete example in this response. However, we would be happy to share it if the reviewer wants, in another response. Our NextCoder models show significant performance gains, as summarized in the tables below.

I suggest the trained model shall be evaluated on some real-world commit datasets like CoEdPilot.

We agree that evaluation on real-world commits is essential -- and we have already done this. A subset of the NoFunEval benchmark [CoLM 2024], which includes five splits assessing a model’s ability to improve code on multiple non-functional requirements (e.g., runtime, safety, etc.), is derived from real-world commits in Android repositories. In particular, the latency and resource-utilization splits are derived from real commits. The results of these evaluations are discussed in Section 5.2 of our paper.

Regarding CoEdPilot, while it represents an interesting benchmark derived from real-world commits, its focus on edit propagation (predicting edits across multiple locations based on patterns of previous edits) differs substantially from our work's objectives. Our approach aims to perform targeted code edits based on natural language instructions, rather than propagating edits across a codebase. Additionally, CoEdPilot's emphasis on fine-grained edit detection and classification at the line level (keep/insert/replace) wouldn't align well with our instruction-following paradigm. The fundamental difference is that our model requires explicit natural language instructions, whereas CoEdPilot infers edit patterns automatically. If required, we will manually write instructions for some of the instances from CoEdPilot and include results in the revised version.

Code editing can involve both edit location and edit generation, as in CoEdPilot. The authors seems to miss the edit generators such as CODIT.

We thank the reviewer for pointing out CoEdPilot and CODIT; however, these methods target distinct scenarios. CoEdPilot focuses on edit propagation, predicting future edits based on past edits, and employs a fine-grained edit detection mechanism (Edit-propagating Line Locator) to classify edit types (keep/insert/replace) at the line level. In contrast, our approach finetunes a code LM to follow natural-language instructions for editing a given codebase, without requiring past edits or explicit edit localization. CODIT, on the other hand, predicts repetitive edits using a tree-based neural machine translation model, focusing on structured edit patterns rather than general-purpose instruction-following for code modifications. Additionally, CODIT predates modern LLM-based approaches, making its methodology different from ours. These distinctions clarify that our work does not overlook edit generators but instead addresses a separate formulation of the code-editing problem. We will incorporate these clarifications, comparing against CoEdPilot and CODIT, into the paper.

How do you evaluate the usefulness of the generated synthetic data?

In Section 5.4, we provide empirical evidence demonstrating the utility of our synthetic data compared to traditional commit-based datasets. Specifically, we observe that fine-tuning DeepSeek-6.7B on our synthetic data yields a performance advantage over CommitPackFT (a filtered dataset of high-quality commit data from GitHub).

We thank the reviewer for the insightful feedback on our work. We answer the questions asked by the reviewer below:

Authors have not done any analysis on generating data overlap with benchmark ...

The benchmarks considered in the paper are based on manually-created coding problems and solutions. Whereas, we used the training split of the StarCoder dataset, which is derived from GitHub commits, as our seed data. The synthetic data pipeline generates samples inspired by the seed data. This reduces the likelihood of unintended overlap with the benchmark data and prevents inadvertent overfitting.

We agree on the importance of this point with the reviewer and quantitatively analyze overlap using the standard approach of decontamination used in StarCoder: bigcode-dataset/decontamination. The decontamination report confirms 0% overlap between our training data and benchmarks.

We will include a discussion on this and report the decontamination measurement in the revised version.

Lack of hyperparam tuning for baselines ...

We have documented the specifics of our SFT and LoRA hyperparameters in Appendix A.1.2. For LoRA, we utilized the hyperparameters from the Qwen model official implementation scripts. To address the reviewer's concern, we have conducted an initial experiment on hyperparameter tuning for baseline methods. Given that SeleKT was trained with a learning rate of 1e-5 and weight decay of 0.0, we explored closer configurations for LoRA and SFT, testing learning rates of 2e-6 and 5e-6 with weight decay values of 0.10 and 0.05. Additionally, for LoRA, we experimented with ranks from 16 to 64 and alpha values of 8 and 16. While this tuning led to some improvements in both SFT and LoRA, a significant performance gap remains between these baselines and NextCoder-7B. We will include these updated results in the final version.

Not clear for how many steps the model was trained.

We trained all our models for 3 epochs (Section 5.1).

Theoretical insights into the optimization dynamics or convergence behavior in non-convex settings

Please refer to a detailed response to reviewer 3QHQ (3rd point).

Lack of data scaling curve

Please refer to a detailed response to reviewer 3QHQ (2nd point).

To measure model's ability to retain generic performance, can you also evaluate it on other tasks such as MMLU?

In addition to (a) the additional experiment on MMLU as suggested by the reviewer, we also conducted (b) experiments on GSM8K to further demonstrate the ability of our method to retain generic performance.

For evaluation, we followed the few-shot setting (N=4) and the same prompt used in the Qwen models' official evaluation script. Given that our model is designed for code-related tasks, we focused on the STEM subset of MMLU, which contains 3.15K problems covering topics such as: Physics, Chemistry, Biology, Computer Science, Mathematics and Engineering. This subset aligns closely with the problem-solving and computational reasoning abilities expected from a code-editing model, making it a more meaningful evaluation of whether fine-tuning on code has impacted general problem-solving performance. For GSM8K, we considered the full benchmark.

The above table presents the accuracy scores for our NextCoder models alongside the Qwen2.5-Coder models. These results substantiate the robustness of our approach, in particular the absence of catastrophic forgetting is evident. We will add these results to the paper.

Underlying LLM for data generation might introduce systematic biases or fail to capture rare-code editing scenarios.

The reviewer raises important issues which can affect effectiveness of synthetic data generation. To mitigate these issues, we implemented a multi-faceted approach to ensure diversity and reduce systematic biases. Our synthetic data generation strategy relies on a diverse set of seed data (which, for example, is significantly larger than WizardCoder’s 20K instances) as the basis (Section 3, point i), ensuring a broad initial representation of code editing contexts. We further enhance the coverage/diversity of our generated scenarios by incorporating three randomly selected improvement areas (Section 3, point i) for each synthetic data instance. This approach helps prevent the model from converging on a narrow set of editing patterns.

By deliberately introducing randomness through multiple improvement areas and using a diverse initial dataset, we have tried to mitigate bias and coverage concerns.

We thank the reviewer for the insightful feedback on our work. We answer the questions asked by the reviewer below:

The scarcity of high-quality fine-tuning data and the risk of catastrophic forgetting during fine-tuning are well-known problems in ML models in general. This raises concerns that the paper’s contribution may not be sufficiently distinct.

While we agree with the reviewer that catastrophic forgetting and lack of high quality finetuning data are well-known problems, we are unable to understand why this implies that our contribution is not sufficiently distinct.

In fact, we do contrast and experimentally compare with some of the recent, key ML work in this space (PEFT/LoRA, model merging/TIES) throughout the paper. Our approach offers a conceptually novel solution. We introduce a novel weight update algorithm specifically designed to mitigate catastrophic forgetting during fine-tuning. Our strong results over model families, sizes, and benchmarks in the code-editing domain show improvements over standard techniques (SFT, LoRA, TIES).

Further, our synthetic data generation pipeline also is a key contribution, that places emphasis on getting high-quality data tailored to the intricacies and diversity of real-world code-editing tasks and scenarios. Overall, our approach goes beyond existing methodologies and offers a nuanced, yet easy-to-implement approach to model adaptation.

The reviewer has examined the proposed SeleKT algorithm for parameter optimization and considers it a general methodology for artificial neural network adaptation. However, the reviewer is uncertain whether its design choices are specifically tailored for the code editing task.

We agree with the reviewer about the potential generality of SeleKT. However, in this work, our motivation and focus is to improve the code-editing performance without sacrificing pre-learned abilities like code generation. The synthetic data generated for finetuning (Section 3) represents the design choices tailored specifically to the code-editing task. Code editing, and more generally coding models, is an important AI domain today, and our work clearly shows the method's potential. We note that in the present form, we have been careful not to make any claims about generality of SeleKT beyond what we demonstrate in the paper. We plan to investigate the applicability of SeleKT to more domains like math and natural language reasoning in the future.

The reviewer has a particular interest in the quality of the generated dataset. In Line 210 (L), the authors state that they perform quality-based filtering by prompting the LLM to select high-quality examples. However, the reviewer believes that this step requires some level of human intervention ... Conducting an empirical study to assess whether the LLM's evaluation standards align with those of human experts.

Doing manual labeling for individual samples is impractical given the size of the synthesized dataset (100K's). We follow a long line of work in the literature on using LLM-as-a-judge to scale quality check.

During the process of designing the synthetic data generation pipeline, we continuously monitored the quality of the generated data. Based on our observations, we implemented a stringent quality check process (Section 3, point iv; Appendix A.2, Figure 7) that filters out low-quality samples. Only instances meeting the specific criteria are retained. Recognizing the significant effort required for human expert labeling, our methodology provides a scalable alternative. Our experimental results clearly show improvements upon finetuning with the synthesized data. Nevertheless, following up on the reviewers' suggestion, we will conduct a study to evaluate agreement between the LLM and human reviewers on sample quality and include our findings in the paper.

A minor concern is whether the prompt length may exceed the model's context window (during data generation).

Thank you for raising this concern. To clarify, we used GPT-4o and Llama-3.3-70B for data generation, both of which have sufficiently large context windows to accommodate our prompts. Therefore, exceeding the model’s context length was not an issue during the data generation process.

We thank the reviewer for the insightful feedback on our work. We answer the specific questions below:

Reason for selecting some specific hyperparameters for seleKT.

Our preliminary experiments indicated that a sparsity value of 5% and periodicity of 1 epoch were effective across model families, sizes, and benchmarks. Due to limited compute, we stick to these values in our experiments. Nevertheless, in Section 5.5, we present ablation on these two key hyperparameters of our method, namely, the sparsity $\alpha$ and periodicity $M$. By separately tuning the hyper-parameters for each model, we can get further improved performance for our approach. As suggested by the reviewer, we will include a discussion of hyperparameter selection and tradeoffs in the revision.

Robustness of the approach across dataset sizes

In addition to answering the reviewer's question on (a) robustness to dataset sizes, we are also including new results on (b) robustness across model sizes.

(a) Robustness across dataset sizes

We appreciate the reviewer’s suggestion to evaluate effect of dataset size. To assess scalability w.r.t. to training data size, we finetuned the QwenCoder-2.5-7B model on varying fractions (random sampling 25%, 50% and 75%) of our dataset which includes both synthetic and CommitPackFT data. The results are presented in the table above. All models were trained for 3 epochs. The results show a clear trend: while performance on CanItEdit and Aider sees a drop at 25% w.r.t to the base model, increasing the dataset size consistently improves performance across all benchmarks (CanItEdit, HumanEvalFix, and Aider).

(b) Robustness across model sizes

Additionally, we are happy to share new results (the tables below) comparing the performance of our SeleKT algorithm across various model sizes against supervised finetuning (SFT) and parameter-efficient low-rank adaptation (LoRA) on multiple code editing benchmarks.

For the smaller 3B model, NextCoder-3B shows significant improvements over the base model across most benchmarks, with a substantial gain on the CanItEdit benchmark (+5.3%).

For larger models, we have also included the latest and more challenging Aider-Polyglot benchmark results (more details in response to reviewer itGF).

Limited theoretical analysis of the mechanism

Though the focus of our paper is on rigorous empirical validation, we have made some progress on theoretical understanding, that we outline next. Under the standard smoothness assumption on $f$ and boundedness assumptions on the gradients and concentration assumption on the task vector stated below, we can show $O(1 /\sqrt{T})$ convergence for SeleKT. Due to char limits, we are unable to give a proof sketch here.

Task Vector Concentration Assumption

The task vector $\tau = \theta - \theta_{base}$ exhibits a concentration property with parameter $r >> 1$, such that the majority of information is contained in the top-$\alpha$ fraction of parameters.

$$\frac{{\sum_{i \in \text{top-}\alpha} |\tau_i|^2}}{\alpha N} \geq r \cdot \frac{{\sum_{i \notin \text{top-}\alpha} |\tau_i|^2}}{(1-\alpha)N} , \quad r >> 1$$

Note that $r$ defines a certain margin between the task-specific parameters (i.e. top-$\alpha$ fraction of parameters) and the rest of the parameters (both suitably normalized).

Besides from codes and instructions, would there be possible application...

Please refer to our response to reviewer UeAD (2nd point).