Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

How does this approach handle a code file that depends on other files in a repository? In such cases, applying the linter only to the code file will raise an error.

Linters like pylint are designed to properly understand and to correctly reflect inter-dependencies between files during error analysis. For datasets containing entire repositories, LintSeq could be run across each file from the “root” repository level, at which other files with potential dependencies would be “visible” to the linter. In this way, synthetic edit sequences could be generated at the repository level. Each edit in the training dataset would need to be modified to also contain information about the name/relative path of the file being changed.

In summary, it is possible to apply LintSeq to entire repositories, though our implementation of the algorithm and the format of the individual “diffs” used during training would need to be modified to support this.

---

Thank you once again for your feedback on our submission. Please let us know if you have any outstanding comments, concerns, or feedback that prevent a recommendation for acceptance.

Dear Reviewer,

Thank you for taking the time to provide comments on our paper. We are glad that you find the research direction explored in our paper to be interesting for future work.

There are a number of important changes that we have made to our paper in response to reviewer comments and feedback, which are reflected in the updated PDF. A complete summary of these changes is provided in our general response.

The authors only evaluated the models on HumanEval and MBPP. While these two benchmarks are popular for most general-purpose LLMs, they are biased toward simple programs and do not cover practical coding scenarios.

To address this limitation, we have added two additional benchmarks to our evaluations of fine-tuned Gemma 2, Phi 3, and Llama 3.1 models that cover both more practical coding scenarios (DS-1000) as well as harder program synthesis problems (CodeContests). Using these new benchmarks, we show that LintSeq also improves LM performance on tasks that are more practical and harder than those in HumanEval and MBPP. We would greatly appreciate it if you could take a look at the updated version of our paper.

The paper used a Python-specific linter pylint to construct pretraining data, but it's unclear how this approach can be easily applied to other languages and be applied at scale.

The Pythonic version of LintSeq that we implemented for our experiments in this paper can be applied at scale – it requires only CPU to run and it is highly parallelizable.

There exist open-source linters like pylint for just about every programming language. As a result, applying LintSeq to code written in languages other than Python can be done simply by adding a dictionary mapping file extensions (e.g. .py) to appropriate linter function calls. We hope to explore processing general code training corpuses (e.g. the entirety of The Stack) with LintSeq in future work.

There is not enough justification for the base models used for finetuning. Why no base code LLMs?

The TinyCodeLM models that we pretrain in the paper are base code LMs. Our experiments instruction finetuning these models demonstrate that training base code LMs on instruction data refactored with LintSeq is effective for improving the quality and diversity of generations (Section 3.3.1).

Our experiments with larger “off-the-shelf” LMs were designed to not only confirm that LintSeq is effective across model scales, architectures, and tokenizers, but also that it is effective for base LMs that were not intentionally pre-trained solely for code understanding. To our knowledge, the Gemma 2, Phi-3, and Llama 3.1 models were pre-trained with general-purpose data mixtures.

We have added additional justification for this design choice to our submission (Section 3.3.2, paragraphs 1 and 2).

There is an overemphasis on pass@k when k is large. While a better pass@ larger k shows the LLM can cover a broader code distribution, by definition, pass@1 is more practical and is what people care about more.

Thank you for this comment. We have updated the framing of our results throughout the paper to emphasize the Pass@1 results for LintSeqInstruct vs Instruct vs RandSeqInstruct (i.e. linter-ablated edit sequence) models.

Besides simple code generation tasks on HumanEval and MBPP, does this approach work for practical coding such as BigCodeBench? Also, does training on edits improve code editing tasks, say CanItEdit?

Our updated evaluations on DS-1000 show that LintSeq models outperform baselines on practical coding tasks on pass@1. The tasks in this benchmark ask LMs to complete partially written code involving data science libraries like matplotlib and numpy.

One limitation of LintSeq synthetic edit data is that it reflects insertion edits only – every edit in a LintSeq edit sequence reflects an existing line of code in the source file used to generate. It is for this reason that we evaluate our models on code generation only, rather than on general purpose code editing tasks that may require models to predict deletions. To adapt LintSeq models for general code editing settings, one could continue to train them to learn to delete and/or re-write code with RL or by mixing synthetic code edits with human edit data. This is another direction that we are very excited about for future work, and we have added additional discussion of this limitation to the “Discussion” section of our updated submission PDF.

What's the intuition on why LintSeq can improve pass@1? It makes sense to me it can improve pass@ big K as the approach optimizes code generation diversity but it's interesting to me that it also gets higher pass@1 (i.e., higher-quality first sample).

There are a few possible explanations for why LintSeq may improve pass@1: (1) Synthetic edit code sequences might act as a data augmentation, since sampling several code edit sequences per program in a corpus shows models “multiple equivalent views” per example; (2) The lint-error-free nature of each sampled edits adds structure that reflects the syntax of a programming language to training data, improving the coherence / quality of the first sample.

---

Once again, we would like to thank you for your review. We would be very grateful if you could let us know if you have any outstanding questions or additional feedback on our submission.

Dear Reviewer,

We are thankful for the time that you have taken to read and review our submission.

Please note that we have made a number of significant revisions to the paper during the rebuttal period, which are reflected in the updated PDF. We would appreciate it if you could also take a look at our general response, which summarizes these changes.

LintSeq reminds me of the code sketch paper [1] which generates code via syntax-validity-preserving edits (i.e., filling grammar holes). This paper should be discussed (and compared if applicable).

Thank you for bringing this paper to our attention. There are several differences between the work of Guo et al. and our method. Namely, in our understanding, Guo et al.:

1. Study code completion in the context of uncertainty. Specifically, they investigate whether a model can be trained to decline to make predictions in parts of code where there is high uncertainty during generation.
2. Train their model (Grammformer) with reinforcement learning and a hand-engineered reward function. This model cannot be used for standard, left-to-right causal generation – instead, it must be run once on each partially expanded sequence during generation.

In contrast, our work refactors existing programs in training corpuses across chunks whose contents and structure reflect the syntax of the programming language. We do not study uncertainty and we use teacher-forced supervised learning only to finetune off-the-shelf decoder-only models for autoregressive, edit-by-edit code generation at scale.

One theoretical limitation of LintSeq is the lack of naturalness of synthetic edits. The data trains the model on random trajectories which might (if not always) not be meaningful.

Our goal in the paper is not necessarily to replace existing “natural” (i.e. human) data, but to study whether training on edits that minimally reflect the syntax of a programming language might improve LM code synthesis.

Code edits in LintSeq are insertion-only -- I can imagine cases deletion/rewriting can be used, such as writing a bunch of def f(): pass when planning the code skeleton and replacing the "pass" when real implementation starts.

We agree that deletion/rewriting are just as interesting as insertion. While it is true that the edits sampled by LintSeq are insertion-only, this limitation stems from the parameter-free nature of the algorithm (i.e. every edit in a LintSeq sequence reflects an existing line of code in the source file used to generate).

We are excited to explore different ways of addressing this limitation in future work. For example, it may be possible to continue to train LintSeq LMs to learn to delete and/or re-write code by using reinforcement learning and/or by mixing synthetic and human edit data (e.g. via datasets like CommitPack) during training.

Overall I feel the evaluation focuses on function-level code generation, which might look a bit contradictory to "inference scaling" -- as I'd expect to use inference scaling for solving much more challenging tasks, e.g., those in SWE-Bench.

We agree and have added evaluations on two additional, challenging benchmarks (DS-1000 and CodeContests) for >1B parameter models to the paper.

We would appreciate it if you could take a look at the results section (especially Figures 4 and 5) in the updated manuscript.

"Open-source edit data with good coverage over the distribution of all error-free code “diffs” is nonexistent" is debatable given high-quality code-edit datasets such as EditPackFT.

Thank you for this comment, we have revised this wording of this point in the paper to be more precise. Please note that we also discuss EditPackFT in the related work section.

Comments on writing: …

We have updated the paper with the suggested changes.

Why disabling chain-of-thought data?

We remove CoT traces from the instruction tuning data used in our experiments purely to simplify our study. We believe that the instruction + program tuning setting is the fundamental setting where LintSeq may be applied. In principle, LintSeq can be easily combined with CoT. For example, models can be fine-tuned to “switch” between predicting code edits vs natural language traces during generation. We agree that this is an exciting direction for future work.

Why pre-training TinyCodeLM instead of reusing existing pre-trained SLMs?

Our goal in this experiment is to confirm that LMs do not need to be explicitly pre-trained on code “diff” data in order for LintSeq edit sequence fine-tuning to be effective.

The off-the-shelf LMs that we fine-tune in the remainder of the paper have open weights but they do not open datasets.

Can you try sampling the model using min-p rather than top-k?

We have added test-time sweeps over min-p to our evaluations in the updated paper draft!

I find the framing of LintSeq or the explanation of it is not easily clear, and even after section 2.2 where it is broken down into phase 1 and phase 2, it is still unclear to me.

Apologies, we have added pseudocode in Figure 2 of the methods section and revised the description of the algorithm. We hope you find the updated text to be clearer.

This led me to inspect the artifact lintseq in the supplementary materials which is >300 lines of code and mentions sampling weights, computing fitness, and "children."

We apologize that the implementation of LintSeq for Python in our submission was not adequately cleaned up and documented. We have updated the code with additional documentation.

Our submitted implementation of the algorithm supports (1) a few additional Python-specific heuristics designed to reduce the number of linter calls made by LintSeq and (2) extra hyperparameters inspired by evolutionary search that can be used to extend the algorithm. For example, these parameters allow a user to up-sample a particular type of edit (e.g. indented blocks) so that such edits occur earlier in training sequences. Similarly, the parameter “children_per_round” allows a user to modulate the number of edits expanded from each sampled intermediate program state beyond 1 at every round of LintSeq.

These additional heuristics and hyperparameters were not used in any of our experiments. They should have been clearly documented in the code (and now are), and we once again sincerely apologize for the confusion.

I'd recommend the authors walk back the claims on their Tiny LMs.

We have adjusted this in the updated paper.

Some papers that came to my mind while reading this were: https://arxiv.org/abs/1810.13337 https://arxiv.org/pdf/1905.11006 (much less related)

Thank you for bringing these papers to our attention. We have added them to our related work section, which we have generally updated to reflect earlier work fine-tuning on edits.

I recommend using a different word from "verify" to not conflate it with formal verification of programs.

We have updated this in the paper.

is it possible to also compare lintseq best@50 to instruction-tuned models best@50, except with Top p = 1.0 and temperatures [1.0, 1.1, 1.2]

We ran this expanded evaluation and updated our reported results. Overall, we found that LintSeq models continue to strongly outperform baselines in diversity even after modulating sampling temperature.

Are there other ways to calculate estimated FLOPs as you have? Would it be possible to also tell me the wall clock time of running inference using the instruction-tuned models and compare that to the wall clock time of using the LintSeq models (e.g. both 14b models compared) just for more transparency.

To our knowledge, estimating total FLOPs with tokens is the standard way to compare total test time costs between models. Our evaluations were conducted across a range of hardware (i.e. nodes with different numbers of available GPUs), so we cannot directly compare wall clock time. In an effort to add transparency to the paper, we have added token-based total FLOPs comparisons to all figures visualizing the benchmark scores of models during repeated sampling (Figure 1(right) and Figure 5).

---

Thank you once again for your very thoughtful review – it greatly helped us improve our paper. We would appreciate it if you could let us know if you have any outstanding questions or concerns.

Dear Reviewer,

We are sincerely grateful for your thorough feedback on our paper. We are glad that you thought our work was very strong, and we agree that the results of our approach speak for themselves. We apologize that you felt misled by our presentation of some of these results. This was not our intention.

Please note we have made a number of changes to the paper during this rebuttal period, as reflected in the revised PDF. These changes are summarized both in our general response and in our responses to your comments below – we believe they substantially improve the clarity and strength of our work.

I found figure 1 to be misleading. It compares leading LLMs on Human Eval with K=1 to LintSeq with K=50.

We apologize that you found the previous version of Figure 1 to be misleading. Our intention had been for this figure to serve as a “teaser” comparing the performance vs total test-time FLOPs used by LintSeqInstruct and Instruct LMs at pass@50. We only included pass@1 from existing, leading, and much larger LMs with corresponding total test-time FLOP estimates to contextualize scores and compute costs.

In the updated version of the paper, we have removed the comparison of LintSeq models to leading LLMs from Figure 1. The new version of the figure shows the effectiveness of our method for improving test-time scaling curves (i.e. pass@k vs. total test-time FLOPs to get k samples) on instruction-tuned model variants on a fixed dataset.

In reality, LintSeq Pass@1 is can equal to or even significantly worse than an instruction-tuned variant of the same base model.…I believe there is no thorough discussion of this phenomenon in the main manuscript and this is an extremely important point.

We respectfully disagree. As described above, we have revised the paper to (1) include estimates of standard error for all computed benchmark scores and (2) reflect more comprehensive evaluations for >1B models. In our updated results, LintSeq statistically significantly improves performance across LMs on all but one of two of the benchmark metrics tested (HumanEval pass@1 and CodeContests pass@1). Notably, even on these metrics, the two least performant LintSeq instruction-tuned models (Llama 3.1 8B and Phi-3 14B) still achieve performance that is statistically indistinguishable from the baseline. On higher pass@k, we see strong improvement on all models and on all benchmarks.

I would also appreciate a comparison with other top LLMs for code (e.g. LLama3 or LLama3.1; CodeLLama, DeepSeekCoder, Gpt3.5) and comparisons with equal pass@1, pass@5, pass@10 to show how these other LLMs scale with more samples. I feel very mislead by Figure 1 comparing Pass@1 for big models vs. pass@50 for LintSeq and I am left feeling such a presentation is problematic.

Again, we apologize that you felt misled. The goal of our experiments was not necessarily to develop the strongest code model possible, but rather to fairly and comprehensively probe whether tuning LMs to generate code edit-by-edit by preprocessing data might improve performance over standard tuning on a fixed dataset.

We would also like to clarify that there are a few other prominent differences between LintSeq models vs the top LLMs for code that you have mentioned (e.g. CodeLlama, DeepSeekCoder, GPT3.5). First, these top LLMs are very likely to have been trained/tuned on code data with chain-of-thought (CoT) traces. In contrast, our goal in this paper was to conduct the simplest possible comparison between LintSeq vs baseline models, and so we did not tune with any CoT data. Second, the hyperparameters that were used during fine-tuning for top code LLMs were likely tuned to improve benchmark pass@1 as much as possible. We did no such tuning in this paper, though we did do some tuning of learning rates in the baseline instruction tuning setting for each model.

For these reasons, we have substantially reframed the discussion of our results in the new draft to center on only comparing LintSeqInstruct to Instruct models and ablations throughout the text, removing direct comparisons to external code LLMs.

We remain open to adding comparisons between LintSeqInstruct and Instruct models vs top LLMs to the Appendix in a future draft, if you believe it would help to improve the reader’s understanding of the method.

I feel uncertain about the motives…putting tables 10 and 11 deep in the appendix

We did not intend to “hide” these tables – the data they contain was present in Figure 4 of the submitted manuscript. In the updated draft, we have moved them to the first few pages of the Appendix and we have prominently highlighted pass@1 scores of LintSeq vs baseline instruction tuned models in Figure 3.

Dear Reviewer,

Thank you for your thoughtful feedback on our submission. We are glad that you thought our paper explores an interesting and important research direction.

Please note we have made a number of changes to the paper during this rebuttal period that we believe improve the strength of our work. These changes are summarized in our general response, and are reflected in the updated PDF.

The comparison between “LintSeqInstruct” and “Instruct” may not be fair enough. While authors have controlled the number of optimizer steps the same, it is also important to make sure that “LintSeqInstruct” and “Instruct” share the same valid training tokens (i.e., the number of tokens in the training sequences that are not padding tokens).

We perform an experiment that controls for total training token count in Section 3.4 of the paper, where we ablate the linter from LintSeq to compare LintSeqInstruct TinyCodeLM models against LMs that are instruction tuned on the same quantity of entirely randomly sampled insertions, i.e. “RandSeqInstruct.” In this experiment, both sets of models are trained for the same number of optimizer steps and on 18e6 valid training tokens. Even so, RandSeqInstruct LMs underperform their LintSeqInstruct counterparts by 24-30% on relative MBPP and HumanEval pass@1. This result suggests that performance gains from tuning on LintSeq vs standard data cannot simply be attributed to extra training tokens.

We have updated our paper to make an explicit note of the token-count equivalence of the LintSeqInstruct and RandSeqInstruct datasets in Section 3.4.

Evaluation is not comprehensive enough. Apart from HumanEval(+) and MBPP(+), more benchmarks should be included to further demonstrate the effectiveness of “LintSeqInstruct”, such as DS-1000, APPS, and LiveCodeBench.

We agree and have updated our paper with additional evaluations on two new benchmarks: DS-1000 and CodeContests.

What are the computational resources required to synthesize edit sequences using LintSeq? If it is cost friendly, will it also be able to improve pre-training performance?

LintSeq is highly parallelizable and requires CPU-only to run. We were able to generate the full training dataset used in our experiments in under 10 minutes on 64 CPU cores.

We do believe that it can also be used to improve pre-training performance, and we are really excited to explore this direction in future work.

The synthesized edit sequences may not be reasonable. While LintSeq ensures that the program at each step of the edit sequence does not contain any static error, it does not necessarily guarantee that this edit sequence is reasonable because a valid edit sequence should reflect the reasoning/thinking process of human developers, which is not guaranteed by LintSeq.

We are somewhat confused by this comment.

Open-source and sequential data that reflects the process by which human developers write code is scarce. The intention behind LintSeq is not to replace this existing human data, but to explore whether generating and training language models on synthetic edits might improve LM performance on code generation tasks. LintSeq can be used to generate pre-training scale synthetic edit sequences.

The contents of insertion edits sampled with LintSeq are informed by the syntax of the programming language that code is written in. Given a target program and a linter for the corresponding programming language, LintSeq edits do represent the minimal granularity at which a developer could make syntactically correct (i.e. linter-error-free) insertion edits to a file while writing this program. In this sense, while LintSeq edits may not reflect the reasoning/thinking used by a developer who wrote a particular file, they do reflect more abstractions of a programming language compared to “raw” code.

---

Thank you once again for the time you have taken to review our paper, and please let us know if you have any outstanding concerns that stand between us and a recommendation for acceptance.