D3: A Dataset for Training Code LMs to Act Diff-by-Diff

Dear Reviewer,

Thank you for your review. We are delighted that you found our paper to be well-written, and the problem & data formulation formulation to be clear and easy to follow.

Although authors have submitted the codebase of this paper, there is no dataset released or provided.

We would like to confirm that we are committed to fully open-sourcing the D3 dataset.

This paper only focuses on Python, raising questions about its generalizability to other programming languages.

All parts of our data preparation pipeline are simple to adapt to other programming languages.

For example, source code filtering with an LLM-Judge is trivial to adapt to another programming language. Furthermore, the LintSeq algorithm that we use for generating synthetic code edit actions throughout the paper can be run on source code written in any programming language, as long as there exists some kind of code linter or verifier for this language.

We focused on Python in this paper due to compute limitations, but we are excited to explore extending D3 to other languages and to repository-level data in future work.

---

Thank you once again for your comments. Please let us know if you have any outstanding questions or concerns that could help us increase our submission’s score.

Dear Reviewer,

Thank you for taking the time to read and review our paper, and for providing your notes!

Not clear what this means at this point in the abstract.

Thank you for this comment, we will adjust the wording in this sentence.

Why is this a point about just validation loss? Does it not also hold for the test sets / benchmarks?

Indeed, we can also report test loss. We will include this in our revision.

Shouldn't T also be indexed by a goal?

The transition function represents the change in state induced by applying an action, i.e. it is a transformation akin to "git apply" and is independent of goal. You may be thinking of the LM policy over the MDP -- the policy indeed depends on the goal.

Some basics about batch size and sequence length would be helpful to see.

We provided all of our training hyperparameters in Appendix tables 4 and 5.

I cannot determine what the SFT dataset is that the paper uses

We describe and cite the SFT datasets that we use in our experiments on lines 207 to 212 in Section 4.

Since the D3 dataset is designed to serve as an initialization for any kind of code LM post-training, our experiments treat SFT data as a hyperparameter. We evaluate LMs with and without mid-training on 4 tasks: (a) vanilla synthesis, (b) completion, (c) debugging, and (d) repository-level editing.

Existing open-source SFT datasets are designed for either synthesis or editing tasks, but not both. Thus, to support our multi-task evaluations, our experiments sweep over data mixes from two sources:

(1) LintSeq-Instruct, an SFT dataset for vanilla program synthesis with diffs, based on the Python subset of the Magicoder dataset and StarCoder2-Instruct.

(2) Python-CommitPackFT, an SFT dataset for program editing. CommitPackFT is sourced and filtered from GitHub commits.

In each experiment, we SFT models on (1), (2), and both (1) & (2). We report the best scores for each benchmark across this sweep. Note that tasks (a), (b), and (c) are in-distribution to our datasets, while (d) tests generalization.

The delta that this paper presents over Piterbarg et al. (2024) is not that large

We respectfully disagree. Piterbarg et al showed that for fixed SFT datasets, training LMs to write code with atomic edits can improve vanilla code synthesis.

In contrast, our submission is a dataset paper. Our work extends Piterbarg et al by:

(1) Merging vanilla code synthesis, code completion, and editing into a single task by training models to natively write code with atomic edits

(2) Experimenting with scalable source file preparation for mid-training, via (a) LLM-judge-based filtering of human-written code and (b) resampling files into error-free edit sequences to get diverse synthesis & completion data

(3) Testing the interaction between Piterbarg et al.-like SFT and mid-training

[The D3 dataset] is curated with standard techniques

Again, we respectfully disagree. To our knowledge, the techniques that we use for curating our dataset have not been explored at-scale for coding LMs in the open-source literature.

For example, though our filtering procedure is indeed inspired by FineMath, this dataset was released in December 2024 and the corresponding paper in February 2025 [5]. Unlike our dataset, FineMath filters natural language data, not code. We believe it is non-trivial that similar, simple LLM-judge filtering is effective for source code.

Table 2: The "compl." columns. I'm not sure whether this is relevant

The completion-style task that we designed here evaluates an important, practical capability for coding LMs: auto-complete of partially-written code given an instruction. This task is integrated into popular coding assistant tools like Cursor.

Though such evaluations are not standard at the moment, we believe that their omission is problematic and is a factor contributing to the capability gap between open- and closed-source models.

Does it make sense to evaluate the base models with the D3 prompt?

In this paper, we opt for a stripped-down evaluation procedure in order to decouple the effects of mid-training on D3 from benchmark “hill-climbing” during SFT or at test-time.

Indeed, there are a few choices that we make that contribute to overall lower Llama 3.2 scores:

• No special stop tokens
• No reasoning traces
• A SWE-Bench-like format unifying editing and synthesis. This format is used in all prompts and completions across our training data and evaluations. Examples of it are shown throughout the paper, e.g. in Appendix H (benchmark evaluations).

Please note that all of our training data (D3 & SFT datasets) reflect these formatting choices -- as a result, we believe that the comparisons between all pairs of trained models here are fair.

---

Thank you once again for your very thoughtful review! We would appreciate it if you could let us know if you have any outstanding questions or concerns.

[1] Ben Alla et al. SmolLM2 (2025).

Dear Reviewer,

We appreciate the time that you have taken to read and provide feedback on our submission. There are a few important points in your review that we would like to respond to and clarify.

It is important that the authors have a human study

Thank you for this comment. We agree that adding a human study to the paper would strengthen our work. We will conduct such a study before revising our paper.

the motivation of the dataset is to enhance the model with the ability to perform iterative and exploratory code generation like human software development.

This is not entirely the case. The core motivation of the D3 dataset is to unify code synthesis, completion, and editing into a single problem by training models to natively code with diffs or atomic edit actions during mid-training or continued pre-training.

D3 is not designed for a hill-climbing on any particular benchmark. Rather, our goal is to develop a source code dataset that can provide a solid initialization for any kind of code LM post-training.

Prior work has shown that for fixed SFT datasets, training LMs to write code with atomic edits (by resampling completions in training data into sequences of synthetic & error-free diffs with LintSeq) can improve vanilla code synthesis [1].

This submission extends prior work by:

(1) Experimenting with source file preparation for mid-training or late pre-training at scale, via (a) LLM-judge-based filtering of human-written code and (b) resampling files into error-free edit sequences to get diverse synthesis & completion data

(2) Studying performance on tasks beyond vanilla code synthesis, like code completion and editing.

To our knowledge, this paper is the first to test any of (1a), (1b), or (2).

On SWEBench, the results seem quite low even in the Oracle setting.

Please note that the scores of our 1b and 3b Llama models on the Oracle SWEBench setting are of a comparable order of magnitude to the scaffold-free scores of SWE-Llama 7b from Jiminez et al [1] (~3% pass@1 Resolved, ~66.78% pass@1 Apply). SWE-Llama 7b is more than 2x as large as our 3b model and was also fine-tuned in-task on the train split of SWE-Bench.

In contrast, our evaluations test generalization to SWEBench. Our models are trained on large-scale atomic edit data but are not fine-tuned on any SWE-Bench-like or even multi-file edit data. To improve the clarity of this point, we will add SWE-Llama 7b scores to Table 3.

We chose to evaluate on the Oracle setting and without scaffolding in order to verify that mid-training on our dataset can improve performance independent of scaffold choice. We agree that it is possible to design a SWE-Agent-like scaffold for our models that would substantially improve pass@1, but we believe this to be outside of the scope of this paper.

results are quite limited with most of the results are on simple tasks such as HumanEval and MBPP

First, we would like to bring your attention to the fact that outside of SWEBench, we also evaluate on both “vanilla” and “completion” variants of HumanEval and MBPP as well as on HumanEvalFix, a popular benchmark for single-file code debugging. Together, these three benchmark variants evaluate our models on three types of coding tasks: synthesis from-scratch, completion, and debugging.

Second, we would like to emphasize that we experiment with 1b and 3b models, due to compute limitations. Models of this scale perform poorly on more challenging competitive coding benchmarks like LiveCodeBench. We could evaluate our models on these benchmarks, but scores would be very low and more highly affected by sampling noise.

Third, we would like to clarify our evaluation philosophy. As stated above, D3 is not designed to hill-climb on a particular benchmark, but rather to provide a good initialization for diverse post-training use-cases.

We believe that the simplest way to evaluate whether the dataset fulfills this objective is by testing the quality of zero-shot and direct generation from LMs, with and without mid-training on D3.

This is why we abstain from using scaffolds, suppress LM reasoning traces, and use no special stop tokens during sampling. These evaluation choices contribute to lower code LM scores, but we believe that they improve the integrity of our results and conclusions.

We would like to assure you that we do believe that scaffolding and reasoning are extremely useful, and we are committed to exploring better post-training for LMs with atomic edit representations in future work.

For other languages such as Java, what tools and steps need to be modified to create a similar dataset?

Please see our general response.

---

Thank you once again for taking the time to review our work. Please let us know if you have any outstanding concerns that stand between us and a recommendation for acceptance.

[1] Jimenez et al. SWE-Bench (2023).