Code diffusion models are continuous human noise operators

Question: Which types of code errors the diffusion model is expected to fix?

We focus on syntax errors and will make this assumption more explicit.

Question: No direct comparison to any other code repair literature makes it hard to evaluate the usability of this approach and the quality of code repair

We do not claim that, in its current state, diffusion should be used as a practical repair approach. Our paper makes fundamental observations about repair behavior of a reverse diffusion process for code, showing that (1) it can repair syntax errors and (2) it can be used to generate training data for a specialized model.

To make these insights stronger and provide insights into the complexity of the benchmarks, we can report the performance of existing repair approaches on these benchmarks. For example, the RING [1] paper that introduced the PowerShell benchmarks, reports a performance of 10% in pass@1 and 27% in pass@50 using the 175B parameter Codex model. We show a repair performance of 18% using pooled diffusion and 34.2% on fine-tuning a codet5-small (60M parameters) on synthetic data.

We include a comparison to baselines below for reference, we will include this in the paper.

Question-3: No details described for the finetuning of the models on the generated training data (4.3) hinders reproducibility.

We will add all the details to the paper. In a high level, we synthetically generate buggy and correct code pairs and then train the model to generate the correct code from the broken code. We employ the standard finetuning described by the authors.

Furthermore, we analyzed the tasks solved by different synthetic data techniques and found that there is a very high overlap between syntactic and GPT-4o and the cases solved by diffusion models are different from these. To give an example, the buggy formula "SUM(A1:A10, B1>10)" is repaired as "SUMIF(A1:A10, ">10")" which is incorrect as it doesnt compare with the B column. The model trained on diffusion generated data, is able to generate the correct repair "SUMIFS(A1:A10, B1:B10, ">10")".

We also include the Pass@K results and will add these to the paper:

Question-4: The assumptions on human cognition patterns (how humans treat code errors) are not supported by any evidence or link to the literature, which also makes the title misleading.

Whereas our intuition was that intermediate code in the reverse diffusion process must have some resemblance to human errors if it can repair them, we indeed do not conclusively show that diffusion models follow human cognition pattern. We will rephrase the abstract, introduction and Section 4.4 to reduce the emphasis on human cognition and focus on the (empirical) overlap in distributions between human errors and intermediate code in the reverse diffusion process.

Previous work has noted human repair patterns in code [1]. We will cite them and clarify the connection and scope of the work.

[1] Dominik Huber and Matteo Paltenghi and Michael Pradel, Where to Look When Repairing Code? Comparing the Attention of Neural Models and Developers, 2023, https://arxiv.org/abs/2305.07287.

Dear authors, thank you for the answers, they resolve part of my concerns and raised my score. I still don't understand some things, could you help me clarify?

Question-7: how any% best% pooled% are exactly computed?

pooled – picking the sample using maximum voting across different noise levels. I still don't understand how this is computed.

Was the manuscript updated already / do you plan to add new things before the end of rebuttle?

Question-1: The methodology, especially, could use some cleaning up. The paper uses quite a few one-letter, some of them overloaded

Thank you for the detailed analysis. We will use them to significantly improve the presentation of our paper.

We include the Pass@k results below and will update the paper to show how multiple generations can improve performance.

We also add baselines to repair for comparison to Diffusion.

Question-2: The motivation doesn't fully connect to the results. The rationale given in the abstract and introduction is that diffusion operations may well resemble human repair actions and suggests the work will investigate if denoising steps are "representative" of human repair steps. The work does not provide much proof for this conjecture

We agree with the high-level observation that the diffusion process does not really resemble how a user repairs code. It is indeed the case that if any buggy code resembles broken code seen during the reverse diffusion process, the model will use its own strategy to solve that bug. This is shown in the results as well: a significant but not perfect proportion of the problems is solved. We will update our motivation and carefully revaluate any claims made, to make sure that they properly align with the results.

Question-3: Given the drop-off between "best" and "any" in Tab. 1, what is the path to a practical approach to using diffusion for repair?

This is an excellent question! There are many possibilities, and turning our fundamental insights into a practical approach is definitely planned for future work. Some possibilities, which we will add to a future work section on the topic of going from insights to practice, are given here.

• Pooling is one small step, which already slightly bridges performance. To continue from here, we can perform two levels of pooling, over different noise levels and over different noise vectors at each level.
• We did not use any syntactic or execution-based filtering of the returned code snippets. If those are available, they allow to improve the pooling performance.
• We did not analyze (yet) if failure cases are changing tokens outside of the actual bug. Instead of pooling over complete repairs, we can also pool over individual edits.

We include results for pooling over different noise levels and \epsilon

Question-4: Fig. 4 and 5 seem to imply that an initial guess of the distance between the broken and repaired code is often needed. It would help to know what the density is in the subplots in Fig. 5.

We agree that since the optimal diffusion noise level depends on the broken formula distance which practically require sampling multiple noise levels. Whereas there is definitely some correlation, we actually expected it to be stronger. We will add the density to Figure 5, to show a more fine-grained insight.

The aim of this paper is not to suggest a new repair method but rather explore latent repair abilities of diffusion models which no one has explored before.

Question-5: Alternatively, do the results in Tab. 2 suggest that it is more effective to simply train auto-regressive models using diffusion-generated data, rather than to use diffusion to repair broken programs?

Without any improvements or deeper insights, that is indeed the more efficient option, both in terms of quality and inference speed. The goal of our research is to explore the potential of using a pre-trained diffusion model for syntactic code repair, and these are exactly the type of insights that we were hoping to get—we will add this more explicitly to the paper.

Question-1: Focus on Program Repair: It would benefit the authors to emphasize the program repair aspect more clearly and consider the implications of their work beyond this specific domain of repair.

We meant generating training data for specialized repair approaches and will clarify that in the text. Our claims are specifically about code repair, where the inverse of repairing code is breaking code—introducing noise. This noise is typically introduced in the discrete token space, by implementing specialized functions that imitate human errors [1, 2]. Our aim is to show that the diffusion process—iteratively removing noise from a latent (continuous) representation—shares enough similarities to the discrete errors that humans make to be useful for repair.

• [1] https://arxiv.org/abs/2106.06600
• [2] https://proceedings.mlr.press/v119/yasunaga20a.html

Question-2: Speed of Diffusion Models: The authors should address diffusion models tend to be slower than other program repair approaches. A comparative analysis of the speed of the method versus existing techniques (e.g., the most similar model without diffusion or just remove the diffusion of your approach) would provide valuable context for evaluating its practicality.

Diffusion models are indeed slower, due to their iterative nature, which we will explicitly mention in the limitation section, along with some potential solutions. This concern motivated us to use the diffusion process to generate data for specialized repair approaches, where codet5-small yields excellent results (Table 2). Note that we are not claiming that, in its current state, diffusion is state-of-the-art for code repair: we are investigating a foundational insight into the potential applications of diffusion for repair. Exploiting these insights into a practical solution leaves a significant potential for future work.

We have included the latency and memory results for Diffusion and Autoregressive techniques and are happy to add this to the paper. To reiterate, we are not claiming that diffusion is state-of-the-art for code repair, rather we are exploring the insight that diffusion models have latent repair abilities which is very interesting.

Question-3: Lack of Baselines: The paper does not consider baseline models to demonstrate the benefits of the diffusion approach (e.g., the baselines used in "https://dl.acm.org/doi/pdf/10.1145/3650212.3680323").

We do not claim that, in its current state, diffusion should be used as a practical repair approach that achieves state-of-the-art results. Our paper makes fundamental observations about repair behavior of a reverse diffusion process for code, showing that (1) it can repair syntax errors and (2) it can be used to generate training data for a specialized model.

To make these insights stronger and provide insights into the complexity of the benchmarks, we can report the performance of existing repair approaches on these benchmarks. For example, the RING [1] paper that introduced the PowerShell benchmarks, reports a performance of 10% in pass@1 and 27% in pass@50 using the 175B parameter Codex model. We show a repair performance of 18% using pooled diffusion and 34.2% on fine-tuning a codet5-small (60M parameters) on synthetic data.

We have included the results for different baselines for repair as reference to how they compare to Diffusion and are happy to add this to the paper. We would again like to highlight the aim of this paper is not to say that diffusion models in its current state are the SOTA in code repair, rather, the paper investigates the latent repair abilities of diffusion models (without training them for this task) and especially since these have not been considered in the context of repair.

Question-4: Closed Source: The closed-source nature of the implementation may limit the reproducibility of results and the ability for the community to build upon this work.

We plan to fully open source our code, models and the generated training data. We have not added the GitHub url for anonymity but will add it in the final version.

Question-1: Limited Significance of Performance Gains: The performance gains achieved using the diffusion-generated data appear to be only marginal. Given that the paper is presenting a new method

We do not claim to present a new method. We show that a diffusion model can be used out-of-the box to generate training data for code syntax repair, because the distribution of “broken” code encountered during training has some overlap with the distribution of syntax mistakes that humans make. Thus, we do not train the diffusion model on repair and only show latent repair capabilities of the models.

The presentation in Table 2 does not properly convey this point, by emphasizing the “best” result for each metric. We will update this in the paper. On Excel formulas, for example, the “syntactic” mistakes required designing 17 operators that break formulas. Getting the inspiration required looking at hundreds of mistakes, and implementing these (potentially) required a lot of formula parsing.

Our approach just requires a corpus of formulas and nothing else and improves by +5.2% (codet5-small) and +2.1% (mistral-7B) on the two models that were not trained by Microsoft. Getting a t5-small model (60m parameters) to outperform phi-3.5-mini (3.8B) and mistral (7B) based on data generated without any supervision is hardly marginal.

Question-2: Incomplete Explanation on GPT-4o Prompting

We will add our prompt and a sample of data to the supplementary material. Note that we only use the model to generate synthetic training data (to break a formula)—not to repair—so it might not benefit from reasoning. Our prompt is a few-shot prompt where we instruct the model to break a formula a human might generate. We do 2 versions of this - (1) We use the syntax operators listed in BiFi [1] and (2) We do not provide guidance so it can cover all types of errors.

Question-3: Unexplored Trade-Offs: The paper removes NL component from the CodeFusion model, turning it into an unconditional model.

CodeFusion did not use NL in the pre-training step, and we are only leveraging the pre-training step. It’s just an artefact of what we are exploring: whether an unconditional code diffusion model, which yields noisy code in intermediate steps that resemble buggy code written by humans, can be leveraged to repair that buggy code or not. Repairing code with a natural language instruction (generated by a model or as a hint provided by the user) is not (supposed to be) part of our current exploration. The main contribution of our work is to show that diffusion models have such latent repair capabilities (without supervising for repair)

Question-4: What are the Y-axes representing in Figure 6?

Figure 6 has distribution plots and the y-axis has the frequency of the bins. We should have specified that these are distribution plots and will update the description.

Question-5: The authors claim that the diffusion process mirrors human debugging behavior. However, there are no concrete qualitative examples supporting this

We will clarify this claim. We show if buggy code is close enough to the types of bugs encountered during reverse diffusion, the model can use its own strategy to arrive at the correct code. We will both update this claim and add some more examples of the diffusion process to the paper. We will also add more citations on how humas repair code and connecting these patterns to diffusion [1].

[1]: Where to Look When Repairing Code?, Dominik Huber and Matteo Paltenghi and Michael Pradel.

Question-6: Are the number of tokens taken by the decoder same as the final generation?

In actual implementation, the decoder is trained to generate N tokens as well, one of which is an end-of-sequence tokens and the rest are padding tokens. We will clarify these details in Section 2.

Question-7: Differentiate from CodeFusion (Singh et al., 2023)

Our work is tangential to CodeFusion as we explore the latent repair capabilities of diffusion models which has not been studied before. The main contribution of our work is to show that diffusion models have such latent repair capabilities (without supervising for repair) which is not explored and is very interesting for these models.

Question-8: Open Source

Yes. Both the model and generated data will be open-sourced.

Getting a t5-small model (60m parameters) to outperform phi-3.5-mini (3.8B) and mistral (7B) based on data generated without any supervision is hardly marginal.

I appreciate your clarification that your approach focuses on the latent repair capabilities of the diffusion model. However, the observation that a small model like t5-small outperforms a model more than 100 times larger, such as Mistral 7B, when fine-tuned on the same synthetic data, is highly counterintuitive. A model of Mistral 7B’s scale should, in principle, leverage the same data far more effectively due to its capacity and generalization abilities. This discrepancy raises concerns about either the quality of the generated data or the experimental setup. Could you provide further insights into why this might be happening?

Note that we only use the model to generate synthetic training data (to break a formula)—not to repair—so it might not benefit from reasoning.

While I understand your point that GPT-4o was used to generate synthetic bugs and not for repair tasks, I still find the restriction to basic syntactic errors problematic. Generating realistic bugs intuitively requires reasoning to mimic the diverse and nuanced mistakes humans make, and larger models like GPT-4o are naturally better suited for this due to their extensive training. Limiting GPT-4o to syntactic errors feels like an artificial restriction rather than a genuine limitation of the model. This choice also seems to undercut the potential of GPT-4o to generate more diverse and complex errors, which would make for richer training data. Including more details about the prompting setup, as you plan to do, is essential to clarify this and ensure reproducibility.

We will clarify this claim. We show if buggy code is close enough to the types of bugs encountered during reverse diffusion, the model can use its own strategy to arrive at the correct code.

Your revised explanation—that the model succeeds when buggy code aligns with reverse diffusion patterns—is significantly different from the original claim in the paper that the diffusion process mirrors human debugging behavior. If this adjustment represents a core shift in the narrative, the paper’s title and framing should also reflect this change to avoid misrepresenting the scope or contributions. A clearer alignment between claims, results, and framing will make the paper easier to follow and review.

Yes. Both the model and generated data will be open-sourced.

I am glad to hear that the model and generated data will be open-sourced. This step will aid reproducibility and future research.

I appreciate your clarification that your approach focuses on the latent repair capabilities of the diffusion model. However, the observation that a small model like t5-small outperforms a model more than 100 times larger, such as Mistral 7B, when fine-tuned on the same synthetic data, is highly counterintuitive. A model of Mistral 7B’s scale should, in principle, leverage the same data far more effectively due to its capacity and generalization abilities. This discrepancy raises concerns about either the quality of the generated data or the experimental setup. Could you provide further insights into why this might be happening?`

While I understand your point that GPT-4o was used to generate synthetic bugs and not for repair tasks, I still find the restriction to basic syntactic errors problematic. Generating realistic bugs intuitively requires reasoning to mimic the diverse and nuanced mistakes humans make, and larger models like GPT-4o are naturally better suited for this due to their extensive training.

Your revised explanation—that the model succeeds when buggy code aligns with reverse diffusion patterns—is significantly different from the original claim in the paper that the diffusion process mirrors human debugging behavior. If this adjustment represents a core shift in the narrative, the paper’s title and framing should also reflect this change to avoid misrepresenting the scope or contributions. A clearer alignment between claims, results, and framing will make the paper easier to follow and review.

We hope we were able to answer the concerns raised. Since today is the last day for submitting questions regarding our paper, we wanted to check in if there are any further clarifications. We appreciate your time, and the valuable feedback provided, which was valuables to improving our work.

We have updated the manuscript to address the concerns raised by reviewers. As a brief summary, we have significantly enhanced the narrative of our paper, emphasizing the applications of diffusion for last-mile repair. We have made a range of improvements, including revisiting our claims, adding results for comparison, and including new visualizations to support our findings. The general comment describes the changes in greater detail

If you have any further questions or require clarification on any aspect of the paper, we would be more than happy to address them before the deadline.