LLM-Assisted Code Cleaning For Training Accurate Code Generators

We thank the reviewer for the constructive comments and feedback.

The proposed code transformations … rely heavily on the capability of ChatGPT (GPT-3.5-TURBO). It is unknown whether the proposed method can be utilized for other, more powerful LLM-based code generation systems.

We acknowledge the reviewer's point that the efficacy of our code transformation approach is influenced by the underlying language model employed. We selected GPT-3.5-Turbo for its balance between transformation accuracy and operational cost. To substantiate the versatility of our method, we have conducted an additional ablation study by substituting GPT-3.5-Turbo with the more recent GPT-4-Turbo model. Owing to the significantly higher cost of GPT-4-Turbo (approximately 8-10 times higher), we use a subset of the original dataset and refer to the modularized version as D_Modularize_. To ensure comparability, we analyzed the same subset of original and transformed programs for both the base and modularized datasets. Further details on the experimental setup can be found in Appendix Section C.2.

The table below illustrates that the D_Modularize_4 dataset yields improved performance over the D_Modularize dataset presented in the main paper. This enhancement not only demonstrates the adaptability but also the potential improvements of our approach using stronger models.

The authors propose two steps of data cleaning (i.e., renaming, and modularizing) to the original source code. However, the authors did not validate the quality of the transformed code (the authors only tested whether the transformed code can be consistent to the original data)....

We appreciate the reviewer's emphasis on the importance of quantifying code quality. We have conducted a thorough analysis of the transformed dataset, as detailed in Section 4.1, where we offer both qualitative and quantitative evidence of quality enhancement. Below, we summarize the insights from Section 4.1 that pertain to code quality:

• Post-transformation, the median number of new functions introduced by the modularization process is three. This indicates that the modularization procedure actively decomposes existing programs to insert new, potentially more coherent functions.
• The helper functions created during modularization typically embody standard algorithms frequently encountered in competitive programming, such as dfs, build_graph, gcd, dp, and binary_search. This suggests that the new helper functions are meaningful and positively contribute to the overall code.
• A qualitative analysis of approximately 100 programs, including successful and unsuccessful transformations is presented with various success and failure modes detailed.

Moreover, we posit that the quality of the dataset can be indirectly inferred from the improved performance of models trained on it. The performance gains achieved through our modularization approach satisfy this measure of quality.

Based on reviewers’ concerns, we have additionally performed an analysis of the generated dataset using GPT-4 as a judge, a method gaining traction for assessing outputs from free-form language models [1,2]. While this approach might have subtle limitations, it still offers valuable insights to address the concerns raised. We tasked GPT-4 to quantitatively assess the transformed programs in regards to the meaningfulness of the transformations and consistency of original and transformed programs (i.e. following the same program logic and structure). Specifically, we asked GPT-4 to rate the transformation on

• variable names (rating between 1-3 corresponding to not helpful, somewhat helpful, very helpful)
• function decomposition (rating between 1-3 corresponding to not helpful, somewhat helpful, very helpful)
• program consistency (rating between 0-1 corresponding to inconsistent and consistent)

Appendix C.1 provides prompts used for GPT-4 jude. Analysis of about 1000 programs reveals that GPT-4 regards 99% of the transformations as helpful (of which only 3-5% of examples can judged as can do better). Similarly, over 99% of the transformed programs are judged as consistent with the original program. The following table provides a distribution of scores annotated

Appendix C.1 provides a qualitative analysis of GPT-4 scores and more details.

Finally, we will release our generated dataset for further analysis for the community.

As observed from the experimental results, the improvement of the proposed method could be insignificant and inconsistent.

Modest Improvements - While the improvements appear modest, this is in large part also an artifact of the algorithmic code generation task being a challenging domain. Following, we provide some context for the two benchmarks:

• APPS. Prior works using the APPS dataset, CodeRL[1] and PPOCoder[2] have reported a similar range of performance improvements – pass@1 improvement from 5.6 to 7.1 by CodeRL and 3.6 to 5.2 by PPOCoder.

• CodeContests. CodeContests dataset is an even more challenging dataset in comparison to APPS and thus is also harder to improve upon. For context, within the AlphaCode models, the 41B model does not exhibit notable improvements over the 9B model. Similarly, even GPT-3.5-Turbo achieves a pass@100 of 18, compared to 12 with our best-performing CodeLLaMa7B model. To our knowledge, the only other study that has tackled this dataset without closed-source API-gated models is “outline then generate” [5]. They proposed a tokenization approach and showed similar improvements against the AlphaCode baseline

Inconsistent Improvements - We observe small performance drops on the CodeContests D_rename experiment. While we cannot ascertain the exact reason, we have the following hypotheses.

Hypothesis 1 - Training examples in the CodeContests datasets on average already consist of good or standard variable naming practices and further renaming does not help (as also evidenced to a lesser degree for the APPS dataset) Hypothesis 2 - Variable names have a lesser impact on code generation capability and the primary impact comes from modularization, our key transformation.

Code generation could be achieved by prompting a general LLM such as ChatGPT directly as well.

We thank the reviewer for suggesting the related work. We have included more detailed related work in code generation with LLMs now.

However, we would like to re-emphasize that in this work our goal is to study and propose methods for improving data quality for code generation. Data quality is an important aspect of training LLMs. We hope our proposed approach and findings will be useful for improving the training process for future (code) language models.

[1] Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Neurips datasets 2023

[2] Large Language Models Are State-of-the-Art Evaluators of Code Generation. Arxiv 2023

[3] Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Neuips 2022

[4] Execution-based code generation using deep reinforcement learning. TMLR 2023

[5] Outline, then details: Syntactically guided coarse-to-fine code generation. ICML 2023

Dear Reviewer DhCg,

Thank you for your time in reviewing our paper and providing insightful and constructive comments. This is just a gentle reminder to revisit our responses and reply as to whether our response and clarifications have addressed the issues raised in the review. If you have any further questions, please do not hesitate to let us know.

We thank the reviewer for the constructive comments and feedback.

However, the efficacy of this approach might be significantly impacted by the presence/absence of an oracle

We agree with the reviewer that the equivalence-checking oracle plays a crucial role in maintaining the data quality. However, our approach can also accommodate learning-based "oracles" to eliminate poor-quality data points. For instance, [1] developed a quality classifier (through the manual annotation of a small data subset) for filtering GPT-3 outputs, while [2] utilized an entailment model to estimate the quality of text summarization pairs.

… clearly outline the filtering criteria that are used for each dataset…

No additional filtering was performed for APPS. Further details are added to the appendix

In many of the tables, there has been no reference or explanation to numbers that show a negative effect on results.

We thank the reviewer for the feedback and have updated the manuscript to reflect this.

Overall, the effect of planning information is a mixed bag, and the conclusions are slightly confusing .....

We agree with the reviewer that results on D_planning do not provide consistent improvements and acknowledge the confusion. We will further clarify the claims about how planning affects performance.

To present our findings and conclusion in perspective, we would like to highlight that improving the reasoning and planning capabilities is an active area of research [3]. Prior works have reported strong performance improvements by using a supervised learning paradigm by training on natural language plans or reasoning chains [4,5,6] albeit typically in less complex domains. In line with these approaches, we adopted the supervised learning paradigm, aiming to refine planning and reasoning for models by training on automatically curated natural language plans using our methodology. We do not assert that supervised learning is the definitive solution here, as discussed in Section 4.1 (final paragraph).

While the results from D_planning are mixed, we believe they raise critical questions for future research. Two particularly promising avenues are:

• Defining "quality" for natural language plans: The mixed success with planning might be attributed to inaccuracies in the automatically generated plans. Future data curation techniques that filter or augment this precision (akin to entailment models used in [2]) are promising.

• Alternative learning paradigms for enhancing planning: The supervised learning paradigm followed in this work might be insufficient for models to generalize planning in complex domains. Future work can explore alternative learning algorithms, possibly over our modularization approach which naturally decomposes programs usually beneficial for solving complex problems.

We have revised the manuscript to reflect this message more clearly.

The improvements on Code-Contests seem to be not as effective as would be expected.

While CodeContests improvements do not seem significant in magnitude, we want to emphasize that the CodeContests dataset is an even more challenging dataset in comparison to APPS and thus also harder to improve upon. For context, within the AlphaCode models, the 41B model does not improve notably over the 9B model, suggesting that even scaling has a limited effect. To our knowledge, the only other study that has tackled this dataset without closed-source API-gated models is “outline-generate” [7]. They proposed a tokenization approach and observed similar improvements against AlphaCode.

What does this line “These cases are sometimes due to incorrect programs but more …. while only a single test solution is provided” mean?

This refers to the observation that certain problems admit multiple correct solutions that are not equivalent to one another. For instance, a problem may allow the output of a list of numbers in any order. Our examination of the datasets revealed that both APPS and CodeContests typically include test cases that cover only one of the many correct solutions (e.g., only one specific ordering of the list). Example - https://codeforces.com/problemset/problem/1512/B.

Minor issues - Figure 1, missing distillation number, prepending plans

Fixed and updated in manuscript.

[1] Symbolic knowledge distillation: from general language models to commonsense models. ACL 2021

[2] Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation. 2022

[3] Towards reasoning in large language models: A survey. 2023.

[4] Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions. 2022

[5] Symbolic Chain-of-Thought Distillation: Small Models Can Also" Think" Step-by-Step. 2023

[6] Orca: Progressive learning from complex explanation traces of gpt-4. 2023

[7] Outline, then details: Syntactically guided coarse-to-fine code generation. ICML 2023

Dear Reviewer FtQG,

Thank you for your time in reviewing our paper and providing insightful and constructive comments. This is just a gentle reminder to revisit our responses and reply as to whether our response and clarifications have addressed the issues raised in the review. If you have any further questions, please do not hesitate to let us know.

We thank the reviewer for the constructive comments.

The improvements in Table 3 are okay, not huge but still noticeable.

While the improvements in Table 3 do not appear to be huge, this is in large part also an artifact of the algorithmic code generation task being a challenging domain. For example, prior works using the APPS dataset, CodeRL[1] and PPOCoder[2] have reported a similar range of performance improvements – pass@1 improvement from 5.6 to 7.1 by CodeRL and 3.6 to 5.2 by PPOCoder. Finally, we study the data quality for improving code LLMs which is an important problem and complements existing works. Similarly, CodeContests is even more challenging and harder to improve upon.

The planning results are also modest, but this is interesting in its own right, and the analysis of how ground truth plans would help considerably is a good way to isolate much of the problem to the plan creation rather than plan execution.

We agree with the reviewer that planning results are modest/mixed bag. To add additional context and perspective – improving the reasoning and planning capabilities of language models is a challenging problem and a research direction actively explored by the community [10]. Our experiment of disentangling plan generation with code implementation shows that even after training on such plans, current models cannot generalize and construct plans for new challenging problems. To provide the insights more clearly, we have added some additional context in the paper summarized below:

We adopted the supervised learning paradigm, aiming to refine planning by training on natural language plans curated using our methodology akin to prior works [4,5,6]. While the results from D_planning are mixed, we believe they raise critical questions for future research. Two particularly promising avenues are:

• Defining "quality" for natural language plans: The mixed success with planning might be attributed to inaccuracies in the automatically generated plans. Future data curation techniques that filter or augment this precision (akin to entailment models used in [2]) is promising.

• Alternative learning paradigms for enhancing planning: The supervised learning paradigm followed in this work might be insufficient for models to generalize planning in complex domains. Future work can explore alternative learning algorithms, possibly over our modularization approach which naturally decomposes programs usually beneficial for solving complex problems.

Clarifications on the distillation experiment

Yes, in the distillation experiment, we take the problem statements from the training dataset and use the GPT-3.5-Turbo model to generate multiple, diverse solutions using a few-shot prompting approach. Next, we filter the generated solutions for correctness based on the ground truth test cases provided in the dataset to ensure we are not training on incorrect hallucinated solutions. This experiment is meant to provide a strong baseline that distills larger-model-generated outputs which has proven to be a very popular approach for both instruction tuning and task-specific finetuning [6,8,9]. In contrast to distillation, our transformation approach enjoys the benefits of performing the simpler operation of transforming (editing) existing solutions instead of directly generating them. Thus it yields about 94% successful transformation rate whilst the distillation baseline can only generate solutions for about 50% of the problems.

Two relevant pieces of work on synthetic data generation …

Yes, these works are relevant. STaR learns natural language reasoning (corresponding to our planning approach) in simpler domains. Haluptzok et al generates datasets using language models and further models on them. We have added them to the related works.

Smaller issues – missing distillation number, typos, usage of the word parallel, table 4 caption

We have fixed these issues and updated the paper.

[1] Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Neuips 2022

[2] Execution-based code generation using deep reinforcement learning. TMLR 2023

[3] Outline, then details: Syntactically guided coarse-to-fine code generation. ICML 2023

[4] Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions. Arxiv 2022

[5] Symbolic Chain-of-Thought Distillation: Small Models Can Also" Think" Step-by-Step. Arxiv 2023

[6] Orca: Progressive learning from complex explanation traces of gpt-4. Arxiv 2023

[7] Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation. 2022

[8] Wizardlm: Empowering large language models to follow complex instructions. Arxiv 2023

[9] WizardCoder: Empowering Code Large Language Models with Evol-Instruct. Arxiv 2023

[10] Towards reasoning in large language models: A survey. Arxiv 2023

Dear Reviewer MNi4,

Thank you for your time in reviewing our paper and providing insightful and constructive comments. This is just a gentle reminder to revisit our responses and reply as to whether our response and clarifications have addressed the issues raised in the review. If you have any further questions, please do not hesitate to let us know.