Learning Code Preference via Synthetic Evolution

Thank you for your response; it has addressed many of my concerns.

However, one of the baselines you compared is the Logprob Sum, which has proven highly inefficient in existing studies (See Figure 7 in https://arxiv.org/pdf/2107.03374) and performs even worse than random. A more reasonable comparison would be using the Logprob Mean, as the log prob sum tends to assign different scores to sequences with different lengths. This is also reflected in your reported results: Logprob Sum only achieves around 30% in correctness, significantly lower than random selection.

In addition, what are the inputs for each baseline? Did they use the same prompt as your method, or only the code itself?

it's unclear whether they took decontamination steps to prevent test set leakage.

Thanks for the question! We have carefully quantified the test set contamination in Appendix (A.5):

• Figure 16 shows that only 0.1% to 1.7% of positive samples in the test-set code pairs can find training-set positive samples with a similarity score above 80.
• As a reference, Riddell et al. (2024) show that 50.8% and 63.4% of code samples in the Stack (Li et al., 2023), can reach over 80 similarity scores with ground-truth code samples in MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021) respectively.

Could you provide the number of comments in positive and negative examples in the training and testing sets?

Thanks for the question! Please kindly note that our empirical observation was that “LLM-generated” comments (not including human comments) may negatively impact LLM’s preference decision due to self-bias. We also run experiments to count the average number of comments between positive and negative samples:

Training set: The following table lists the comment distribution in the two training sets. It shows that our observation is right – even if positive samples overall have a bit more comments, they do not seem to help CodeFavor models achieve better preference accuracy.

Test-set: The following table shows the comment distribution of the raw data in our test-set – please note that our evaluation setup removes all comments when evaluating a model; therefore, imbalances in the raw data (if any) won’t impact the evaluation results presented in the paper.

For the human preference, efficiency, and security categories, the average amount of comments is balanced. For the correctness category, comments amounts are unbalanced – after a closer investigation we found this is because (i) our code samples are generated from both base and instruction-tuned models; (ii) base models perform completion which preserves the docstring from the task description; (iii) instruction-tuned models tend not to repeat the docstring in the task description; (iv) instruction-tuned models are generally stronger than base models, leading to the imbalance.

Our implementation to count the number of comments is listed below:

import re

def count_comments(code_snippet):
    single_line_comment_pattern = r"#.*"
    multi_line_comment_pattern = r'"""(.*?)"""|\'\'\'(.*?)\'\'\''

    # Find all single-line comments
    single_line_comments = re.findall(single_line_comment_pattern, code_snippet)
    # Find all multi-line comments
    multi_line_comments = re.findall(multi_line_comment_pattern, code_snippet)
    # Flatten multi-line comments and remove empty strings
    multi_line_comments = [
        comment for group in multi_line_comments for comment in group if comment
    ]
    return len(single_line_comments) + len(multi_line_comments)

(Author response to be continued in the next reply)

How do the experiments demonstrate that CodeFavor’s performance gains are due to the framework itself rather than simply distilling knowledge from a stronger LLM (Llama-3-70B-Instruct)?

Great question! Looking at Table 3 – the original Llama3-70B-Instruct’s overall score is 76.1 and our smaller models trained on data partially derived from Llama3-70B-Instruct can achieve a score of 77.7, which is even better than the derived model at a much smaller size. In theory, if we solely distill Llama3-70B-Instruct, it is unlikely to achieve a competitive score within a much smaller model parameter scale.

Why is CodeFavor better than distillation? Please note that instead of solely distilling stronger models, our generated data is also derived from real-world code commits which brings additional information and weak labeling.

Could you provide more details on the inference process for the evaluation results in Table 3? Specifically, how many samples were created for each problem, what temperature was used, and are the results statistically significant?

Thanks for raising the question! We use greedy decoding (mentioned in Section 3.1) for LLM generation and the prompt is listed in Listing 1 (mentioned in Appendix A.3). We implemented a post-processing method (Appendix A.3) to extract LLMs’ preferences from their generations. As we use greedy decoding, the result is theoretically deterministic.

Could you elaborate on any aspects that emphasize the novelty of your work compared to previous studies?

Thanks for raising the concern. We argue that our paper is novel from various perspectives:

• Study-wise, we invested significant efforts to study human annotation for code at scale and demonstrated various new insights.
• Benchmark-wise, to our knowledge, CodePrefBench is the first benchmark to evaluate code preferences from models and non-model approaches, covering 4 detailed dimensions. We also provided a line of insights by conducting comprehensive evaluations.
• Technique-wise, to our knowledge, we are the first work to construct synthetic code preference data, based on which we train competitive models to efficiently predict fine-grained code preference.

…creating datasets from git commits [1,6] and evolving from sampled code[2,3] are common practices in the field.

While we greatly appreciate the reviewer for the question and references, we’d like to point out that our synthetic data generation has completely different purposes and implementations compared to the mentioned prior work.

• We focus on code preference training data, which is a pair of code snippets where one is better than the other, alongside detailed criteria for comparison.
• The mentioned work [2,3] synthesizes coding prompts and solutions, targeting the different applications of code generation. Meanwhile, the mentioned work [1,6] focuses on creating evaluation tasks rather than generating training data.
• Meanwhile, we use the data (e.g., commits) differently: SWE-Bench [1] uses the pull request (which is also a commit) partially by directly using added test cases as an oracle. Our technique Commit-Instruct fully leverages the whole commit, including the pre-and post-commit code and the commit message, by rephrasing the noisy commits into a format that focuses on the actual change.
• Lastly, the mentioned work [6] does not use nor mention git commit at all in their whole paper.

…synthetic data generation may not fully ensure code correctness….

We agree that synthetic data can be low-quality if not well-created! We argue that our technique is empirically robust. This can be exemplified by Figures 3, 4, and 5 in the paper, where the preferred code snippets improve the quality of earlier code.

Also please kindly note that our technique is not “pure distillation” and thus does not solely rely on LLM’s capability. For example, in Commit-Instruct, the synthesized code pair is inherited from code commits — empirically the post-commit code improves the pre-commit code; otherwise, they would have been filtered out.

Llama3-70B-Instruct is relatively weak compared to state-of-the-art models…

Thanks for pointing this out!

Please kindly note that our method can be applied to any model and in this paper, our choice of the critic model comes from legal reasons (Llama3 is license-friendly). Even though it's not the SOTA model, results still show significant improvements which proves that our method does not have to rely on proprietary models.

To validate the effectiveness of the developed training framework, might it be helpful to add some baseline training approaches which also train the LLMs using the same training data used by CODEFAVOR?

Thanks for the question! Please kindly note that our framework is a combination of (i) training data generation; and (ii) model training scheme – the effectiveness is demonstrated through the co-design of both components.

Our understanding of the reviewer’s question is to study the (ii) component, i.e., the model training schemes (Please kindly let us know if we misunderstood).

• In Table 5 from the paper, we thoroughly studied various training schemes including classification, generation, and model merging.
• To further improve the study thoroughness, we follow the RLHF literature and additionally trained a Bradley–Terry model (Dong et al, 2024) for comparison: Surprisingly, while the overall performance of Bradley-Terry modeling is suboptimal to the classification modeling (4% weaker), its preference accuracy on code correctness beats all evaluated LLMs approaches including Llama-3.1-405B-Instruct.

Dong et al. RLHF Workflow: From Reward Modeling to Online RLHF. TMLR, 2024.

In Table 1… whether CODEFAVOR can further improve these Open-Weight Models with larger sizes?

Great suggestion! With our best computing budget (8xA100@80G), we additionally trained a CodeFavor classifier based on Gemma2-27B-IT with data mixture.

We show that:

• Our method can further improve the larger 27B open-weight models by 9.4%.
• Meanwhile, compared to the 9B CodeFavor model, the 27B CodeFavor classifier can further improve it by ~5%.

… effectiveness … when the approach is applied to some other LLMs for code related tasks (e.g., Code Llama, Lemur).

Thanks for the great suggestion! We extended the new experiments below by training CodeLlama 13B based on CodeFavor and were able to achieve an overall improvement of 16-17%.

We are especially grateful for this suggestion as we found CodeLlama achieved a positive default correctness score without tuning compared to other general models (such as Mistral Nemo 12B) which mostly randomly guess. This might indicate that code instruct models might have a better sense of code correctness in their intrinsic preference.

what are the backgrounds of the developers participating in the evaluation and data annotation, and how their potential biases may have affected the soundness of the approach and the evaluation?

Thanks for the question! Section 3.2 provides a detailed description of the annotators’ backgrounds: “Our annotation team consists of 18 software developers, two-thirds of which hold degrees in computer science, and 95% of them have over two years of programming experience. For Python proficiency, 43% of them self-rate as advanced, while the rest consider themselves middle-level.”

Specifically, we mitigate potential individual biases with our best efforts:

1. We engage with as many as 18 developers
2. We have each task annotated by three different developers through major voting
3. We comprehensively evaluate over a thousand tasks

Notably, to ensure the quality of human evaluation, we iteratively refined the annotation guidelines, carefully communicated with the annotators, and performed post-annotation inspection.

…the larger LLMs have clear edges over the CodeFavor-improved small models... Why would they not be a better option than using CodeFavor to improve the smaller models?

Users should always use CodeFavor models in place of the raw model as CodeFavor can improve the preference accuracy of both small and large models.

*Our existing experiments have shown that it achieves up to 28% improvement for 8-12B models.

• Additionally, with our best computing budget, we apply CodeFavor to Gemma2-27B-IT (data mixture), where an improvement is also observed by as much as 9%:

We initially focused on small models as they are more efficient to train and deploy.

The paper mentions that the reliability of using large language models (LLMs) as evaluators often hinges on their reasoning capabilities… A more thorough examination of potential biases and their impact on the findings would strengthen the paper's arguments.

Thoroughly studying the LLM evaluators’ bias is definitely important!

That is exactly why in Appendix (A.4) we put nearly 10 pages of case studies and summaries to understand the preference and bias patterns of evaluated models. For example, Gemini 1.5 Pro seems reluctant to sharpen its preference for security-related code snippets. We hope our analysis can provide insights and references for future research.

… the prohibitive costs and limitations of human-based code preference assessments … suggests that human evaluators may struggle with certain tasks... The paper could benefit from a more in-depth exploration of these limitations and their implications for the overall findings.

Thanks for the suggestion! At a high level, our paper explores human-based code preference in various dimensions (Section 3.2), including expertise, confidence, overhead, etc. Additionally, at a lower level, in Appendix A.4, we provide 10 case studies that compare model and human behaviors in detail.

Overall we found:

1. Human preference for code is expensive and slow
2. Human preference strongly aligns with code correctness but can struggle with non-functional properties
3. The 2nd conclusion can come from the fact that generalist developers are familiar with writing test cases to validate code but they might not have enough background in code optimization and code security, suggesting that domain experts or LLM assistance are generally needed when assessing non-functional code properties

in what scenarios would these two versions be available?

Great question! Accurate code preference can be applied in various scenarios including (i) quality assurance of code commits, where we detect if post-commit code is better than pre-commit one; (ii) inference-time sample ranking; (iii) training reward models (i.e., our experimental setting); and (iv) provide data and signal for preference optimization algorithms such as DPO.

While synthetic data can be useful, it may not fully capture the complexities and nuances of real-world coding scenarios.

Thanks for the comment! Maintaining the real-world complexities in synthetic data is definitely important! That is exactly why we specifically leveraged weak supervision in data generation rather than performing pure distillation:

• In Commit-Instruct, the synthesized data comes from rephrasing diverse, real-world commits collected and cleaned carefully from GitHub to make sure the training data reflects real-world code usage.
• Beyond human-created commits, we also want to cover code that is directly generated by LLMs, given that nowadays lots of GitHub code can be partially generated by LLMs. As such, Critic-Evol constructs code preference pairs by capturing and fixing code defects generated by smaller models.