Does Instruction Tuning Reduce Diversity? A Case Study Using Code Generation

5. Experiments Limited to Python

-Reviewer Comment: "The experiments are Python-only. The authors should conduct evaluations on more programming languages."

• Action: While resource constraints limit our ability to conduct extensive experiments in other languages, we will openly discuss this limitation in the paper and suggest it as an avenue for future work.
• Clarification: To our knowledge it is often the case that diversity in Natural Language is only evaluated in English and also that many properties of program synthesis research is only evaluated on one language (usually and often Python). For example HumanEval was Python-only [3] and the APPS dataset is Python-only [4]. Additionally, our work makes empirical contributions from our findings and a methodological contribution in leveraging program execution to measure diversity. Taken together, we believe there is a significant contribution.

[3] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

[4] Hendrycks, Dan, et al. "Measuring Coding Challenge Competence With APPS." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

6. Suggestion Regarding Paragraph in Section 4.2

Reviewer Comment: “The conclusion that "Program semantics may be harder to model than natural language." is a bit wrong, as BERTScore itself is not robust [4-6]. There is no reason why CodeBERTScore should accurately reflect the program's correctness.”

Clarification: Perhaps there is a misunderstanding in terms of our motivation of this paragraph; we do not intend to highlight this as a major contribution. We try to provide a rationale and discussion for the results that we see. As we discuss in the paragraph, we make the same statement that you make about CodeBertScore, that it correlates relatively poorly with functional correctness: “CODEBERTSCORE reports a moderate correlation with accuracy” (line 329-330).

However, this fact may not be obvious to researchers with a background on diversity measurement in natural language. In natural language, it may be actually more reasonable to use neural networks to measure diversity than in code: neural measures of diversity have been reported to be competitive with human evaluations of diversity in natural language [3]. In this paragraph, we are only trying to explain why the nature between CodeBertScore and diversity is different for code than it is for natural language. While it may be obvious to some of us, it is intended to help make sense of why this is the case for researchers less familiar with work in the code space.

[5] Stasaski, Katherine, and Marti A. Hearst. "Semantic Diversity in Dialogue with Natural Language Inference." Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

• Action: In the revised paper, we will discuss the limitations of current metrics like CodeBERTScore in accurately reflecting program semantics, and how this impacts the evaluation of diversity. We will also try to change language such that it is more clear this is not a contribution per-se, but rather a rationalization / explanation. If you like, we can remove this paragraph from the manuscript, however, we’re hesitant to do so if it will make the paper less appealing to a broader audience or if other reviewers object. If the reviewers agree that removing this paragraph is beneficial, we are happy to make this change.

7. Related Work Section

Thank you for this suggestion.

• Action: We will expand our related work section to include recent studies on code language models, code generation, and code benchmarks, situating our work within the broader research context.
• Clarification: We originally prioritized the framing of our work as a paper within the research context of the diversity literature, and so we chose to prioritize this first.

8. Questions

• Column N in Tables: The column 'N' represents the sample size available/used for statistical tests.

• Motivation for Code Generation over Text Generation: We will enhance our motivation section to explain the significance of studying diversity in code generation, including practical applications and theoretical implications.

Coherence vs Diversity Matrix (informal and conceptual)
---------------------------
                 Low Diversity   |   High Diversity 

High Coherence | Mode collapse   |   Ideal output
               |                 |       
               | Well-formed but |   Diverse, generally
               | repetitive      |   well-formed solutions
---------------|-----------------|------------------
Low Coherence  | Poor output     |   Spurious output
               |                 |       
               | Often erroneous |   Usually erroneous,
               | not varied      |   can be varied

We sincerely thank you for your thoughtful evaluation and for acknowledging the strengths of our paper in terms of structure, experimentation, and the multidimensional evaluation of diversity. We appreciate the opportunity to address some concerns and clarify any misunderstandings.

1. Undocumented Training Setups

Reviewer Comment: "The training setups are not documented, and the results could be questionable."

We apologize for any confusion. We believe there is a core misunderstanding.

• Clarification: We utilized off-the-shelf models without any additional training. We will make sure this is clear in the paper to avoid misunderstanding.

2. Lack of Correctness Metrics Evaluation

Reviewer comment: ”There is no evaluation of correctness metrics. Representative benchmarks like HumanEval [1] and BigCodeBench [2] should be included.”

• Clarification: We believe there may be a core misunderstanding: in our work we only evaluate off-the-shelf LLMs (e.g. LLama3.1), we do not do any specific fine-tuning on our own. Generally in the works we cited for these models, correctness metric evaluation has already been reported (e.g. HumanEval). Our focus was to evaluate diversity rather than correctness. In addition to this, we introduce the metric coherence to measure the quality and well-formedness of programs: we can think of this exactly as a correctness metric except allowing for open-endedness. We use coherence to determine if increased diversity comes at the expense of quality (coherence) and we report coherence in all our tables.
• Question: Given this clarification, would you still like us to report correctness metrics like HumanEval for these models? We could retrieve these metrics from the respective publications and include them in an appendix sections. Otherwise, we believe that the coherence metric measures the important dimension of quality and well-formedness.

3. Limited Diversity of Models Evaluated

We agree that including more models would enhance our study.

• Action: We will include CodeQwen in our evaluations to increase the diversity of our analysis.
• Clarification: We are limited in including StarCoder2 because it does not provide separate pre-trained and instruction-tuned models, which is important for our experimental design.

4. Appropriateness of CodeBERTScore

• Clarification: We believe that CodeBERTScore is an appropriate and current metric for our study, yet we will augment our study with ICE-Score. We chose CodeBERTScore, it was the most analogous model to those used in recent diversity works BertScore and Sentence-BERT [1][2], making it a suitable starting point for assessing neural diversity in our study. Additionally, given its recent publication (published in December, 2023) and has been widely downloaded and utilized in the community (>400K downloads), we thought it reasonable to begin with.

[1] Tevet, Guy, and Jonathan Berant. "Evaluating the Evaluation of Diversity in Natural Language Generation." Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021

[2] Kirk, Robert, et al. "Understanding the Effects of RLHF on LLM Generalisation and Diversity." The Twelfth International Conference on Learning Representations.

While ICE-Score offers some advantages, it has primarily been evaluated for assessing functional correctness or usefulness rather than diversity. Moreover, ICE-Score's reported correlations with correctness are moderate (e.g., correlations < 0.35 on Python HumanEval).

We also note that there is limited literature on using ICE-Score or similar LLM-based evaluators specifically for measuring diversity. Our choice of CodeBERTScore aligns with standard practices in the diversity evaluation literature, which often employs BERT-like models due to their established use and validation.

Action: We will also try to include ICE-Score in our analysis if feasible, to offer a comparative perspective; but we may not complete these experiments by the rebuttal deadline. In the revised paper, we will expand discussion of the limitations of CodeBERTScore and acknowledge the potential of alternative metrics. Additionally, we believe that exploring and validating new models for diversity measurement and assessing functional correctness is an important direction for future work, and we aim to motivate further research in this area.

I think reporting the functional correctness of these models will make the findings more concrete.

I second this.

2. Generalizability and Broader Impact

Reviewer Comment: ”Whether the conclusions obtained generalizable to the general-domain openended generation? I think it's hard to evaluate, and at the same time, the answer might be no.”

While we're careful not to over-claim in our paper, we believe our findings have important implications beyond code generation. Here's why:

• Our N-gram diversity results align with previous work in other domains (e.g. natural language summarization)

• The discovery that preference tuning increases semantic diversity while reducing diversity in form (lexical/syntactic diversity) is plausibly a fundamental pattern in how instruction tuning affects model behavior. Anecdotally this resonates with our experience that preference-tuned models have a certain “voice” or “style” but are often capable of generating diverse content. More future work should be done to systematically evaluate these questions.

• Our insights challenges existing assumptions and opens new research directions/problems for studying diversity in other domains

Do you think it would be good to add reasons why our paper could have more generalizability beyond code to our discussion section? We would also want other reviewers to be on board as well.

3. Importance of Diversity When One Correct Output Suffices

Reviewer Comment: ”Code generation is a special case in open-ended generation: it usually has a "correct" output. As the semantic diversity is measured by considering the model output, then why it matters when it generates diverse outputs while only one of them is correct? Isn't it true that we only care about whether it covers the correct output?”

• Clarification: Please correct us if we have not understood your question. Many real world programming tasks have multiple correct outputs as we mention in the previous response: we gave examples in data science, website design, and writing unit tests. However, these programming tasks generally have not been reflected in benchmark design. Diversity is crucial in helping address ambiguity in tasks, exploring different algorithms/techniques, optimizing for various constraints, and providing alternative solutions that may be more efficient, readable, or in better style.
• Correctness in our Paper: In our paper, we introduce the term coherence to measure the quality of our generations. If a program is syntactically correct, if it encounters no type error or runtime error, and it prints output to the console, we consider the program coherent. We like to think of coherence similar to accuracy, but it allows for multiple semantically different answers that all are sufficient.
• Action: We will include this discussion in the paper to highlight the significance of diversity even when “correctness” is achieved.

4. Reviewer's Questions

• Benefits of Higher Response Diversity Beyond Creative Writing: As mentioned above, we will elaborate on practical applications of diversity in programming, such as improving inference-time compute, supporting ambiguous user requirements, and facilitating more comprehensive testing.

1. Meaningfulness of Studying Diversity in Code Generation

Reviewer Comment:: ”Why is it meaningful to study diversity for LLM open-ended generation? From my perspective, it may contribute to more effective solution searching if we scale up the inference computation (like o1).“

We agree that the practical benefits of diversity need to be clearly articulated in the paper. We will improve on our presentation and writing. And you raise an excellent point about improving LLMs like O1 or scaling up inference-time compute - this was actually one of our initial motivations in measuring diversity. The importance of how we study diversity extends across several key areas:

Immediate Practical Applications:

• Enhancing best@K inference optimization (a form of scaling inference time compute as you noted)

• Supporting/Enhancing model bootstrapping for models like O1 with better “exploration.” More diverse generations / reasoning trajectories are necessary for improving models with RL. Whereas, if models lack diversity, it is hard to bootstrap for improvement. Towards this, measuring semantic diversity can be used while tuning foundation LLMs to debug/monitor for instability or if mode collapse occurs. Additionally insights from our work can have practical implications for which model configurations and sampling strategies improve the pareto frontier exploration/exploitation trade-offs: if preference-tuned LLMs are indeed capable of high semantic diversity, we can feel confident sampling from them when trying to bootstrap for models like O1.

• In programing: Generating comprehensive test suites that cover different edge cases and test for different program behavior

• In programming: Handling ambiguous user specifications in real-world programming tasks. Here are some examples:

  • For example if we ask an agent to try to draft a website N times, we want all N to be meaningfully diverse and high quality

  • If we ask an agent to improve our code with N suggestions, we want all N suggestions to be diverse, high quality, and insightful

  • If we ask an agent to perform a data science task with a certain dataset, there are numerous ways of pre-processing the data, fitting different models, and ensembling different models. Asking an agent to provide “insights” from data is even more open-ended; yet these are all programming tasks.

In real-life applications of agents, humans will almost always provide vague instructions: agents must be capable of diverse generations. Our evaluation is supposed to measure the ability for LLMs to make “diverse generations with sufficient quality” which is not measured by normal programming tasks like HumanEval where there is only 1 “correct answer.” Unlike benchmarks like HumanEval, we make a contribution by creating a methodology to cheaply and systematically measure semantic diversity.

Theoretical/Intellectual Understanding of LLMs:

• It is important to know how instruction tuning affects different aspects of diversity. If we can measure diversity better, we can judge if LLMs are capable of diverse generation like humans. We believe that as ML researchers, this is a worthy end of research in its own right, without needing practical application.

In our work we challenge existing understanding about the effects of preference tuning on diversity: compared to prior-work which argues preference-tuning reduces diversity, we find that instruction-tuning actually increases semantic diversity. And traditional diversity metrics breakdown, because when comparing instruction-tuned and preference-tuned models to pre-trained base models, not only are traditional diversity metrics poorly correlated… they’re negatively correlated.

• Action: We will enhance the motivation section to discuss the importance of diversity in code generation, including applications in creative problem-solving, code synthesis, and improving model robustness.

• Examples: We will provide concrete examples where multiple correct solutions are valuable, such as generating different implementations for ambiguous specifications or creating diverse unit tests.

Thanks again for reviewing our paper! We thank you for your recognition of our soundness, noting the significance of our methodological contribution of semantic diversity, and that the insights from the experimental results are interesting. We have uploaded an updated draft to OpenReview that also addresses the feedback you had and some clarifications we made. Key changes are highlighted in blue. While we've been engaged in extensive discussions with all reviewers and have conducted additional experiments requested by others (including evaluating LLM-based diversity metrics and adding QWen-Coder 2.5), we’d like to highlight the changes most relevant to your feedback:

1. Background and Related Work on the Importance of Studying Diversity

Thank you for your suggestion to explain more of why it is meaningful to study diversity for LLM open-ended generation. We revamped our background and related work section (Section 2) to provide a much stronger background on the context in which we write our work, and we explain that there are a few crucial reasons to focus on diversity. Diversity is crucial for exploration, which is a necessary condition for improving models with RL or DPO, and the future of stronger LLMs. Additionally, as we discussed with you, there are numerous practical applications and real-world reasons why diversity matters: in the updated paper we mentioned extracting insights from data science, proposing website front-ends, scientific discovery, and software testing as a few examples of open-ended tasks that are important.

2. Broader Impact and Generalizability

In terms of the broader impact, in the paper we also explain that diversity is important for improving LLMs with RLHF or DPO, and that our work challenges the prevailing narrative about preference tuning uniformly reducing diversity [1]. As we explained in our comment to you, we personally believe that our results may generalize, but we are careful not to over-claim in our paper. We added to Section 6 a section on Future Work outlining different directions for future research that we believe would be interesting for exploration. Additionally, we’ve added experiments where we evaluate ICE-Score, an LLM-based approach for measuring program correctness [2], and also added QWen-Coder 2.5 as another preference-tuned model to lead to more robust results. These additional models followed patterns similar to those made in our original claims. We believe the new additions will help improve the robustness of our results and the validity of our findings.

[1] Kirk, Robert, et al. "Understanding the Effects of RLHF on LLM Generalisation and Diversity." The Twelfth International Conference on Learning Representations.

[2] Zhuo, Terry Yue. "ICE-Score: Instructing Large Language Models to Evaluate Code." Findings of the Association for Computational Linguistics: EACL 2024. 2024.

3. Importance of Diversity Instead of Improving Accuracy with Best@K Sampling

We hope that our last comment resolved some questions and we also believe that our expanded background and related work section motivate that existing LLM program evaluation benchmarks do not evaluate diversity. We also hope that the additional discussion in the paper’s Background section about real-world applications, how diversity is important for exploration in RLHF, and prior work in the diversity literature also help explain why diversity and open-ended settings are important. Lastly, we would also like to add an additional point, that in our work, because of our use of the Coherence Metric and the Diversity Metrics, we can separate the effects of quality from diversity to analyze them (Section A.3 in the Appendix). Whereas with Best@K sampling, it may be difficult to disentangle the effects of higher quality models from models capable of diversity in open-ended or ambiguous scenarios.

We also hope that in our response to you, we have clarified all questions you have had. If they are not clear, we would like to use the remaining time we have to provide additional clarity. We've worked diligently to address concerns from all reviewers, and the constructive exchanges have helped strengthen the paper significantly. Your feedback about the motivation for our paper and important applications was particularly important for our background section. We believe we've addressed the core concerns raised among all reviewers, and we are still waiting to hear back from everyone. We would appreciate your thoughts on whether these changes adequately address the specific concerns you had in your review.

I hope this message finds you well. As we approach the end of the review period, I wanted to gently follow up regarding our revised submission. We greatly appreciated your thorough initial review and have made significant changes based on your valuable feedback, including:

• Expanding our background and related work section to better justify studying diversity in LLM generation
• Adding concrete examples of practical applications in programming, data science, and website design
• Including additional experiments with ICE-Score and QWen-Coder 2.5 to strengthen our findings and improve generalizability
• Clarifying the importance of diversity beyond improving accuracy with higher inference time compute

Given today is the last day for reviewer responses, we would be very grateful if you could take a look at our revisions and provide any final thoughts. Your insights would be particularly valuable in assessing whether we've adequately addressed your initial concerns. Thank you again for your time and thoughtful consideration of our work

4. Degree of Contribution

Reviewer Comment: ”The contribution appears insufficient for a comprehensive paper, but it could form the basis of a shorter piece, such as a research agenda exploring this topic further.”

We respectfully believe that our work makes a significant contribution.

• Clarification: As noted in the previous paragraph where we clarify our contributions, our paper challenges the dominant narrative on how preference-tuning impacts the diversity of generations. It is non-trivial, because we introduce a methodology to measure semantic diversity that is substantially different from lexical diversity or neural diversity metrics (Section 4.2 Diversity Metrics Can Fail to Reflect Execution Semantics) and furthermore is a methodological contribution to the community that can be cheaply re-evaluated in the future or expanded to be even more comprehensive.

• Action: We will try to articulate some of these points better in an updated draft.

5. Question Regarding "Case Study" Presentation

• Clarification: We are not committed that we have to use “case study” in the title, and we will reconsider the language. After internal debate on this exact question, we chose it, because we believe that a case study is enough to show that the story of how preference-tuning impacts diversity is more complex than we thought. The dominant narrative is that preference-tuning reduces diversity, and we find a strong pattern that lexical/syntactic diversity and semantic diversity can actually be negatively correlated when comparing pre-trained to instruction-tuned models. However, we are careful to not over-claim that this is the case over all domains, hence the choice of “case study” for now. Unofficially I think that it is likely that more domains will behave similarly, but we believe that it is best for future work to address this.

• Action: We will put more thought into articulating the framing of our study, possibly removing the term "case study" to better reflect the scope and impact of our research or explain this point further.

All your points are really thought out, reasonable and very constructive, and we really appreciate the care you’ve taken in reviewing.

1. Need for Detailed Dataset Construction Explanation

Reviewer Comment: ”The dataset construction could benefit from more detailed explanations.”

• Action: We will provide a detailed description of our dataset construction process in both the main paper and the appendix.

• Addition: A flowchart or diagram will be included to visually represent the dataset creation and evaluation pipeline.

2. Improvements in Presentation and Clarity

Reviewer Comment: ”The overall presentation of the study requires improvement to enhance clarity.”

Thanks for the constructive suggestions regarding the presentation, we do want the present the paper as well as possible.

• Action: We will reorganize the paper to include a stronger Background and Related Work section, thoroughly introducing all key concepts and terminologies.

• Language Refinement: We will perform a comprehensive proofreading to correct typos and improve narrative flow.

3. Regarding Conclusive Findings

Reviewer Comment: ”The study's findings regarding the research question, "Does instruction tuning reduce diversity?", are not conclusive.”

• Clarification: Let us clarify our contribution relative to the title: Our work provides strong empirical evidence that challenges the dominant narrative about preference tuning and diversity. Previously the dominant narrative was that preference tuning reduced diversity [1]. In our experiments we find strong evidence via systematic measurement of semantic diversity that preference tuning increases semantic diversity while still reducing lexical and syntactic diversity. This is a non-obvious finding, because it requires defining a semantic diversity metric. Our methodology is also noteworthy, because it can be reproduced cheaply by other researchers; whereas, manual human evaluation would be extremely costly to reproduce.

[1] Kirk, Robert, et al. "Understanding the Effects of RLHF on LLM Generalisation and Diversity." The Twelfth International Conference on Learning Representations.

This finding isn't just interesting - it suggests that previous work using only lexical or neural metrics may have missed important aspects of how instruction tuning affects generation diversity. We hope that we can convince you of this perspective, and perhaps our exposition did not clearly articulate this.

• Action: We will strengthen our discussion to clearly highlight how our results provide evidence that challenges existing narratives about instruction tuning reducing diversity.

Thank you again for your thoughtful initial review, we particularly appreciate your recognition of our work's relevance and the value of providing empirical demonstrations in a space with limited evaluation frameworks. We have uploaded an updated draft to OpenReview that addresses your feedback with highlighted changes in blue. While we've been engaged in extensive discussions with all reviewers and have conducted additional experiments requested by others (including evaluating LLM-based diversity metrics and adding QWen-Coder 2.5), I'd like to highlight the key changes most relevant to your feedback:

1. Dataset Construction Documentation

• We've added comprehensive documentation of our dataset construction process in the Appendix (pages 15-16)
• This includes a detailed flowchart visualizing the complete pipeline
• These additions provide transparency into our methodology and enable better reproducibility
• We believe this documentation better demonstrates the careful consideration that went into our data collection process

2. Paper Structure and Presentation

• We've substantially expanded the Background and Related Work sections to better contextualize our research
• The improved structure better frames our key contributions, particularly our finding that preference tuning produces an interesting trade-off: increasing semantic diversity while reducing lexical/syntactic diversity
• This finding challenges the prevailing narrative about preference tuning uniformly reducing diversity [Kirk et al., 2024]

3. Contribution and Findings

• We've clarified how our work advances the field by:
  • Providing a reproducible methodology for measuring semantic diversity
  • Demonstrating that our semantic diversity metric measures properties of LLM generations that are not reflected in Neural or Lexical diversity metrics
  • Showing that previous conclusions about diversity reduction may need to be reconsidered and studied further

We've worked diligently to address concerns from all reviewers, and the constructive exchanges have helped strengthen the paper significantly. Your feedback about presentation and dataset documentation was particularly valuable in improving the paper's clarity. We believe we've addressed the core concerns raised in all reviews, and we are still waiting to hear back across the board, and would appreciate your thoughts on whether these changes adequately address your specific concerns.

I hope this message finds you well. As we approach the end of the review period, I wanted to gently follow up regarding our revised submission. We greatly appreciated your thorough initial review and have made significant changes based on your valuable feedback, including:

• Expanding the dataset construction documentation with a detailed explanation and a flowchart
• Restructuring the paper with enhanced Background/Related Work sections
• Clarifying our contributions
• Addressing the presentation issues you identified

Given today is the last day for reviewer responses, we would be very grateful if you could take a look at our revisions and provide any final thoughts. Your insights would be particularly valuable in assessing whether we've adequately addressed your initial concerns. Thank you again for your time and thoughtful consideration of our work.

7. Questions

• Best Semantic Diversity in Figure 1: We have clarified what our takeaways for this were and we will update our draft to also expand the discussion around this point. We also believe that our dataset will enable important future work on these questions.
• Increasing Number of Problems: As mentioned, we believe increasing samples per problem enhances the robustness of our experiments more effectively than adding more problems.
• Comparing SFT and PT on Identical Base Models: As addressed above, we will include experiments comparing SFT and PT using the same LLama3.1. Additionally, we have explained that this only affects conclusions for SFT, and that we have already tried to mitigate for this effect by using more than one model family for PT as well as direct and paired comparisons for model size and prompting experiments.
• Controlled Comparison with Varying Training Set Diversity: We will acknowledge this as an important area for future work and discuss how our benchmark could facilitate such studies.
• Adopting CodeExecutor for Neural Diversity Evaluation and Adding LLM-Based Neural Diversity Metrics: We do not believe that Chase-SQL may be directly utilized for diversity evaluation without some custom modification and adaptation. We’ve mentioned that we used CodeBertScore as it was published in 2023 and bears high resemblance to models that have been evaluated for diversity. We will try to the best of our ability to evaluate another neural metric, ICE-Score as reviewer ASEt suggested by the rebuttal deadline, and at least one more for the camera-ready version.

5. Findings on Execution-Based vs. Lexical Diversity

Reviewer Comment: “The finding that execution-based diversity differs from lexical/syntactic diversity is unsurprising.”

We appreciate your observation regarding the N-Gram-based metrics and execution-based metrics for functional accuracy. We believe there may be a misunderstanding around the purpose of this section of the paper.

• Clarification: While previous studies have highlighted that execution-based accuracy and N-Gram metrics like BLEU do not correlate perfectly, we believe there may be a misunderstanding here. Our intention was not to present our correlations as a novel finding but rather to use it as a motivation for our methodology and validation that our implementation does what it is supposed to do. We are introducing a novel methodology to measure diversity, and it is important that it measures something that N-Gram metrics and Neural Metrics do not. By quantifying the significant differences between these metrics, we aim to convince readers—especially those who might be significantly more skeptical—that execution-based evaluation measures something substantially different in code generation compared to N-Gram/syntactic metrics. Moreover, there is a significant difference between reference-based accuracy metrics where there is only one correct answer and our methodological proposal for execution-based diversity which allows for a wide range of acceptable solutions: it is novel to use execution traces from programs as a method to measure diversity and it is not exactly straightforward to implement.

• Contribution: Our results strengthen the case for utilizing execution-based metrics when evaluating diversity. While prior work has focused on functional correctness, our study validates that the disparity between execution-based and lexical measures extends to diversity assessments. This underscores the limitations of relying solely on lexical or syntactic metrics and highlights the necessity of execution-based evaluation to measure semantic diversity in generated code. While this fact may be obvious to some of us, leveraging this advantage for diversity measurement is non-trivial. Our methodology is novel, and as a result, we believe this is an important contribution to the body of work on measuring LLM diversity where no equivalent methodology has been proposed.

6. Elaboration on Temperature Settings (Figure 1)

“Strengths: Figure 1’s temperature plot reveals potential improvement over CodeT's [1] default T=0.8, suggesting that T=0.9 or T=1.0 may yield higher diversity.”

• Clarification: Perhaps there is a slight misunderstanding about the motivation behind including the figure. In Figure 1, we observed that increasing the temperature leads to higher semantic diversity up to a certain point before outputs begin to degenerate in quality. We did not aim to make "the optimal temperature" parameter scope of our work, but we believe these are interesting directions for future work. Instead, we included this figure as a proof-of-concept to validate that our implementation of semantic diversity is sound. It captures the idea that there is a "sweet spot" where the temperature maximizes diversity without compromising the coherence of the generated code. Our findings indicate that semantic diversity metrics effectively capture this phenomenon, whereas N-gram or syntactic measures do not.

• Reasoning: We agree that the question of optimal temperature sampling parameters is interesting: this was a major motivation for our work, but we later found that answering questions about model size and instruction tuning were more urgent to the community and where our findings were most interesting. A short answer to your question about CodeT's default of 0.8: the optimal sampling temperature for open-ended diversity may vary from model to model. Furthermore, sampling for diverse outputs may require a higher temperature than sampling for “correct” outputs as the nature is different. As a result, we think that our dataset could be an interesting tool for researchers interested in choosing optimal sampling parameters for open-ended generation tasks.

• Future Work: We think that more detailed future work should be done to analyze how additional sampling parameters affects diversity such as nucleus sampling and top-p sampling: for our draft our primary focus was on the effects of instruction tuning, model size, and prompting on diversity, where we believe our findings are particularly insightful and break new ground. If you think it would further strengthen our paper, we could investigate more questions if you think that would be valuable for the camera ready version if they are feasible.

• Action: In response to your suggestion, we will expand our discussion of temperature settings in the revised paper.

4. Use of CodeBERTScore

Reviewer Comment: “CodeBERTScore is outdated. More current options include UniXCoder [5] or CodeExecutor [6]. Another approach could leverage LLM-based evaluation, such as in CHASE-SQL [7]”

We appreciate your suggestions regarding evaluation metrics and understand the importance of using appropriate tools to assess neural diversity.

• Clarification: We respectfully believe that CodeBERTScore is an appropriate and current metric for our study, yet we will try to add at least another related model-based approach. We chose CodeBERTScore, it was the most analogous model to those used in recent diversity works BertScore and Sentence-BERT [1][2], making it a suitable starting point for assessing neural diversity in our study. Additionally, given its recent publication (published in December, 2023) and has been widely downloaded and utilized in the community (>400K downloads), we thought it reasonable to begin with.

[1] Tevet, Guy, and Jonathan Berant. "Evaluating the Evaluation of Diversity in Natural Language Generation." Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021

[2] Kirk, Robert, et al. "Understanding the Effects of RLHF on LLM Generalisation and Diversity." The Twelfth International Conference on Learning Representations.

• LLM-Based Evaluators like CHASE-SQL: We acknowledge the potential of LLM-based evaluators for code assessment. However, CHASE-SQL is specifically designed for evaluating SQL code and may not be directly applicable to Python code generation, which is the focus of our work. Adapting CHASE-SQL to Python would involve substantial modification and validation, which is outside the scope of this paper. Lastly, it seems that Chase-SQL is still under review and we were unable to find a reference implementation on Github. Instead, we will try to utilize ICE-Score [3], which is recent and LLM-based, as reviewer ASEt suggested. [3] Zhuo, Terry Yue. "ICE-Score: Instructing Large Language Models to Evaluate Code." Findings of the Association for Computational Linguistics: EACL 2024. 2024.

• Regarding UniXCoder and CodeExecutor: While UniXCoder (2022) is a valuable model for code representation, we found that CodeBERTScore (2023) to be recent, rigorously evaluated on functional correctness (CodeBertScore Section 4), and bears resemblance to models used in the literature [1][2]. CodeExecutor, on the other hand, is specialized for code execution tasks and, to our knowledge, has not been evaluated in terms of its correlation with functional correctness or diversity metrics for code generation.

• Action: While we recognize that exploring additional models could provide further insights, including every possible model may not be practical within the constraints of this work. Given the short time of response, we will add analysis using ICE-Score as reviewer ASEt suggested. For our camera ready we can commit to adding analysis for either UniXCode or CodeExecutor, but we cannot guarantee that we will add these by the rebuttal deadline. Additionally, we will expand our related work section to discuss alternative models like CHASE-SQL, UniXCode, and CodeExecutor.

• Future Work: We consider the exploration of alternative models and more specialized LLM-based evaluators for Python code diversity as valuable directions for future research.

Thank you for your time in reviewing our paper, we appreciate the time you put into it and your feedback. Hopefully we can make some clarifications, and we also have some quick questions (e.g. on the appropriate SFT dataset) we’d appreciate a reply to which may affect changes you requested.

1.Open-Ended Dataset Size

We understand the concern regarding the generalizability due to the number of open-ended prompts in our dataset.

• Action: In the camera ready version, we propose to increase the number of samples per problem from 100 to 200, resulting in a total of 4,200 programs. This enhancement will improve the robustness of experiments and reduce variance. Additionally, we will include examples in the supplementary material to illustrate the extensive variability within individual problems
• Justification: Our problems are open-ended and designed to allow a vast space of valid solutions, each capable of exhibiting significant diversity.

For example, Problem 02280 (prompt provided in the supplementary materials) in our dataset prompts the language model to generate a function that processes a list of nodes. This prompt allows for a very diverse set of generations. For example, when we looked at some of LLama3.1 70B Instruct’s submissions here were some examples of the diverse generations it created:

• Find all articulation points in a graph using Tarjan’s Algorithm
• Find the number of connected nodes in a graph using Union-Find
• Print the number of nodes in a graph with multiple valid non-cyclical paths in a specific and certain manner
• Print out the count of unreachable and reachable nodes from node 0 in the list
• Print out all nodes in the graph using DFS
• Check if a tree is a valid binary tree
• Check if the structure is a binary search tree (with some additional explicit constraints)
• Remove nodes / relations in a graph with certain cyclical properties
• Find the maximum depth in a tree-like structure

Given the time it takes to sample from so many language models, we do not believe that we can complete this by the rebuttal deadline. However, we don't anticipate our results to change dramatically.

Given that the prompts are so abstract and open-ended and having manually constructed this dataset from CodeNet, we believe the abstracted prompts are already quite comprehensive, and we understand that may sound very unintuitive. However, most competitive programming problems rely on relatively simple data structures like lists of integers. Despite this limited set of possible input types, competitive programming problems can ask for an extremely wide range of programming solutions. This is why we think taking higher samples will still achieve this goal.

2. Comparison Between Different Base Models

We acknowledge that comparing CodeLlama with Llama3-based models may introduce confounding factors due to differences in base architectures, and we have tried to mitigate this effect on our conclusions and findings. Nevertheless, we commit to an SFT experiment with LLama3.1 8B for further robustness.

• Mitigation: To address this, we have conducted paired statistical tests and observed consistent trends, particularly for the preference-tuned models across different model families. This indicates that the effects of preference tuning on diversity are robust despite differences in base architectures.

• Action: We will also conduct additional experiments by performing SFT on a Llama3.1-base model and compare it to Pre-Trained to ensure a fair comparison.

• Feasibility: While we may not complete these experiments before the rebuttal deadline, we commit to including them in the revised paper.

• Action Item: We propose using the Magicoder OSS Instruct fine-tuning dataset [1] for this SFT experiment, we would like to confirm that you believe this is a sufficient approach. We’d like to know as early as possible to give us as good a chance to turn around results within the window.

[1] Wei, Yuxiang, et al. "Magicoder: Empowering code generation with oss-instruct." Forty-first International Conference on Machine Learning. 2024.

• Clarification: We will revise our draft to acknowledge that our claims regarding supervised fine-tuning (SFT) on their own are not as strong due to this confounding factor. However, we believe that our findings for preference tuning (e.g., PPO/DPO), model size, and prompting are robust due to the use of multiple model classes.

3. Influence of Training Set Diversity

We agree that training set diversity can impact model diversity.

• Action: We will clarify in the paper that while retraining large models to vary training set diversity is beyond our current resources, our methodology provides a framework for such studies.
• Future Work: We will suggest that future research could leverage our benchmark to investigate the effects of training data diversity on model outputs.

... We chose 21 competitive programming problems from CODENET and manually abstracted the problem descriptions. ...

... we randomly generated 100 additional test cases by writing a property-based testing ...

### Input Description:

An integer $ n $ (1 ≤ $ n $ ≤ 25), representing some quantity or size.
A list of $ n $ tuples, where each tuple contains three integers $ id $, $ left $, and $ right $:
- $ id $ (0 ≤ $ id $ < n), representing a unique identifier.
- $ left $ (−1 ≤ $ left $ < n), representing a relationship or connection.
- $ right $ (−1 ≤ $ right $ < n), representing another relationship or connection.

### Example Input:


9
0 1 4
1 2 3
2 -1 -1
3 -1 -1
4 5 8
5 6 7
6 -1 -1
7 -1 -1
8 -1 -1


### Function Signature:
Write a function `f(n, nodes)` that takes in the input:
def f(n: int, nodes: List[Tuple[int, int, int]]):
    '''
    n: an integer
    nodes: a list of tuples, where each tuple contains three integers

As I reread lines 030-032:

Many real-world tasks are open-ended to some degree with many possible answers—e.g.,
writing convincing essays, suggesting cooking recipes, writing unit tests, etc. Evaluating diversity
can also provide insights into the nature of language models, especially their creative capabilities.

I think 'many possible answers' implies that there exist some constraints or conditions for expected outputs (as random word sequences cannot be considered valid answers). The main problem in this paper is that the vagueness in problem descriptions. This makes it hard to scale up because problems need to be manually annotated, and makes it difficult to evaluate functional correctness (as the problem allows different functionalities), leaving the evaluation metrics reliant on either human efforts or model-based alternatives which could be erroneous.

One way to tackle the original question—evaluating the model's diversity—is to narrow down the constraints that the expected outputs would have. That is, if we fix the expected functionality, then we can ensure the validity of answers while measuring lexical and syntactic diversity through n-grams and AST-based similarity metrics. This direction is scalable, as we can directly employ existing code generation benchmarks (e.g., BigCodeBench [1] of 1K problems), and we can also ensure functional correctness.

Compared to this possible direction, why should we use the proposed approach, which is not scalable (only 21 problems) and does not ensure the validity of outputs relative to the problem description?

[1] Anonymous (2024). BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In Submitted to The Thirteenth International Conference on Learning Representations.

My response will be imperfect, because we're in a time crunch between the paper revision and other things happening this week. But I appreciate the thinking you're doing about the paper, they're very good questions. We hope that we could convince you that we have thought about similar questions and yet we strongly believe the work has substantial merit.

1. Choosing Open-Ended Questions Instead of "Directed" (i.e. either Correct or Incorrect) Questions

"That is, if we fix the expected functionality, then we can ensure the validity of answers while measuring lexical and syntactic diversity through n-grams and AST-based similarity metrics. This direction is scalable, as we can directly employ existing code generation benchmarks (e.g., BigCodeBench [1] of 1K problems), and we can also ensure functional correctness."

Actually, early on, we initially intended to analyze how lexical and syntactic diversity would change when looking at directed experiments where there is only 1 correct answer. We have working code to do this, and we are happy to provide these in our supplementary .zip if you like; we plan to open source them on acceptance.

However, we didn't think that the questions we could ask with such a setup were going to make any strong methodological contributions nor noteworthy empirical findings. Methodologically, evaluating lexical and syntactic diversity is a weak contribution at best. Prima-facie, increasing/decreasing lexical or syntactic may not have a great degree of practical importance. And some preliminary work on natural language summarization (where the semantic content generally will be the same) has already probed the effects of preference-tuning on lexical diversity [1].

We strongly believe the most interesting contributions and research questions were related to open-ended generation. This is because code can measure semantic diversity at scale. The questions about automatically differentiating semantic vs. diversity in form (lexical and syntactic diversity) is a methodological contribution and fills a missing need in the community. And the empirical questions we probe, the evidence we find that preference-tuning increases semantic diversity and decreases diversity in form (lexical and syntactic diversity), these challenge the dominant narrative on how preference-tuning (e.g. RLHF) impacts diversity [1]. As a result, we strongly believe the methodological and empirical findings are both noteworthy.

[1] Kirk, Robert, et al. "Understanding the Effects of RLHF on LLM Generalisation and Diversity." The Twelfth International Conference on Learning Representations.

2. Potential Lack of Constraints

"I think 'many possible answers' implies that there exist some constraints or conditions for expected outputs (as random word sequences cannot be considered valid answers)...."

I appreciate you bringing up these points, and we've thought hard about similar points while working on this, and I can understand that without clarification I'd also have the similar questions.

I would respectfully disagree, and strongly argue that our semantic diversity places constraints on programs. We think Figure 1 in the paper does a great job of showing how semantic diversity as we measure it drops when we increase Temperature to be very high. As the generations become more like random strings the semantic diversity rapidly decreases. This is because nearly all generations become syntactically incorrect, and therefore are incapable of meaningful semantic diversity. Our implementation captures this phenomenon. However, we see that as temperature becomes high, the lexical diversity becomes very high. On the "temperature test" our semantic diversity is much more robust to the effect of random / spurious strings

In general, if all the generations are syntactically incorrect or the LLM consistently misunderstands the prompt and is not capable handling types in the right way, the semantic diversity will suffer. The syntactic constraints, type constraints, and the requirement to print results to the console are constraints on outputs. Generally, most of our models achieved Coherence (Section 3 for description) scores below 50%, implying that often LLMs are incapable of following the instructions in the prompt. We think future work can and should explore adding even more constraints on outputs, (e.g. requiring only 1 integer printed out between a certain range), but we did not feel it was necessary nor impacted the validity of the empirical questions we ask in our paper. For example, a major narrative in our paper is how preference-tuning impacts diversity. We believe we find enough evidence as it is to challenge the dominant narrative that preference-tuning reduces diversity: instead, we find it can reduce diversity in lexical form while increasing semantic diversity. Setting a low temperature value (e.g. 0.2) is much more likely to constrain semantic diversity than opting for preference-tuning.