ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments

We sincerely thank you for your detailed review.

Q1-1. How are you detecting whether ground truth code is leaked by the GPT 4o based expert? From reading A.3, I couldn’t quite understand this. Is the ground_truth_code function doing an exact match lookup in GPT 4o’s feedback? Or is there another implementation?

Q1-2. Table 7 seems to be justifying that the verbal feedback doesn’t leak ground truth code because of downstream task performance differences, instead of a direct examination of the verbal feedback’s content.

A1-1: Ground truth code leakage detection is done by canary sequence:

• We considered as leakage if the feedback simulator includes a canary sequence (while it is misinterpreted as a function in the review). This use of canary sequences—ground_truth_code given in the prompt (see Example 1)—is common in testing for training data or prompt leakage in LLMs [1, 2, 3, 4, 5] (see response towards Reviewer cHoy for a leakage example).
• How ConvCodeWorld minimizes the leakage?: We chose GPT-4o for the simulator since its significantly lower rate of ground truth code mention (2.5%) compared to GPT-4-0613 (51.1%) and GPT-4-Turbo-2024-04-09 (31.4%), and only 0.1% of refined code inclusion (see Table 8).

We will clarify this behavior in Appendix A.3 and provide examples to further illustrate these observations.

A1-2: Table 8 is a direct examination

• Table 8 provides a direct examination of the verbal feedback's content. In Table 8, we report the instances of leakage as defined by the occurrence of ground_truth_code in the feedback.
• From the direct analysis the content of the verbal feedback for specific terms, we selected GPT-4o-2024-05-13 as the expert feedback simulator, which has significantly lower ground truth code leakage compared to other models.

Example 1. Prompt used for expert feedback generation in the feedback combination $\langle f_c, [f_e|f_e^*], f_v^* \rangle$.

You are given input, previous_code, execution_feedback to simulate user feedback that compares the `previous_code` and the `ground_truth_code`.
Your task is to provide the simulated `user_feedback` that highlights specific areas where the `previous_code` deviates from the `ground_truth_code` and suggests improvements or corrections.
- You SHOULD NOT leak `ground_truth_code` in the simulated user feedback.
- Do not generate updated code.
- Do not reveal that you can access the `ground_truth_code`. Only indirect information is allowed.

Q2-1. What does Table 9 mean? When it is first referenced in Section 2.1.1, the phi character does not appear to be defined at that point in the paper.

Q2-2. Also, GPT 4/4o models are tested exclusively. Any reason why Claude / Llama / other models were not used?

A2-1: Clarification on Table 9:

• Table 9 evaluates different models as potential verbal feedback simulators. In this table:
  • Each row represents a model used to provide verbal feedback.
  • Each column represents a model that utilizes this feedback to refine code.
• By comparing the performance across columns, we assess the effectiveness of the feedback provided by each simulator.
• Regarding the symbol $\phi$, we will provide a clear explanation in the caption of Table 9 in the next version.

A2-2: Why GPT models?:

• Previous findings in studies like MINT (see Table 4 in the MINT paper) demonstrated that model performance is crucial for effective feedback.
• During the setup of our experimental settings, there was no such powerful open-weight models (see response towards Reviewer cHoy for using open-source models as verbal feedback simulators).
• Based on that, we decided to consider high-performing closed-source models, such as GPT-4 and GPT-4o, which allowed us to achieve reliable results.
• Claude models were not considered due to resource constraints and computational credits supported by our organization.

We sincerely thank you for your detailed review.

Q1. Could you clarify that the number of samples for the single-turn baseline (i.e., first column in Table 3 and Table 4) is the same as the number of maximum turns?

In our experiments, the single-turn baseline generates a single code sample via greedy decoding, following [1]. This approach mirrors the typical user experience in code generation tools, where a user receives a single initial suggestion and may refine it through feedback in subsequent turns. We will explicitly state this in the revised version.

Q2. Currently, it seems that for a given example, the same combination of feedback is given at each turn. Have you considered varying the type of feedback at different turns (e.g., compiler only in turn 1, compiler and execution in turn 2, compiler and novice feedback in turn 3, compiler and expert feedback in turn 4). Perhaps certain types of feedback or more useful (and more practical) at different steps?

Our framework accommodates varying feedback types across turns, though we opted for consistent feedback combinations to maintain result clarity. Future work will explore a feedback recommender system to optimize feedback source selection per turn.

W1. While I agree that having a static benchmark like ConvCodeBench is useful, I am not convinced that it can be interpreted as a multi-turn evaluation set. Namely, my understanding is that it does not evaluate whether a model can iteratively improve its own prediction. The output from the previous turn is pre-defined, and so this seems to simply be trying to repair a program (which the model under test did not generate). If it fails to do it at a given turn, the output of that turn is ignored, and in the next turn, a fixed program is provided, which could be very different from the program the model under test generated in the previous turn. This setting seems to be a bit unnatural to me.

While awaring these discrepancies, we found that appropriate reference model selection produces high correlations with live results (we provide detailed analysis in our response to 3Fw3).

W2. The environment is constrained to Python only.

• Although ConvCodeWorld extends BigCodeBench, a benchmark focused on Python, it is not restricted to Python. Specifically:
  • Compiler feedback can be generated using the compiler of any target language.
  • Execution feedback is universally applicable across different programming languages.
  • Verbal feedback remains independent of the programming language in use.
• As ConvCodeWorld is open-sourced, it is easily adaptable for future efforts to incorporate additional programming language benchmarks.

S1. It would have been interesting to have an ablation without the compiler feedback.

While we plan to include full ablation results in the camera-ready version, our current findings from Tables 3 and 4 show minimal performance differences between no feedback and compiler-only feedback, as most evaluated models achieve nearly 100% compilation success. This suggests that results without compiler feedback will likely remain consistent.

S2. Other relevant paper to cite: https://arxiv.org/pdf/2306.09896

Thank you for bringing this relevant paper to our attention. We will include it in our revised manuscript.

[1] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, & Weizhu Chen (2023). CodeT: Code Generation with Generated Tests. In The Eleventh International Conference on Learning Representations .

W4-1: The claim that "Weaker Models with Feedback Surpassing Single-Turn SOTA Models" seems an unfair comparison. While I understand that the aim is to highlight the value of multi-turn interactions, weaker models with multi-turn feedback inherently benefit from additional input context compared to single-turn, stronger models. It is therefore unsurprising that the weaker model with multi-turn interactions outperforms single-turn, stronger models.

Weaker models with additional context not always outperform SOTA

• Prior research reported that smaller models may struggle with additional context [3].
• Our empirical results reflect this mixed reality, showing that while some models benefit from additional context, others do not (see Table 4):
  • Llama-3.1-8B-Instruct improves significantly from 31.4 to 51.8 Recall with feedback ($\langle f_c, f_e^*, f_v \rangle$), surpassing GPT-4o-2024-05-13's no-feedback score (50.8).
  • DeepSeek-Coder-6.7B-Instruct, despite a higher baseline performance (35.2 vs. 31.4), only reaches 48.2 with identical feedback, failing to surpass GPT-4o-2024-05-13.
• These findings highlight the challenge of utilizing multi-turn feedbacks and justifies our evaluation to better understand this dynamic.

W4-2: A fair comparison would evaluate the same LLMs in both single-turn and multi-turn scenarios (I thought this could also be concluded from the results section). The statement about using the same LLM would more effectively emphasize the role of multi-turn interactions in enhancing performance.

Meanwhile, comparison of same LLMs in single-turn and multi-turn is also reported

• In our paper, Figure 3 illustrates how the Pass@1 score evolves over multiple turns for the same model, underscoring the impact of multi-turn interactions on performance.

[1] Chen, Mark, et al. "Evaluating large language models trained on code." arXiv preprint arXiv:2107.03374 (2021).

[2] Li, Yujia, et al. "Competition-level code generation with alphacode." Science 378.6624 (2022): 1092-1097.

[3] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

We sincerely thank you for your detailed review.

Q1: Could the author add more comparisons between ConvCodeWorld and MINT to enhance the clarity?

W1: In the MINT paper (Wang et al., 2024), the multi-turn interaction (Figure 1, page 3) already includes execution results and human feedback. The human feedback in MINT covered both novice feedback (referred to as "lazy user" feedback in MINT) and expert feedback (referred to as "natural language" feedback in MINT). Given this, the novelty of this paper appears limited. Could the author include more comparisons between ConvCodeWorld and MINT to enhance clarity and novelty?

A1: Novelty from MINT

• We respectfully disagree that MINT's "lazy user" feedback serves the purpose of introducing "engaged" novice feedback. ConvCodeWorld's novice feedback provides verbalized explanations of other feedback (e.g. execution feedback), as illustrated in Figure 16:

    "It seems like there is an issue with the socket connection or the way the code is handling the socket. The OSError exceptions are being raised during the execution of the task_func function."
  

  In contrast, MINT only provides generic response:

    "Your answer is wrong."
  

• Another key distinction is, supporting partial to full test coverage in execution feedback:

  • ConvCodeWorld allows isolated assessment of an LLM's test generalization and test utilization:
    • Test generalization: The model's ability to produce code that passes comprehensive tests when only partial tests are provided.
    • Test utilization: The model's capability to leverage test results for code refinement.
  • Meanwhile, MINT (partial test only) provides an entangled evaluation of test generalization and test utilization. We revisit this point in our response to the third question (A3-2).

Further detailed contrasts are summarized in Table 6 in the draft and also in response to cHoy.

Q2: Could the author address the discrepancy in feedback simulation compared to actual human feedback?

W2: Figure 13 (appendix) shows one example of the simulated expert-level verbal feedback, which is overly detailed and well-structured. I am concerned that it might not accurately reflect how real experts provide feedback when using LLMs in coding. An analysis comparing simulated feedback to actual human feedback would be both helpful and necessary. A similar concern was found in the simulated novice-level verbal feedback (Figure 12, appendix). The simulated novice-level feedback includes information that seems unrealistic for novice programmers (e.g., "Also, I think there exists a simpler way to sort a list in Python."). How could a novice programmer provide such feedback based on the compilation and execution results? As suggested above, an analysis of how the simulated feedback relates to actual human feedback is necessary.

A2: Figures 12 and 13 are in-context examples, not model-generated feedback

• Model-generated examples of novice and expert feedback is presented in Figures 16 and 17 respectively, not in Figures 12 and 13. Particularly, in Figure 16, the novice feedback primarily consists of verbalized explanations of the execution feedback, which we might expect from novice programmers.

• We acknowledge the value of conducting a large-scale HCI study with real experts to provide deeper insights. To support such analyses in future research, we have open-sourced both ConvCodeWorld and ConvCodeBench, enabling researchers to compare simulated feedback with actual human feedback in subsequent studies.

• Additionally, as reported from a human study in the MINT paper (Table 5), ground truth code-conditioned expert feedback simulations are generally considered human-like, indirectly supporting the validity of simulated feedback in large-scale evaluations.

W1: Novelty: While this paper offers valuable insights into evaluating conversational code generation, the topic itself is not new. Prior works like InterCode and MINT have explored compilation, execution, and verbal feedback mechanisms (see Table 6).

We elaborate distinctive implications from InterCode and MINT:

1. Comparative Analyses of Partial to Full Test Coverage in Execution Feedback enables to evaluate both:

   • Test generalization: A model's ability to produce code that passes full tests even when only partial tests are provided.
   • Test utilization: A model's capability to leverage given test results for code refinement.

   a) MINT (partial test only): an entangled evaluation of test generalization and test utilization.

   b) InterCode (full test only): evaluates test utilization only.

   c) ConvCodeWorld, by providing both partial and full test, enables isolated evaluation of each test generalization and test utilization as we illustrate below.

   For instance, in Table 4:

   • DeepSeek-Coder-6.7B-Instruct:
     • Modest test generalization ($\langle f_c, \phi, \phi \rangle \rightarrow \langle f_c, f_e, \phi \rangle$: 35.2 $\rightarrow$ 37.7)
     • But limited test utilization ($\langle f_c, f_e, \phi \rangle \rightarrow \langle f_c, f_e^*, \phi \rangle$: 37.7 $\rightarrow$ 37.5)
   • In contrast, Qwen1.5-72B-Chat exhibits strong capabilities in both aspects:
     • Test generalization: $\langle f_c, \phi, \phi \rangle \rightarrow \langle f_c, f_e, \phi \rangle$: 33.2 $\rightarrow$ 39.9
     • Test utilization: $\langle f_c, f_e, \phi \rangle \rightarrow \langle f_c, f_e^, \phi \rangle$: 39.9 $\rightarrow$ 47.5

2. ConvCodeWorld simulates an "engaged" user, offering verbalized explanations of test results, as illustrated in Figure 16:

    "It seems like there is an issue with the socket connection or the way the code is handling the socket. The `OSError` exceptions are being raised during the execution of the `task_func` function."
   

   In contrast, InterCode lacks verbal feedback, and MINT provides only generic feedback:

    "Your answer is wrong."
   

   Full execution feedback + novice feedback scenario in ConvCodeWorld effectively evaluates how verbalized explanations enhance models' test utilization capabilities. In Table 4:

   • Full test coverage execution feedback ($\langle f_c, f_e^*, \phi \rangle$): Llama-3.1-8B-Instruct's test utilization capabilities (40.0) are weaker compared to CodeQwen1.5-7B-Chat (41.1).
   • However, the inclusion of novice feedback ($\langle f_c, f_e^*, f_v \rangle$): significantly improves Llama-3.1-8B-Instruct's performance, surpassing CodeQwen1.5-7B-Chat (51.8 vs. 49.5).

3. Covering comprehensive combinations of feedback types, ConvCodeWorld analyzes previously underexplored cases, such as:

   • Full execution feedback vs. partial execution feedback + novice feedback
   • Partial execution feedback + expert feedback vs. full execution feedback + expert feedback
   • Full execution feedback + novice feedback vs. expert feedback

4. Cost-Effective Static Benchmark (ConvCodeBench): ConvCodeBench correlates strongly with online evaluation while reducing costs. Neither MINT nor InterCode provide such a static benchmark.

W2-1: Clarity of Writing: The paper’s clarity could be improved, particularly in the introduction. Expanding the background on conversational code generation tasks and typical settings would help readers appreciate the unique contributions of this work.

We will expand the introduction with comprehensive background on conversational code generation, and relocate related work to follow it—better highlighting our novel contributions.

1. **Configuration File Reading**: The `previous_code` correctly reads the configuration file using `configparser`. However, ensure that the configuration file path is valid and exists before attempting to read it. This is not explicitly checked in the `previous_code`.

2. **Directory Existence Check**: The `previous_code` uses `os.path.exists(project_dir)` to check if the project directory exists. While this works, it is more appropriate to use `os.path.isdir(project_dir)` to specifically check for directory existence, as it is more semantically correct.

3. **ZIP Archive Creation**: The `previous_code` attempts to create the ZIP archive using `shutil.make_archive(project_dir, 'zip', archive_dir)`. This is incorrect because `shutil.make_archive` expects the base name of the archive and the root directory to archive. The correct usage should be `shutil.make_archive(base_name=os.path.splitext(zip_file_path)[0], format='zip', root_dir=project_dir)`.

4. **Exception Handling**: The `previous_code` raises a generic `Exception` if the ZIP archive creation fails. While this is acceptable, it is better to provide a more specific error message indicating the failure reason. Additionally, ensure that the ZIP file is actually created by checking its existence after the `shutil.make_archive` call.

5. **Return Value**: The `previous_code` correctly returns `True` if the ZIP archive is successfully created. However, it should also ensure that the ZIP file exists before returning `True`.

6. **Code Simplicity and Readability**: The `previous_code` includes a detailed docstring, which is good practice. However, the actual implementation can be simplified and made more readable by following the correct usage of `shutil.make_archive` and ensuring proper exception handling.

Overall, the `previous_code` has the right structure but needs corrections in the directory existence check, ZIP archive creation, and exception handling to function correctly.

1. **Class Name**: The class name in the `previous_code` is `EmailHandler`, but it should be `EmailRequestHandler` to match the `ground_truth_code`.

2. **Content-Type Check**: Instead of directly checking the `Content-Type` header, use `cgi.parse_header` to parse the header and then check if `ctype != 'application/json'`.

3. **Error Handling for Content-Type**: When the `Content-Type` is not `application/json`, simply send a 400 response and end headers without writing a message to the response body.

4. **Reading Content-Length**: Use `length = int(self.headers.get('content-length'))` instead of `content_length = int(self.headers.get('Content-Length', 0))`.

5. **JSON Decoding**: When catching `json.JSONDecodeError`, send a 400 response and end headers without writing a message to the response body.

6. **Missing Fields Check**: When required fields are missing, send a 400 response and end headers without writing a message to the response body.

7. **SMTP Authentication Error Handling**: When catching `smtplib.SMTPAuthenticationError`, send a 535 response and end headers without writing a message to the response body.

8. **General Exception Handling**: Remove the general exception handler that sends a 500 response, as it is not present in the `ground_truth_code`.

By making these changes, the `previous_code` will align more closely with the `ground_truth_code`.

We sincerely thank you for your detailed review.

Q1. While the authors evaluate verbal feedback with proprietary GPT models, it would be beneficial to include SOTA open-weight models as well. Given that GPT models evolve frequently, this reliance may risk data leakage. Testing open-weight models, even if currently less performant, could strengthen the reproducibility and discussion of findings.

A1: Using open-weight models as verbal feedback simulators is also effective

• Table A supports the feasibility of using Llama-3.1-70B-Instruct as a verbal feedback simulator, replacing GPT-4o-2024-05-13.
• At the time of setting up our experiments, we considered open-weight models. However, given MINT's findings that model performance significantly impacts feedback quality (Table 4 in the MINT paper), and the unavailability of powerful models like Llama-3.1-70B-Instruct at the time, we opted for GPT models.

In the camera-ready, we will also report the main results of ConvCodeWorld and ConvCodeBench where verbal feedback is simulated by concurrent open-source models like Llama-3.1-70B-Instruct.

Table A. : Pass@1 results over different expert-level verbal feedback $v_f^*$ generation on ConvCodeWorld where $\Omega = \langle f_c, \phi, f_v^* \rangle$, the total number of turns $n=1$.

Q2. The claim in Appendix A.2 that the approach achieves "1.5% of the cost of human annotation" seems optimistic. Beyond the token generation cost, quality and accuracy of the model-generated content should also be factored in.

A2: 1.5% is not optimistic

• We respectfully disagree 1.5% is an optimistic estimate, as the hourly rate of expert programmers required for $f_v^*$ annotation is indeed higher than the base rate used in the calculation.

• While other costs like quality are not included in our numeric cost analysis, recent work evidences that top-performing LLMs demonstrate skills comparable to median human programmers [1], significantly reducing dependence on human experts. Furthermore, MINT’s human evaluation (Table 5) confirms that expert feedback simulations conditioned on ground truth code are capable of producing feedback that is both helpful and human-like, underscoring the feasibility of automating this process.