Revisit Self-Debugging with Self-Generated Tests for Code Generation

• If I understand the method correctly, the way you are generating the trace for the "in-execution" method requires treating dynamic loops as one atomic unit. For example, if you have for loop which an unknown number of iterations, you can't statically insert a breakpoint halfway through it, so instead you have to break before or after. I would like to see this limitation discussed in the paper; do you, for example, have any thoughts on whether this hurts performance in practice, e.g. if the bug often lays within such a loop?
• I think the paper could do a better job emphasising that a +X.Y% increase in pass rate after 3 iterations of self-debugging does NOT mean that's what you want to do in practice, since it is compared against a baseline with 1 sample - not a baseline with the same budget as you used up during self-debugging. I apreciate that the point of this paper is not to chase numbers on these benchmarks but to actually do some cool science so we can understand why these things work the way they do, but again I am worried that a less familiar reader might mistakenly see this as a raw performance increase (neglecting the cost).
• Finally, I would encourage you to double check your references list. I was confused why you kept citing "Chen (2023)" when I was pretty sure that self-debugging paper was at ICLR 2024; indeed your references list it as ICLR 2023, which I do not believe to be correct. Probably worth double checking that you don't have any more mistakes in there.

An important findings from this paper is that the in-execution state can help reduce the bias of wrong self-generated tests. However, the section of in-execution is short and not very detailed, which I think need more details:

1. A case study of how the bias is offset by in-execution information.
2. Since the in-execution helps offset the bias from wrong self-generated tests, it could be better presented in a comparison figure with post-execution information. This can show that although in-execution does not significantly improve the original performance, it does not cause harm and at least sometimes help.
3. Though this paper provides many interesting findings, the technical side is relatively weak. It would be great if the author can provide some potential solutions on improving test generation accuracy.
4. The iteration number of the experiments is limited, which might not fully support the authors' findings.

• Limited Evaluation Scope: The paper's evaluation is constrained to a relatively small set of models, potentially limiting the generalizability of the findings.
• Novelty Concerns: While the paper presents a novel application of self-debugging to LLMs, the core idea of leveraging runtime information for debugging is not entirely new and has been explored in previous work, as cited in [1].
• Inconsistent Evaluation Setup: The paper's experimental setup may introduce bias due to the different prompts used for post-execution and in-execution self-debugging. A more consistent approach would involve using the same prompts for both methods to ensure a fair comparison.

[1] Ni, Ansong, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. "NExT: Teaching Large Language Models to Reason about Code Execution." arXiv preprint arXiv:2404.14662 (2024).

• Limited number of study LLMs:
  • The paper aims to provide a detailed investigation to evaluate the slef-debugging capabilities of LLMs by using generated tests
  • However, the paper only evaluates two different LLMs: GPT-4o and LLaMA-3-70B-Instruct
  • How would the findings highlighted in the paper translate to other LLMs, for example LLMs that are not instruction-tuned or LLMs that are specifically fine-tuned for code
  • Given the landscape of current SOTA LLMs, evaluate only on a few LLMs is not enough to showcase the investigation findings
• Lack of more comprehensive study settings can be applied:
  • The authors evaluate some basic settings with respect to code generation
  • Given the setup of the paper to study existing settings without providing any new technique or improvement, I would expect a more systematic settings
  • For example, the authors only use greedy sampling, what if we study the effect of sampling (for example in the classic self-debug study paper: Olausson et al. 2024) this is a key finding:
  • For example, what if we vary the number self-generated tests we produce? what if we use the tests from a larger LLM to debug the smaller LLM?
  • I believe the paper lack more interesting findings that can be obtained with more detailed experiments.
• Evaluation on more realistic coding benchmarks
  • The authors evaluate on some basic coding benchmarks including HumanEval, MBPP, and LiveCodeBench.
  • However, these are self-contained benchmarks that are not in fact realistic
  • I am not sure if the findings in this paper would translate to more realistic repository level coding benchmarks
  • This limitation is not neccesarily the fault of the authors as it may be difficult to perform very extensive evaluation on those benchmarks, however I would still like to see some prelimeary experiments if possible and discussion
• Limited direction and discussion for future work:
  • While I really like the insight of additional bias (i.e., giving more detail may not be good) in the findings and discussions, I find the discussion section after the findings rather weak.
  • I believe the paper would benefit a lot from either a small technical contribution of testing some new techniques given the findings or offer more in terms of future discussion to improve future work along this direction

References: Olausson, Theo X., et al. "Is Self-Repair a Silver Bullet for Code Generation?." The Twelfth International Conference on Learning Representations. 2023.