REDO: Execution-Free Runtime Error Detection for Coding Agents

I thank the authors for the responses. However, the repsonses are overall unsatisfactory, since they largely fail to address the core concern or even raise further questions.

Lack of precision

The authors decide to not provide a score for the in-class precision of their method, and instead argue that no further differentiation is needed as i) precise errors types would be too noisy and ii) the detected errors still help agents at improving the results.

i) sounds like pure speculation to me, to consolidate this claim I would expect some (manual?) analysis on whether reported errors of LLMs are actually present and how often these are not triggered by unit tests. The final metric also only compares to triggered errors by unit tests - if the model reports an error that is not the cause of the failure, it is not at all clear whether the error was correctly predicted.

ii) The experiments in Appendix G do precisely not involve any information flow from the detected error to the agent, beyond the classification as safe/unsafe. The models developed by Bieber et. al. or static-analysis-only error detection provide the same necessary information for the provided experiment. So indeed this part somewhat justifies that "unsafe/safe" is a useful-enough classification, but it is unclear if the proposed method is superior in this setting (i.e. instead of F1 scores, for this setting precision or recall might be more important, additionally derived details could provide crucial insight to aid agents in repairing the previous patch).

No generalized / fair comparison

It is not clear at all from the provided experiments that the reasoning provided by the LLM is actually helpful to the agents in any way (refer to ii) of the previous answer). Moreover Table 3 only compares F1 scores - the additional reasoning is not relevant for this metric. That LLMs can make better use of the surrounding context to achiever overall higher F1 score is not based on these results.

Lack of precision in assessing method prediction in SWEDE

We acknowledge that the binary classification-like analysis is relatively limited. The decision to label each instance as 'safe' or 'unsafe,' rather than by specific runtime error types, stems from the fact that we can only observe runtime errors that fail the unit tests. However, other runtime errors induced by the patch might remain undetected if not triggered. Consequently, labeling instances by their error types would result in noisy labels, making it difficult to fairly evaluate the error detection algorithms.

To further explore the utility of the predicted errors, we refer to the patch ensemble results (link: https://ibb.co/McZh0Kr and also reported in Appendix D and discussed in Section 5.1) and the qualitative examples in Appendix G. The patch ensemble results demonstrate that leveraging REDO's detection outcomes allows us to ensemble patches from different coding agents and generally improve overall performance. Additionally, the qualitative examples in Appendix G illustrate how LLM reasoning about detected runtime errors can guide coding agents to refine patches and resolve runtime errors.

Lastly, we note that we predicted execution errors based solely on the outputs of existing coding agents, without actually running the agents, as most state-of-the-art (SOTA) coding agents are proprietary. By directly analyzing their results, we could evaluate REDO's performance across a wide spectrum of methods.

No attempt at generalization of the proposed method

First, we thank the reviewer for acknowledging the generality of our approach, which combines a precise but conservative method with an imprecise yet more sensitive learning-based model. We would like to point out that the method proposed by Bieber et al. (2022) is limited in the context of coding agents, as their model only takes generated code as input and outputs an error type. It lacks the ability to comprehend the runtime context or provide reasoning for its detection.

In contrast, LLMs can incorporate additional information, such as the problem statement and the original implementation, while also generating reasoning for their detection. As demonstrated in Appendix G, this reasoning can assist coding agents in refining their current patches and addressing runtime errors more effectively.

Presentation

We thank the reviewer for providing constructive suggestions on improving the presentation. We will incorporate these recommendations into our revision.

Questions

• Please refer to the response to W1.
• Please refer to the response to W2.
• We reported the corresponding error margins in Table 6 and will include them in Figure 1 in our revision.
• We apologize for the confusion. The STA dataset refers to the dataset used in Bieber et al. (2022), which is built upon CodeNet.
• In general, we found static analysis tools to be most effective in detecting errors where function or variable types can be inferred. This insight motivated us to use Pyright, which excels at type inference, as our static analysis tool. LLMs, on the other hand, are particularly helpful when the running context is unclear, making the types of related variables or functions uncertain. In such cases, LLMs can reason about potential running contexts and detect possible runtime errors. Furthermore, for languages with strong type safety guarantees, we expect static analysis tools to perform significantly better.
• For SWEDE, we consider the label noise discussed in response to W1. For metrics on CodeNet, we followed the approach in Bieber et al. (2022) to ensure a fair comparison.

I acknowledge that the authors want to contribute to a relevant field, concretely to more closely analyze how error detection in LLM Agents can be improved to correct patches swiftly. However, a meaningful contribution would require more thorough analysis than what is presented in the paper.

• The example from Appendix F is a single example that could well be cherry picked. It further does not feature an instance where the LLM correctly predicts an error that is not triggered by the unit tests, which is what I am concerned about. A manual investigation of several (at least 20), randomly drawn samples would be required to alleviate this concern.
• I am sorry about the confusion regarding App. G. The prompt from App. E.6 appears to be the prompt used for the single example presented in App. G and uses the additional reasoning by the LLM. However, this is again a single example and could be cherry-picked. The quantitative experiments meant to corroborate that REDO is useful for patch fixing (Ensemble Patch, App. D, Algorithm 2) do not appear to use this prompt or any reasoning generated by REDO.

Further I highly recommend the authors to present results as requested.

1. What is the performance of Bieber et. al. combined with PyRight on the STA dataset?
2. How likely is it that the LLM predicts an error that is present in the code but not triggered by unit tests (manual analysis of 20 samples)? or alternatively: What is the performance of the LLM in predicting precisely one of the error types triggered in the unit tests?
3. What is the performance of other approaches (Bieber et al, PyRight, Bieber + PyRight) in the "Ensemble Patching" setting?

The previous discussion did not address any of my raised concerns, either unintentionally or intentionally. I do recommend the authors to not answer unless they can provide concrete numbers, since the content of the paper is not sufficient to alleviate my concerns. These are mainly, that LLM + PyRight is not superior to e.g. Bieber et. al. + PyRight, and that REDO is not quantitatively more helpful for guiding Agents in patch correction than only Static Analysis or Bieber et. al..

1 Overclaim

We thank the reviewer for highlighting the broad applicability of our method. While our primary focus is on coding agents, we agree that REDO has potential applications in other scenarios, such as generating static code reviews.

2 Limited Novelty

The core challenge of REDO lies in balancing the high reliability of static analysis tools with the broader detection scope of LLMs. To address this, we deliberately designed the differential analysis algorithm for static analysis tools to minimize false alarms and crafted prompts for the LLM to detect potential runtime errors more aggressively, thereby maximizing recall. While no explicit information is exchanged between the static analysis tool and the LLM, these design choices collectively optimize the trade-off between reliability and detection scope.

3 Inconclusive Performance Improvement

We acknowledge that false alarms may still occur with REDO, particularly in cases where the LLM overanalyzes and identifies runtime errors that are unlikely to happen. However, we emphasize that the experimental results in Tables 2 and 3 demonstrate that REDO effectively leverages both static analysis tools and LLMs to achieve improved F1 and weighted F1 scores, indicating enhanced error detection performance. Additionally, the patch ensemble experiments show that by using REDO to ensemble patches from different coding agents, we can generally improve the overall performance of the coding agents.

Questions

• Please refer to the response under "Overclaim."
• Please refer to the response under "Limited Novelty."
• We would like to emphasize that when designing the prompt for the LLM, we specifically instruct it to identify all potential runtime errors, thereby maximizing its recall. This approach is intended to address the limitations of static analysis tools, which may fail to detect certain runtime errors.

1 The integration of static analysis tools with LLM

We note that the core challenge of REDO lies in balancing the high reliability of static analysis tools with the broader detection scope of LLMs. To address this, we deliberately designed the differential analysis algorithm for the static analysis tool to minimize false alarms and crafted prompts for the LLM to aggressively detect potential runtime errors, thereby maximizing recall. While no direct information is passed from the static analysis tool to the LLM, these designs collectively optimize the trade-off between reliability and detection scope.

2 Static analysis tool limitation

First, we agree that more sophisticated tools could further enhance performance. However, there are two key motivations for using relatively simple static analysis tools like Pyright.

• Lack of unit tests: Unit tests for the generated code may not always be available. For example, in SWE-Bench, unit tests are inaccessible to coding agents.
• Efficiency and cost-effectiveness: Since REDO is designed as a plug-and-play tool for coding agents, minimizing its runtime and cost is a priority. To this end, we selected Pyright, which is well-known for its fast runtime, and limited the LLM to a single prompt to reduce API costs.

3 Technical contributions

We acknowledge that our algorithm is straightforward. However, we emphasize that augmenting program analysis tools to enhance both their reliability and capabilities is a novel approach that could inspire solutions to other challenges, such as security and privacy. Moreover, while we agree that more sophisticated integrations of LLMs and static analysis tools could further improve performance, our current approach already demonstrates significant improvements to static analysis tools, as evidenced in Table 2.

Question:

1. What are the benefits of being execution-free? While we agree that execution feedback is often informative, executing generated code introduces several challenges:

• Lack of unit tests: Unit tests for the generated code may not always exist. For example, in SWE-Bench, unit tests are inaccessible to coding agents. Even if coding agents generate unit tests, as shown in [1], these tests may not accurately represent the practical runtime context of the generated code.
• Legal, privacy, and cost concerns: As discussed in [2], executing code can raise legal, privacy, and security concerns, particularly in real-world applications like cloud services. Additionally, execution-based error detection is costly, as only one error can be identified per run; the program halts at the first encountered error. In contrast, REDO can detect multiple potential runtime errors in a single analysis.
• Complexity of environment setup: Setting up the execution environment can be challenging, often requiring dependency installations and significant computational resources. Moreover, an execution-free algorithm is easier to integrate into state-of-the-art coding assistants, such as Copilot, Amazon Q, or Cursor, which lack the capability to execute the underlying codebase. By being execution-free, REDO circumvents these challenges while remaining efficient and scalable.

[1] N Mundler, et al. Code Agents are State of the Art Software Testers, 2024. [2] M Lacoste, V Lefebvre. Trusted execution environment for telecoms: Strengths, weakness, opportunities, and threats, IEEE Security & Privacy, 2023.

2, Can LLMs work more tightly with static analysis tools for this task, for instance, LLM-based static analysis or static analysis for LLMs?

We would like to highlight that in our current algorithm, the differential analysis of static tools and the prompt design for the LLM are carefully crafted to maximize the strengths of both tools and achieve an optimal balance between the high reliability of static analysis tools and the broader detection capabilities of LLMs. Since REDO is designed as a tool for coding agents, the algorithm is optimized to detect runtime errors efficiently, avoiding long delays or high costs. However, we acknowledge that if more time and resources were available, a deeper integration of these tools could further enhance performance. For instance, one potential approach could involve using the LLM as an agent capable of multi-turn reasoning to analyze and interpret the outputs from static tools.

3, What are the key challenges? The core challenges are as follows:

• Integrating static analysis tools and LLMs effectively: Achieving a balance that combines the high reliability of static analysis tools with the broader detection scope of LLMs.
• Ensuring efficient detection performance: Maintaining strong detection capabilities while minimizing runtime delays and reducing API costs.
• Developing a fair evaluation framework: Designing a fair and robust method to assess the performance of error detection algorithms on state-of-the-art (SOTA) coding agents.

1, How does REDO help coding agents?

We would like to highlight the patch ensemble results (https://ibb.co/McZh0Kr and reported in Appendix D and referenced in Section 5.1, as well as summarized below) and the qualitative examples in Appendix G. The patch ensemble results demonstrate that by leveraging the detection outcomes from REDO, we can effectively combine patches generated by different coding agents to enhance overall performance. As shown in the qualitative examples in Appendix G, the reasoning provided by the LLM for detected runtime errors can guide the coding agent in refining patches and resolving runtime errors. Additionally, we only predicted execution errors on the outputs of existing coding agents, without directly running these agents, as most state-of-the-art coding agents are proprietary. By directly analyzing their outputs, we were able to evaluate the effectiveness of REDO across a diverse range of methods.

2, Motivation for execution-free runtime error detection

First, we agree that execution feedback is often informative; however, executing generated code comes with several challenges:

• Lack of unit tests: Unit tests for the generated code may not always be available. For instance, in SWE-Bench, unit tests are inaccessible to coding agents. Even if coding agents can generate unit tests, as shown in [1], these tests might fail to reflect the practical execution context of the generated code.
• Legal, privacy, and cost concerns: As discussed in [2], executing code can raise legal, privacy, or security issues in real-world scenarios, such as in cloud services. Additionally, execution-based error detection can be costly since only one error is identified per run, as the program halts upon encountering the first error. In contrast, REDO can identify multiple potential runtime errors in a single pass.
• Environment setup challenges: Setting up the execution environment can be complex, often requiring dependency installations and consuming significant computational resources. Moreover, an execution-free algorithm like REDO can be more easily integrated into state-of-the-art coding assistants such as Copilot, Amazon Q, or Cursor, which do not execute the underlying codebase."

[1] N Mundler, et al. Code Agents are State of the Art Software Testers, 2024. [2] M Lacoste, V Lefebvre. Trusted execution environment for telecoms: Strengths, weakness, opportunities, and threats, IEEE Security & Privacy, 2023.

3 SWEDE an afterthought?

We note that REDO can be seamlessly used as a plug-and-play tool for coding agents. As mentioned, the primary motivation for proposing SWEDE is that most state-of-the-art (SOTA) methods are proprietary. Consequently, to evaluate error detection algorithms on these SOTA agents, we must rely on the output from these agents rather than directly running them. We acknowledge that the binary classification-like analysis is relatively limited. The decision to label each instance as “safe” or “unsafe,” rather than categorizing specific runtime error types, stems from the observation that we can only detect runtime errors that fail the unit tests. However, other runtime errors introduced by the patch might remain undetected because they were not triggered during execution. As a result, labeling instances by error types could produce noisy labels, leading to an unfair evaluation of error detection algorithms. Finally, we emphasize that in addition to the binary classification-like analysis, we also propose the patch ensemble metric, which offers a practical means to assess how error detection algorithms enhance coding agent performance. We believe that combining the binary classification-like metrics with the patch ensemble evaluation provides a more comprehensive and fair assessment of error detection algorithms.

To Question:

While we emphasize that REDO is primarily designed for coding agents, as noted by reviewer mtjz, it can also be applied to other tasks, such as code generation. Another potential application is leveraging LLM feedback to automatically generate static code reviews.