Large Language Model Critics for Execution-Free Evaluation of Code Changes

There are several major weaknesses in this paper:

1. The requirement for gold patches in the proposed method limits its applicability in practice. Although it also discusses a reference-free situation, the accuracies below 0.5 are far from practical use.
2. The proposed method is only compared to a simple Edit Distance baseline in the experiments. If given the gold patches, various metrics like CodeBLEU and ROUGE could also provide a way to evaluate the similarity between the deleted and inserted code snippets in the generated output and gold patches. However, such baselines are missing from the experiments. Including these comparisons in the experiments and discussing the results would provide a more comprehensive evaluation of the proposed method's advantages.
3. To evaluate the proposed method across different LLMs, this paper uses three variants of Claude models. However, the choice of using models from the same series may limit the generalizability of the proposed method. It would be beneficial to evaluate the method on more diverse LLMs, such as GPT-4 or other open-source models, to demonstrate its effectiveness.
4. In what scenarios would the proposed method fail? There appears to be a lack of case studies or discussions on the success and failure cases of the proposed method. It would be helpful to provide examples that illustrate the benefits and limitations of the proposed method compared to other baselines. Additionally, I look forward to seeing example inputs and outputs of the LLM evaluator for code patches to better understand the proposed method.

And some minor issues:

1. The organization of Section 2 is somewhat confusing and lacks a figure or algorithm to illustrate the proposed method, making it difficult for readers to understand. There is only one subsubsection under each subsection, which obscures the structure of the section. I recommend that the authors consider adding a figure or algorithm table to help readers better grasp the proposed method and reorganize the section for improved clarity.
2. The authors claim to open-source the code in Lines 25 and 111 as an important contribution. However, the code is not provided in the supplementary material or the paper with anonymous links, making it difficult for me to evaluate the artifacts.
3. As claimed in Line 192, the proposed method prompts LLMs to generate evaluations of code patches. However, the details of how the LLMs are prompted are not clear. It would be helpful to provide the exact prompt in the Appendix.
4. The left figure in Figure 4 is too small, and there are timestamps in the caption that seem unnecessary. It is also recommended to use a larger font size for the text in Figures 3 and 4.

1. Technical Description: The technical pipeline of the test-centric framework is intricate, and certain steps could be explained with more clarity and visualization. For example, the mechanisms by which LLM critics predict test pass/fail outcomes based on context enhancement are briefly mentioned but should benefit from further elaboration and details, especially regarding how these critics handle complex test cases, also the correctness of CoT of LLM when evaluating the code patch.

2. Limited Discussion on Scalability Constraints: While the authors demonstrate strong results on SWE-bench, the generalizability of the approach to a broader set of software repositories and programming languages is not deeply explored. This leaves open questions regarding the framework’s ability to adapt to repositories that may require unique dependencies or multilingual support. Discussions on granularity (explanatory power), code size (upper limit of capability), and multi-domain (characterizing domain performance of LLM) of LLM in reviewing code patches are crucial but missing in this paper.

3. Reliance on Gold Patches: The framework's dependence on a reference (gold patch) for optimal accuracy raises potential issues in scenarios where a ground-truth patch may not exist. Although the authors attempt to address this with reference-free baselines, the performance drop observed here indicates that further research may be needed to refine reference-free evaluation methods.

• Accuracy of LLM critics. The paper evaluates gold-patch-guided LLM critics aggregated over test cases; however, LLM-based judges and verifiers are usually quite inaccurate and miscalibrated, even for simple programming problems like those in HumanEval or LeetCode. For example, [1] reported about 50% accuracy for open-source models serving as LLM critics, while GPT-4 achieves only 70-80% accuracy. This raises doubts about the feasibility of execution-free approaches to more complex software code changes studied in this paper.

• Baseline experiment on standard programming evaluations. Building on the previous point, standard programming benchmarks like HumanEval or programming contest problems can be formulated as 'code change' problems—for example, given a function with a docstring, insert the necessary code. Understanding the effectiveness of the approach on such "simpler" settings might provide a more grounded understanding of the strength of the approach.

• Overfitting to SWEBench. The authors use SWE-Bench-Lite as the sole evaluation benchmark; however, as they acknowledge, SWE-Bench-Lite is imbalanced, with the majority of tests passing. This raises concerns about the generalization of the approach. Specifically, in the micro-evaluations, authors observe a 98% recall -- potentially due to bias in LLMs to respond correct [1]. This aligns with the evaluation suite with high positives potentially inflating the results. It is unclear if this approach will generalize to more challenging benchmarks where a smaller fraction of tests pass and should be evaluated (say on SWEBench full suite).

• Calibration on the test benchmark. Authors recalibrated model confidences -- capping a 65% confidence on model responding YES. However, this number uses private knowledge about the correctness of the patches during micro-evaluation recalibration on the SWEBench test set. This raises concerns about the potential for data leakage and the validity of the evaluation results.

• Pass to Fail tests. In many cases, a programming agent solution can fail if it introduces a bug in an already passing testcase. It seems this is not handled since the approach only handled "newly introduced" tests in the PR

• Access to clean pull requests (PRs) is assumed. SWE-Bench, problem instances (PRs) are cleaned via execution to map the tests and code changes using a Fail-to-Pass strategy. From my understanding, the authors assume access to this information for setting up LLM critics. While this approach works for benchmarks, real-world PRs can be messy, containing unrelated changes to tests and code. This complexity may require at least one execution round to collect the necessary information, limiting the real-world applicability of the work focusing on execution-free nature.

• Undefined or inconsistent Terminology.

  • 'build status' is never defined and it is unclear if authors mean simply "all tests passing" or something beyond that
  • In Table 1, settings in the candidate patch column are not properly explained -- +- function-level supposed to mean context-enhanced patches is not clear from the table.
  • In Table 2, change-aware vs test-aware is not immediately clear from the description and could be explained with examples.
  • 'code changes,' 'patches,' and 'edits' are used interchangeably without clarification and should be normalized to one aspect

References.

1. The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?

• I believe an execution-free metric is most useful in situations in which you do not have a test suite at all or when the existing test suite has low coverage. However, this work requires having a high-quality test suite. The only dataset that the authors evaluate on is SWE-Bench, which comes with Docker images corresponding to the test environments, and so it seems like it is rather straightforward to just execute the tests in the test suite. Therefore, it seems that the impact of this work will be fairly limited.
• Additionally, from Table 2, the best accuracy that is attained is 71.4% (which is incorrectly claimed as 82.1% in the abstract). From the paper alone, I am not convinced that we can simply replace the execution-based metric with this. Perhaps if the authors had demonstrated that the rankings of the top ~10 models on the SWE-Bench leaderboard remained identical when using the LLM-based metric, it would have been more convincing. Currently, the best approach nearly matches the random baseline in terms of precision (i.e., the LLM-based approach will often say the patch passes tests when it actually does not). And if it is possible to obtain the execution-based score, then this LLM-based metric will likely not be needed at all since it serves to approximate the execution-based metric.
• A lot of details seem to be missing, misleading, inconsistent, and sometimes incorrect. This makes it difficult to follow and at times even assess the paper:

1. In the abstract, the authors claim an accuracy of 91.6% at the micro-level and 82.1% at the macro-level. However, these are F1 scores, and not accuracy, based on Tables 1 and 2.
2. For the majority of the first half of the paper, it seems that the authors use the gold code patch which resolves the issue (L017-018, L080-081, L137-141). However, it becomes clear later that the authors only consider the gold test patch and the gold code patch is not used in their approach and only in their baselines.
3. The prompts that the authors use are not given. Additionally, not a single example is provided.
4. The notion of "function-level" is in multiple tables but this is never explained. This also seems to be the best performing, so it is not clear what the best-performing method is actually. Namely, Table 1 has only two rows for "Isolated, Test-Aware", one corresponding to "post-commit functions" and one augmenting that method with "function-level." However, the two methods that are introduced in the main text are "context enhancement" and "source code" (which is also referred to as post-commit functions). In L259, it says "We can see that the LLM critic with context-enhanced patches performs the best, outperforming other baselines by 7.4% and 12.7%". But "context-enhanced" is not in Table 1 and there is only 1 baseline.
5. In the abstract, the authors motivate this work as an "intermediate/step-level" evaluation methodology. This suggests that the intermediate steps in the LLM-based agentic framework could be evaluated. However, this does not seem to be demonstrated in the paper. It is not obvious, but if Figure 4 (right) was intended to demonstrate this, it is not clear. Additionally, it is not clear why this was presented as a plot rather than an aggregate spearman's ranking coefficient. It looks like there is a decent chunk of the density in the negative range here too.