We sincerely thank the reviewers for their constructive feedback! We are pleased that most reviewers have a clear positive view of our submission, praising it as a "​​Novel Approach" (Reviewer RQoh), of "great importance to industry community" (Reviewer jQdC), is "​​lightweight and easy to apply" (Reviewer 4J1w), and "presents a solid execution of a straightforward idea" (Reviewer pDaa), while also highlighting the "thorough empirical study across multiple established benchmarks" (Reviewer RQoh).

We below address the individual concerns of Reviewer RQoh. All minor points will be directly incorporated into the revised manuscript.

Q1. [Dependence on Capable Critique LLMs: The method's performance is highly tied to the capabilities of the critique LLM as shown in Table 3. While predicting code performance without real execution is a promising direction, it remains unclear how well these predictions align with ground-truth performance metrics.]

First, while the capabilities of the critique LLM do influence performance, we emphasize that all standard evaluation results in our paper use the same model for both code generation and critique. This design ensures that our improvements do not rely on access to a stronger or external model, and thus impose no additional requirements.

We provide supplementary empirical evidence showing that the LLM-based reward used during decoding aligns well with real execution efficiency. Specifically, we report the average Pearson correlation coefficients between the LLM reward scores and measured execution times across samples from multiple datasets for the studied models, achieving correlations of -0.78, -0.69, and -0.74 on Effibench, Mercury, and COFFE datasets, respectively. These results indicate a strong and consistent alignment between the LLM-generated reward signal and ground truth performance, supporting the reliability of our critique mechanism.

Q2. [LLM Critique Generalization: Current evaluation is performed on self-contained and established benchmarks. It is highly probable that this data was involved in the training corpora of most LLMs. Therefore, it raises a concerns about whether the LLM's ability to critique code efficiency stems from truly code comprehension or merely from memorization of the benchmark solutions.]

We acknowledge the concern regarding potential data contamination in benchmark evaluations, which is a challenge that is nearly impossible to eliminate entirely, given that LLMs are typically trained on large-scale web-scraped corpora. However, our results suggest that even when LLMs possess the necessary knowledge to generate correct and efficient code, they often fail to do so. This is likely due to limitations in autoregressive decoding and the reliance on perplexity as the primary generation objective. In this context, our approach of steering the LLM at inference time using feedback signals (derived from the same model) should offer meaningful practical value by promoting generation that better aligns with both correctness and efficiency.

Thank you for the clarifications. Could you elaborate on how you calculate the Pearson correlation coefficients? I'm also curious if you used an isolated environment for code execution and whether the results would be consistent if run on different machines.

We sincerely thank the reviewers for their constructive feedback! We are pleased that most reviewers have a clear positive view of our submission, praising it as a "​​Novel Approach" (Reviewer RQoh), of "great importance to industry community" (Reviewer jQdC), is "​​lightweight and easy to apply" (Reviewer 4J1w), and "presents a solid execution of a straightforward idea" (Reviewer pDaa), while also highlighting the "thorough empirical study across multiple established benchmarks" (Reviewer RQoh).

We below address the individual concerns of Reviewer pDaa. All minor points will be directly incorporated into the revised manuscript.

Q1. [The related work section is adequate, but I would have expected more discussion of refactoring systems. What was the advantage of baking code-efficiency into the reward function and applying it at the statement level?]

We thank the reviewer for highlighting this point. We agree that automatic refactoring systems are worth discussing. Traditional refactoring tools (such as those in IDEs or static analyzers) aim to improve code quality or structure while preserving behavior. While recent work has explored learning-based approaches, most existing methods remain largely template-driven and rely on executability and test cases to validate behavior-preserving transformations. As a result, they often focus on stylistic or structural improvements and are less effective at producing efficiency-oriented rewrites, particularly without access to execution or profiling.

In contrast, our approach incorporates efficiency considerations directly into the sampling objective through an inference-time reward function. By applying this at the statement level, it enables fine-grained control over decoding, helping the model explore its learned code manifold and generate efficient solutions as part of the construction process, rather than just refactoring them post hoc.

We also believe that insights from refactoring systems could inform future improvements to our refinement stage, and we will revise the related work section to reflect this connection more clearly.

Q2. [Breakdown of the impact of varying the alpha and beta parameters in the general formulation. Is having the secondary LLM consider AST factors redundant with the empirical AST measure?]

We present detailed results on hyperparameter tuning using CodeLlama-7B-Instruct on the HumanEval+ dataset. While it is challenging to draw definitive trends given the interplay of three correlated reward components, we generally observe that incorporating all terms leads to the best overall performance. Among them, slightly emphasizing the LLM-based reward ($\beta$) over the perplexity-based term ($\gamma$), e.g., using a weight ratio of 3:2 to 2:1 tends to yield better results. Additionally, modestly increasing the weight of the AST term ($\alpha$) also leads to slight improvements in final performance.

Regarding the AST term, our design leverages AST signals in two complementary ways. First, they provide knowledge guidance to the LLM reward model via prompting, as incorporating AST information helps the LLM reason about structural aspects of efficiency—something we find empirically beneficial. Second, because AST alignment is closely tied to structural correctness, where deviations may compromise functionality, it serves a role akin to a hard constraint. This contrasts with the LLM-based critique, which reflects a soft, semantic preference. This separation motivates us to modulate their respective influences via the $\alpha$ and $\beta$ parameters, ensuring that structural correctness is explicitly prioritized alongside more flexible semantic signals.

Q3. [Understanding why the method is more effective than alternatives. Why is correctness improved? Why the method find performance gains that a system like Effi-Learner misses?]

We believe that incorporating AST patterns into the decoding-time reward plays an important role in guiding the model to be aware of both correctness and efficiency during generation. In contrast, previous refinement-based methods typically rely on passively re-feeding profiling signals and simply filtering out incorrect code that fails test cases for further refinement. However, this process lacks active guidance and can lead to instability—new errors may emerge when attempting to fix earlier ones, as seen in approaches like Self-Debug and Self-Refinement. Without an explicit feedback signal guiding the generation holistically, these methods risk oscillating between incorrect variants rather than converging on efficient and correct solutions.

Q4. [Is shorter code generally more likely to be correct? What is the advantage of performing multi-dimensional analysis after every statement?]

Regarding code length: We compute the Pearson correlation between average code length (in lines) and correctness across our experiments and find a coefficient of 0.06, indicating a negligible relationship between the two. The observed code lengths are likely an artifact of our example selection for visualization purposes, as we deliberately chose shorter examples to fit within space constraints.

Regarding the advantage of performing analysis after every statement: We present additional results comparing two variants: one that performs selection only at the end using standard perplexity-driven token generation during intermediate decoding (End-Selection), and our default setting that applies selection after every statement (Ours). The results show that our method consistently outperforms End-Selection, suggesting that intermediate feedback more effectively guides generation and promotes greater diversity in the outputs. As briefly mentioned above, we hypothesize that applying the reward at the statement level provides fine-grained control during decoding, helping the model better explore its learned code manifold and construct efficient solutions incrementally, rather than relying solely on post hoc refinement.

Q5. [How well do the individual estimates made on predicted performance compare to the actual results?]

We provide supplementary empirical evidence showing that the LLM-based reward used during decoding aligns well with real execution efficiency. Specifically, we report the average Pearson correlation coefficients between the LLM reward scores and measured execution times across samples from multiple datasets for the studied models, achieving correlations of -0.78, -0.69, and -0.74 on Effibench, Mercury, and COFFE datasets, respectively. These results indeed indicate a strong and consistent alignment between the LLM-generated reward signal and ground truth performance.

Q6. [There is essentially no commentary on how the method might generalize to other programming languages or larger code problems.]

We appreciate the reviewer’s comment and will include a discussion on generalization in the revised version. To our knowledge, there is no fundamental limitation that prevents our method from extending to other programming languages. High-level AST patterns are typically identical or highly similar across languages, and the LLM-based reward component is designed to be language-agnostic and should remain effective, assuming the LLM is appropriately trained. While the AST extraction module would need to be adapted for each target language, this is a relatively straightforward engineering task. Additionally, the processing time of our method scales linearly with code length and, in principle, should not form a bottleneck for larger code problems.

We sincerely thank the reviewers for their constructive feedback! We are pleased that most reviewers have a clear positive view of our submission, praising it as a "​​Novel Approach" (Reviewer RQoh), of "great importance to industry community" (Reviewer jQdC), is "​​lightweight and easy to apply" (Reviewer 4J1w), and "presents a solid execution of a straightforward idea" (Reviewer pDaa), while also highlighting the "thorough empirical study across multiple established benchmarks" (Reviewer RQoh).

We below address the individual concerns of Reviewer 4J1w. All minor points will be directly incorporated into the revised manuscript.

Q1. [While using LLMs as critics or evaluators is an emerging trend, applying them to assess fine-grained, precision-sensitive attributes (e.g., efficiency, memory use) raises concerns. The paper assumes LLMs can act as reliable efficiency evaluators, but this assumption lacks empirical validation or calibration against ground truth measurements. Unlike prior works that rigorously validate LLM-as-judger setups, this paper does not justify the trustworthiness of its critique model with sufficient rigor.]

First, we would like to clarify that our approach does not rely on LLMs as the final evaluators for measuring code efficiency. All evaluation results reported in the paper are based on real code execution, ensuring objectivity, verifiability, and immunity to potential biases inherent in LLM-as-judge evaluations. Therefore, our findings are grounded in actual runtime behavior, not subjective estimations.

In addition, we provide supplementary empirical evidence showing that the LLM-based reward used during decoding aligns well with real execution efficiency. Specifically, we report the average Pearson correlation coefficients between the LLM reward scores and measured execution times across samples from multiple datasets for the studied models, achieving correlations of -0.78, -0.69, and -0.74 on Effibench, Mercury, and COFFE datasets, respectively. These results indeed indicate a strong and consistent alignment between the LLM-generated reward signal and ground truth performance.

Q2. [The proposed method hinges on the assumption that LLMs can consistently and accurately judge code efficiency without actual execution. However, inspection of examples and appendix results reveals that the judgments may be inconsistent or even contradictory across similar samples. For a critique system to guide generation, its correctness must be established—something not thoroughly validated here.]

We appreciate the reviewer’s concern and would like to clarify that the examples provided in Figures 2–5 actually demonstrate a consistent alignment between the LLM reward signal and real code execution performance. Specifically, higher LLM rewards (0.89, 0.81, 0.78, 0.43) closely correspond to lower execution times (1.64E-4, 6.45E-4, 5.97E-4, 1.52E-3), indicating the LLM’s ability to reasonably infer code efficiency. The slight variance between some scores (e.g., 0.81 vs. 0.78) is expected, as the LLM reward captures a combination of both time and space complexity, which may not align perfectly with a single metric like execution time.

To our knowledge, Figures 2–5 are the only examples in the paper that include both LLM rewards and real execution metrics, and we are not aware of any other results that could suggest inconsistency or contradiction.

Furthermore, as elaborated in our response to Q1, we provide supplementary empirical validation showing that the LLM reward signal indeed strongly correlates with ground-truth execution time across multiple datasets.

Q3. [The decoding configuration described in lines 212–214 (“best-of-n=50 with top-p=0.95 sampling”) is claimed to follow [33], but [33] in fact uses different sampling parameters. Moreover, the search space size (n=50) is unusually large and can significantly impact performance, especially under resource constraints. This discrepancy is not analyzed or ablated, making it difficult to disentangle gains from prompt-based critique versus aggressive sampling.]
In [33], the authors indeed claim to use top-p = 0.95 in Section 3 (page 4). In our experiments, we set best-of-n = 50 to ensure a fairer comparison and to isolate the impact of our proposed method from potential gains that could result simply from using a large n.

Below, we present results on Qwen2.5-Coder-33B using our approach under varying values of n. The results show that increasing n leads to improved generated code efficiency and accuracy, albeit with higher computational cost. Overall, we choose n = 50 as our default setting, as it provides a strong balance between generation performance and computational efficiency.

Q4. [The authors propose that "time complexity, space complexity, runtime performance, memory usage efficiency, and syntax correctness" can be assessed by simple speed test scripts. I am not convinced of the rationale behind introducing an LLM to perform this evaluation.]

First, as noted above, we do not use the LLM for final evaluation. All final assessments are performed via actual code execution, which is straightforward and typically requires only one or a few runs per sample. In contrast, the LLM is used during the refinement stage, which often involves repeated sampling and evaluation to guide the model toward generating correct and efficient code. Replacing such repeated ground-truth executions in this stage with a lightweight LLM-based proxy results in substantial computational savings.

Additionally, the human effort and infrastructure overhead involved in setting up execution environments are far from negligible. In this regard, our method presents a practical and scalable alternative “for free”: it enables iterative improvement toward efficient and correct code using only the same LLM, without the need for repeatedly executing the code. We believe this makes our approach particularly valuable in resource-constrained or rapid-development settings.

Thank you for the detailed rebuttal.

While I appreciate the clarification that all final evaluations are based on real execution, my concern still lies with the motivation and reliability of using LLMs as efficiency critiques during decoding (I know the grounded in actual runtime behavior). Recent works [1, 2] (not only two works) have challenged LLM-as-judge approaches for fine-grained tasks which covers the metric in this paper, and which are often hardware-sensitive and not easily inferred from code area alone. Correlation with execution time is helpful but not sufficient to ensure trustworthiness without stronger calibration or adversarial validation. While the reported correlation scores are moderately strong, Pearson correlation alone is a necessary but insufficient condition to claim reliable alignment. It does not guarantee local consistency, accurate ranking, or robustness under distribution shifts. For a reward signal to guide decoding effectively, stronger calibration and per-sample analysis would be necessary. I believe the core assumption that LLMs can reliably guide efficiency without execution remains under-justified and weakens the motivation of the proposed approach.

As the these concerns, I maintain the original score.

[1] Preference Leakage: A Contamination Problem in LLM-as-a-judge

[2] Adversarial ML Problems Are Getting Harder to Solve and to Evaluate

Thank you for the detailed rebuttal. Most of my concerns have been addressed.

I have raised my score to weak acc. Wish the authors good luck!

We sincerely thank the reviewers for their constructive feedback! We are pleased that most reviewers have a clear positive view of our submission, praising it as a "​​Novel Approach" (Reviewer RQoh), of "great importance to industry community" (Reviewer jQdC), is "​​lightweight and easy to apply" (Reviewer 4J1w), and "presents a solid execution of a straightforward idea" (Reviewer pDaa), while also highlighting the "thorough empirical study across multiple established benchmarks" (Reviewer RQoh).

We below address the individual concerns of Reviewer jQdC. All minor points will be directly incorporated into the revised manuscript.

Q1. [The novelty of the paper is limited. It introduces a set of reward functions as the critique to speed up the code efficiency. However, as it is only conducted on the decoding stage, it cannot boost the performance of the original coding model. If the coding model itself cannot generate both efficient and effective codes, the proposed algorithm will fail as well.]

To our knowledge, we are the first to move away from the dominant paradigm of relying on labor- and time-intensive real execution for refining code generation, and instead propose an alternative that uses a set of lightweight, efficiency-oriented reward functions directly during decoding.

While it is difficult, if not impossible, to make definitive claims about the inherent capabilities of a code generation model, what we do observe is that even when a model has the capacity to produce correct and efficient code, it often fails to do so (likely due to the nature of autoregressive generation being guided by token-level perplexity rather than functional objectives). Our method effectively steers the generation process toward more efficient and reliable outputs without modifying the base model or relying on ground-truth execution. We believe this shift in methodology offers substantial practical value, particularly in scenarios where real-time execution feedback is costly or infeasible.

Q2. [The proposed algorithm will obviously increase the decoding time of the coding model. How about a comparison with the COT/RL model like Deepseek R1 for both inference computational cost as well as the code efficiency.]

While DeepSeek-R1 (with a model size exceeding 600GB) is beyond the hardware constraints of typical academic research environments, we conduct experiments using the distilled variant (the DeepSeek-R1-Distill-Qwen-32B) on the EffiBench dataset. As shown in the results below, although the COT/RL model improves correctness over the base Qwen2.5-Coder-32B model (~59% vs. ~53%, see Table 1 in the main paper), our method achieves further gains in both correctness and efficiency. Also, the computational cost comparison is in line with the trends and analysis reported in Table 2 of the main paper.

Q3. [How about the setting of the hyperparameters (\alpha, \beta, \gamma) in General Formulation reward signals?]

The hyperparameter values used in our main experiments are provided in Appendix C. Below, we present detailed results on hyperparameter tuning using CodeLlama-7B-Instruct on the HumanEval+ dataset. While it is challenging to draw definitive trends given the interplay of three correlated reward components, we generally observe that incorporating all terms leads to the best overall performance. Among them, slightly emphasizing the LLM-based reward ($\beta$) over the perplexity-based term ($\gamma$), e.g., using a weight ratio of 3:2 to 2:1 tends to yield better results. Additionally, modestly increasing the weight of the AST term ($\alpha$) also leads to slight improvements in final performance. Notably, a broad range of hyperparameter configurations already outperform all baselines, indicating the robustness of our approach. For the final evaluations across different datasets and models, we fix the combination $\alpha = 1.3$, $\beta = 1$, and $\gamma = 0.4$, which achieved the best results in this setting.