Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:

W1 & Q1 & W5: Baseline selection can be largely improved.

Thanks for your suggestion. We provide the evaluation results of SOAP and suggested baselines in Author Rebuttal Table 1.

• For the first suggested method, we named it Directly Efficiency, where LLMs are required to generate efficient code for the task description.

• For the second suggested method, we name it Self-Refine Efficiency, where we ask LLMs to generate code and then directly optimize it. Then, since PIE fine-tunes LLMs to improve the efficiency of LLM-generated code, which is orthogonal with SOAP, we conduct experiments for PIE and PIE + SOAP for five different prompts to demonstrate that with SOAP, PIE fine-tuned LLMs generated code could be further improved.

As shown in Author Rebuttal Table 1, we can observe that SOAP achieves SOTA results compared with Directly Efficiency and Self-Refine Efficiency. For example, Directly Efficiency and Self-Refine Efficiency decreases NTMU from 19.65 to 2.88 and 2.65, while SOAP decreases from 19.65 to 1.64. Besides, we can observe that with SOAP, the efficiency of PIE-generated code would further improve, e.g., decreasing the average execution time from 0.74s to 0.41s for PIE+Dynamic Retrieval,k=4.

W2: Compare the models under a set of mutually passing coding tasks.

Thanks for your suggestion. The evaluation results are shown in Author Rebuttal Table 2, where we conduct experiments on seven closed-source LLMs with 210 tasks, which have higher pass@1 compared with open-source LLMs, and then can collect massive tasks addressed by all closed-source LLMs. As shown in Table 8, we can observe that SOAP with GPT-3.5-turbo-0301 achieves SOTA efficiency compared with other LLMs. For example, the average execution time decreases from 0.37s to 0.28s.

W3: Follow PIE to use architecture simulators or other better-defined metrics.

We provide the evaluation results of SOAP and the initial code in Author Rebuttal Table 3. We can observe that in the PIE-provided default simulator, SOAP can also improve the efficiency of LLM-generated initial code. For example, the average execution time of OpenCodeInterpreter-1.3B decreases from 0.80s to 0.52s.

W4: Unclear if Self-Optimization can work for performance-demanding languages such as C/C++. To address the reviewer's concern, we conduct experiments for SOAP in the HumanEval-ET (C++) dataset. Our evaluation results are shown below, where we can observe that SOAP can also improve the efficiency of LLM-generated C++ programs. For example, the average execution time of CodeLlama-70B-Instruct-hf decreases from 667.9ms to 459.5ms.

Q2: Are profilers necessary? What if we just replace profilers with self-reasoning or self-reflection?

Profilers are necessary for the code efficiency optimization process. Specifically, to demonstrate the importance of profilers, we conduct experiments for straightforward code optimization (Unsupervised Self-Refine and Result-Aware Self-Refine), reasoning-based code optimization (self-reasoning and self-reflection), single profiler-based code optimization, and provide both memory profiler and execution time profiler (SOAP).

The evaluation results are shown below. Our experiments demonstrate that by providing both memory profiler and execution time profiler (SOAP), the code efficiency achieves SOTA results compared with other methods. For example, SOAP decreases the average execution time from 0.36s to 0.28s.

Q3: How's the cost (token consumption and running time) of SOAP and baselines?

To address the reviewer’s concern about the cost of SOAP and baselines, we provide the evaluation results in Author Rebuttal Table 4, where we can observe that the initially generated code requires 1.1M token usage and 76.81s for all tasks. While SOAP requires 5.4M token usage and 566.67s for all tasks. Although SOAP requires more tokens compared with the initial version, the execution time and memory usage (See paper table 1) are largely improved. We believe that this exact token usage is affordable.

Q4: When running and measuring the code execution for N times, what is the overall coefficient of variation?

The overall coefficient of variation (CoV) for each model and metric in the SOAP benchmark is presented in Author Rebuttal Table 5, which is calculated by running and measuring the code execution 5 times for each model, then computing the mean and standard deviation of each metric. From the table, we can observe that all of the metrics' coefficients of variation are lower than 3% for all models, suggesting that the performance of these models is relatively consistent across multiple runs.

Thank you for your reply. SOAP is effective for both scenarios (i.e., improving LLMs' own tasks and improving common correct tasks).

To address the concern of the "apple-to-apple" results of our experiments in the Paper's main flow, in this thread, we provide results based on common tasks for Table 2 - Table 4's results, which provide consistent results for the two configurations. (Table 1's results are already demonstrated in a previous thread.)

For Paper Table 2, where we discuss how SOAP's effectiveness would be affected by the optimization results, we provide the evaluation results in Reviewer pc3i Table 2 (common tasks). First, we find the tasks addressed by both CodeLlama-70B-Ins and GPT-3.5-turbo-0301 with the Initial version (Step 0) and SOAP optimization process (Step 1-5). Then, we collect 46 tasks and measure the efficiency of LLM-generated code in these tasks. The evaluation results in Reviewer pc3i Table 2 (common tasks) show that CodeLlama-70B-Ins achieves SOAP efficiency results with one-step SOAP optimization, while GPT-3.5-turbo-0301 continuously improves the efficiency for the evaluated 46 tasks from Step 0 to Step 3. Then, the efficiency of GPT-3.5-turbo-0301 maintains efficiency for the other two optimization steps.

Next, for Paper Table 3, where we discuss how different prompt optimization steps affect the efficiency of LLM-generated code, we provide the evaluation results in Reviewer pc3i Table 3 (common tasks). First, we find the tasks addressed by both CodeLlama-70B-Ins and GPT-3.5-turbo-0301 for six different prompts and then get 40 tasks. Then, we provide the efficiency results in Reviewer pc3i Table 3 (common tasks), where we can observe that SOAP also achieves SOTA efficiency results compared with other baselines. For example, SOAP achieves 3.72 (Mbs) for TMU, while baselines only achieve 3.84 (Mbs) in the TMU metric for GPT-3.5-turbo generated code.

Finally, for Paper Table 4, where we discuss the differences in the efficiency of SOAP and the initial version of the CodeLlama family, we provide the evaluation results in Reviewer pc3i Table 4 (common tasks). First, we detect correct tasks addressed by both SOAP and the Initial version for four CodeLlama models and then get 3 tasks. Next, we provide the efficiency results of CodeLlama family-generated code in Reviewer pc3i Table 4 (common tasks). We can observe that for four CodeLlama models, SOAP continuously improves the efficiency of LLM-generated code.

We hope that our newly provided results can address the reviewer's concern.

Note: We provide the Tables in the next thread.

Thanks for the reply. I am sorry for not explaining my decision enough and here are my detailed reasons for my suggestion of decision:

1. First, the preliminary experimental results from the authors quickly cleared my concerns about approach effectiveness regarding weaknesses in #1, #4, and #5.
2. I won't reject the paper just for #1, #4, and #5 because those results are complementary to what the paper already had, meaning that they could be easily added to the camera-ready version if we end up accepting it.
3. Yet, weakness #2 still holds as existing results of the paper are mostly not represented in an "apple-to-apple" fashion. It is not even right to compare the efficiency of LLM codegen over different sets of tasks, making the efficiency score unreasonable to compare. For an extreme case, two LLMs may solve completely different sets of solutions, and computing efficiency scores over two totally different task sets cannot lead to clean and reasonable results regarding code efficiency in general.
4. Because it is problematic to compare LLM efficiency over different sets of tasks, ideally the author should globally correct the evaluation methodology, i.e., removing results made by inconsistent comparison and enforcing correctness alignments. This is crucial before accepting this paper because if not future work might inherit such a misevaluation methodology, leading to more inconsistent comparison in general.
5. Yet, doing reason 4 in rebuttal is quite hard objectively speaking. Meanwhile, given it's a major revision, it might be more rigorous to apply and reorganize the experiments and resubmit a clean version for another review, rather than accepting the paper through some rough views over the set of preliminary results that we are unsure if those can be systematically adopted in the camera-ready.

To summarize, I deeply appreciate the efforts of the authors in the rebuttal; however, for soundness, the required revision goes beyond "convincing the reviewers through proof-of-concept experiments" that it is crucial to globally correct the evaluation methodology for positively leading future research in this thread. Last but not least, I strongly feel this work can be accepted after doing a major revision.

Reviewer pc3i Table 2 (common tasks) Effect of the number of self-optimization steps in SOAP (46 tasks). The evaluation results are used to replace the results of Paper Table 2 with the same correct tasks addressed by two LLMs for all steps.

Reviewer pc3i Table 3 (common tasks) Effect of the number of self-optimization steps in SOAP (40 tasks). The evaluation results are used to replace the results of Paper Table 3 with the same correct tasks addressed by two LLMs for all steps.

Reviewer pc3i Table 4 (common tasks) Effect of the number of self-optimization steps in SOAP (3 tasks) for CodeLlama family.

Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:

W1 & W2 & Q3: The multi-iteration optimization method requires a large token overhead.

To address the reviewer's concern about overhead and context window limitations, we provide detailed metrics on execution time, token usage, per iteration input/output token usage， and efficiency in the Table below:

Table: Overhead of different code efficiency optimization methods for GPT-3.5-turbo.

We can observe the following:

• Execution Time and Token Overhead: SOAP requires approximately 8x more execution time compared to the Initial Version. However, this overhead is justified as the optimized code resulting from SOAP significantly reduces the execution time and memory usage for real-world software that could be executed millions of times. Besides, while the optimization process itself is resource-intensive, it yields substantial efficiency gains in deployment scenarios. For example, the average memory peak (MU) of SOAP-generated code only requires 40% compared with the initially generated code, which can help source code applied in memory-constrained environments, such as embedded systems or mobile devices. Furthermore, the reduced memory footprint and improved execution speed of the optimized code can lead to better overall system performance, especially in scenarios where the software is frequently used or runs on resource-limited hardware. As a result, the upfront computational cost of the optimization process is offset by the long-term benefits of more efficient and lightweight code in real-world applications.

• Context Window and Scalability: SOAP requires an average of 1.78K tokens for each input+output task iteration. Given that existing LLMs such as GPT-3.5 and GPT-4 have context windows of 4K tokens or more, they can effectively handle the global profiler information provided by SOAP. This capability allows the LLMs to understand comprehensive profiling data and perform global code optimizations, addressing concerns about scalability within the context window limitations.

By leveraging the available context window efficiently and optimizing the code based on detailed profiling data, SOAP manages to enhance the performance and scalability of the generated code, making it a valuable tool despite its initial overhead.

Q1: Include optimization options (similar to -O3 or -Oz in GCC) to cater to different preferences.

Thanks for your suggestion. As shown in Table 3 in the paper, the memory profiler and line profiler focus on the memory usage and execution time individually. To specify the optimization options, developers can directly revise Lines 508-509 in the Link to specify the optimization object (e.g., execution time and memory usage). In our final version, we will refine our source code argparse to provide optimization options for SOAP in the version.

Q2: Mechanisms to prevent code deterioration?

To mitigate this risk and ensure the reliability of our experiments, we have implemented a robust two-step verification process:

• Initial Verification: During the optimization process, if the refined code fails to pass the public test cases, we revert to using the initial version of the LLM-generated code for that specific task. This ensures that any code considered in our experiments at least meets the baseline correctness as defined by the public test cases.

• Comprehensive Filtering: In cases where the code passes the public test cases but fails the private test cases, we exclude the entire task from our experiments for that respective model. Consequently, we do not calculate the optimized code's efficiency for any step of that task. For example, if the code optimized in step 3 is correct but the code in step 4 is incorrect for a specific task, we exclude all steps of that task from our efficiency calculations for the corresponding model.

By adhering to this two-step process, we ensure that only fully verified and correct code is included in our evaluations. This approach allows us to provide a fair and accurate comparison of code efficiency across different optimization steps and models, thereby preventing any deterioration in code quality through the iterative optimization process.

Thank you for your thoughtful review and insightful comments.

W1 & Q1: Impact of self-optimization steps

In our paper, the self-optimization steps are constraint as 5 Section 4.2, consistent in our experiments. 5 is the default setting for many existing self-optimization methods [1, 2].

[1] Islam M A, et al. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving[J]. arXiv preprint arXiv:2405.11403, 2024.

[2] Zhou, Andy, et al. "Language agent tree search unifies reasoning acting and planning in language models." arXiv preprint arXiv:2310.04406 (2023).

Based on your suggestion, we require GPT-3.5-turbo-0301 to consider stopping the SOAP self-optimization process and only constrain the max step as 5. As shown in the Table below, the adaptive self-optimization process improves the efficiency of LLM-generated code. However, it does not achieve SOTA performance compared with SOAP. For example, when optimizing its initially generated code, it still needs 0.35s to execute, on average. SOAP only requires 0.28s. One key reason is that in each step, the efficiency of most code can be optimized. If the self-optimization stops early, the code’s efficiency remains in the sub-optimal results.

W3 & Q3: Semantic degradation in code correctness after refinement where test cases do not cover all edge cases.

We clarify that most edge cases have been covered in SOAP. For EffiBench, the private test cases covered most edge cases for each canonical solution. For the other two datasets, HumanEval and MBPP, we also utilize the test cases generated by EvalPlus to cover more edge cases.

On the other hand, the degradation is small (0.5% pass@1 for EffiBench, 0% pass@1 for HumanEval+, and 0.84% pass@1 for MBPP+ on average as follows). We believe that the potential degradation in correctness after refinement is manageable.

W4: Section 3.4, is verbose

Thanks for your suggestion. We will revise it accordingly.

Q2: Analysis of the profiling overhead

We provide the breakdown of the overhead for SOAP and other prompt engineering in Author Rebuttal Table 4. Compared with the initial version, SOAP would increase the total token usage from 1.1M to 5.4M.

Q4: Certain types of optimization where the method may fall short

There are certain types of optimization where the SOAP method may fall short. Specifically, tasks that do not require algorithmic improvements or have already been implemented with optimal time/space complexity may not benefit significantly from SOAP. One example is that as shown in Paper Fig. 12 to Fig. 18, when LLM first generates code for the task to sort two arrays, if the code already has an optimal time/space complexity, SOAP may not optimize the code into a more efficient version. We will add this discussion in the revised version.

Q5: Could rule-based post-processing techniques improve or accelerate the self-refinement?

Based on your suggestion, we conduct experiments for the SOAP by extracting the Top 5 lines with the highest execution time and memory usage and then use these profiler parts to guide LLM to optimize its previously generated inefficient code.

As shown in the Table below, rule-based optimization improves the efficiency of LLM-generated code compared with the initial version. For example, the average execution time (ET) of GPT-3.5-turbo-0301 decreases from 0.68s to 0.50s.

Results for GPT-3.5-turbo-0301

However, the efficiency is lower than SOAP. For example, SOAP further decreases the average execution time for GPT-3.5-turbo-0301 from 0.50s to 0.39s. Only providing part of the overhead profile information may not make LLMs understand how to optimize the code to improve efficiency as the efficiency problems are usually based on all code.

Q6: Refined code fails the test cases during optimization?

In our experiments, we follow a two-step process:

First, if the refined code does not pass the public test cases during the optimization process, we directly use the initial version of the LLM-generated code for that particular task.

Second, if the code passes the public tests but fails the private tests, we filter out the entire task from our experiments for the respective model and do not calculate the optimized code's efficiency for any step of that task.

For example, if step-3 optimized code is correct, but step-4 optimized code is incorrect in private test cases, we will not calculate the efficiency for any step of that task for the corresponding model, ensuring that we only consider code that passes all test cases in the evaluation and provide a fair comparison across different optimization steps and models.

Q7: Memory optimization is easier in the benchmark

One reason for memory usage being hard to optimize is that most of the tasks already achieve SOTA memory usage compared with execution time. Specifically, as seen in Table 1 in Section 4.2, most of the experiments achieve 1.00 NMU for LLM-generated initial code, which causes the code to be hard to further optimize compared with execution.

Thank you for your response to our rebuttal.

We have conducted additional experiments and provided further clarification to address the concerns raised in the follow-up questions. We hope that these efforts have adequately addressed Reviewer 5c1j's concerns. If Reviewer 5c1j thinks that the issues have been satisfactorily resolved, we would greatly appreciate it if they could consider improving the overall assessment for SOAP, as the current average overall assessment is in a borderline scenario.

Q1

To address your concern, we first optimized the code based on the steps outlined in Paper Table 2 and then continued to optimize for an additional five steps to analyze the efficiency of LLM-generated code after a total of 10 optimization steps. Our evaluation results are shown below. We can observe that increasing the number of optimization steps decreases the overhead, but after five steps, the efficiency metric improvements are minimal. Therefore, in our paper, we mainly set the number of steps to 5. However, if developers want to optimize their code further, they can increase the optimization steps at the cost of some additional token usage. We believe this token usage is worthwhile, as the optimized code would be deployed in the software and run thousands to millions of times.

Q2

Response:

The difference between Initial pass@1 v.s. SOAP pass@1 is only the SOAP-generated incorrect code. While the address method discussed in Q6 does not reflect in W3 & Q3, which means that with our solution in Q6, the incorrect code generated by SOAP in SOAP pass@1 will be replaced by the Initial generated code.

Q3.

We understand your concern about filtering out failed code in our evaluation. However, our approach is consistent with existing works in this field [1-6]. The primary reason for focusing only on correct code is that incorrect code may cause test cases to fail and terminate the execution process prematurely. Including such cases would lead to inaccurate results when measuring the efficiency of LLM-generated code.

[1] Huang, Dong, Jie M. Zhang, Yuhao Qing, and Heming Cui. "EffiBench: Benchmarking the Efficiency of Automatically Generated Code." arXiv preprint arXiv:2402.02037 (2024).

[2] Qiu, Ruizhong, Weiliang Will Zeng, Hanghang Tong, James Ezick, and Christopher Lott. "How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark." arXiv preprint arXiv:2406.06647 (2024).

[3] Shi, Jieke, Zhou Yang, and David Lo. "Efficient and Green Large Language Models for Software Engineering: Vision and the Road Ahead." arXiv preprint arXiv:2404.04566 (2024).

[4] Zheng, Jiasheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. "Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models." arXiv preprint arXiv:2407.11470 (2024).

[5] Waghjale, Siddhant, Vishruth Veerendranath, Zora Zhiruo Wang, and Daniel Fried. "ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?." arXiv preprint arXiv:2407.14044 (2024).

[6] Du, Mingzhe, Anh Tuan Luu, Bin Ji, and See-Kiong Ng. "Mercury: An efficiency benchmark for llm code synthesis." arXiv preprint arXiv:2402.07844 (2024).

Dear Reviewer 5c1j,

Thank you for your review and comments on our paper on 13 Aug, we have provided the clarification and other experiment results to address your concern.

Since the discussion period will be closed in the next six hours, while we do not receive the message about whether we have addressed all of your concerns, we would greatly appreciate it if you could take the time to review these results and consider improving your overall assessment of our paper based on these revisions.

Thank you once again for your valuable feedback and for considering our request.

Thank you for your thoughtful review and insightful comments. We hereby address your concerns below:

W1 & Q1: Novelty and Similarities with Self-Edit and Critic

To address the reviewer’s concern about the novelty of SOAP in comparison to Self-Edit and Critic, we provide detailed evaluation results in Table below. These results demonstrate that SOAP outperforms both Self-Edit and Critic significantly.

SOAP introduces a unique overhead profiler that significantly enhances the efficiency of LLM-generated code by optimizing both execution time and memory usage. In contrast, existing methods focus primarily on execution time and cannot comprehensively address efficiency concerns. Specifically, Critic requires progressive amendment of outputs, which does not effectively tackle efficiency-critical issues.

The novelty of SOAP lies in two aspects:

• Dual Optimization Focus: Unlike existing works that primarily target execution time, SOAP optimizes both execution time and memory usage, providing a more holistic approach to code efficiency.

• Overhead Profile-Guided Optimization: SOAP utilizes overhead profiles, including execution time and memory usage profiles, to guide the optimization process. This allows SOAP to achieve SOTA results in efficiency, as demonstrated by our ablation studies.

We would like to highlight that self-refinement is a broad direction with many published papers exploring various novelties. The specific challenge addressed by SOAP is improving the efficiency of LLM-generated code, a critical and valuable contribution to the field. Our experiments clearly show that the inclusion of overhead profiles significantly enhances optimization effectiveness compared to existing methods.

W2 & Q2: Discuss and compare transformers or LLMs for code optimization.

We would like to highlight that SOAP and these methods [1,2,3] are orthogonal, which means we can combine SOAP and these efforts.

LMEA [1] uses evolutionary search to generate more efficient code compared with the initial code, which requires 250+ solutions for each task to achieve optimal results, whereas SOAP only generates 5 solutions. LMEA requires massive token usage and execution time consumption. Due to limited time and resources during rebuttal, we are not able to conduct experiments with LMEA. However, we commit to include the results for LMEA in the revised manuscript.

For Supersonic [2] and PIE [3], we conduct experiments in Author Rebuttal Table 1. For PIE, we conduct experiments for five different prompts. As shown in Table 1, the performance of these methods is further boosted when SOAP is applied to them. On average, PIE+Few-Shot requires 0.82s for each generated code to execute. While PIE+SOAP+Few-Shot generated code only requires 0.41s to execute.

Ref:

[1] Large language models as evolutionary optimizers. 2023

[2] Supersonic: Learning to generate source code optimizations in c/c++. 2023

[3] Learning performance-improving code edits. 2023

W3 & Q3: Prompts for Self-Refine and Reflexion

For a fair comparison, we conduct experiments for baselines by revising the prompt used by SOAP in our experiments, which specifically requires LLMs to refine the coding efficiency. We provide the detailed prompts used in the paper in the Link: https://anonymous.4open.science/r/SOAP-FF6C/prompts.py

W4: Step-optimization steps only include two big models.

We provide three smaller models, i.e., OpenCodeInterpreter-1.3B, DeepSeek-1.3B-Ins, and CodeLlama-7B below. We can observe that SOAP consistently optimizes the efficiency of LLM-generated code for each optimization step.

W5: Pass@1 on HumanEval and MBPP.

As shown in the tables below, we provide the pass@1 for HumanEval+ and MBPP+, where the evalplus version is used in our experiments to measure the efficiency of LLM-generated code as it contains hundreds of tests. We can observe that the pass@1 of LLM-generated code only has merely changed for these benchmarks. The performance of our evaluated LLMs on HumanEval+ does not show a decrease. On MBPP+, the pass@1 decrease is only 0.0% and 0.84% for HumanEval+ and MBPP+, on average. Notably, for more powerful LLMs like GPT and Claude, the performance remains essentially unchanged.

Dear Reviewer H3VZ,

Thanks for your response before the discussion period deadline, which allows us to further clarify the potential misunderstanding and concerns that would affect your overall assessment.

First, compared with self-refine, SOAP utilizes line_profiler and memory profiler to catch the execution time and memory usage of each line in the LLM-generated code.

Second, for the pass@1 of LLM-generated code of pre-SOAP and post-SOAP, we want to mention that in https://anonymous.4open.science/r/SOAP-FF6C/src/SOAP.py (Lines 177-Lines 185), we provide tolerance for SOAP that requires the SOAP optimized code can pass public test cases and then replace original code. If the optimized code can not pass the public test cases, SOAP optimized code will not replace the original code, and we will use the original code for the next step of optimization (Lines 184-185). In this step, the coverage of the public test cases would largely affect the correctness of the final SOAP-optimized code. After the empirical study, we observe that the code line coverage of EffiBench for public test cases achieves 99.24%, which makes sure that the pass@1 after the self-optimization process (SOAP) only has a few decreases, and even in some models, the pass@1 would not decrease.

Third, we need to mention that PIE already trained CodeLlama with its provided code, which means that the pass@1 of PIE+CodeLlama-7B may be different, and the tasks addressed by PIE and the original CodeLlama are different. Then, some tasks with higher execution time (e.g., 4.7s) may not addressed by PIE-CodeLlama, and the default (initial) ET and other metrics are different. We hope that Reviewer H3VZ can understand this difference between the original CodeLlama and PIE-CodeLlama.

We hope that our clarification can address Reviewer H3VZ's concern about I feel these results require another round of careful reviews, especially given that some results look quite surprising and counterintuitive.

Dear Reviewer H3VZ,

Thank you for your review and comments on our paper.

Since the discussion period will be closed in the next six hours, while we do not receive the message about whether we have addressed all of your concerns, we would greatly appreciate it if you could take the time to review these results and consider improving your overall assessment of our paper based on these revisions.

Thank you once again for your valuable feedback and for considering our request.

I appreciate your response. I think this is a good metric.

However, could you clarify the numbers in your table?

I would think that "Optimal ET" would have initial 0.29 and after-SOAP 0.25 (DeepSeek-33B-Ins and OpenCodeInterpreter-33B are most efficient based on ET pre-SOAP, I picked DeepSeek-33B-Ins since SOAP improves its code more), "Optimal MU" would have initial 24.31 and after-SOAP 24.28 (StarCoder2-3B is most efficient based on MU), "Optimal TMU" would have initial 13.06 and after-SOAP 11.54 (OpenCodeInterpreter-33B is most efficient based on TMU). Am I misunderstanding your metric? (I did not look up other columns in the table just the ones corresponding to the Optimal metric in your subheading)

Thank you again.

Dear AC,

Thanks for your efforts to remind reviewers to participate in the discussion process.

To address the concern about the global efficiency results, we conduct experiments with our previously generated source code and SOAP-optimized code. We follow the below instructions to calculate the overall efficiency. First, we collect all correct LLM-generated code for each task and then select the most efficient code generated by LLMs based on ET, MU, and TMU. Then, we calculate the overall efficiency metrics for each metric-guided selected code for the initial code and SOAP-optimized code. The evaluation results for EffiBench are shown below. We can observe that in this scenario, SOAP largely improved the efficiency of LLM-generated code. For example, in the Optimizal ET results (i.e., we collect source code for both initial and SOAP optimized code for each task), SOAP decreases the average ET from 0.97 (s) to 0.23 (s). In Optimizal MU, SOAP decreases the MU from 59.16 (Mb) to 21.13 (Mb). In Optimial TMU, SOAP also decreases the TMU from 137.18 (Mbs) to 10.14 (Mbs). We hope our experiments can address your concern.