Effi-Code: Unleashing Code Efficiency in Language Models

Dear Reviewer HPnS,

Thank you for recognizing the importance of studying the efficiency of code generated by LLMs. We are pleased that you appreciate our proposed technique for generating efficient code snippets that can be used for fine-tuning existing LLMs. Your acknowledgment of the significant increase in the performance of LLMs, both in terms of efficiency and correctness, as demonstrated by our empirical results, validates the effectiveness of our approach. To address your concerns regarding the weaknesses and questions you have raised, we will provide a detailed point-by-point response below. We hope that our clarifications, additional experiments, and thorough explanations will help alleviate your concerns and demonstrate the robustness and potential of our approach.

False claims / Repetitive work

We would like to clarify that the steps such as Section 3.3 using SOAP [2] to optimize the efficiency of LLM-generated code and Step 3 generating test cases to measure the efficiency of LLM-generated code, are introduced as the detailed component of our framework to construct the Effi-Code dataset. We do not treat these as our key contribution.

We clarify that the key contributions of our work are as follows:

• We provide a framework to inspire researchers to construct code generation datasets containing efficient solutions for each code generation task (task description), which is versatile and can be adapted to different programming languages and leverage various existing data sources. Unlike some other code generation datasets that rely on powerful models (e.g., GPT-4), our framework can be implemented using more accessible models like DeepSeek-Coder. The framework provides a systematic method for researchers to enhance existing datasets or create new ones focused on code efficiency across different languages and domains.

• Based on our proposed framework, we release the Effi-Code dataset, which is the first instruct-tuning dataset that focuses on improving the efficiency of LLM-generated code. The primary purpose of Effi-Code is to instruct and fine-tune LLMs to ensure that the LLM-generated code for each task is more efficient than the code generated by the original LLM.

• We use Effi-Code to fine-tune several widely used LLMs and will release these models on the Hugging Face website in the final version. Different from existing datasets that are used to finetune the LLMs to improve the pass@1 of LLM-generated code, our evaluation results demonstrate that both the pass@1 and the efficiency results would be improved for LLMs finetuned on our Effi-Code dataset.

Each step in the Effi-Code construction framework demonstrates how we process candidate tasks, filter out lower quality and risky tasks, and construct an efficient solution for each task. We introduce these techniques used in our Effi-Code construction framework. However, it is important to note that the individual steps in our construction framework are not considered our primary novelty or contribution; rather, they serve to illustrate our methodology.

W1: In the abstract, the authors falsely claim

Thank you for your comment. We apologize for the confusion caused by our wording in the abstract. Our intention was not to claim the Self-Optimization process based on Overhead Profiling (SOAP) as a new method, and take it as our core contribution. We acknowledge that SOAP was proposed in the previously published paper and already accepted by NeurIPS. In our manuscript, we have also cited the aforementioned paper in both the Introduction and Method sections to give proper credit to the original work. We will revise the abstract to clarify that our core contribution is the generation of the Effi-Code dataset using the SOAP method, rather than proposing SOAP itself.

Next, in our work, we aim to express that we generate a dataset (Effi-Code dataset) based on the SOAP method, which ensures the efficiency of the generated code is better than the original solution provided by each task in our collected candidate tasks. Similar to SOAP, some methods such as Mercury and PIE can be used to generate more efficient solutions compared to the original solution provided by each task in our collected tasks. In contrast, we use SOAP to optimize the efficiency of the original solution are due to SOAP can generate the most efficient solutions and more correct solutions compared to other methods in our preliminary experiments.

The key contributions of our paper are

• We provide a framework to construct a code generation dataset (Effi-Code) that contains efficient solutions for each task. Based on our proposed framework, we release the Effi-Code. To the best of our knowledge, it is the first instruct tuning dataset that focuses on improving the efficiency of LLM-generated code.
• We reveal that fine-tuning LLMs using Effi-Code (efficient solutions for each task) ensures that LLMs generate more efficient solutions compared to baselines.

After carefully reading author responses, I am not convinced with their answers and still have my questions. I am not going to raise my scores. Please see my responses about unresolved issues. Thanks.

Dear Reviewer 5L15,

We greatly appreciate your positive feedback on our work, particularly your recognition of the importance and practical relevance of optimizing efficiency in LLM-generated code. Your acknowledgment of EFFI-CODE's rigorous data curation process, which includes iterative optimization cycles and runtime profiling, highlights the value of our approach in achieving measurable improvements in real-world benchmarks. We are pleased that you find our work valuable for performance-critical applications in resource-constrained environments. We are committed to open-sourcing EFFI-CODE to foster reproducibility and encourage broader research on efficiency-focused fine-tuning for code generation. We believe that this will contribute to the advancement of the field and enable other researchers to build upon our work.

To address your concerns regarding the weaknesses and questions you have raised, we will provide a detailed point-by-point response below. We hope that our clarifications, additional experiments, and thorough explanations will help alleviate your concerns and demonstrate the robustness and potential of our approach.

W1: Limited comparison with baselines

Response: 

Thank you for your suggestion! We provide the evaluation results of Supersonic, PIE, Mercury, and Effi-Code in Rebuttal Table 1. We currently only have the inference results of Mercury in the DeepSeek-Coder-6.7B-base, so we compare the efficiency of Mercury and Effi-Code in the DeepSeek-Coder-6.7B-base. For Supersonic and PIE, we compare the efficiency results in CodeLlama-7B-hf. Furthermore, as the training set of Mercury contains some tasks in EffiBench, for a fair comparison, we evaluate the efficiency results in the HumanEval dataset.

As shown in Rebuttal Table 1, we can observe that for both models, Effi-Code achieves state-of-the-art (SOTA) performance compared to the baselines. For example, in CodeLlama-7b-hf, the average execution time for Supersonic decreases from 1.40s to 1.24s on average for all overlapping correct tasks, while Effi-Code further decreases the average execution time from 1.24s to 1.21s. Compared to the solution generated by CodeLlama-7b-hf, the average execution time was reduced by 16.7%.

Rebuttal Table 1: Efficiency comparison of different methods on the HumanEval dataset.

W2: It would be good to check the generalizability of compiled languages (e.g., C++) and managed languages (e.g., Java) with distinct memory management and execution models.

Response:

Thank you for your suggestion. We agree that focusing solely on algorithmic performance may be too narrow, and efficiency gains could also be achieved by addressing issues related to third-party library usage. We also acknowledge that EFFI-CODE is currently focused on Python, and it would be beneficial to check the generalizability of compiled languages (e.g., C++) and managed languages (e.g., Java) with distinct memory management and execution models.

To address this concern, we have conducted additional experiments using the HumanEval-X dataset, which includes tasks in C++ and Java. We fine-tuned the DeepSeek-Coder-6.7B-base model using the Effi-Code dataset and compared its performance with the original model on the HumanEval-X tasks. 

Rebuttal Table 2: Efficiency comparison of different methods on the HumanEval-X (CPP) dataset.

Rebuttal Table 3: Efficiency comparison of different methods on the HumanEval-X (Java) dataset.

As shown in Rebuttal Tables 2 and 3, we can observe that the results show that the model fine-tuned with Effi-Code achieves better efficiency compared to the base model on both C++ and Java tasks. For C++, the execution time (ET) is reduced by 27%, and the total memory usage (TMU) is reduced by 25%. For Java, the execution time is reduced by 21%, and the total memory usage is reduced by 12%.

Q1: Can the authors please further expand the evaluation process?

Response: Thanks for your reminder. We have revised the manuscript for Section 3.2 Step 4 Lines 202-236 and Section 3.4 Lines 266-287 to further expand the evaluation process. For more details, Reviewer can check the response for W4.

Q2: What percentage of solutions improved in efficiency but showed degraded correctness, if any?

Response: That's a great question! To address this, we provide the evaluation results of degraded correctness (tasks that were correct in the original LLM but became incorrect in the Effi-Code fine-tuned LLM) and upgraded correctness (tasks that were incorrect in the original LLM but became correct in the Effi-Code fine-tuned LLM) in Rebuttal Table 3. We can observe that for all LLMs in the two evaluation datasets, the first scenario, i.e., degraded correctness, is very low. For example, in DeepSeek-Coder-6.7B-base, no tasks went from correct to incorrect after the Effi-Code fine-tuning. However, we can also observe that a large number of incorrect tasks in the original LLMs were correctly addressed by the Effi-Code fine-tuned LLMs. For instance, in DeepSeek-Coder-6.7B-base, an additional 52.5% of tasks were addressed by the fine-tuned version.

Rebuttal Table 3: Evaluation results of degraded and upgraded correctness after Effi-Code fine-tuning on various LLMs.

Minor comments: In Table 3, the pass@1 results do not consistently increase as the proportion of Effi-Code used for fine-tuning grows. Could the authors elaborate on this inconsistency?

Response:

The minor inconsistency in the pass@1 results as the proportion of Effi-Code used for fine-tuning increases can be attributed to the inherent randomness in the fine-tuning process. When fine-tuning LLMs on the same tasks, the resulting pass@1 scores may vary slightly between different runs. To illustrate this, we conducted additional experiments where we fine-tuned DeepSeek-Coder-Base using the same subsets of Effi-Code. As shown in Rebuttal Table 4, the pass@1 scores for three separate fine-tuning runs exhibit slight variation.

Rebuttal Table 4: Pass@1 scores for three separate fine-tuning runs using the same subsets of Effi-Code.

Returning to the results in Table 3 for DeepSeek-Coder-6.7B-base, we observe that the pass@1 score increases significantly from 7.3% to 55.5% when using just 25% of the training subset. When the entire fine-tuning dataset is provided, the pass@1 score further improves to 59.8%. This suggests that the gains in pass@1 scores when using 50% and 75% of the fine-tuning subset may be relatively small compared to the substantial improvement achieved with only 25% of the subset. Consequently, due to the inherent randomness in fine-tuning, the pass@1 scores for models fine-tuned on 50% and 75% of the subset may not consistently surpass the results obtained with the 25% fine-tuned model.

References:

[1] Mercury: An efficiency benchmark for llm code synthesis. NeurIPS 2024.

[2] Effibench: Benchmarking the efficiency of automatically generated code. NeurIPS 2024.

[3] SOAP: Enhancing Efficiency of Generated Code via Self-Optimization. NeurIPS 2024

Initial Code

Execution Time: 1.100s

def levenshtein(a, b):

    if not a: 

        return len(b)

    if not b: 

        return len(a)

    if a[0] == b[0]:

        return levenshtein(a[1:], b[1:])

    return 1 + min(levenshtein(a[1:], b), 

                levenshtein(a, b[1:]), 

                levenshtein(a[1:], b[1:]))



Optimized Code

# Execution Time: 0.00012s

def levenshtein(a: str, b: str) -> int:

    distance_matrix = [[0] * (len(b) + 1)\

     for _ in range(len(a) + 1)]

    for i in range(len(a) + 1):

        distance_matrix[i][0] = i

    for j in range(len(b) + 1):

        distance_matrix[0][j] = j

    for i in range(1, len(a) + 1):

        for j in range(1, len(b) + 1):

            if a[i-1] == b[j-1]:

                distance_matrix[i][j] = \

                distance_matrix[i-1][j-1]

            else:

                distance_matrix[i][j] = min(

                distance_matrix[i-1][j] + 1,

                distance_matrix[i][j-1] + 1,

                distance_matrix[i-1][j-1] + 1)

    return distance_matrix[len(a)][len(b)]

W3: The profiling metrics setup lacks robustness.

Response: Thank you for raising this concern. We agree that execution time and memory usage can be affected by different platforms and system noise. To address this issue, we have conducted additional experiments and provided more robust evaluation results.

Firstly, we have evaluated the effectiveness of Effi-Code on seven different software-hardware setups, as shown in Rebuttal Table 2. The results demonstrate that Effi-Code fine-tuned LLMs achieve higher efficiency than the original LLMs across all setups. For example, in the environment of Python 3.11.10 - Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz, the average execution time decreases from 0.59s to 0.40s when using Effi-Code to fine-tune Qwen2.5-Coder-7B, reducing the average execution time by 32%.

Secondly, we clarify that for the same setup, where we evaluate the efficiency of LLM-generated code several times, the efficiency results are consistent. As shown in Paper Table 8, where we execute the LLM-generated code five times, the standard deviation of execution time (ET) is 0.00548 (s), indicating that the evaluation results are consistent and reliable for a given setup.

Finally, our evaluation setup follows the practices established in recent works on benchmarking the efficiency of automatically generated code, such as Mercury [1], Effibench [2], and SOAP [3]. By adhering to these benchmarks, we ensure that our evaluation is in line with the current standards in the field.

Rebuttal Table 2: Evaluation results of Effi-Code's effectiveness on different software-hardware setups.

Dear Reviewer wYvD,

Thank you for your appreciation of our work in improving the efficiency of LLM-generated code. To further address your concerns regarding the weaknesses and raised questions, we address them point by point below. We hope that our clarifications, additional experiments, and responses can alleviate your concerns and lead you to consider increasing your rating of our work.

W1: The paper can benefit from a more extensive comparison of Effi-Code Benchmarks across different open-source LLMs.

Response: Thank you for your valuable suggestion. To address your concern, we have conducted additional experiments by fine-tuning Effi-Code on five more open-source LLMs. We have carefully selected these LLMs based on their popularity and performance in code generation tasks. However, if there are any specific models the reviewer would like us to evaluate, we would be more than happy to include them in our analysis and report the results.

The results are presented in Rebuttal Table 1, demonstrating the effectiveness of Effi-Code in improving the efficiency of the generated code across various LLMs. We can observe that all the evaluated LLMs exhibit improvements in both code efficiency and pass@1 metrics after fine-tuning with Effi-Code. For instance, CodeLlama-13B-hf shows a significant reduction in execution time (ET) from 0.86s to 0.13s on average for correctly overlapped tasks, which reduces execution time by 84.88%. In addition, we can also observe that the pass@1 of CodeLlama-13B-hf generated code increases from 7.9% to 28.8%, which also increases pass@1 by 20.9% compared to the original LLM. These additional experiments on a diverse set of open-source LLMs further validate the generalizability and effectiveness of our proposed Effi-Code dataset.

Rebuttal Table 1: Comparison of Effi-Code across different open-source LLMs.

W2: The paper can benefit from a concrete example to show how SOAP’s iterative refinement improves the quality of the solutions.

Response:

Thank you for your suggestion. We have provided a case example below (also added in Paper Appendix Figure 4, Lines 1091-1119) to demonstrate how SOAP's iterative refinement improves the quality of the solutions.

In this example, the initial code generated by DeepSeek-Coder-V2-Lite calculates the Levenshtein distance using a recursive approach, which has an exponential time complexity of O(3^(m+n)). For longer strings, this recursive method becomes highly inefficient due to the large number of function calls.

To optimize the code, the refined version employs dynamic programming, which avoids redundant calculations by filling a distance matrix to compute the Levenshtein distance. The time complexity of the dynamic programming approach is O(mn), where m and n are the lengths of strings a and b, respectively. By filling the distance matrix in a single traversal, the optimized code eliminates redundant calculations, resulting in improved efficiency.

The dynamic programming solution leverages the characteristics of optimal substructure and overlapping subproblems, decomposing the problem into smaller subproblems and storing intermediate results to avoid redundant calculations, thereby improving the efficiency of the algorithm.

In the provided example, the initial recursive code takes 1.100s to execute, while the optimized dynamic programming code completes execution in just 0.00012s, demonstrating a significant improvement in execution time.

Furthermore, to investigate whether the efficiency of the code generated by Effi-Code fine-tuned LLMs can be further enhanced once we add additional efficient C++ code into Effi-Code dataset, we have followed the pipeline of Effi-Code and constructed an Effi-Code (C++) subset containing 3,322 C++ tasks. We then fine-tuned LLMs using three different setups: Effi-Code (Py), Effi-Code (C++), and Effi-Code (C++) + Effi-Code (Py). The evaluation results, presented in Rebuttal Table 2, reveal several interesting findings.

Firstly, LLMs fine-tuned on the Effi-Code datasets generate more efficient code compared to the original LLM-generated code. For example, the average execution time for Qwen2.5-Coder-7B generated code is 0.35s, while the Effi-Code (Py) fine-tuned LLMs require only 0.17s on average for overlapped tasks, resulting in a 51.4% reduction in average execution time.

Secondly, when we utilize Effi-Code (C++) and Effi-Code (Py) + Effi-Code (C++) to fine-tune LLMs, the overhead of LLM-generated code is further decreased. The average execution time for overlapped code decreases from 0.17s to 0.16s, and the memory peak (MU) also decreases from 46.71MB to 43.72MB. These results indicate that by incorporating C++ source code to guide LLM fine-tuning, LLMs may learn additional optimization strategies.

Due to the limited time and computational resources of rebuttal, we can only do a small number of experiments in other languages. However, these experiments above illustrate the generality and effectiveness of effi code, and we will add more experiments in other languages to test more models and datasets.

W3: Broadening the scope to include non-algorithmic optimizations will not only improve the soundness of the paper but can also strengthen the significance of Effi-Code in terms of real-world implications.

Response:

We appreciate the reviewer's insightful suggestion regarding the potential for optimization in non-algorithmic tasks. To address this concern, we have conducted additional experiments and provided the evaluation results in Rebuttal Table 3, which compares the performance of the original Qwen2.5-Coder-7B, the model fine-tuned on Effi-Code, and the model fine-tuned on Effi-Code + non-algorithmic tasks (optimized).

We can observe that when we fine-tune Qwen2.5-Coder-7B on either Effi-Code or Effi-Code + non-algorithmic tasks, the efficiency of LLM-generated code improves. For instance, the average execution time for overlapped correct tasks decreases from 0.49s to 0.19s for both Effi-Code and Effi-Code + non-algorithmic tasks fine-tuned Qwen2.5-Coder-7B.

However, we also observe that the TMU of the Effi-Code fine-tuned Qwen2.5-Coder-7B is lower than the model fine-tuned on Effi-Code + non-algorithmic tasks. Specifically, the Effi-Code + non-algorithmic tasks fine-tuned Qwen2.5-Coder-7B decreases the average TMU for overlapped correct code from 10.75 MBs to 4.17 MBs. In contrast, Qwen2.5-Coder-7B fine-tuned only on Effi-Code further reduces the TMU from 4.17 MBs to 4.07 MBs.

Our results indicate that while incorporating non-algorithmic tasks in the fine-tuning process can lead to improvements in code efficiency, focusing solely on algorithmic tasks, as done in Effi-Code, may yield even better results. Nonetheless, we acknowledge the potential benefits of broadening the scope to include non-algorithmic optimizations, as it can enhance the real-world implications of Effi-Code. In future work, we plan to explore the integration of non-algorithmic tasks more comprehensively while maintaining the focus on algorithmic optimization.

Rebuttal Table 3: Efficiency results on the EffiBench dataset with different fine-tuning setups.

Dear Reviewer 4rzw,

Thank you for your appreciation of our work and for recognizing that the Effi-Code framework exhibits significant absolute improvements. To further address your concerns regarding the weaknesses and raised questions, we will address them point by point below. We hope that our clarifications, additional experiments, and responses can alleviate your concerns and lead you to consider increasing your rating of our work.

W1: I would be glad to reconsider and potentially raise these scores if the authors could provide further justifications for the unique methodologies or modifications that set Effi-Code apart from prior work, especially SOAP [2].

Thanks for your appreciation that our work consolidates established methods to synthesize efficiency-enhanced code. We would like to clarify that the steps such as Section 3.3 using SOAP [2] to optimize the efficiency of LLM-generated code, Step 1 collecting candidate tasks from open-source datasets, and Step 3 generating test cases to measure the efficiency of LLM-generated code, are introduced as the detailed component of our framework to construct the Effi-Code dataset. We do not treat these as our key contribution.

We clarify that the key contributions of our work are as follows:

• We provide the first framework to inspire researchers to construct code generation datasets containing efficient solutions for each code generation task, which is versatile and can be adapted to different programming languages and leverage various existing data sources. Unlike some other code generation datasets that rely on powerful models (e.g., GPT-4), our framework can be implemented only using open-sourced LLMs like DeepSeek-Coder. The framework provides a systematic method for researchers to enhance existing datasets or create new ones focused on code efficiency across different languages and domains.

• Based on our proposed framework, we release the Effi-Code dataset. To the best of our knowledge, it is the first instruct tuning dataset that focuses on improving the efficiency of LLM-generated code. The primary purpose of Effi-Code is to instruct and fine-tune LLMs to ensure that the LLM-generated code for each task is more efficient than the code generated by the original LLM.

• We use Effi-Code to fine-tune widely used LLMs and will release these models on the Hugging Face website in the final version. Different from existing datasets that are used to finetune the LLMs to improve the pass@1 of LLM-generated code, our evaluation results demonstrate that both the pass@1 and the efficiency results would be improved for LLMs finetuned on our Effi-Code dataset.

Each step in the Effi-Code construction framework demonstrates how we process candidate tasks, filter out lower quality and risky tasks, and construct an efficient solution for each task. We introduce these techniques used in our Effi-Code construction framework. However, it is important to note that the individual steps in our construction framework are not considered our primary novelty or contribution; rather, they serve to illustrate our methodology.

W2: Evaluating the models on a C/C++ benchmark could provide a more accurate and comprehensive assessment of their code optimization capabilities. It would also be interesting to see how the model's performance increases on a Python benchmark when fine-tuning on a C/C++ dataset.

We appreciate the reviewer's insightful suggestion. To address the concerns raised, we have conducted additional experiments on the HumanEval-X (C++) dataset and provided the efficiency results in Rebuttal Table 1. We can observe that the efficiency of LLM-generated code also improved with Effi-Code fine-tuned LLM. For instance, the average execution time (ET) for the overlapped code decreases from 0.44s to 0.32s, resulting in a 27% reduction in execution time.

Rebuttal Table 1: Efficiency results on the HumanEval-X (C++) dataset.

Rebuttal Table 2: Efficiency results on the EffiBench dataset with different fine-tuning setups.

We appreciate the reviewer's thoughtful comments and address the key points below.

Regarding the novelty of our framework, while PIE also provides a dataset containing efficient code solutions, it was curated by collecting existing efficient human-written programming submissions. In contrast, our framework focuses on taking an inefficient candidate solution for a task and optimizing it to construct the dataset. This key difference sets our work apart and supports our claim of being the first framework to inspire researchers to construct code generation datasets containing efficient solutions for each code generation task. Our approach motivates researchers on how to construct efficient code solutions for each task, rather than directly collecting tasks from human-written solutions. As the number of tasks becomes very large (e.g., 700K+ candidate tasks in our collected tasks), the overhead for constructing an efficient solution for each task by human-written means would be significantly time-consuming.

Next, thank you for pointing out the limitations of our current approach to non-algorithmic optimization tasks. After further analysis, we believe the lack of improvement on these tasks is due to the absence of suitable benchmarks that reflect the nature of non-algorithmic optimizations. The benchmarks we used for evaluation, specifically HumanEval and EffiBench, have a distribution gap compared to non-algorithmic tasks, which makes it hard to show the real impact of our approach. To better evaluate Effi-Code's performance on non-algorithmic tasks, we need a wider range of benchmarks that are more like real-world scenarios where these optimizations can make a big difference. Unfortunately, we do not have access to such benchmarks right now, which limits our ability to show the usefulness of non-algorithmic optimization tasks. Given the time constraints and this lack of benchmarks, we may not be able to complete more experiments or provide convincing evaluation results for non-algorithmic tasks within the rebuttal period. We acknowledge this as a limitation of our current work and are committed to addressing it in future versions of Effi-Code.

While some existing works also employ open-source LLMs to synthesize datasets, their primary focus is on generating solutions for tasks without explicitly considering efficiency. For instance, CRUXEval requires LLMs to generate Python functions ranging from 3 to 13 lines of code, and StarCoder2-Instruct uses StarCoder2 to generate thousands of instruction-response pairs. However, these works do not provide a systematic framework and dataset specifically designed to address code efficiency optimization. In contrast, our work aims to complement and build upon these existing efforts by introducing a novel approach that prioritizes the generation of efficient code solutions. By focusing on code efficiency optimization, we provide a unique contribution to the field, offering a framework and dataset that can help developers and researchers create more efficient and optimized code. Our work bridges the gap between code generation and efficiency optimization, providing a valuable resource for the programming community to improve the performance of their software solutions.

We hope this clarifies the key concerns raised. Please let us know if you have any other questions or feedback. We are committed to refining this work to provide a meaningful contribution to the community.