EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning

We want to thank the reviewer for his insightful comments and suggestions. We provide detailed responses point by point. We hope our responses can address your concerns.

W1: LLM-generated code may be inefficient

Thank you for raising this important point about the efficiency of our candidate code generation. To address this concern directly, we evaluated both DeepSeek-Coder (Lite) and GPT-4o on the ENAMEL benchmark [1]. As shown in Table 1, these models achieve higher effi@k scores compared to most LLMs reported in Qiu et al. [1], confirming their capability to generate sufficiently efficient code candidates.

Our optimization methodology further enhances this efficiency. After applying our techniques, we observed substantial improvements: average ET decreased from 1.14s to 0.27s, while average TMU dropped from 26.24MBs to 5.13MBs—representing 75-80% improvements across efficiency metrics (Figure 4 in our paper).

These results demonstrate that our approach effectively generates and optimizes code that meets high efficiency standards, even when evaluated on benchmarks specifically designed to assess code efficiency.

Table 1: Evaluation results of DeepSeek and GPT4o generated code in ENAMEL dataset [1]. Due to OpenAI token limitations, we then only provide the results of GPT4o for k=1 and 10.

W2: This work uses canonical solutions to evaluate efficiency. The authors evaluate the generated code on the benchmark of [1]

Thank you for highlighting this important methodological consideration regarding canonical solution efficiency. We address this concern in two ways:

First, EffiBench, our primary evaluation benchmark, uses optimized canonical solutions provided by the dataset constructors and employs sufficiently large test inputs to meaningfully differentiate between algorithmic complexities (e.g., O(n) vs O(n²)). This allows reliable measurement of efficiency differences across implementations.

Second, to further validate our approach, we conducted additional experiments using the ENAMEL benchmark [1], which specifically emphasizes efficient canonical solutions and large-scale inputs. Table 2 presents these results, comparing baseline models against SwiftCoder fine-tuned versions.

The results show substantial improvements across all metrics. For example, eff@1 (measuring both efficiency and correctness on a single generation) increases from 0.373 to 0.458 (+22.8%) for Qwen2.5-Coder and from 0.179 to 0.393 (+119.6%) for DeepSeek-Coder.

In our final version, we will include these ENAMEL benchmark results to provide a more comprehensive evaluation of SwiftCoder.

Table 2. Evaluation results of baseline models vs. SwiftCoder on ENAMEL benchmark.

W3 Discussion on negative results

We appreciate your noting the absence of discussion on negative results. You correctly identified that, in some instances, fine-tuning on SwiftCode produced negligible improvements or slight performance decreases in certain metrics, as shown in Table 3.

These variations stem from our optimization approach using the TMU metric for code selection. When optimizing for this composite metric, improvements in TMU may occasionally come at the expense of a slight degradation in individual metrics (either execution time or memory usage). For example, a solution might achieve substantial memory efficiency while slightly increasing execution time, resulting in an overall improved TMU score but showing a negative trend in the execution time metric when viewed in isolation. We'll include a detailed discussion of these cases to better illustrate the complex relationships between efficiency metrics in the revised version.

T1: line 72 typo

Thank you for identifying this error. We will correct it in the revised version.

Q1: Space-time tradeoff

When selecting the final code for our dataset, we addressed the space-time tradeoff using the TMU (Total Memory Usage) composite metric that combines both execution time and memory consumption into a single evaluation metric.

Rather than treating time and memory as separate dimensions requiring individual optimization, TMU provides a holistic efficiency measure that accounts for both resources simultaneously. This approach acknowledges the inherent tradeoffs in algorithm design (such as dynamic programming vs. brute force approaches) and allows us to identify solutions that achieve the most favorable overall resource balance.

overhead, memory_usage, execution_time, memory_peak = calculate_code_execution_efficiency(dataset[i])
if dataset[i]["memory_usage"] > memory_usage:
      dataset[i]["memory_usage"] = memory_usage
      dataset[i]["execution_time"] = execution_time
      dataset[i]["memory_peak"] = memory_peak

We want to thank the reviewer for his insightful comments and suggestions. We provide detailed responses point by point. We hope our responses can address your concerns and lead you to consider increasing your rating of our work.

C1 & W1 Relation & Limitation.

Thank you for acknowledging the practical value of our work on code efficiency optimization. While the concept may seem intuitive retrospectively, SwiftCoder's implementation required significant technical innovation to overcome multiple challenges.

First, our multilingual efficiency dataset development required specialized infrastructure to measure code efficiency across five programming languages with consistent metrics, control for system-dependent execution variations, maintain consistent evaluation environments, and scale our pipeline to handle 70,000+ diverse code samples. Second, we established a previously unconfirmed relationship between training data efficiency and LLM-generated code efficiency. Unlike discrete correctness metrics, efficiency exists on a continuous spectrum with complex interactions with functionality, creating unique optimization challenges beyond merely applying the "quality in, quality out" principle. Finally, while substantial research has focused on code correctness, efficiency optimization remains critically underrepresented despite its practical importance. SwiftCoder provides both methodology and dataset contributions to this nascent research area.

We believe these contributions collectively advance the state of efficiency-aware code generation and establish foundations for future research in this important domain.

C2 & W4 Additional references

We couldn't find the latest Effi-Code [1] on ArXiv. If the reviewer provides a link, we'll gladly compare it with SwiftCoder.

We've added EffiLearner [2] comparison in the Table below, showing SwiftCoder's superior performance. On EffiBench, SwiftCoder reduces average execution time more effectively than EffiLearner. Unlike EffiLearner, which decreases pass@1 rates, SwiftCoder consistently improves pass@1 across all evaluations.

We also evaluated SwiftCoder on ENAMEL [3], which tests efficiency on enhanced HumanEval test cases. As shown in the Table below, SwiftCoder consistently improves both pass@k and effi@k metrics across all models and k values (1, 10, and 100).

Coignion et al [4] focus on LeetCode efficiency evaluation, which is partially covered in the evaluated EffiBench (See paper Table 2) dataset.

Jiang et al [5] collected 52 bilingual programming questions (Chinese and English) from existing benchmarks. Their work is under review, and the dataset is not publicly available yet. We will evaluate their dataset when it becomes available and include all related discussions and results.

W2 & Q1 Test cases & ground truth.

All tasks in our datasets have verified solutions (ground truth) that correctly fulfill their descriptions. For example, SelfCodeAlign only includes code that passes all test cases in a controlled environment.

Since most datasets lack test cases, we used GPT-4 to generate them based on task descriptions and solutions. We validated these tests by running them against the original solutions from the datasets. Only tests that executed successfully were retained; failed tests were filtered out. Tasks without valid tests were removed entirely.

During optimization, we ran LLM-generated code against these validated tests. Solutions failing any test were eliminated. From the remaining valid solutions, we selected the most efficient implementation as the final optimized code.

W3 & Q2 Data Contamination

We addressed data contamination concerns in our SwiftCoder dataset. Analysis shows no exact duplicates between training and evaluation sets, with only 0.20% of evaluation samples having minimal vocabulary overlap (5-10%). Our dataset construction included decontamination processes, as did our source datasets like Evol-Ins, which removed content from common benchmarks. Experiments on EvoEval, designed to avoid benchmark leakage, still demonstrate our approach produces more efficient code than the original LLM-generated solutions.

Table 1 Data Overlap Analysis

Table 2 Vocabulary Overlap Analysis

We want to thank the reviewer for his insightful comments and suggestions. We provide detailed responses point by point. We hope our responses can address your concerns and lead you to consider increasing your rating of our work.

W1 & Q4 Novelty and Comparison with PIE and Mercury

Our paper's primary contribution is application-driven: we introduce the first multilingual code efficiency instruction tuning dataset. As the ICML guidelines note, "originality need not mean wholly novel methods… a novel dataset... match the needs of the user." Our dataset addresses the critical real-world need for efficient code generation, enabling researchers to fine-tune models for improved performance.

Next, compared to existing works:

1. SwiftCoder introduces a fully automated code optimization framework that transforms initial task descriptions into efficient solutions without human intervention. Unlike PIE, which relies on human programmers to write efficient solutions, or Mercury, which selects the most efficient solution from pre-existing human-written code, SwiftCoder can optimize code starting from just a task description. This automation enables researchers and developers to enhance their existing code generation datasets with minimal manual effort, making efficiency optimization more accessible and scalable.

2. SwiftCoder offers broader language coverage and greater generalizability by including optimized tasks across multiple programming languages (C++, Python, Java, Rust, and Go), contrasting with PIE's focus on C++ and Mercury's focus on Python, allowing models fine-tuned on SwiftCoder to perform effectively across diverse language environments. Additionally, SwiftCoder's significantly larger scale—comprising 65,710 unique tasks compared to Mercury's 1,889 and PIE's 1,474—provides more comprehensive training data, resulting in superior pass@ and efficiency.

W2 Rejection sampling in math and code domain

We agree that rejection sampling has been widely used in the mathematical domain. However, this technique is not our key contribution. Our primary contributions are our empirical study establishing the correlation between training data efficiency and generated code efficiency, the development of a multilingual code efficiency dataset, and the end-to-end pipeline for improving code efficiency across multiple languages.

Unlike prior code generation work focuses on binary correctness metrics, efficiency in our work is a continuous metric requiring different optimization strategies. We would appreciate it if the reviewer could suggest specific code-related rejection sampling techniques for efficiency optimization. We would be happy to include them in our paper.

W3 Sampling params

We conduct an ablation study on the sampling size. The evaluation results are shown in Reviewer jr1T Ref Table 1, where we evaluate the efficiency results for the most efficient code with 1 and 10 samples. Our results reveal that SwiftCoder consistently achieves higher results.

We also compared baselines against those fine-tuned on the original dataset and SwiftCoder. The table below shows that models fine-tuned on the original dataset decreased efficiency. For example, average execution time increases from 0.33s to 0.37s for DeepSeek.

Q1&Q2 Inconsistency of column names

The purple percentages in paper Table 7 indicate the reduction in overhead metrics compared to CodeLlama-7B-hf without instruction tuning from PIE, Mercury, or SwiftCoder. We provide the detailed results in the Table below and we will add it in our paper in camera ready.

Q3 PIE and Mercury with Deepseek and Qwen

All comparative experiments were conducted with fairness as a priority. For Table 7, we used CodeLlama because PIE only provides fine-tuned CodeLlama. For Qwen and DeepSeek, we compared SwiftCoder with Mercury only. The table below shows that both Mercury and SwiftCoder improve the efficiency of LLM-generated code, while SwiftCoder achieves better results compared to Mercury for all LLMs.

We would like to thank you for your insightful comments and suggestions. We provide detailed responses point by point below. We hope that our clarifications, additional experiments, and responses can address your concerns and lead you to consider increasing your rating of our work.

W1 Limited novelty

Our paper's primary contribution is application-driven: we introduce the first multilingual code efficiency instruction tuning dataset. As the ICML guidelines note, "originality need not mean wholly novel methods… a novel dataset... match the needs of the user." Our dataset addresses the critical real-world need for efficient code generation, enabling researchers to fine-tune models for improved performance.

Next, our empirical study provides valuable insights by establishing the correlation between training data efficiency and LLM-generated code efficiency - a relationship not previously well understood in the literature. This finding can inspire future construction of SFT datasets that optimize for both correctness and efficiency. Unlike prior methods, PIE [1], which relies on human programmers to write efficient solutions, or Mercury [2], which selects efficient solutions from pre-existing human-written code, SwiftCoder introduces a fully automated framework that transforms task descriptions into efficient solutions without human intervention.

Finally, SwiftCoder offers greater generalizability by including optimized tasks across 5 programming languages, contrasting with PIE's focus on C++ and Mercury's focus on Python. SwiftCoder's significantly larger scale—comprising 65,710 tasks compared to Mercury's 1,889 and PIE's 1,474—provides more training data, resulting in models that demonstrate superior performance.

[1] Learning performance-improving code edits. ICLR 2024

[2] Mercury: A code efficiency benchmark for code large language models. NeurIPS 2024

W2 Clarify the method in PIE was compared

We compared against the best-performed All strategy fine-tuned LLM by PIE in the paper. Now, we included comparisons with other PIE strategies in the table below. SwiftCoder outperforms all PIE variants on ET and TMU (e.g., further reduces ET and TMU from 19.6% and 72.3% to 22.5% and 75.9% for All strategy).

W3 Results with more samples

Our evaluation follows the setup of prior works (EffiLearner, Mercury), where they use greedy decoding. We conducted additional evaluations for pass@10 (T = 0.8) and will add pass@100 results in our camera-ready manuscript due to rebuttal time limitations. The table below presents our findings across different sampling scenarios. SwiftCoder maintains its efficiency advantage across all configurations. Even when baselines are sampled 10 times, they still cannot match the efficiency of 1 SwiftCoder sample. When both approaches use the same number of samples, SwiftCoder consistently generates more efficient solutions. Specifically, when both use 10 samples, SwiftCoder improves pass rates by 4-24 percentage points while maintaining better efficiency.

Reviewer jr1T Ref Table 1

W4 High variance on local machine

We use an open-sourced code efficiency platform Monolith for evaluation. Similar to the PIE system simulator, Monolith provides consistent performance evaluation but with broader language support. We also report variability results in the Appendix Table 8, where we execute multiple tasks (32 concurrent tasks) from EffiBench 5 different times on Monolith. Our results demonstrate consistent performance across different execution runs even under concurrent program load, with a coefficient of variation below 3% across all metrics. This consistency shows that our measurement approach provides reliable results without requiring specialized system simulators.