Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization

Dear Reviewer UfKJ,

Thank you for your constructive and insightful reviews. We hope we can address your concerns as follows:

Q1: Would a history-aware model perform better?

We conduct the experiment as you suggested, and show results in the following table. The results below show that the history-aware model did not outperform the single-turn model in the current setting.

We attribute the underperformance of history-aware model to two primary factors: 1) Distribution Shift: The performance drop is likely caused by a distribution shift, as the training format (single-turn) differs from the multi-turn, history-aware format used during inference. 2) Training Challenges: While fine-tuning the model with multi-turn format could mitigate the distribution shift, this approach introduces the significant challenge of long-term credit assignment, a well-known difficulty in Reinforcement Learning. We agree that history-aware modeling is a valuable and promising research direction. We appreciate the insightful feedback and plan to explore this more thoroughly in our future work.

Q2: Model Scale Generalization

While this work focuses on exploring the methodology of iterative efficient optimization rather than the scalability of larger models, we do agree that it is critical to evaluate our frame on larger models. Due to the rebuttal time limitation, we conducted additional experiments on Qwen2.5 7B as you suggested. The result demonstrates our iterative optimization framework can be generalized to larger models.

Q3: Baseline Comparisons in IOF Iterations

We thank the reviewer for this insightful suggestion. To better isolate the benefits of our framework versus our model training approach, we have conducted new experiments applying IOF to several baseline models as requested. Specifically, We ran the IOF process for 8 iterations on the vanilla Qwen-2.5-3B, Qwen-2.5-7B, and OpenAI GPT-4o models. The results are presented below:

As the data shows, larger models do achieve better performance through the IOF iterations. However, the improvement in these vanilla models is significantly less than that of our Afterburner models. This demonstrates that while the IOF framework itself is beneficial, our proposed training paradigm are the primary drivers of the substantial performance gains we report. (Note: As closed-source models have not disclosed their training details for, we cannot determine if similar reinforcement learning techniques were already incorporated into their development pipeline.)

Q4: Venus Justification

We recognize the importance of a clear comparison with prior work and will clarify this in the revised paper.

• Multilingual Coverage: A primary contribution is that Venus is the first multilingual code efficiency benchmark, covering six languages. Prior work, including Mercury [1], EffiBench [2], and EvalPerf [3], focuses exclusively on Python.
• Solutions Diversity: Venus offers a significantly larger and more diverse set of reference solutions. While EffiBench uses a single baseline and Mercury averages 18.4 solutions per problem, Venus provides an average of 79.3 solutions per problem for each language. This massive increase directly contributes to the statistical diversity of performance metrics, as shown in the time and memory distributions in Appendix Figure 7.
• Evaluation Dimensions: As a direct extension of Mercury, which only measures execution time, Venus evaluates execution time, memory usage, and their integral, providing a more holistic assessment of code efficiency.

We will incorporate this detailed comparison, currently summarized in Appendix C, into the main body to make the justification for Venus more clear.

Q6: IOF Efficiency Spectrum

It's a good suggestion! We conduct the experiment to start Afterburner SFT and Afterburner GRPO with the efficiency spectrum. Since we are not allowed to upload images in the rebuttal response, we present iterative Beyond-I results beginning from the inefficient (0% Beyond-I) to the efficient (50% Beyond-I) solutions in the following table. I will update the complete efficiency spectrum figure in the revision.

We argue that a key advantage of RL is exploring more sufficiently in the exploitation space than SFT or DPO. Therefore, it is more robust if the inference distribution aligns with the training distribution.

Q7: Performance Ceilings

Yes, In our experiments, performance ceilings are observed from IOF round 7 on both the Venus and APPS benchmarks, as shown in Figures 5 and 9. We hypothesize that these ceilings result from the trade-off between task complexity and the model capability. We conduct an additional experiment on three difficulty categories. For a model with a fixed capability, more difficult tasks require more iterations to converge on an optimal solution, whereas on simpler tasks, performance saturates and hits this ceiling much earlier. The following table shows that the iteration of performance ceilings (less than 5% efficiency gap with the final iteration) on Venus and APPS:

L1: Limitation Location

Thank you for your suggestion. We will move it from the appendix to the main body.

We hope our explanation can address your concern. Please ask follow-ups if you have any other questions. Thank you!

[1] Mercury: A code efficiency benchmark for code large language models.

[2] Effibench: Benchmarking the efficiency of automatically generated code.

[3] Evaluating language models for efficient code generation.

I thank the authors for taking the time to conduct additional experiments to address my concerns. However, I still have remaining reservations regarding Q1 Q5 and Q6, which I elaborate on below:

Q1: Would a history-aware model/framework perform better?

It remains unclear whether a history-aware model would yield better performance. The current experiment evaluates a single-turn trained model in a multi-turn setting, introducing distribution shift and making the results difficult to interpret. A more direct experiment would involve explicitly including the history of code improvements and corresponding efficiency feedback from the iterative optimization trace in the prompt. Specifically, to predict $C_i$, the model could be conditioned on {C_0,M_0,…, C_{i−1}, M_{i−1}.

A comparison between this history-aware design and the current approach could yield important insights into the potential benefits of leveraging longer optimization traces—insights that are currently missing from the paper.

Q5: Missing Benchmark Analysis on existing Datasets like Mercury and EffiBench

Q6: IOF Efficiency Spectrum

While the additional results show efficiency gains for both less and more efficient initializations, it remains unclear how far these improvements can be pushed with additional iterations. Notably, the last two columns indicate there is still substantial headroom. It would be helpful to understand whether starting from a less efficient code leads the IOF toward a global minimum or if the model tends to get stuck in local optima. Exploring this behavior over extended optimization horizons would help contextualize the limits and strengths of the proposed framework.

Dear Reviewer z3gd:

Thank you for your constructive and insightful reviews! We hope we can address your concerns as follows:

Q1: Scaling to API-dependent Real-world Code

Thank you for this insightful question. While this paper focuses on establishing a robust method for function-level optimization, we agree that scaling to repository-level benchmarks like SWE-bench is a critical next step. We view our current work as the foundational layer for this larger goal and introduce the following strategy to address the main challenges:

• API Dependencies: To handle API-dependent code, we may integrate Retrieval-Augmented Generation (RAG) and static analysis. This allows our model to dynamically pull in relevant API definitions, usage patterns, and code snippets from the broader repository, providing the necessary context for effective optimization.

• Context Management: Real-world repositories are too large for current model context windows. We could employ a divide-and-conquer approach. This involves identifying and isolating the most relevant functions and modules for optimization, making the task computationally tractable.

• Test Generation: Verifying changes across an entire repository is notoriously difficult. Our strategy would focus on generating targeted unit tests specifically for the modules undergoing optimization. This ensures the correctness and efficiency of our changes in a localized, verifiable, and recursive manner.

In summary, while scaling our method to repo-level presents a significant research challenge, we hope that structured approach provides a clear path forward.

Q2: Hyperparameters Sensitivity

In our experiment, we observed that a dominant weight for functional correctness (beta_c​ ≥ 0.5) is essential for stable training, as values below this threshold often led to training crash. To balance our objectives, we set a fixed beta_c ​​= 0.5 while gradually increasing the code efficiency weight (beta_e) from 0.3 to 0.5. This strategy ensures the model first learns to produce correct code before optimizing for efficiency.

Q3: Statistical Significance

We employ bootstrap sampling to construct 95% confidence intervals for our efficiency metrics. Here is the detailed procedure:

1. Data Collection: For each individual problem, we generate 4 unique solutions (temperature=1) and measure the performance of each one 16 times. This results in a performance set of 64 measurements (4 solutions×16 measurements).
2. Bootstrap Sampling: We then create 128 bootstrap samples. Each sample is formed by drawing one measurement with replacement from the original performance set.
3. Distribution Generation: We generate an empirical sampling distribution for each efficiency metric by aggregating the 128 bootstrap samples over all problems.
4. Score Reporting: We report the mean of these bootstrap distributions along with their 95% confidence interval. This method allows for a robust estimation of efficiency performance and its statistical uncertainty. We provide a detailed explanation in Appendix G.

Q4: Contamination-free Evaluation

We evaluate our methods on LiveCodeBench and report the pass rate below. The results confirm that our conclusions hold. Afterburner-GRPO shows substantial and consistent improvement as the number of iterations increases, while Afterburner-SFT plateaus quickly. This trend is consistent with our findings on the APPS and Venus benchmarks, demonstrating the general effectiveness of our approach.

Q5: Some typos like "paassed" in line 135.

We will fix this typo in the revision.

We hope our explanation can address your concern. Please ask follow-ups if you have any other questions. Thank you!

Dear Reviewer xncp,

Thank you for your positive feedback!

We have prepared the artifacts of Venus, Afterburner, and Monolith. Due to the new rebuttal policy this year, we cannot share external links at this time. We are committed to reproducibility and will update these artifacts in the final version immediately after the review process.

We hope it can address your concern. Thank you!

I am glad to see that the authors commit to release their data and artifacts. I am keeping my score to 5.

Dear Reviewer ovks,

Thank you for your constructive and insightful feedback. We are encouraged that you found our work valuable. Below, we address per questions and suggestions raised.

Q1 & Q2: Solution Correctness in Venus

All Venus code snippets are exclusively collected from accepted (including the official ground truth) solutions on Leetcode, meaning they passed all test cases. We agree that this validation process is a critical detail. Due to space constraints, we included the full pipeline in Appendix C.1. For improved clarity, we will add a brief explanation and a direct reference in Section 5.1 Dataset Recipe:

“Venus Python subset contains 2,181 algorithmic problems, each accompanied by a validated test case generator and an average of 106.6 validated human solutions... For each LLM-generated test case input, we follow the paradigm of Mercury, where we execute them through all collected solutions from LeetCode, and only keep those cases having consistent outputs over all correct solutions. More details can be found in Appendix C.1 Rigorous Validation.”

Q3: Model Generalization

We evaluated our Venus-tuned model on another dataset APPS [1]. Unlike existing code efficiency benchmarks such as Mercury [2], EffiBench [3] and Venus, which consist of LeetCode-style questions, the APPS dataset features distinct problem formats. As shown in the tables below, Afterburner demonstrates a consistent performance improvement pattern on the APPS dataset. Please refer to Appendix Figure 9 for a complete visualization. We also conducted an additional experiment on LiveCodeBench [8] to demonstrate the consistent performance gain on the contamination-free benchmark.

We acknowledge that we missed the reference to the APPS results in the main body. Thank you for pointing it out! We will add [5,6] into related work and append the following statement in Section 6.2 to reference these findings in the main text:

“...To verify whether our models can generalize to out-of-distribution questions, we evaluated Afterburner on the APPS dataset [1]. As illustrated in Appendix Figure 9, Afterburner demonstrates a similar pattern of performance improvement, confirming its effectiveness on problems with distinct data distribution.”

Q4 & L4: Native "SFT" Baseline We agree that a comparison with a native SFT baseline is valuable. We interpret "native SFT" in two paradigms and address both below:

• Code Refinement SFT: trains a model using (problem_description, original_code) to generate an improved_code. This is conceptually very similar to our Afterburner SFT baseline, which augments the input with performance feedback from the original code. We provide its results in Figure 5.
• Direct Generation SFT: trains a model using (problem_description) to directly generate the improved_code. We assume your question refers to this paradigm. We initially omitted this baseline because it lacks a performance feedback mechanism, making a direct comparison with our feedback-driven method is unfair. Furthermore, as noted by prior work Mercury [2], this direct approach can be susceptible to catastrophic forgetting, especially on smaller models.

We agree including the native SFT baseline in this work could serve as a straightforward baseline. Prompted by your suggestion, we have conducted this additional experiment to fine-tune the 'Qwen2.5-3B' model using the same hyperparameters as our main experiments. The results are presented in the table below. We observed model collapse on the native SFT baseline.

Q5 & L3: Model Generalization

Due to the rebuttal time limitation, we conducted additional experiments on another model ‘Qwen2.5 7B’ and 'GPT-4o'. As the data shows, larger models do achieve better performance through the IOF iterations. However, the improvement in these vanilla models is significantly less than that of our Afterburner models. This demonstrates that while the IOF framework itself is beneficial, our proposed training paradigm are the primary drivers of the substantial performance gains we report. For other model architectures, such as Llama, we will complete its experiment results in the revision.

Q6 & L2: Where do the test cases used in the iterative improvement stage come from?

We use the same pre-generated test cases for both iterative improvement and evaluation. This approach is valid because the model only receives high-level execution feedback—such as pass status, running time, and memory usage—without ever seeing the specific contents of the test cases. This design prevents information leakage and ensures a fair evaluation.

Q7: When computing the efficiency metric during evaluation, did you filter out incorrect code snippets, or did you only consider correct ones?

During the efficiency evaluation, we didn’t filter out incorrect code. The monolith sandbox will return the default maximum metric values (time: 90 s, memory: 1,048,576 kb, integral: 90 * 1,048,576 = 94371840), so their percentile ranks will be 0. You can consider these efficiency metrics (time, memory, integral) as efficiency-weighted functional correctness (pass).

L1: Evaluation Metric Rationale.

Thank you for this insightful suggestion! We agree that using absolute efficiency metrics has its advantages, especially 1) it is easy to calculate; 2) it doesn't rely on the distribution of human solutions; 3) it may work if we evaluate all models on the same machine. We appreciate your perspective and we have revised the paper to clarify our rationale per your suggestion:

• Benchmark Standardization: We follow the relative efficiency metrics used in most established code efficiency benchmarks, such as Mercury [2], EffiBench [3], EvalPerf [4] and ENAMEL [7].
• Result Comparability: Since Venus is a code efficiency benchmark, we hope the reported score is comparable. While absolute efficiency metrics may be infeasible to compare across machines, relative metrics can handle this issue. For example, a solution that is faster than 80% of human solutions on one machine is likely to be similarly performant relative to the same set of solutions on another machine.
• Performance Normalization: Absolute runtimes can vary by orders of magnitude across different test cases. This large range means that a model's aggregate score can be dominated by a few long-running outliers, masking subtle but significant improvements. Our relative metric normalizes performance against the distribution of human-written code, providing a more stable and fine-grained measure of efficiency improvement.
• Training vs. Evaluation: You correctly note that we use absolute metrics internally. We do this specifically during model training, where the raw execution time and memory provide simple and effective reward signals. Since training occurs in an isolated sandbox on a single hardware setup, comparability is not a concern. Furthermore, for reinforcement learning, an unbounded absolute metric allows the model to explore solutions that surpass the efficiency of any known human code. However, for the final evaluation, the need for a normalized, comparable, and robust benchmark makes the relative metric the superior choice.

We hope our explanation can address your concern. Please ask follow-ups if you have any other questions. Thank you!

Reference

[1] Measuring coding challenge competence with apps.

[2] Mercury: A code efficiency benchmark for code large language models.

[3] Effibench: Benchmarking the efficiency of automatically generated code.

[4] Evaluating language models for efficient code generation.

[5] DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination.

[6] DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation

[7] How efficient is llm-generated code? a rigorous & high-standard benchmark.

[8] LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.

Dear Reviewer ovks,

Thank you for your insightful follow-up questions. We are pleased that our previous response resolved most of your concerns, and we appreciate the opportunity to provide further clarification.

Q6 & L2: Where do the test cases used in the iterative improvement stage come from?

You raised an important point regarding the source of test cases used during the iterative inference stage. We'd like to clarify our original process and present a new experiment to demonstrate its robustness.

In the current iterative inference stage, we ran pre-generated 100 test cases on the original solution and fed its holistic performance metrics (Passed, Time, Memory, Integral) back into the Afterburner prompt template. For each iteration, we aggregate these metrics as the evaluation results. Notably, since ‘Passed’ is True if and only if the original solution passed all the test cases, Afterburner is not able to get individual test case information and hardly overfit all of them during the inference stage. In the real-world applications, we can further eliminate the potential bias (or overfitting) by increasing the number of test cases.

<…>
## Problem Description
{problem_description}

## Original Solution
{original_solution}

## Original Performance
Passed: {original_passed} / Time: {original_time} / Memory: {original_memory} / Integral: {original_integral}
<…>

While the chance of test case leakage is rare, we agree that using different test cases for inference and evaluation is more rigorous. To address this concern, we set the existing test cases in the dataset as the public test cases. Then we follow the setup of EffiLearner [2] to generate 100 private test cases using corresponding test case generators. To avoid potential data leakage, we also filter out identical cases in the public test cases. During the iterative optimization stage, we use the public test cases to get the overhead information and optimize the inefficient code. After generating the improved code, we then use the private test cases to measure the performance of the improved code. The evaluation results are shown below:

Afterburner-GRPO (inference on public test cases and evaluation on public test cases)

Afterburner-GRPO (inference on public test cases and evaluation on private test cases)

We can observe that the performance between the current evaluation (using private test cases) and the previous evaluation (using public test cases) is very similar. Their Pearson correlations (Pass: 0.9939, Beyond-T: 0.9852, Beyond-M: 0.9955, Beyond-I: 0.9986) demonstrate that our presented evaluation is not biased to the public test cases, i.e., only providing the overhead information to LLMs would not leak the test cases and cause the Afterburner-generated code to be biased to these test cases. We will update these evaluation scores using private test cases in the revision. We hope the results and explanation can address your concern.