Execution Guided Line-by-Line Code Generation

Thank you for your encouraging and thoughtful review. We particularly appreciate your recognition of the method’s novelty and the clarity of its presentation. Your feedback has been highly motivating, and we’ve made several meaningful revisions to further improve the paper. Below, we address your comments point by point.

Significance: Benchmark Diversity

We have substantially expanded our evaluation beyond MBPP to address concerns about benchmark diversity and broader applicability. Our updated experiments include the additional five benchmarks:

• HumanEval: A foundational and widely adopted benchmark.
• MBPP-ET and HumanEval-ET: Extended test suites that add 100 hidden test cases per task, offering a more rigorous evaluation of generalization and robustness - helping mitigate overfitting to visible tests.
• CodeContests: A challenging competitive programming benchmark that requires multi-step reasoning and broader generalization.
• DS-1000: A domain-specific benchmark targeting data science tasks. In this dataset, the problem is defined without heavily relying on test cases (one sample is shown as part of the problem prompt).

Notably, the benchmarks HumanEval, CodeContests, DS-1000 contain hidden test cases which help mitigate overfitting to visible tests.

Our method achieves state-of-the-art performance across all major benchmarks: MBPP (96.6%), MBPP-ET (73.0%), HumanEval (99.4%), HumanEval-ET (89.0%), CodeContests (60.6%), and DS-1000 (69.9%). We compared against every strong baseline we could find, both from PapersWithCode and directly from papers. We weren’t just relying on reported results, we actively tried to reproduce methods ourselves wherever possible. In many cases, we reran existing methods on the model (DeepSeek‑V3‑0324), to ensure a fair comparison. When code didn’t work with DeepSeek or wasn’t available, we adapted or re-implemented it, and documented any limitations.

These results show that our method is not limited to MBPP. It generalizes effectively across diverse levels of complexity, from foundational problems to competitive programming tasks and domain-specific benchmarks - demonstrating the broader applicability of our method. Due to character limits, we have distributed the result tables across fegN and FLp2.

Effect of More Trials on Accuracy

Thank you for the suggestion. We agree that showing how accuracy scales with increased inference-time compute would provide valuable insight into the efficiency of our method. In our revised version, we will include such a plot.

Questions

How does the computational cost (e.g. API-call tokens) of the EG-CFG method compare with other baselines? If the costs differ significantly, the comparison may not be fair.

First, native parallelization is a key strength of our method. We leverage the low price of existing APIs and the ability to parallel them, which is a growing trend among today's agents. Other baseline methods mentioned in our paper can also have access to full parallel compute, but by design, they cannot fully leverage it. They rely on sequential retries, where each iteration depends on the output of the previous one. This makes it impossible for them to parallelize inference on a single task, even if the computational resources are available. Moreover, in (5/6) of the benchmarks: HumanEval, HumanEval-ET, MBPP-ET, CodeContests, DS-1000 there are hidden-test cases. There are hidden test cases. This means our method cannot rely on extensive token usage to overfit the public tests in order to succeed. Additionally, In order to rigorously make sure the comparison is fair, we ran trials using the DeepSeek-V3 model with two baseline methods: MGDebugger and MapCoder, on the MBPP benchmark, using a much higher number of retries which leads to significantly more extensive token usage. For MGDebugger, we increased the retry count from 5 to 200 - x40 increase, resulting in ~x40 tokens. This resulted in only an improvement of 34 out of 500 (6.8 percent), out of the 86 examples it originally got wrong. For MapCoder, we similarly increased retries from 5 to 200 (40x tokens), and saw an improvement of just 8 out of 500 (1.6 percent), out of 64 examples it originally failed. In both cases, the improvement was achieved with a 40x increase in runtime and is still considerably below our performance on MBPP. This supports our conclusion that the advantage of our method is not due to higher token usage, and therefore the comparison is fair. In order to clarify this issue more we will add discussion about this point in our revised version.

The MBPP dataset contains relatively simple tasks with few variables. How does the method perform when applied to problems with more variables and inputs? This could increase the complexity of inline execution feedback and potentially result in an overly long prompt.

Our method is completely agnostic to the number of variables in the input or whether there are variables at all in the input. This is since it is agnostic to the format of the execution feedback (line 196). It relies on implicit supervision, which is an important part of our novelty, as we treat execution feedback as a soft guidance signal. The model interprets execution behavior on its own, without being explicitly told what is correct or relying on any specific format. This allows it to self-verify regardless of the quality of the test cases. Moreover, we believe that in many cases, it's actually easier for the model to self-verify logic when it is broken down into smaller, more atomic steps. To encourage this, we introduced the “long-code” prompt template (see Figure 4), which encourages the model to produce longer, step-by-step solutions with simpler lines of code. Regarding rigorous evaluation, we have addressed your point in “Significance: Benchmark Diversity”. The benchmarks CodeContests and DS-1000 are indeed more complex, with many more inputs resulting in longer prompts, and our method still outperforms all known results on these benchmarks.

We sincerely appreciate the time and care you put into your review. Your feedback helped us improve the clarity and depth of the paper. If you feel the changes address your concerns, we would be grateful if you consider updating your score. Thank you again for your thoughtful input.

We sincerely appreciate your detailed and constructive review. Your positive remarks on the novelty of our approach were encouraging. Your suggestions helped us identify key areas for clarification and refinement of our work. We've made substantial efforts to improve the paper based on your feedback. Below, we respond to each of your comments in detail.

Executable Extraction Generality

EG-CFG is language-agnostic. It only requires the ability to extract executable code segments, execute them, and collect execution feedback. The current implementation is specific to Python and uses the ast library for extracting executable code and a Python debugger that we modified specially for extracting the execution feedback. These can definitely be implemented in other languages.

Benchmark Diversity

We have substantially expanded our evaluation beyond MBPP to address concerns about benchmark diversity and broader applicability. Our updated experiments include the additional five benchmarks:

• HumanEval: A foundational and widely adopted benchmark.
• MBPP-ET and HumanEval-ET: Extended test suites that add 100 hidden test cases per task, offering a more rigorous evaluation of generalization and robustness - helping mitigate overfitting to visible tests.
• CodeContests: A challenging competitive programming benchmark that requires multi-step reasoning and broader generalization.
• DS-1000: A domain-specific benchmark targeting data science tasks. In this dataset, the problem is defined without heavily relying on test cases (one sample is shown as part of the problem prompt).

Notably, the benchmarks HumanEval, CodeContests, DS-1000 contain hidden test cases which help mitigate overfitting to visible tests.

Our method achieves state-of-the-art performance across all major benchmarks: MBPP (96.6%), MBPP-ET (73.0%), HumanEval (99.4%), HumanEval-ET (89.0%), CodeContests (60.6%), and DS-1000 (69.9%).

We compared against every strong baseline we could find, both from PapersWithCode and directly from papers. We weren’t just relying on reported results, we actively tried to reproduce methods ourselves wherever possible. In many cases, we reran existing methods on the model (DeepSeek‑V3‑0324), to ensure a fair comparison. When code didn’t work with DeepSeek or wasn’t available, we adapted or re-implemented it, and clearly documented any limitations.

These results show that our method is not limited to MBPP. It generalizes effectively across diverse levels of complexity, from foundational problems to competitive programming tasks and domain-specific benchmarks - demonstrating the broader applicability of our method.

We include detailed benchmark comparison tables below (other tables can be found in reply to FLp2). These present accuracy and retry success rates (RSR) across all datasets and baselines, including both our runs on DeepSeek‑V3‑0324 and results reported in the literature.

Table: Performance on the CodeContests benchmark.
Our proposed EG-CFG achieves a new state-of-the-art overall accuracy on CodeContests.
The runs on DeepSeek-V3-0324 are by us, and all other results are quoted from the literature.
*LPW results were obtained on a custom test-set; the published code was not compatible with evaluating on DeepSeek.
**Reported by the MapCoder paper. ***Reported by the CodeSim paper.

Extending Beyond Self-Contained Functions.

In order to evaluate our method we used the benchmarks that we mentioned earlier. All of them indeed build upon the assumption of a single entry point “self-contained” function setting. As we explicitly wrote in the limitations section (line 326) our method does not exploit a task-decomposition strategy. We believe the core idea of guiding generation via execution feedback can be extended to more complex settings of self contained function and can be combined with task-decomposition and sequential refinements approaches (lines 327-329). Regarding the dependency on unit-tests, our method is less dependent on test cases than the existing methods, since in most stages of the generation, it relies on implicit supervision, which is a central part of our method. The execution behavior is used as a soft feedback signal, interpreting outcomes without being explicitly told what is correct. This allows it to self-verify regardless of the quality of the test cases. We explicitly discuss this in the Introduction section (51-55) and highlight it as a key differentiator from prior approaches in the Related Work section. We note in the limitation section that the method is applied to benchmarks that have executable test cases. This is the most direct way to compare with the existing relevant methods. However, it is a soft limitation since test cases can also be generated by the model itself from the task instruction. We also note that in DS-1000, each problem includes only a single input-output example, making it impractical to rely heavily on such examples during code generation.

Questions

Line 158 - p_inst is p_0 and p_task combined? Line 170 - where does p_sol fit into this whole process? This is referring to what is eventually regarded as the solution generated by the LM, which is then evaluated for correctness?

Correct. p_0 is a prompt template. We incorporate the task components (text definition of the task, public tests, function name) to the prompt template in-order to form the instruction prompt p_inst. We explicitly explain about the prompt template types in lines 160-163 and include examples of these prompt templates in the appendix A (Figures 3 and 4). Thank you for pointing this out - in our revised paper, we will clarify that and make it more explicit. p_pre refers to the full model output in response to the instruction prompt p_inst. It includes both the intermediate reasoning and the solution block (which contains the “initial solution”). We define i_sol as the position of the last code-start token (which is defined in p_0) in p_pre, marking the beginning of the solution block. The solution prompt p_sol is constructed by taking the prefix of p_pre up to i_sol and appending the new tokens generated by our decoding algorithm, until a code-end token is generated. The tokens generated beyond i_sol form the final solution returned by our algorithm that is evaluated for correctness.

Figure 2 – What is the highlighted “...” character referring to?

The “...” in Figure 2 indicates parts of the execution trace that were omitted to simplify the figure.
Below is the complete version of the trace (marked with **)

Invocation:

first_non_repeating_character("aabc")

Execution Trace:

s = 'aabc', char_count = {}

char = 'a' -> char_count = {'a': 1}

**

char = 'a' -> char_count = {'a': 2}

char = 'b' -> char_count = {'a': 2, 'b': 1}

char = 'c' -> char_count = {'a': 2, 'b': 1, 'c': 1}

char = 'a' -> count = 2 → skip

char = 'a' -> count = 2 → skip

**

char = 'b' -> count = 1 → return 'b'

Is there a reason you didn’t compare against agent scaffolds such as SWE-agent, InterCode, CodeAct, or OpenHands? It feels like those works explore the inference-from-execution-across-turns approach that maybe interesting to compare against. At this point, a lot of the code completions benchmarks (e.g. MBPP, HumanEval) feel very saturated, as a strong model like the GPT o series or Claude 3.7/4 could zero-shot get 90%+ on the benchmark. I feel like the performance gains could be more meaningful if evaluated in a more challenging setting.*

As described in the “benchmark diversity” we addressed your point regarding evaluation in more diverse and challenging settings of code generation tasks. Regarding the benchmark that you proposed, as we understand, these benchmarks (SWE-Agent, InterCode, CodeAct, and OpenHands) are not purely code generation benchmarks in the traditional sense. Rather than focusing solely on evaluating a model’s coding or algorithmic skills, these benchmarks assess broader agentic abilities that reflect task-solving ability in interactive environments, such as interacting with external tools or completing multi-step workflows. As we wrote in the conclusion section (lines 310-313), we believe that the core idea of guiding the inference process by the execution feedback can extend to broader settings, which involve combined textual and visual feedback from interactive environments and external programs.

What is “minimal trace” in Table 3?

“Minimal trace” refers to a minimal execution trace that captures only the final values and types of all variables in scope after the last line of the solution function has completed execution. We will extend the current clarification in lines 288-289.

Thanks again for really diving into our paper. We have done our best to improve the work based on your suggestions. If you feel the updates address your concerns, we’d truly appreciate it if you considered updating your score.

Thank you for your thoughtful and detailed review. Your comments helped us identify several areas that we want to clarify and we have worked hard to improve the paper accordingly. Below, we address each of your points in turn.

Clarity Related Concerns

Ambiguity Between Token Steps and Candidate Extensions

On the one hand, we used token notation in Eq. (3) and (7), as these parts are related to token sampling and indexing of specific tokens. On the other hand, we chose to use the granularity of code snippets in the candidate continuation sampling in Eq. (4), because after sampling candidate continuation programs we perform deduplication and execution which are both operating at the granularity of full code snippets rather than individual tokens. For consistency and extra clarity, we will revise Eq. (4) to use token notation as well.

Outer Loop Behavior and Termination Conditions

The outer loop is the execution of each agent as an independent inference run, which terminates (returns) when the first correct solution is found. Otherwise, inference continues across the remaining agents (lines 229 to 231). Each inference run is independent and allows parallel execution (line 274). We did not describe a retry mechanism because there is no such mechanism, i.e., each agent runs exactly once. To make this clearer, we will revise the Method section to describe the outer loop execution as such and will include corresponding pseudocode.

Execution Trace Format Was Unclear

Our approach is agnostic to the format of the execution feedback (line 196). This is an important part of our novelty, as we treat execution feedback as a soft guidance signal. The model interprets execution behavior on its own, without being explicitly told what is correct or relying on any specific format (lines 141–142). The structure of the execution trace used in our implementation is described in lines 197–200. To further illustrate this, we included Figure 2, which shows a concrete example of a trace. In our revision, we will formalize the trace structure so that the exact format is fully explicit.

Clarifying the Inference Procedure

In our revision, we will include full pseudocode outlining the complete procedure. We will split it into two algorithms: Algorithm 1 describes the outer loop that executes agents in parallel, each executing our core algorithm “Execution-Guided Inference Loop” (Algorithm 2). In addition, we will add more prompt examples and a diagram.

Benchmark Diversity

We have substantially expanded our evaluation beyond MBPP to address concerns about benchmark diversity and broader applicability. Our updated experiments include the additional five benchmarks:

• HumanEval: A foundational and widely adopted benchmark.
• MBPP-ET and HumanEval-ET: Extended test suites that add 100 hidden test cases per task, offering a more rigorous evaluation of generalization and robustness - helping mitigate overfitting to visible tests.
• CodeContests: A challenging competitive programming benchmark that requires multi-step reasoning and broader generalization.
• DS-1000: A domain-specific benchmark targeting data science tasks. In this dataset, the problem is defined without heavily relying on test cases (one sample is shown as part of the problem prompt).

Notably, the benchmarks HumanEval, CodeContests, DS-1000 contain hidden test cases which help mitigate overfitting to visible tests.

Our method achieves state-of-the-art performance across all major benchmarks: MBPP (96.6%), MBPP-ET (73.0%), HumanEval (99.4%), HumanEval-ET (89.0%), CodeContests (60.6%), and DS-1000 (69.9%). We compared against every strong baseline we could find, both from PapersWithCode and directly from papers. We weren’t just relying on reported results, we actively tried to reproduce methods ourselves wherever possible. In many cases, we reran existing methods on the model (DeepSeek‑V3‑0324), to ensure a fair comparison. When code didn’t work with DeepSeek or wasn’t available, we adapted or re-implemented it, and clearly documented any limitations.

These results show that our method is not limited to MBPP. It generalizes effectively across diverse levels of complexity, from foundational problems to competitive programming tasks and domain-specific benchmarks - demonstrating the broader applicability of our method.

We include detailed benchmark comparison tables below (other tables can be found in reply to fegN and FLp2). These present accuracy and retry success rates (RSR) across all datasets and baselines, including both our runs on DeepSeek‑V3‑0324 and results reported in the literature.

​​Fairness of Computational Comparison

First, native parallelization is a key strength of our method. We leverage the low price of existing APIs and the ability to parallel them, which is a growing trend of today's agents. Other baseline methods mentioned in our paper can also have access to full parallel compute, but by design, they cannot fully leverage it. They rely on sequential retries, where each iteration depends on the output of the previous one. This makes it impossible for them to parallelize inference on a single task, even if the computational resources are available. Moreover, in (5/6) of the benchmarks: HumanEval, HumanEval-ET, MBPP-ET, CodeContests, DS-1000 there are hidden-test cases. There are hidden test cases. This means our method cannot rely on extensive token usage to overfit the public tests in order to succeed.

Additionally, In order to rigorously make sure the comparison is fair, we ran trials using the DeepSeek-V3 model with two baseline methods: MGDebugger and MapCoder, on the MBPP benchmark, using a much higher number of retries which leads to significantly more extensive token usage. For MGDebugger, we increased the retry count from 5 to 200 - x40 increase, resulting in ~x40 tokens. This resulted in only an improvement of 34 out of 500 (6.8 percent), out of the 86 examples it originally got wrong. For MapCoder, we similarly increased retries from 5 to 200 (40x tokens), and saw an improvement of just 8 out of 500 (1.6 percent), out of 64 examples it originally failed. In both cases, the improvement was achieved with a 40x increase in runtime and is still considerably below our performance on MBPP.

This supports our conclusion that the advantage of our method is not due to higher token usage, and therefore the comparison is fair. In order to clarify this issue more we will add discussion about this point in our revised version.

Questions

How was beam search performed with lines of code as the units for the length of the samples, since a fixed number of lines of code doens't translate into a fixed number of tokens?

Sampling is performed auto-regressively with a stopping condition based on the number of newline characters (\n) encountered. Specifically, generation stops once the target number of lines, denoted by d, is reached. We do not assume a fixed number of tokens per line. We explicitly wrote in lines 175-176.

In equation 10, what happens if some value of gamma causes M_CFG to become a positive number? I think this can happen unless.

Thank you for clarifying this point. For some values of gamma, the guided logits M_CFG​ can include positive values and won’t sum to a valid probability distribution. When using argmax decoding rather than sampling, normalization isn’t necessary. Softmax wouldn’t change the selected token, so the behavior of the algorithm remains the same. However, it is possible to normalize the distribution to make it compatible with the standard inference loop that allows sampling.

When does the loop end if a correct solution is not found (i.e. how long do you try before you give up)?

We do not use a retry mechanism within a single agent. Instead, we run multiple agents in parallel, each following a different reasoning path by varying dynamic signal parameters. As soon as one agent finds a correct solution, the others are terminated, and that solution is returned. If none succeed, no solution is reported for that task - “not found”. We can add a retry mechanism, which was shown to be beneficial to iterative refinement methods.

What happens to the initial solution $w^0_{pre}, \cdots, w_ What is the relationship between p_sol and p_pre ?

p_pre refers to the full model output in response to the instruction prompt p_inst. It includes both the intermediate reasoning and the solution block (which contains the “initial solution”). We define i_sol as the position of the last code-start token in p_pre, marking the beginning of the solution block. The solution prompt p_sol is constructed by taking the prefix of p_pre up to i_sol (We ignore the tokens representing “initial solution” which come after that prefix) by appending the new tokens generated by our decoding algorithm, until a code-end token is generated. The tokens generated beyond i_sol form the final solution returned by our algorithm.

Line 168: "last few example inside the template" should be "last few-shot example inside the template"?

Correct - fixed.

In conclusion, we sincerely thank you for your thoughtful and detailed review. We've worked hard to address each of your concerns through significant clarifications, expanded experiments, and improved presentation of our method. We hope these revisions make the contribution and impact of our work clearer. If you find the updated version compelling, we kindly ask you to consider updating your score. We truly appreciate your time and feedback.

Thank you again for taking the time to review our paper.

We kindly ask for your response to our new comments before the end of the rebuttal period.

We have worked extensively to address your primary concerns, including clearing various aspects of our method and massively extending our evaluation beyond the MBPP dataset.

Please let us know if our responses address your concerns or if we can clarify any remaining issues. Thank you once again for your time and consideration.

Thank you for your thoughtful review and constructive feedback.

Evaluation Diversity and Generalization

We have substantially expanded our evaluation beyond MBPP to address concerns about benchmark diversity and broader applicability. Our updated experiments include the additional five benchmarks:

• HumanEval: A foundational and widely adopted benchmark.
• MBPP-ET and HumanEval-ET: Extended test suites that add 100 hidden test cases per task, offering a more rigorous evaluation of generalization and robustness - helping mitigate overfitting to visible tests.
• CodeContests: A challenging competitive programming benchmark that requires multi-step reasoning and broader generalization.
• DS-1000: A domain-specific benchmark targeting data science tasks. In this dataset, the problem is defined without heavily relying on test cases (one sample is shown as part of the problem prompt).

Notably, the benchmarks HumanEval, CodeContests, DS-1000 contain hidden test cases which help mitigate overfitting to visible tests.

Our method achieves state-of-the-art performance across all major benchmarks: MBPP (96.6%), MBPP-ET (73.0%), HumanEval (99.4%), HumanEval-ET (89.0%), CodeContests (60.6%), CodeContests (60.6%) and DS-1000 (69.9%).

We compared against every strong baseline we could find, both from PapersWithCode and directly from papers. We weren’t just relying on reported results, we actively tried to reproduce methods ourselves wherever possible. In many cases, we reran existing methods on the model (DeepSeek‑V3‑0324), to ensure a fair comparison. When code didn’t work with DeepSeek or wasn’t available, we adapted or re-implemented it, and clearly documented any limitations. These results show that our method is not limited to MBPP. It generalizes effectively across diverse levels of complexity, from foundational problems to competitive programming tasks and domain-specific benchmarks - demonstrating the broader applicability of our method.

We include detailed benchmark comparison tables below (due to character limit, CodeContests table can be found in reply to fegN). These present accuracy and retry success rates (RSR) across all datasets and baselines, including both our runs on DeepSeek‑V3‑0324 and results reported in the literature.

Table: Performance on the MBPP and MBPP-ET benchmarks.
Our proposed EG-CFG achieves a new state-of-the-art overall accuracy.
The DeepSeek–Coder-1.3B and –V3-0324 results for all baselines were obtained by our study using the official implementations provided by each baseline method.
The results below the double separator were collected from the respective papers.

Table: Performance on the HumanEval and HumanEval-ET benchmarks.
Our proposed EG-CFG achieves a new state-of-the-art overall accuracy on HumanEval-ET.

Table: Accuracy results from the official DS-1000 leaderboard.

Dependence on Test Cases

Our method is less dependent on test cases than the existing methods, since in most stages of the generation, it relies on implicit supervision, which is a central part of our method. The execution behavior is used as a soft feedback signal, interpreting outcomes without being explicitly told what is correct. This allows it to self-verify regardless of the quality of the test cases. We explicitly discuss this in the Introduction section (51-55) and highlight it as a key differentiator from prior approaches in the Related Work section (lines 141-143). We note in the limitation section that the method is applied to benchmarks that have executable test cases. This is the most direct way to compare with the existing relevant methods. However, it is a soft limitation since test cases can also be generated by the model itself from the task instruction. Finally, in five out of six benchmarks we evaluate on (HumanEval, HumanEval-ET, MBPP-ET, CodeContests, and DS-1000), the test cases are hidden. This means the model never sees them during inference. It helps ensure that our method’s strong performance is not due to overfitting to public test cases and further supports its generalization capabilities. We also note that in DS-1000, each problem includes only a single input-output example, making it impractical to rely heavily on such examples during code generation.

Computational Overhead

First, native parallelization is a key strength of our method. We leverage the low price of existing APIs and the ability to parallel them, which is a growing trend of today's agents. Other baseline methods mentioned in our paper can also have access to full parallel compute, but by design, they cannot fully leverage it. They rely on sequential retries, where each iteration depends on the output of the previous one. This makes it impossible for them to parallelize inference on a single task, even if the computational resources are available. Moreover, in (5/6) of the benchmarks: HumanEval, HumanEval-ET, MBPP-ET, CodeContests, DS-1000 there are hidden-test cases. There are hidden test cases. This means our method cannot rely on extensive token usage to overfit the public tests in order to succeed. Additionally, In order to rigorously make sure the comparison is fair, we ran trials using the DeepSeek-V3 model with two baseline methods: MGDebugger and MapCoder, on the MBPP benchmark, using a much higher number of retries which leads to significantly more extensive token usage. For MGDebugger, we increased the retry count from 5 to 200 - x40 increase, resulting in ~x40 tokens. This resulted in only an improvement of 34 out of 500 (6.8 percent), out of the 86 examples it originally got wrong. For MapCoder, we similarly increased retries from 5 to 200 (40x tokens), and saw an improvement of just 8 out of 500 (1.6 percent), out of 64 examples it originally failed. In both cases, the improvement was achieved with a 40x increase in runtime and is still considerably below our performance on MBPP. This supports our conclusion that the advantage of our method is not due to higher token usage, and therefore the comparison is fair. In order to clarify this issue more we will add discussion about this point in our revised version.

Thank you once again for taking the time to review our paper.

Following your feedback, we have worked diligently to address your primary concerns, and in particular, present a much more diverse body of empirical results.

We would greatly appreciate it if you could respond to our comments before the end of the rebuttal period so that we could address any remaining issues.

Thank you for your time and consideration.

The first table lists the mean performance (z-values, lower is better) per metric for each method.

The second table shows the weighted score per sample, comparing the EG-CFG method against the baseline LLM.

The third table compares the weighted performance scores of EG-CFG against Gemini 2.5 Pro.

EG-CFG consistently outperforms the baseline LLM across all CUDA code generation tasks, achieving a significantly lower overall performance score (−0.32 vs −0.02) under the normalized weighted scheme. This gain reflects improvements in key hardware-level metrics such as L2 and L1 cache hit rates, occupancy, and SM utilization, leading to more efficient memory access and thread scheduling. While warp cycles per instruction are slightly worse (0.02 vs −0.10), IPC remains equivalent, and the overall efficiency is clearly improved.

Per-task results show EG-CFG outperforming the baseline on all eight examples, with gains ranging from +2.6% to +77.0% and a +36.4% average improvement. These results, achieved with the same underlying LLM, highlight the impact of execution guidance in enabling hardware-aware optimizations beyond the reach of standard prompting.

Compared to Gemini 2.5 Pro, EG-CFG achieves better performance in 7 of 8 tasks, with a +31.4% average improvement-despite using an open-weight model rather than a proprietary one. The first table confirms EG-CFG’s advantage across key metrics, often achieving the best or near-best values.

Altogether, these results demonstrate EG-CFG’s ability to reliably produce more efficient CUDA code than both a naive baseline and a commercial LLM, without retraining or specialized infrastructure. This affirms its practicality and generality for real-world performance-critical code generation.

def solution(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
    out = torch.zeros_like(a)
    for i in range(a.shape[0]):
        if a[i] > b[i]:
            out[i] = a[i] ** 2
        else:
            out[i] = b[i] ** 2
    return out

__global__ void solution(const float* a, const float* b, float* out, int N) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx >= N) return;
    float a_val = a[idx];
    float b_val = b[idx];
    out[idx] = (a_val > b_val) ? a_val * a_val : b_val * b_val;
}

__global__ void solution(const float* a, const float* b, float* out, int N) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx >= N) return;
    
    float ai = a[idx];
    float bi = b[idx];
    float max_val = fmaxf(ai, bi);
    out[idx] = max_val * max_val;
}

__global__ void solution(const float* a, const float* b, float* out, int N) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx >= N) return;
    float val = fmaxf(a[idx], b[idx]);
    out[idx] = val * val;
}

def solution(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
    out = torch.zeros_like(a)
    for i in range(a.shape[0]):
        temp = a[i] * 0.1 + b[i] * 0.1
        out[i] = temp + temp
    return out

__global__ void solution(const float* a, const float* b, float* out, int N) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx >= N) return;
    float temp = a[idx] * 0.1f + b[idx] * 0.1f;
    out[idx] = temp + temp;
}

__global__ void solution(const float* a, const float* b, float* out, int N) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx >= N) return;

    float temp = (a[idx] + b[idx]) * 0.1f;
    out[idx] = temp + temp;
}

__global__ void solution(const float* a, const float* b, float* out, int N) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx >= N) return;
    out[idx] = 0.2f * (a[idx] + b[idx]);
}

Metrics

We evaluate GPU performance of the generated CUDA kernel functions using NVIDIA's profiling tool Nsight Compute (NCU), with input tensors of size N = 256 populated with random values generated using a fixed seed. The following hardware-level metrics are collected: executed IPC, warp cycles per instruction, SM utilization, occupancy, L1/L2 cache hit rates, and branch efficiency.

Formulation

All metrics are normalized using z-score normalization:

$$x_i^{\text{norm}} = \frac{x_i - \mu_i}{\sigma_i}$$

where $\mu_i$ and $\sigma_i$ are the global mean and standard deviation of metric $i$, computed across all samples (baseline LLM and our method combined so that the normalization is shared). If $\sigma_i = 0$, we define $x_i^{\text{norm}} = 0$.

A scalar performance score is then computed as a weighted sum over the normalized metrics:

$$\text{score} = \sum_i w_i \cdot x_i^{\text{norm}}$$

Lower scores indicate better GPU performance.

To obtain a single score, we propose the normalized_weighted_sum variant, which uses the following weights (lower score = better):

Metric Weights Used in Normalized Scoring

Notes:

• Positive signs indicate metrics where higher is better.
• Negative signs indicate metrics where lower is better.
• Norms (magnitudes) are tuned to balance importance - IPC and warp latency dominate, while memory and branching metrics are secondary.

Results

The tables below present performance results comparing EG-CFG to the Baseline LLM across all evaluated tasks. Both methods use the DeepSeek-V3-0324 LLM. The baseline LLM was run multiple times with a temperature of 0.6 to create a diverse set of solutions. For broader comparison, we also evaluated CUDA code generated by an additional frontier model: Gemini Pro 2.5 with its default settings and the same prompt used in the baseline LLM. Scores are computed using a normalized weighted sum over profiling metrics. Each metric is normalized using z-score normalization, as described in the Metrics section. Lower performance scores indicate better GPU efficiency.