Q1: ..examples to analyze when more powerful models generate slower code

Thanks for the question! We case study some a few samples where DeepSeekCoder-6.7B-Instruct (perf-instruct) outperforms DeepSeekCoder-33B-Instruct (perf-instruct):

Mbpp/732: Replace certain characters to ":".

# The 6.7B model uses the simpler and faster "replace" function:
def replace_specialchar(s):
    return s.replace(' ', ':').replace(',', ':').replace('.', ':')

# whereas the 33B model tend to use "re.sub" which is more powerful but slower as parsing regular expressions is costly:
import re
def replace_specialchar(s):
    return re.sub(r'[ ,.]', ':', s)

HumanEval/24: Finding largest divisor.

# The 6.7B model smartly cut half of the computation in the beginning
def largest_divisor(n: int) -> int:
  i = n // 2
  while i > 0:
    if n % i == 0:
      return i
    i -= 1

# The 33B model
def largest_divisor(n: int) -> int:
  for i in range(n-1, 0, -1):
    if n % i == 0:
      return i

Q2: ..Do bigger values of scale lead to slower reference times across all tasks?

Thanks for the great question! We did see cases where input samplers produce "smaller"/"faster" inputs given a larger scale factor, e.g., sometimes GPT-4-turbo ignores the monotonicity requirement and simply generates random samplers. Notably, when making EvalPerf we add monotonicity assertions to falsify samplers that give "faster" inputs over larger scale factors.

To study the frequency of monotonicity violation, we ran input samplers by setting the scale factor from 2 to 10 without monotonicity checking. Among the 563 initial tasks, samplers from 486 tasks can generate valid test inputs and 327 (67%) of them generate inputs that always respect computational monotonicity. We will add this in our revision.

Q3: ..if we simply asked for test generators parameterized with a random seed?

Asking LLMs to produce random test generators has limitations in both theory and practice.

In theory, when random sampling the tests, the scale/complexity of the test input is bounded by the range of randomness hardcoded by the LLM, e.g., the "1000" in return [randint(0, 1000) for _ randint(0, 1000)]. Such hardcoded values can still be too small to fully exercise the code solutions or too large to run. Meanwhile, for finding the largest computable input within a user-given time limit, searching inputs uniformly and randomly is less targeted and thus slower than searching inputs in monotonic scales.

C1: ..there is no analysis of the variance..

Please kindly note that Table 2 shows a negligible variation even when replicating the experiments over four different test beds.

To study statistical significance, we rerun the experiment 40 times using the "instruct" and "perf-instruct" versions of DeepSeekCoder-6.7B (2 settings X 40 repeats X 50 samples X 121 tasks). Next, we use the Wilcoxon signed-rank test to check if the difference in the normalized DPS between the two settings is statistically significant, showing a P-value of 0.000379, which is much smaller than 0.05 (the most common significance level).

C2: ..application of DPS to problems other than HumanEval and MBPP..

Please refer to C4 for reviewer 6zda.

C3: ...no qualitative analysis

Please note that we did show a few qualitative conclusions, e.g., instruction tuning implicitly improves efficient code generating. One reason for limited qualitative analysis is that code efficiency is not optimized in training, thus we see more randomness than scaling laws, calling for research to optimize this direction. Nonetheless, Q1 for reviewer Jb72 includes some qualitative case studies.

Q1: Why ... unify both of these under one benchmark?

EvalPerf evaluates end-to-end efficiency where both aspects can equally exhibit efficiency advantages.

Why not put code-level efficiency under a program rewriting setting? Besides post-implementation optimizations, experienced developers can often directly write code of efficient paradigms. For example, in Python, it's faster to run built-in APIs like filter/map/reduce than vanilla loops. In C++, memory pre-allocation is a common efficiency tip where calling vector::reserve before vector::push_back reduces costly runtime allocations.

Q2: ..contributions of your benchmark from those of PIE

PIE treats the efficiency problem in a program repair setting where a base solution is available, while DPE targets the direct generation of efficient code, as is exemplified in Q1.

The unique contributions of DPE are to systematically create efficiency benchmarks and reliably evaluate them. As shown in "Limitation #1" (Section 1), prior work does not manage to use compute-costly tests and DPE specifically curates such tasks/tests to exhibit code efficiency. As shown in "Limitation #2", traditional speedup has unbound scales and we develop DPS to fix this issue. These can also be applied to the PIE setting. We will discuss more in our revision.

C1: ... models would have to be compared across the same set of programs...

Thanks for the great suggestion! Indeed, it is important to compare model pairs over the same set of validated solutions. So we draw heatmaps in the following anonymous links:

• Heatmap of $DPS$ comparison: https://ibb.co/TBQvFV0
• Heatmap of $DPS_{norm}$ comparison: https://ibb.co/5TkQbfY

Specifically, we sort the models by their global scores. Each block in the figure shows the performance score of the Y-axis model and its improvement against the X-axis model (if the difference > 1pp). Overall, most higher-ranking models still outperform lower-ranking ones on the same set of solvable tasks. We will discuss more in our revision.

C2: ..hard to follow due to their complexity and brief description..

Thanks for the suggestion! Indeed we should improve the methodology section by adding more explanation and examples. Regarding the detailed questions, "base" is a typo for "bias" and we should explain "cumulative ratio" in its definition. We will fix these in our revision!

C3: ..consider also generating programs that exercise the LLM-generated solutions

Thanks for the question! We did study a handful of cases where we saw both ground truth and slower implementation can lead to similar test generators.

Let's case-study HumanEval/39 which computes the n-th prime Fibonacci number. The ground truth tests primality using the Miller-Rabin algorithm with O(k logN), while LLMs can use a vanilla method with O(N) or O(sqrt(N)). Using our prompts, all of the three lead to the same input sampler:

def perf_input_gen(scale: int):
  return (scale,)

This might be reasonable as LLMs mainly use the reference solutions to ground code semantic understanding and their complexity might not lead to significantly different generators.

C4: ..testing more complex programs that rely on external libraries

Thanks for the suggestions! We initially applied DPE on close-domain benchmarks as proof of concepts. We believe our technique can be applied to open-domain benchmarks such as DS-1000. We will include this in our future work.

Q1: ..results of your method vary depending on the architecture..

Yes, hardware profilers are architecture-dependent. We were saying our "scoring mechanism", i.e., the metric of "differential performance score" (in S2.5), is agnostic to measurements (physical time, #instructions, etc.). We will fix the confusion in our revision.

C1: ... a more extensive comparison ... across different open-source LLMs.

Thanks for the suggestion! For a more extensive comparison, in our rebuttal, we include evaluations of some recent open-source code models:

• CodeQwen1.5-7B: DPS: 72.1 (%) / DPS_norm: 65.6 (%) / Pass@1: 68.2 (%)
• CodeQwen1.5-7B-Chat:
  • Instruct: DPS: 78.9 (%) / DPS_norm: 71.8 (%) / Pass@1: 78.8 (%)
  • Perf-Instruct: DPS: 79.3 (%) / DPS_norm: 73.0 (%) / Pass@1: 75.2 (%)
  • Perf-Cot: DPS: 74.2 (%) / DPS_norm: 69.3 (%) / Pass@1: 72.2 (%)
• starcoder2-15b-instruct-v0.1:
  • Instruct: DPS: 66.9 (%) / DPS_norm: 63.2 (%) / Pass@1: 71.9 (%)
  • Perf-Instruct: DPS: 69.2 (%) / DPS_norm: 65.9 (%) / Pass@1: 72.5 (%)
  • Perf-Cot: DPS: 67.5 (%) / DPS_norm: 62.8 (%) / Pass@1: 72.5 (%)

Meanwhile, we include more experimental insights (e.g., pairwise comparison) for these models in C1 for Reviewer 6zda. In our next revision, we will try to include more if new code models are coming out.

C2: ... releasing the code used to reproduce the results would be beneficial.

Thanks for the suggestion! We are enthusiastic about open research and have already open-sourced and packaged the code and the dataset. However, for review anonymity, we cannot disclose the pointers now but will include them in the final version.