AuPair: Golden Example Pairs for Code Repair

1. discuss related work that uses adversarial approaches for reference.

We will include references to adversarial approaches such as [1,2] in the Related Work section.

2. Only unit test scores as the metric to evaluate correctness of code repairs

In this work, we have used unit test scores for assessment of fix quality and for guiding the exploration phase, such as has been done in prior work [3,4,5]. However, there is nothing fundamentally in the approach that prevents the use of other feedback mechanisms, such as reward models, or other sources of feedback to boost its performance. We leave this set of experiments for future work since we believe it is orthogonal to the key ideas presented.

3. Does not has good analysis on failure cases or situation that the method not working well

We refer the reviewer to point 1 in our rebuttal address to reviewer oGUo: we provide concrete failure modes in which using AuPairs can lead to fixes that are potentially worse than the initial guesses, and this is observed more frequently for AuPair than other baselines on average, since AuPair is a diversity-boosting algorithm.

4. Is the unit score the only metrics you used?

No, in addition to the test pass rate for unit tests, we also report the strict accuracy metric [6], which is the percentage of test problems that are fully solved. We report this metric for every single ablation in the paper; we refer the reviewer to Section A.2 in the Appendix, as well as Figs. 10, 11, 12 for plots showing this metric.

5. Consider discuss potential avoid-situation that may cause failures or lower performance.

We refer the reviewer to Section A.4 of the Appendix, where we have a detailed discussion on the impact of smaller datasets on the efficacy of our proposed approach. Our results indicate that even though AuPair works well in the small dataset regime, curating the AuPairs on a larger dataset and then applying them to a smaller dataset, even though out-of-distribution, recovers in-distribution performance; it can also potentially lead to stronger scaling, since there will be fewer AuPairs extracted on smaller datasets. Furthermore, we have additional results for this in point 4 of our rebuttal to reviewer oGUo.

[1] Adversarial patch generation for automated program repair, Alhefdhi et al. 2023

[2] Learning to repair software vulnerabilities with Generative Adversarial Networks, Harer et al. 2018

[3] Code Repair with LLMs gives an Exploration-Exploitation Tradeoff, Tang et al. 2024

[4] Cycle: Learning to self-refine the code generation, Ding et al. 2024

[5] Teaching LLMs to self-debug, Chen et al. 2023

[6] Measuring coding challenge competence with apps, Hendrycks et al. 2021

1. Limited exploration of failure cases

While AuPairs have been shown to significantly boost performance, they can occasionally have unintended impacts as well. The following table contains the percentage of CodeForces problems in which some fixes had a decrease in fix score compared to the initial guess. Note that this does not affect any of the performance results in the main paper, since for measuring performance, the best scoring response is selected.

As we can see from the above table, in most cases, using AuPair results in an increase in the number of problems for which a fix is worse than the initial guess. This is to be expected since AuPair is an algorithm that in addition to boosting performance also boosts diversity of the generated responses.

2. The evaluations focus predominantly on competitive programming tasks. It remains unclear whether the impressive results would translate directly into other code repair domains.

Competitive programming tasks are a canonical domain for code repair for two reasons: these tasks are rigorous (can be precisely evaluated, cannot be gamed) and they are hard (supposed to differentiate between top human coders). Both of these make them appropriate to research and test new methods, the crisp, reliable evaluation of challenging tasks means that even differences of a few percent are meaningful capability improvements. But indeed, competitive programming tasks are not the bulk of what users might request. While we see no reason to doubt that our method will translate to broader, less well-defined, or easier tasks, we do not want to make a strong claim about this transferability; producing that evidence is out of scope for this paper. We will expand on this in the final version of the paper.

3. The paper's method strongly relies on predefined test cases and evaluations on these tests to judge fix quality.

Indeed, in this work, we have used test cases for assessment of fix quality and for guiding the exploration phase, such as has been done in prior work [1,2,3]. However, contrary to other approaches for code repair, where failed test cases are given to the model in the prompt, we do not include test cases in the prompt; we use test cases solely to build the AuPairs.

Furthermore, we would like to highlight that there is nothing fundamentally in the approach that prevents the use of other feedback mechanisms, such as reward models, or other sources of feedback to boost its performance. We leave this set of experiments for future work since we believe it is orthogonal to the key ideas presented.

4. How sensitive is AuPair's effectiveness to the size and diversity of the validation set used for AuPair selection?

This is a great question; we conducted experiments with smaller validation sets to curate AuPairs and report the results below. We also include the results of the random baseline for calibration:

The larger the validation set, the more distinct complementary improvements can be observed, and hence the larger the maximal set of AuPairs that can be discovered. So larger validation sets make it possible to effectively scale up to more inference compute. However, even just looking at just the top 32 AuPairs (which is apples-to-apples for varying validation set sizes), we find that their quality increases monotonically with the size of the validation set. We have also conducted additional analysis on the impact of smaller datasets, for which we point the reviewer to section A.4 in the paper.

[1] Code Repair with LLMs gives an Exploration-Exploitation Tradeoff, Tang et al. 2024

[2] Cycle: Learning to self-refine the code generation, Ding et al. 2024

[3] Teaching LLMs to self-debug, Chen et al., 2023

1. Including a more diverse set of models would strengthen evaluation

We have included 5 models spanning 3 model families across 7 datasets in the existing results – Gemini, GPT, Gemma; the results clearly indicate that AuPair works across models. We understand the reviewer's concern and agree that showing results on more models would strengthen evaluation, but getting the full set of results with another model within the short rebuttal period would not be feasible. What we can do, however, is include new transfer results with GPT-4o (suggested by reviewer) using AuPairs from GPT-4o-mini and Gemini-1.5-Pro.

Even when using AuPairs from other models, we see 11% and 20% absolute performance gain over baselines, thus solidifying our claim that even for more code-competent models such as GPT-4o, AuPair gives large performance gains.

2. RAG over the candidates dataset pairs is not considered as a baseline

We implemented a RAG baseline, choosing the top 32 pairs for each problem, retrieved from the candidate pair dataset, as the reviewer suggested:

We see that across all models, AuPair, which uses a fixed set of in-context examples, outperforms RAG, which requires the entire set of candidate pairs to be compared with each test problem to choose the in-context examples.

3. Another analysis on the diversity aspect which includes the type of repairs these pairs help in

We show a breakdown of the repairs generated into 4 categories: 1) %problems with improved fix score, 2) %problems in which code was reformatted to obey constraints, 3) %problems in which fix was improved by just changing the logic, 4) %problems in which score remained unchanged. Composite changes (formatting + logical) are reported in formatting fixes.

Some insights:

• Since Gemini-1.5-Pro guesses have more formatting bugs, % problems with formatting fixes using AuPair is high.
• AuPair also helps the model repair solutions with logical errors.
• AuPair yields responses that are more diverse than baselines in the test case scores, indicated by the lower value of "Unchanged".

Example of composite fixes:

Guess:

def solve(s: str):
   n = len(s)
   a = int(input()) - 1
   b = int(input()) - 1
    cost = 0
   for i in range(a, b):
     if s[i] != s[i + 1]:
       cost += 1
   print(cost)

Fix:

def solve(s: str):
   n, a, b = map(int, s.split('\n')[0].split())
   companies = s.split('\n')[1]
   cost = 0
   if companies[a - 1] != companies[b - 1]:
     cost = abs(a - b)
   print(cost)

4. Does AuPair utilize a slightly higher computational budget compared to baselines?

This is partially correct: there is a one-time amortised cost that our algorithm incurs to construct the AuPairs, but the cost at test-time is identical. Note that the same fixed set of AuPairs boosts performance across models and datasets (Fig. 6, 7 show out-of-distribution generalisation), indicating that the upfront cost is easily amortised.

5. The phrase "in conjunction with high performing" in line 260 is somewhat confusing

We explain this in line 250: "since the code generated has to adhere to a specific format to allow successful execution, we observe that many initial guesses of generated code fail because they do not obey these conditions". Moreover, our experiments on GPT-4o above also show that the insight stands, since we observe absolute improvements of 11% and 20% using AuPair compared to self-repair (strongest baseline).

6. The necessity of the subtraction step is not intuitively clear

The fix quality matrix contains the score for each candidate pair for each validation problem. After picking the pair with the best mean score, we subtract this score from the matrix because only after removing the previous best pair can we find the complementary next best pair.

7. Missing references

We will include the mentioned references in the Related Work section.