Discovering Preference Optimization Algorithms with and for Large Language Models

We thank the reviewer for their thoughtful and extremely helpful review. We’re glad that the reviewer finds our approach to be both novel and useful, though we agree that more analysis would improve the paper significantly.

The majority of insights about DiscoPOP are presented as hypotheses or intuitive insights...further experimentation would be necessary

This is a great point and we’re grateful to the reviewer for bringing it up. We’ve performed further analysis that considerably improves the paper and our understanding of the loss function. Previously, we merely hypothesized that the local optimum of the DiscoPOP loss could catch noisy or incorrect data points. We now have empirical evidence for this and include the results in the rebuttal PDF.

In short, we identified which data points end up between the local optima after training, and found that 1.35% of points fall there (see Figure 1 in the PDF, where they are clearly visible). Although we use the binary preference labels from the IMDb dataset for training, the dataset also includes a reward score for each completion. When we analyze the data points that are between the local optima, we find that the positive and negative completions are significantly closer in absolute reward difference than the data points outside the local optimum (See Table 1 in PDF). This implies that the preference labels on those points are more difficult to distinguish and helps empirically validate our hypothesis. Thanks to the reviewers, we will be adding this analysis to the paper.

Further experiments or details on...refinement vs regurgitation

This was another great suggestion. We’ve included an ablation on the CIFAR-10 results in which we don’t provide fitness feedback. Thus, the model is unable to perform refinement (as it does not know what performs well) and can only regurgitate ideas. This is a key baseline to compare to to validate our discovery method.

In the attached rebuttal PDF, you can see that, without the fitness score, the LLM is unable to refine its ideas and can only regurgitate, thus leading to fewer steps of improvement.

On LLM Discovery vs. DiscoPOP as the main focus

Our main focus was on DiscoPOP, as that is where the majority of our resources and efforts were focused. We will adjust the writing to make this clearer.

We believe both parts are significant contributions of our paper, though we understand both could use more analysis. Thanks to the reviewer’s feedback, we’ve managed to significantly strengthen evidence for both.

what properties an optimal objective function should possess

Thanks for bringing this up. This was imprecise language on our end. We agree, and have now removed any claims about what an "optimal" objective should possess, instead replacing the word optimal with the word "good" in the conclusion (line 277 and in line 110).

How often are unit tests for the generated code failed?

For GPT-4 the unit tests failed ~5% of the time; however, the model usually fixes it upon the feedback. We briefly experimented with weaker models and found the failure rates to be significantly higher.

---

We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.

We would like to once again thank the reviewer for their time and feedback. We've incorporated it into our manuscript and we believe it has strengthened our paper.

We hope our rebuttal, which includes substantial additional results and analysis, has addressed the reviewer's concerns.

Seeing as the discussion period is coming to a close, could the reviewer please let us know if they have any further questions or concerns about our submission?

We would like to thank the reviewer for their concise feedback. We’re happy that the reviewer finds the paper innovative and the results impressive. We understand the reviewer has concerns around prompt sensitivity. We would like to point the reviewer to a feel ablations we’ve run on this.

In Appendix D3 of the paper, we show that the discovery process is very robust to different sampling parameters, prompts, and techniques from the LLM. We find little change in performance or behavior when we change the sampling temperature (unless it is set unreasonably high), telling it to “think” first, or sorting the input context.

Notably, in our attached rebuttal PDF we show that providing the fitness in context is important to allow the LLM to refine its proposed ideas.

---

We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.

We thank the reviewer for their constructive feedback. We’re glad the reviewer finds the approach and results promising and effective.

does proposed LLM-driven discovery method generalize to other domains?

In our submission, we demonstrated that it works well on CIFAR-10 in the small case study in Figure 2, where it discovers loss functions that outperform standard cross-entropy loss and transfers to other models and hyperparameters. We will adjust the writing to make this clearer.

There were no significant modifications needed to adapt this method to other fields, the only change needed is to adapt the text in the prompt (Appendix A) to contain information and examples for the new setting. We chose preference optimization, as it is a new exciting area with great potential for novelty, though we agree that adapting our LLM-driven discovery method to other domains/tasks is an exciting future direction of work.

reliance on a complex method that involves detailed tuning...could limit the method's applicability unless the process can be generalized

We’re not sure exactly which hyperparameters the reviewer is referring to, but Algorithm 1 does not contain any hyperparmaeters and is very easy to apply to other domains.

Theoretical justifications for DiscoPOP

To understand DiscoPOP more from a theoretical perspective, we can closely compare it to the standard DPO loss. At the limits (eg when the relative preferences $\rho$ are very high or low), it has similar behavior to DPO. However, it mostly differs by its non-convex region, which we analyze in the rebuttal PDF. Previously, we merely hypothesized that the local optimum of the DiscoPOP loss could catch noisy or incorrect data points. We now have empirical evidence for this.

In short, we identified which data points end up between the local optima after training, and found that 1.35% of points fall there (see Figure 1 in the PDF, where they are clearly visible). Although we use the binary preference labels from the IMDb dataset for training, the dataset also includes a reward score for each completion. When we analyze the data points that are between the local optima, we find that the positive and negative completions are significantly closer in absolute reward difference than the data points outside the local optimum (See Table 1 in PDF). This implies that the preference labels on those points are more difficult to distinguish and helps empirically validate our hypothesis. Thanks to the reviewers, we will be adding this analysis to the paper.

We would like to further emphasize that, in the current offline preference optimization paradigm, training and evaluation differ significantly. In most settings in machine learning, the loss usually corresponds directly with some desired metric, such as the accuracy. However, in our setting, while we train on an offline set of preference pairs, we ultimately evaluate the model using MT-Bench and GPT-4 as a judge. Thus, it’s not very clear how the loss function used necessarily corresponds to the downstream task. In fact, recent works [1, 2] show that optimizing the DPO loss function too much can lower the quality of the model. Thus, “theoretical” justifications are not as useful in this setting.

what strategies do you suggest to avoid local minima during optimization?

We only found the local minima to be a problem when we set $\beta$ to be very low ($\leq 0.01$). Note that, while local minima are a problem when the loss function directly corresponds to the task, it’s not immediately clear if that’s the case in offline preference optimization (see our comment on this above).

[1] Feng, Duanyu, et al. "Towards analyzing and understanding the limitations of dpo: A theoretical perspective." arXiv preprint arXiv:2404.04626 (2024).

[2] Chen, Angelica, et al. "Preference Learning Algorithms Do Not Learn Preference Rankings." arXiv preprint arXiv:2405.19534 (2024).

---

We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.

We thank the reviewer for their thorough and in-depth review. We are pleased that the reviewer finds our approach interesting and effective. We address the weaknesses outlined by the reviewer individually below.

The objective function is discovered with a different placement of $\beta$

Thank you for highlighting this. We assume that the reviewer is referring to discrepancies between the code for the LLM sampler in Appendix E.6 and the formula presented in Equation (28), as explained in Appendix E. As a result of the reviewer’s comment, we found have now corrected a typo in the LRML formula, which now reads:

$f_{lrml}(\beta\rho) = (1-\sigma(\beta\rho/\tau)) \cdot f_{dpo}(\beta\rho) + \sigma(\beta\rho/\tau) \cdot f_{exp}(-\beta\rho)$

The error occurred in the written mathematical notation, and the implementation remains the same.

We introduce the $\tau$ parameter largely to retain the original semantic meaning of “beta” used in most papers, which is to adjust the KL-divergence regularisation. Since we use $\tau = 0.05$ they are equivalent.

We have now corrected the mathematical notation for DBAQL, AQL, AQFL, and PFL as well.

It is unclear if the LLM's capability used in the discovery process can change the duration for finding a good objective function or if it will cause the search process to fail.

This is a good point. We briefly tried using LLama3 8b-instruct and found that it rarely responded with the correct format and working code, rendering it largely unusable. Should the reviewer request, we can do a more thorough sweep over base models. However, we do not expect this to fundamentally change our results.

it would be nice to...perform some traditional searching algorithms

Unfortunately, this exact experiment would be very hard to implement, as we would need to design a domain-specific language to search over for the objective function or parameterize it in some way. Parameterizing the objective with a neural network is possible, but would require far too many inner loop training runs to be optimized. However, inspired by the reviewer’s feedback, we’ve designed an experiment that is similar in spirit to validate the effectiveness of the LLM-Driven Discovery method. In particular, we’ve added a version of the CIFAR-10 experiment where we do not return the fitness to the LLM. We show the results in Figure 2 of the attached PDF and confirm that, without the fitness, the LLM is unable to refine its ideas, as it does not have knowledge of which ones worked.

---

We hope that most of the reviewer’s concerns have been addressed and, if so, they would consider updating their score. We’d be happy to engage in further discussions.