Confronting Reward Model Overoptimization with Constrained RLHF

Thank you very much for your careful review and feedback! We are glad that you appreciated the simplicity of our approach, the identification of the importance of the interaction among component RMs, the breadth of our experimentation, and the efficiency of NM-PPO. We hope to address your concerns below:

Weaknesses

• Multiple runs + Ground Truth: We absolutely agree, as we note in the Discussion section of the paper. However, we stress that the need for some access to ground truth evaluation is required of all methods, including standard PPO, as it needs multiple runs to tune the fixed weights used on the component RMs. In this sense, we can interpret the results as follows: $\xi$-PPO (and $\mu$-PPO) improve performance relative to PPO for the same number of queries to the evaluation metric. NM-PPO also improves performance relative to PPO, although to a lesser degree than $\xi/\mu$-PPO, while dramatically reducing the number of required queries to the evaluation metric as well as the number of overall training steps. Detecting overoptimization without any queries to an evaluation metric would be a significant advance for any RLHF approach, but we believe it is orthogonal to our method and outside the scope of this paper.

Questions

• Constraint Satisfaction: The answer to this depends to a large extent on the definition of “necessary.” Our results indicate that the best performance is obtained when all constraints are satisfied. If the true evaluation performance is a smooth function of the component RM scores, then a subset of component RMs deviating slightly from their constraints would likely not produce a significant drop in performance. This is what appears to have occurred with $\mu$-PPO in our case (although it should be stressed that because it is using inequality constraints, it’s not violating its constraints by exceeding the intent threshold). However, this is hard to predict and would likely vary greatly across problems.
• $\mu$-PPO: This is likely related to the issue you raise in the previous question (and our response). $\mu$-PPO doesn’t dramatically exceed the Intent threshold, and it matches the METEOR threshold fairly accurately. In this problem, the predicted evaluation surface (Fig 3.2) is rather smooth, so this likely explains why its behavior doesn’t produce a dramatic drop in performance.

Thank you very much once again for your time and consideration. We hope that we’ve managed to address your questions and concerns. If not, please let us know and we’d be more than happy to continue our discussion!

Thank you very much for your response and for helping to improve the paper!

Thank you very much for your detailed review! We’re glad you liked the paper and found it to be well-written, novel, containing thorough analysis, and of importance to the field. We hope to address your concerns below:

Weaknesses

• Proxy Point Identification: The difficulty of finding proxy points is likely more a function of the number of component RMs and their interaction with each other rather than dataset or model size. The first method in the paper (find proxy points, then optimize) is simple but expensive due to the number of runs required. (Regular PPO is also expensive in this way, because we need to search for the best fixed weighting on RMs.) NM-PPO makes this process much more efficient, but since it’s a local hill-climbing algorithm, there’s no guarantee that it’ll reach the globally optimal set of proxy points. We think improving this is an exciting direction for future work!
• Paper Length and Sample Outputs: We absolutely agree regarding the length of the paper! Is there anything in particular that’s in the appendix that you would like to see in the main text? Regarding the sample outputs, we were simply hoping to highlight the fact that high METEOR and intent scores do not necessarily correlate with high evaluation scores, a sign of overoptimization. We have added outputs from NM-PPO, as well as the component RM and evaluation ratings for each response to emphasize this.

Questions

• Transferability of Results: We don’t know exactly how GPT-4 was finetuned, but in principle this method should be effective for any RLHF approach. Methods like NM-PPO which eliminate the need for multiple runs and reduce the overall number of queries to the ground truth evaluation metric would be especially helpful at that scale.
• Weaknesses: We hope we have!

Thank you very much once again, and we hope we have adequately addressed your questions and concerns!

Thank you very much for your careful review and helpful feedback! We are glad that you found the paper to be an original contribution, of high quality, and a significant stride towards improving RLHF. We aim to address your concerns below:

Weaknesses

• Querying the Evaluation Metric: We absolutely agree that the need to query a ground truth model is a weakness. However, as we note in the paper, this requirement is common to all the methods considered, including PPO (which requires access to the evaluation metric to tune the RM weightings). There’s currently no way to determine if overoptimization is happening without some access to ground truth evaluation. One possible route would be to make use of AI feedback (RLAIF). Such an advance would be orthogonal to the aim of this paper (and would benefit both our approach and standard PPO), and we believe it is outside the scope of this work.
• Contextualizing the Evaluation Metric: We thank the reviewer for the reference. We found the paper insightful and have worked it into Section A.2 with a more in-depth discussion. We'd like to emphasize that the main desideratum for our evaluation metric was that it could be overoptimized in the RLHF process. In the context of the findings of Maynez et al., we note our work differs in considering RLHF on dialogue tasks, rather than open-ended natural language generation. Due to the structured nature of the task, models rarely break from the format (see generation examples in Appendix D), which permits the use of overlap metrics. To ensure relevance to the ground truth outputs, we used ROUGE2 (like Maynez et al.) in combination with some other metrics. The main difference in our evaluation metrics are the diversity metrics, which were chosen since degeneration is a common problem when finetuning language models [1]. We are excited by future extensions of our work to larger models, and more sophisticated tasks (like summarization), which would necessitate evaluations like those suggested by Maynez, et al.
• Detailing Methodologies: Thank you for this observation! Algorithm 1 in the paper appendix comprises a detailed procedure which can be used to implement $\mu$-PPO and $\xi$-PPO, and Algorithms 2 and 3 describe the overall two-phase procedure for identifying and leveraging proxy points and NM-PPO, respectively. In addition, we included the code implementing these algorithms in the supplementary material. In the revision, we have added additional references to the algorithm descriptions in the main text as well as a description of PPO-SAT in Algorithm 2. We hope this improves the clarity of the proposed approaches!

Questions

• Evaluation Metric: Thank you for this question! As noted above, the evaluation metric was selected because it was overoptimized (i.e., obeyed Goodhart’s Law) with respect to each of the component RMs. That is, we wanted an evaluation metric that would initially increase as METEOR and intent scores increased, but would begin to decrease after a certain point, much as human evaluation ratings initially correlate with RM scores and then decline. To design this metric, we tested a number of combinations of other metrics (those included in Figure D.1), and found that the combination described in Appendix A.2 most closely adhered to the pattern of overoptimization with respect to METEOR and intent. That is, the evaluation metric was not directly chosen to simulate human preferences, but rather the way human ratings change as component RMs increase.
• Value Function: The value expression is correct, as it measures the expected, discounted, cumulative reward with respect to the Lagrange multipliers. Furthermore, $\sum_{i=1}^N \mu_i \theta_i$ is not a (directly) policy-dependent term.
• PPO-SAT: We apologize for the misunderstanding! PPO-SAT is not a Lagrangian method, it sets the reward signal equal to the weighted sum of the squared errors of each RM from its corresponding proxy point, and the weights on each of these errors is fixed. It can be seen as standard PPO with a modified reward function that takes into account proxy points. We have added Algorithm 2 to the appendix, which describes this procedure in greater detail.
• Figure D2: Thank you for catching this mistake! We mistakenly applied the wrong transformation to the saved Lagrange multiplier data, but the result is consistent. We have updated the plot in the revision accordingly and added results from $\mu$-PPO and $\xi$-PPO for comparison.

Thank you very much once again for your feedback and review! We hope that the changes made and responses above have addressed your concerns. If that’s not the case, we’re more than happy to continue discussing.

[1] Khalifa, M., Elsahar, H., & Dymetman, M. (2020, October). A Distributional Approach to Controlled Text Generation. In International Conference on Learning Representations.

Thank you very much! We fully agree that reducing dependence on costly evaluations is important for RLHF, and hope to address this problem further in future work. To further study this dependence in our setting, we added an additional plot (Figure D.7) which examines the effect of the number of queries per run (in the non-Nelder Mead-based approach) on the predicted evaluation surface. We appreciate your time and help in making the paper better.

Thank you very much for your detailed and helpful review! We’re glad that you appreciated our analysis of composite reward functions, the efficiency and novelty of NM-PPO, and found the paper to be well-written. We hope to address your concerns below:

Weaknesses

• Joint Proxy Point Identification: The idea behind our approach was to increase each individual RM and observe the corresponding change in the others as a way to measure their interaction. If the agent were maximizing both component RMs, then a change in one couldn’t be ascribed purely to the change in the other. Additionally, the fact that NM-PPO appears to hill-climb along the topography of the predicted surface (Fig. 5.4, and Fig. D.3) can be seen as empirical validation that our approach is in fact effective at approximating the evaluation surface. Finally, given that (to our knowledge) overoptimization has not been studied in this setting before, we believe that simple yet effective heuristics offer a useful starting place for future work, both to understand their efficacy and to design improvements. We hope to continue working on this topic, as well as inspire broader follow-up work!

Questions

• Proxy Points: The evaluation function is queried 20x per run in Figure 3.1. Figure 3.2 is generated using the results from the runs in Figure 3.1 (no additional queries). The algorithms do not make additional queries to the evaluation function for the results in Figure 5.1. We tested two methods for finding the maximum of the fitted evaluation function surface. First, we used the built-in implementation of L-BFGS-B in the SciPy optimization package using the results from KDE to provide bounds. Second, since the polynomial surface is approximated using a discrete mesh, we also simply picked the coordinates corresponding to the plotted maximum. There was a negligible difference between methods, so we used the second in practice. We have added this detail to the appendix. We do not use KL regularization to determine the proxy points because our goal is to isolate the influence of each component RM on the others, and KL regularization would be an additional factor influencing behavior.
• Long Training Performance: This is a good question! Upon investigation, we found that all runs managed to continually satisfy the intent reward constraint across training, but that a single run suffered a loss of stability with respect to the METEOR reward after roughly twice the baseline training time (stability was maintained in the other seeds). This aligned with a drop in evaluation performance. We added a figure showing this, Figure D.3.

Minor points:

• Stability: We agree! We’ve added these results to the figure.
• All-PPO: Yes, it has the same KL regularization mechanism as standard PPO. We have added text clarifying this.
• NM-PPO Curve: We agree, and have added this in Figure D.5.
• NM-PPO Sample Outputs: Good catch, we have added this.
• Sample Output Rationale: We have added the reward and evaluation scores for each output. Our main point of emphasis was that high METEOR and intent scores do not necessarily correlate with high evaluation scores. Because the evaluation score was selected because it can be overoptimized by METEOR and intent, rather than any specific suitability as a measure of text quality, higher evaluation scores should not be interpreted as such. We have added text to clarify this.

Thank you once again for your detailed review and comments! We hope our responses and the changes made address your concerns. If not, we’re more than happy to continue the discussion!

We have uploaded a new revision of the paper with several additions:

• To evaluate whether NM-PPO is able to scale effectively to more than two component RMs, we added BLEU score as a third RM and compared the performance to PPO. Both NM-PPO and PPO (with tuned coefficients) were run for 3 seeds. Our results show that NM-PPO is again able to outperform PPO while using 1/3 the number of total training steps. This result is shown in Figure D.2.
• Because of the additional RM in the above result, we allowed PPO five additional tuning runs (15 total) to adjust the coefficients on the component RMs. These runs are shown in Figure D.9.
• To visualize the effect of the number of queries to the evaluation function per run on the estimate of the evaluation function surface, we created contour plots of the predicted surface for three different query frequencies: 20 (the default), 10, and 5. This was done using the same data used to generate Figures 3.1 and 3.2. In Figure D.7, we can see that the basic structure of the predicted surface is preserved even down to 5 queries/run, indicating that much of the performance can be preserved with even more limited access to the evaluation function.

We'd like to thank all the reviewers once again for their time and help in reviewing the paper!