Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF

Thanks for your detailed review! We are happy to address your questions as follows.

1. The discussion of the expectation over prompt

Thanks for the good question! Our goal is to find the model $\pi$ that maximize the total expected reward $u(\pi)=\sum_{i=1}^m(\mathbb{E}\_{x\sim \rho, y\sim \pi}[r_i(x,y)])^p,$ where the aggregation is applied to the expected reward $u_i(\pi) = \mathbb{E}\_{x\sim \rho, y\sim \pi}[r_i(x,y)]$, rather than the reward for each prompt. In practice, the utility function (i.e., performance) of the language model is always measured over the entire prompt distribution, and our aggregation is applied to this overall utility.

We acknowledge that our introduction to multi-objective learning (namely, Eq.(1), (2) and the beginning of Section 3.1) may have been unclear and misrepresented our motivation, potentially leading to this misunderstanding. We can clarify it as follows: For a language model $\pi$ (and single group), the utility for the model and the objective $i$ and the group $n$ is defined as $u_i(\pi) = \mathbb{E}\_{x\sim \rho, y\sim \pi}[r_i(x,y)]$, and the model aims to maximize the aggregated social welfare $(\sum_{i=1}^m \alpha_i (u_i(\pi))^p)^{1/p}$. To make this problem more tractable, we try to minimize the distance between target set and current expected reward vector $S(\pi) = (u_1(\pi), u_2(\pi), \cdots, u_m(\pi))$ (ignore the regularizer for clarity), which is a reasonable alternative. (See Example 3.1 and 3.2 in our paper).

2. The experimental evaluation and discussion

We conduct our experiments on the Anthropic-HH dataset using the objectives Helpful, Harmless, and Humor, which are popular choices in previous MORLHF papers. We provide comprehensive results across different weights and objectives in both single-group and multi-group settings. Additionally, we include further experiments in the additional experiment part. We also provide an analysis for hyperparameter selection (the selection of $c$) in response to Reviewer MMQx.

The experimental results demonstrate that our approach achieves a smaller distance between the target set and the expected reward vector, and also achieves balance between diverse objectives. For example, with the weight (0.7,0.3), objectives 'helpful' and 'harmless', and the aggregation parameter $p=0.5$, MOPO get the utility $u_{MOPO}=(0.123,0.412)$, while $u_{RS}=(1.203,-0.707), u_{MOD}=(1.295, -0.8)$. This shows that MOPO not only achieves a smaller distance to the target set but also better balances multiple objectives, aligning with our theoretical motivation of striking a balance between linear aggregation and max-min RLHF.

Since the offline RLHF is more common, practical, and convenient (we can access offline dataset and reward-model easily), we only perform our method in offline RLHF. We believe that the online MOPO can also achieve better expected utility and better performance compared to linear aggregation, and we leave it as future work.

The AR baseline means that we directly aggregate the reward to $r(x,y) = (\sum_{i=1}^n \alpha_i r_i^p(x,y))^{1/p}$, and train the reward directly using PPO. Since it requires aggregating the individual reward, it cannot be applied to the negative reward setting, while our algorithm can also work for the negative reward. In the experiment, we perform $r_i(x,y) = \max\{r_i(x,y),0\}$ to change the reward into non-negative.

3. The paper is overly dense in many places.

Thanks for the reminder. We will improve our writing and proof to make it easier to follow in our final version.

4. Additional experiments.

We also perform experiment results on the Summarize-from-Feedback dataset. We fine-tune ALPACA-7B for objectives Summary and Faithful and get two policies $\pi_1, \pi_2$. Then, we perform experiments on these two policies. We use $\alpha=(1/2,1/2)$ and $p=0.5$ for the aggregation, choose $c$ adaptively by $c=u(\pi^t) + 0.01$ where $u(\pi^t)$ is the expected utility for $\pi^t$ at round $t$ (See Response for Reviewer MMQx for the selection of $c$). The results are shown in the following table, which shows that our MOPO performs better than RS and MOD.

5. Typos and definitions Thanks for the reminder. Eq. (3) should be $\theta_i$ instead of $\theta$, and $\Pi_W(V)$ means the projection of the point $V$ on the set $W$, which can be written as $\Pi_W(V) = \arg \min_{x \in W} \|\|V-x\|\|_2^2$.

6. Missing references

Thank you for providing additional relevant references! We will include a detailed discussion in our next version.

Thanks again for your time and effort! We will be happy to answer any further questions you may have.

Thanks for your positive response and time in reviewing our paper! We will address your questions as follows.

1. The paper assumes that the type/group of each human is known in advance.

In this paper, we assume the group information is known. However, if the group information is unknown, we can use EM or other clustering algorithms to first get an estimation of the group indexes. This approach is also used in previous work [1].

2. Does the paper assume that human provides pairwise feedback for all objectives?

Yes. We assume that humans can give feedback for all objectives. However, this is not a strong assumption. In fact, it is a common assumption that we have feedback or the reward model for all objectives in previous MORLHF works [2,3,4,5]. Also, in our online setting, we do not need the human to give all objectives for one response. Instead, we only need users to give pairwise feedback on the objectives they consider most important. Specifically, we ask them to first select the most important objective and provide a comparison between them. The selection of the objective is modeled by a softmax distribution based on the weighted reward gap. $\alpha_i \cdot (r_i(x,y_1)-r_i(x,y_2))$ (See Line 392 in our paper).

3. The assumption that objectives are explicitly available may not always hold in practice.

That's a great question! One possible way to address this challenge is to require a "reason" alongside the human's response. This reason could be a property or a short explanation, such as "helpfulness" or "Response 2 is too long". When the model encounters a reason that differs from the previous ones, it can identify this reason as a potentially new factor and add it as a new objective. This is an exciting and meaningful direction for future research, which we leave for future work.

4. The assumption 5.3 seems too strong.

In fact, the distribution of $y_1$ and $y_2$ will not be close as training advances: $y_1\sim \pi^*(y\mid x)$, which is the optimal policy, while $y_2\sim \pi_{ref}(y\mid x)$ only follows the reference policy (typically the policy after SFT). Therefore, we simply assume that the performance gap between the optimal human-aligned policy and the reference policy remains constant, which is generally a reasonable assumption in practice.

Thanks again for your time and effort! If you have more questions, we are happy to solve them.

[1]. Chakraborty et al. 2024: MaxMin-RLHF: Alignment with Diverse Human Preferences

[2]. Shi et al. 2024: Decoding-Time Language Model Alignment with Multiple Objectives.

[3]. Yang et al. 2024: Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment.

[4]. Wang et al. 2024: Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

[5]. Mukherjee et al. 2024: Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization

Thanks for your positive response and time in reviewing our paper! We will address your questions as follows.

1. Analysis of hyperparameter impacts and selection.

Since $\alpha$ and $p$ is assumed to be given in the experiment, the only hyperparameter specifically included in our algorithm is the parameter $c$, which represents the requirement by human.

If $c$ is too small, the target set is easily achieved, causing MOPO to lose the motivation to change direction. (In this case, the induced LLM is enough for achieving the human's requirement.) On the other hand, if $c$ is too large, the target set becomes distant, and the projection direction will tend to balance $[1/m,1/m,\cdots,1/m]$, which is also not conducive to maximizing the weighted p-norm expected utility. Hence, a value that is slightly larger than the expected utility could be suitable for $c$, for maximizing the expected utility function.

In practice, it can be selected in different ways: (a) Since it represents a requirement set by the human, it can be provided directly by the human. Note that in this case, if LLM already satisfied the requirement, it will have no motivation to further enhance the expected utility. (b) If the goal is to maximize the expected utility function $u(\pi)=\sum_{i=1}^m(\mathbb{E}\_{x\sim \rho, y\sim \pi}[r_i(x,y)])^p,$ $c$ can be adaptively chosen to be slightly larger than the current expected utility, such as $c = u(\pi^t) + 0.01$ after getting the policy $\pi^t$ at each round $t$. Then, the model will tend to maximize the expected utility by approaching the target set.

In experiments in our original paper, we manually choose the parameter $c$ that is slightly larger than the previously calculated expected utility. We also provide additional experiments using the adaptive approach (b) to validate our statements. See the additional experiment for details.

For other hyperparameters like $\beta$ and $\eta$, these are standard hyperparameters used in other works ($\beta$ in [1], $\eta$ in [4]). We can just use the standard configuration.

2. The lack of analytical experiments on multiple LLMs may limit the universality of the findings.

We conduct experiments on LLAMA2-7B, which is a commonly used model. We use three objectives to evaluate our algorithm since they are classical metrics that are used in previous works [1,2,3]. Since the primary contribution of this paper is theoretical, we provide preliminary experiments on popular benchmarks.

We also perform experiment results on the Summarize-from-Feedback dataset. We fine-tune a different LLM ALPACA-7B for objectives 'Summary' and 'Faithful' and get two policies $\pi_1, \pi_2$. Then, we perform experiments on these two policies. We use $\alpha=(1/2,1/2)$ and $p=0.5$ for the aggregation, and $c=u(\pi^t) + 0.01$ for round $t$. The results are shown in the following table, which shows that our MOPO performs better than RS and MOD, with adaptively selected parameter $c$.

3. Computational efficiency compared to non-linear aggregation baseline.

The nonlinear aggregation baseline in our paper is Aggregated Reward (AR), which means that we directly aggregate the reward to $r(x,y) = (\sum_{i=1}^n \alpha_i r_i^p(x,y))^{1/p}$, and training the reward directly using PPO. It cannot be directly applied to the negative reward setting because of p-norm aggregation. To address this, we transform the rewards into non-negative values by applying $r_i(x,y) = \max\\{r_i(x,y),0\\}$.

For computational efficiency, AR needs retraining each time we change the aggregation method or the weights, which takes more than 7 hours in our experiment and 30 minutes evaluation times per different weight. In contrast, our training-free algorithm does not require retraining. Instead, we use pre-calculated optimal policies for each objective, which only requires less than 30 minutes for evaluation to get the expected reward for each weight.

We will release our code later in our revised version. Thanks again for your time and effort! If you have more questions, we are happy to solve them.

[1]. Shi et al. 2024: Decoding-Time Language Model Alignment with Multiple Objectives.

[2]. Zhou et al. 2024: Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization.

[3]. Yang et al. 2024: Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment.

[4]. Cen et al. 2024: Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF.

Thanks for your positive response and meaningful review! We will address your question as follows.

1. If negative values occur, the paper should discuss how p-norm aggregation applies.

Both in theory and practice, our algorithm can handle both the negative reward and the positive reward. In fact, by shifting the objective from maximizing the aggregated expected reward to minimizing the distance between the target set and the expected reward vector, we can effectively handle situations where the expected reward is negative (as the expected reward vector can be negative while the target set remains in the positive range). In this case, the goal is to make the estimated reward vector (which may be negative) closely align with the target set. For the theoretical part, note that the p-norm aggregation appears in the definition of the target set, and the expected reward vector $S(\pi)$ can be negative. In practice, MOPO performs well in experiments where the reward is negative, whereas truncating rewards to ensure positivity and directly aggregating them leads to failure (AR in our table).

Moreover, the boundedness of the reward function is a common assumption and holds for most reward models.

2. The minimax formulation in Equations (6) and (7) is a central theoretical contribution. However, solving it might be computationally challenging; the paper would benefit from a more concrete explanation of how one can efficiently compute or approximate the solution in large-scale settings.

This is an important question. We have two approaches to make it practical. The first way is to use a reward-free DPO algorithm, as in Eq.(8) in our paper. We can further simplify the Eq.(8) by replace $\log \pi_\theta(y\mid x)$ to $\alpha_i \log \pi_{\theta_i}(y\mid x)$, and Eq.(8) becomes decentralized optimization, where $\\theta_i = \arg\min_{\theta}\beta \alpha_i\mathbb{E}\_{\pi_{base}} [\log \pi_{\theta}(y\mid x) - \eta \ell(D_i, \theta_i)]$ for each objective $i$, and we can perform VPO [1] for estimating $\theta_i$.

Second, one can ignore the first exploration term, and Eq.(6) and Eq.(7) become traditional MLE for estimating the reward model. Also, one can use a pre-existing reward model if it is available. In our practical MOPO algorithm, we use a pre-existing reward model and use PPO to get the optimal policy $\pi_{\theta_i}$ for each objective. Then, Line 2 and Line 3 in Algorithm 1 can be implemented by previous linear MORLHF algorithms like MOD and RS. We will discuss it in detail in our next version.

3. While the results on anthropic HH illustrate the method’s viability, this dataset is somewhat outdated. Testing on newer datasets, such as HelpSteer2, with higher-quality multi-objective labels and more categories would better demonstrate robustness and broad applicability.

We also perform experiment results on the Summarize-from-Feedback dataset [3]. We fine-tune a different LLM ALPACA-7B for objectives 'Summary' and 'Faithful' and get two policies $\pi_1, \pi_2$. Then, we perform experiments on these two policies. We use $\alpha=(1/2,1/2)$ and $p=0.5$ for the aggregation and choose $c$ adaptively by $c=u(\pi^t) + 0.01$ where $u(\pi^t)$ is the expected utility for $\pi^t$ at round $t$ (See Response for Reviewer MMQx for the selection of $c$ and Response for reviewer uhy5 for the definition of $u(\pi)$).

The following table shows the aggregated expected utility, which shows that our MOPO performs better than RS and MOD. Note that we do not select the HelpSteer dataset since the variance of its reward model is too small (see Appendix F.4 in [2]), making it less suitable for demonstrating the differences between linear and non-linear aggregation.

Thanks again for your time and effort! We will be happy to answer any further questions you may have.

[1]. Cen et al. 2024: Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF.

[2]. Shi et al. 2024: Decoding-Time Language Model Alignment with Multiple Objectives.

[3]. Stiennon et al. 2024: Learning to summarize from human feedback.