MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

We greatly appreciate your constructive and insightful feedback! Here we provide a detailed response to address all of your concerns below.

Confusion of Algorithm 1 and Figure 1.

Thank you for the question and apologies for any confusion. As shown in Thm3.4, we have $\pi^*(y|x) \propto \prod_{k=1}^K\left(\pi_k(y|x)\right)^{\lambda_k^*},$ which can be seen as a logit-level aggregation of multiple LLMs. However, our main goal is to compute $\lambda^*$ that maximizes the minimum reward, which balances the trade-offs among different objectives. Based on your comment, we will revise Figure 1 to better highlight the role of $\lambda$ in shaping the final policy.

Evaluation of $\pi_{helpful/harmless/humorous}$ in Table 1.

Thank you for your feedback.

1. $\pi_{helpful/harmless/humorous}$ in Table 1 were trained by our own, and the results demonstrate a significant surplus in the corresponding rewards—indicating the effectiveness of our single-objective policies.
2. However, our primary goal is to balance multiple objectives rather than to optimize individual objectives. Given the differences in training data, we believe that direct comparisons of single-objective policies may not be entirely fair.
3. Nonetheless, we have added further comparisons with the PKU-SafeRLHF model with MPO under $\beta=0.1$ in Eq.10. As shown in FIgure 4, our $\pi_{MPO}$ mostly rely on $\pi_{helpful}$ and $\pi_{harmless}$, which aligns closely with the objectives considered in the PKU-SafeRLHF. The results below show that MPO still achieves the highest minimum win rate.

Table : Win rate(%) against the Reference Model

4. Moreover, the normalized reward for $\pi_{MPO}$ and $\pi_{PKU}$ are listed below:

Here, larger normalized rewards indicate better alignment with the corresponding objective. These results indicate that the PKU-SafeRLHF model places a greater emphasis on helpfulness, which is likely due to its constrained optimization loss. We will add more discussion with PKU-SafeRLHF in the revision.

Aligned model is only marginally better than the baseline. Are such results reasonable?

Thank you for the question.

1. Table 1 demonstrates MPO's optimality among all methods under the max-min setting, with the aligned model outperforming the baseline on all individual objectives.
2. The improvement appears only marginal because of the inherent conflict between objectives such as helpfulness and harmlessness. Multi-objective alignment tasks involve complex trade-offs, and simultaneously improving all conflicting objectives is nearly impossible [1]. This challenge is precisely why we consider the max-min setting in our work.

References: [1] Rame, A, et al. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. NeurIPS 2023.

What is the role of the $\beta$ during the alignment process?

Thank you for your question.

1. While a larger $\beta$ encourages policy to stay close to the reference model, it plays a critical role in balancing conflicting objectives in our multi-objective setting. Stronger regularization helps stabilize aggregation and maintain desirable baseline behaviors while optimizing the minimum reward.
2. $\beta$ is a tunable hyperparameter. As shown in Table 1, $\beta=0.5$ outperforms $\beta=0.1$ and $\beta=\infty$. In particular, the case of $\beta=\infty$ collapsed to the reference policy with a 50% win rate. This highlights the importance of tuning $\beta$ for optimal performance.

In practice, the reviewer thinks it is quite difficult for us to serve multiple LLMs at the same time. Especially when the model is large. If the authors adopt the model merging method, this should not be a problem.

1. We understand the concern about serving multiple LLMs concurrently. However, the efficiency of MPO primarily comes from the training phase. In Section 4.2 of our experiments, training policies using PPO-based approaches (e.g., MaxMin-RLHF, MORLHF) requires approximately 10 A100 GPU hours, whereas MPO only requires around 2.5 A100 GPU hours, since it avoids reinforcement learning step.
2. In the inference phase, instead of running full inference on all LLMs simultaneously, we can compute single-objective policy outputs logits in parallel, then aggregate logits using the MPO framework.
3. Additionally, as shown in Table 1, our method achieves better alignment than parameter merging baselines like reward soups and future work will focus on further optimizing the inference pipelines to enhance the scalability of MPO in real-world applications.

We greatly appreciate your constructive and insightful feedback! Here we provide a detailed response to address all of your concerns.

Differences between main theorem and the one in Shi et al., 2024

Thank you for your feedback. Compared to Thm 1 in Shi et al., our approach differs in both objective and applicable setting, with some overlap in a special case:

1. Our Thm 3.4 addresses the max-min setting without explicit preference weights, whereas Shi et al. uses predefined weights. In the special case of linear reward aggregation, Lem 3.9 shows that the optimization leads to a closed-form solution, which indeed reaches same conclusion in Shi et al.
2. We introduce an auxiliary normalizing operator for rewards, which is crucial for transforming the optimization over $\lambda$ into Eq12 (line 199). Without it, the reward function cannot be avoided in optimization.
3. In terms of the derivation, Shi et al. uses a Legendre transform, converting the problem into $\max_y \pi_{ref}(y|x) \text{~~such that~} r(y∣x)>C,$ where $C$ is unspecified. In contrast, our proof uses a direct reward–policy mapping, leading to the closed-form expression for $\pi^*$ directly, providing a more interpretable and transparent theoretical derivation.

The use of mirror descent for weight adjustment closely resembles the approach in Ramesh et al., 2024.

Indeed the high-level idea of Ramesh et al. is to perform robust alignment, which is similar to us. However, we highlight several differences in terms of less resources required, applicable settings, and methodology developed.

1. Ramesh et al. aims to derive a group robust preference optimization objective and conduct robust alignment from scratch, which is computationally expensive and requires extensive hyperparameter tuning. In contrast, we could directly use existing single-objective policies, avoiding full retraining and significantly reducing computational cost.
2. By reusing pretrained or open-source LLMs, we simplify robust training to only updating preference weights—a lightweight post-processing step that aligns better with practical academic and industry use cases. Ramesh et al. 's method is better suited for settings where LLMs must be trained from scratch.
3. Methodologically, Ramesh et al. uses a gradient descent-mirror ascent method to update the policy and the weight simultaneously as its objective is to solve a min-max optimization. In addition, it has access to unbiased gradient estimators. In our case, since we do not train policies, weight optimization becomes a conditional stochastic problem without unbiased gradient estimators. To tackle this, we designed a biased mirror descent method and analyzed its convergence.

Could the authors provide further clarification on why MPO is not computationally expensive?

Thanks for your question.

1. While standard RLHF requires training multiple reward models, this can be avoided by using DPO, which is mathematically equivalent and was adopted in our experiments for efficiency.
2. For $dim(\lambda=3)$, training the policy via MORLHF (or maxmin-RLHF) for a fixed $\lambda$ takes approximately 10 A100 GPU hours, whereas solving for $\lambda$ in MPO only takes 2.5 A100 GPU hours.
3. Unlike standard MORLHF, which re-runs PPO for different $\lambda$, MPO avoids this overhead by efficiently combining logits. While Reward Soups has similar cost with predefined weights, MPO achieves better alignment (shown Table 1), highlighting its effectiveness in balancing multiple onjectives.

Comparisons with more prior works.

Thank you for the suggestion, we have added further comparisons with prior works:

1. We compare our approach with MORLHF, which employs linear aggregated rewards using PPO, as well as with reward soups that utilize learned weights. As shown in Table 1, our MPO method still achieves the highest minimum win rate, where $\pi_{Weighted RS}$ corresponds to the Reward Soups with learned weights

2. While MOD employs a linear combination of logits, in Figure 2(b), we have evaluated $\pi(y|x) \propto \pi_1(y|x)^{\lambda_1}\pi_2(y|x)^{1-\lambda_1}$ with different $\lambda_1 \in \Lambda_{grid}= \\{0.0, 0.2, 0.4, 0.6, 0.8, 1.0\\}$. Our results show that the policy obtained via MPO achieves the best objective performance across this grid, outperforming MOD with predefined weights in the max-min setting.

We greatly appreciate your constructive and insightful feedback! Here we provide a detailed response to address all of your concerns.

How can the correctness of Eq. 10 be verified?

Thanks for the question. As we explained after Eq. 10 in our original submission, the result follows directly from Sion’s minimax theorem. The objective function is convex in $\lambda$ ($\pi_{\theta}$ fixed)​ and concave in $\pi_{\theta}$​ ($\lambda$ fixed), which satisfies the conditions for the theorem and guarantees the interchange of max and min in Eq. 10.

$\pi^*$ is a global and certain optimality of the policy, but $r^*$ is obtained by stochastic descent in Alg. 2. It is unclear how the estimation of $r^*$ can approximate optimize the objective of Eq. 10.

1. If $r^*$ refers to the reward model, our approach does not involve reward training; Instead, we use an auxiliary normalizing operator to directly derive the optimal policy $\pi^*(y|x) \propto \prod_{k=1}^K\left(\pi_k(y|x)\right)^{\lambda_k^*}$
2. If $r^*$ refers to the preference weight $\hat{\lambda}$ as solved by Alg. 2, then Thm 3.8 provides a KL-based error bound between the learned policy $\hat{\pi}$ and $\pi^*$. The theorem formally states that $\hat{\pi}$ closely approximates $\pi^*$ under mild conditions.

The optimization of $\pi^*$ is with respect to the input-output (x,y). However, the optimization of $r^*$ is input-output agnostic.

Thank you for your question. Again, we believe you are referring to $\lambda$. As shown in Thm 3.4, our main task is to solve for $\lambda^*$ via Eq .12. Once obtained, the optimal policy is effectively a linear combination of the logits from the single-objective policies. Additionally, when applying Alg 2, we only need the individual policies and a set of prompts.

Can we improve the size of $\lambda$ in the experiments?

Thank you for your comment. We would like to point out that $dim⁡(\lambda)=3$ is consistent with prior works [1,2,3], which also adopts up to three objectives. This dimensionality has proven effective for capturing the scalability of multi-objective alignment, balancing soundness and computational efficiency.

References:

1. Chakraborty, S, et al. MaxMin-RLHF: Alignment with diverse human preferences. ICML 2024.
2. Yang, R, et al. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. NeurIPS 2024.
3. Zhou, Z, et al. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708 (2023).

The comparison does not consider other important RLHF baselines such as PPO-based approaches.

Thank you for the comment. Standard MORLHF approaches [1,2] optimize linearly aggregated reward functions. while Maxmin-RLHF perform PPO to obtain the policy by optimizing: $\max_{\pi} \min_k E\left[r_{k}(x,y) \right] - \beta D_{\mathrm{KL}}\left[\pi \Vert \pi_{\text{ref}}\right].$ In light of your comments, we have added more comparisons of such methods. Notably, our MPO method still achieves the highest minimum win rate.

References:

1. Ji, J, et al. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. NeurIPS 2023.
2. Wu, Z, et al. Fine-grained human feedback gives better rewards for language model training. NeurIPS 2023.

The analysis of training costs is missing in the experimental analyses.

Thanks for the suggestion. For computation cost, training the policy using MORLHF (or maxmin-RLHF) with aggregated reward models takes approximately 10 A100 GPU hours, since both methods rely on PPO for policy optimization and differ only in how they aggregate reward functions, In contrast, our approach avoids PPO entirely, solving for the preference weights $\lambda$ via Alg 2 requires only about 2.5 A100 GPU hours, offering a significant reduction in training time while still achieving competitive performance.

Where are the key novelty and soundness compared with MORLHF and MaxMin-RLHF?

Thanks for the comment, We have stated the key novelty and soundness of MPO compared to MORLHF and MaxMin-RLHF in both the Introduction and Conclusion sections. In summary:

1. MPO connects reward aggregation to policy aggregation, yielding a closed-form aggregation rule. 2. Our approach operates directly on single-objective policies, avoiding extra RL updates and reward model training, thereby significantly reducing computational overhead. 3. The method is backed by rigorous theoretical error bounds that ensure robustness relative to the optimal policy.

We greatly appreciate your constructive and insightful feedback! Here we provide a detailed response to address all of your concerns below.

It would be better to add more validations on the GPT-based evaluation.

Thanks for the suggestion. When utilizing GPT-based evaluations, we have experimented with multiple GPT versions and leveraged prompts similar to previous works such as [1] and [2]. We found that while there are some variations in the output, the overall performance trends remain stable, which gives us confidence in the reliability of this evaluation method. We are also open to further validations and comparisons to ensure that our evaluation framework is as comprehensive and robust as possible.

References:

1. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS 2023.
2. Zhou, Z., Liu, J., Shao, J., Yue, X., Yang, C., Ouyang, W. and Qiao, Y., 2023. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708.

It would be better to include a discussion on the scalability of MPO with respect to computation resources when increasing the number of objectives.

Thanks for the question. From our observation, computational cost increases approximately linearly with the dimensionality of $\lambda$. Specifically, obtaining an approximate optimal $\lambda$ (using 600 iterations of Algorithm 2) requires:

This linear scaling indicates that the computational burden remains manageable when increasing the number of objectives.