One Model for All: Multi-Objective Controllable Language Models

W3. Limited novelty: The idea is similar to RiC (Yang et al., 2024).

A3: While both RiC and MOC aim to personalize LLMs, the following significant differences highlight the novelty and distinct contributions of MOC:

• Formulation and Approach:

  • MOC formulates controllability as a multi-objective policy optimization with preference-based constraints, solved via the proposed MOC algorithm.
  • A key novelty is the surrogate problem (Equation 12), which reduces computational cost to near that of a single-objective PPO.
  • In contrast, RiC relies on rejection sampling and lacks an explicit policy optimization framework, making its methodology fundamentally different.

• Explicit Policy Optimization and Controllability:

  • MOC explicitly optimizes the policy to align model behavior with user preferences, establishing a rigorous and systematic controllability framework.
  • RiC does not perform explicit policy optimization, limiting its ability to maximize reward while aligning with preferences.

• Performance Advantages:

  • Thanks to its principled design, MOC significantly outperforms RiC in controllability, solution quality, diversity, and generalization, as substantiated by quantitative results in Tables 2–6.

We summarize these key differences below for clarity:

W4.1 Weak experimental design: The paper lacks a comparison with the MODPO method, which could provide valuable insights into the proposed method's effectiveness against a strong baseline.

A4.1 Thank you for highlighting the relevance of MODPO to our work. We have added a comparison with MODPO in Table 1 and included another strong baseline, MetaAligner (in NeurIPS 2024 [1]).

However, it is important to emphasize that MOC focuses on controllability with two critical goals:

• Training one LLM to generate personalized outputs for diverse user preferences.
• Generalizing to unseen preferences with the once-trained LLM.

In contrast, MODPO:

• Trains M separate LLMs (where M corresponds to the number of preferences), targeting only a fixed set of pre-defined preferences.
• Does not consider generalization to unseen preferences.

These differences highlight the fundamental distinctions between MODPO's methodology and objectives and those of MOC.

W4.2 The experiments are limited to the Llama2-7B model. Testing the proposed method across a broader range of model sizes and types would strengthen the validity of the findings.

A4.2: Thank you for suggesting an expansion of our evaluation. In response, we conducted additional experiments to address this concern:

• New Dataset: We included the Reddit Summary dataset [2].
• Additional Baseline: We compared against MetaAligner [1].
• Model Variants and sizes: We tested MOC on Llama-3-8B [3].

We have included these experiments in Appendix H of our revised paper.

Experiment: HH-RLHF

• Setup: Llama-3-8B, with MetaAligner added as a new baseline. MetaAligner seeks to optimize all rewards in this setting.
• Results: As detailed in the table below, MOC-Llama3-8B dominates MetaAligner and other baselines in hyper-volume:

• Visualization: The results are presented in this link: https://anonymous.4open.science/r/MOC-9E51/llama3_8b_helpful_harmless.svg , highlighting MOC's superior performance compared to other methods

Experiment: Reddit Summary. Llama-3-8B with Summary (measure the quality of the summary) and Faithful (measures the faithfulness of the summary for the original post) reward models.

• Results: MOC outperforms the baseline (RiC) significantly in terms of rewards:

• Visualization: https://anonymous.4open.science/r/MOC-9E51/RiC_Faithful_Summary.svg , showing MOC significantly outperforms baseline in terms of the rewards.

Dear Reviewer gxJD,

We sincerely appreciate your positive feedback on the clarity of our writing, the importance of the problem, and the efficient optimization we proposed. Thanks for your thoughtful comments. Below, we address each concern in detail.

W1. Lack of Clarity in Formulas: The mathematical formulation is unclear, particularly in Equation 5, where the reward model $R^k$ is fixed. The sampled response $y$ does not incorporate the parameter $\theta$, so how to compute its gradient with respect to $\theta$?

A1: We apologize for any confusion caused by the notation. In the paper, as explained in lines 144–145, $x$ represents the prompt/query, $y \sim \pi(x;\theta, \mathbf{p}^i)$ is LLM-generated response, and $\mathcal{D}$ is the prompt dataset. The reward vector $\mathbf{R}(x,y) = (R^1(x,y), R^2(x,y), \cdots, R^N(x,y))$ is associated with the $N$ optimization objectives $\\{J^i\\}_{i=1}^N$.

This setup is consistent with standard RLHF. The response $y$ depends on the policy parameters $\theta$, which propagates through the sampling process $y \sim \pi(x; \theta, \mathbf{p}^i)$. This dependency allows the gradient of $R^k$ with respect to $\theta$ to be computed using standard policy-gradient techniques. To address this concern more explicitly, we have revised the manuscript to more clearly emphasize the dependency of $y$ on $\theta$ and to highlight its equivalence to standard RLHF practices.

W2. Marginal Improvement: The experimental results show that the Multi-Objective Control (MOC) does not significantly outperform other baselines in terms of solution quality, particularly in the Harmless & Helpful setting (as indicated in Figure 2(b)). The performance is also not clearly superior in the Humor & Helpful setting.

A2: We respectfully argue that MOC provides significant and consistent advantages across key dimensions: controllability, solution quality, diversity, and generalization. Below, we provide additional evidence to support these claims:

• Controllability: MOC excels in aligning model behavior with diverse preferences, as measured by the local order rate, which quantifies the proportion of adjacent data points maintaining a monotonic relationship with input preference vectors. As reported in Table 2, MOC achieves a local order rate of 0.889, substantially outperforming baseline methods, all of which fail to preserve this order. This result highlights MOC’s superior ability to adapt to preference.

• Solution Quality: MOC produces higher-quality solutions by converging more closely to the Pareto front, as assessed using the hyper-volume indicator, a standard metric for multi-objective optimization. Table 3 shows that MOC achieves a hyper-volume score of 10.91, significantly surpassing the baselines (RiC: 7.97, MORLHF: 7.90, Rewarded Soups: 7.50). These results confirm that MOC produces high-quality solutions and balances multiple objectives effectively.

• Diversity of Solutions: MOC generates a broader range of trade-offs, as reflected by higher entropy values. In Table 4, MOC achieves an entropy score of 1.76, outperforming baselines (RiC: 1.51, MORLHF: 1.47, Rewarded Soups: 1.63). This underscores MOC’s ability to produce a more diverse solution set, catering to varying preferences.

• Generalization to unseen preferences: MOC demonstrates robustness in generalizing to unseen preferences. In Figure 3, MOC's solutions dominate RiC's solutions in all cases. Furthermore, MOC consistently outperforms baseline methods under varying conditions, as quantified by hyper-volume (Table 5) and local order rate (Table 6). These results validate MOC’s adaptability and its robustness in handling diverse scenarios.

Thanks for the clarifications. After reading the response, I have decided to keep my rating.

Dear Reviewer gxJD,

We conducted additional experiments to include MODPO [1] as a baseline, and we updated the results in Appendix H. The new results demonstrated that our proposed method, MOC, significantly outperforms MODPO and other baselines, particularly in terms of hyper-volume:

• Visualization: The results are presented in this link: https://anonymous.4open.science/r/MOC-9E51/llama3_8b_helpful_harmless.svg , highlighting MOC's superior performance compared to other methods

• Setup: Llama-3-8B, with MODPO added as a new baseline. MODPO seeks to maximize all rewards in this setting.

We are grateful for your feedback, as it allowed us to strengthen our comparisons and further validate the contributions of our method.

Thank you again for your comments.

Best wishes,

The authors

[1] Zhou Z, Liu J, Shao J, et al. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization[C]//Findings of the Association for Computational Linguistics ACL 2024. 2024: 10586-10613.

Dear Reviewer gxJD,

We hope this message finds you well. As the rebuttal period concludes in less than 24 hours, we wanted to kindly remind you of our responses to your comments, particularly addressing the comparison with MODPO. In our revised paper, we have included additional experiments demonstrating that MOC significantly outperforms MODPO and other baselines.

Your feedback has been invaluable in strengthening our work, and we are eager to address any remaining concerns you may have. If there are additional clarifications or points you need further clarify, please let us know.

Thank you again for your time.

Best regards,

The Authors

Dear Reviewer BCRA,

Thank you for your detailed and constructive feedback on our submission. Below, we provide clarifications, additional experiments, and evidence to address the concerns you raised.

W1&2: The evaluation could be broadened by including more datasets and comparing MOC against additional baselines like Personalized Soups and MetaAligner. The experiments are conducted solely with Llama 2 7B. Testing the approach on other models like Mistral or Gemma, and at different scales such as 13B, would strengthen the claim.

A1&2.

We appreciate your suggestion to expand the evaluation. In response, we have conducted additional experiments:

• New Dataset: We included the Reddit Summary dataset [1].
• Additional Baseline: We compared against MetaAligner [2].
• Model Variants and sizes: We tested MOC on Llama-3-8B [7].

These experiments are detailed in Appendix H of our revised paper.

Experiment: HH-RLHF

• Setup: Llama-3-8B, with MetaAligner added as a new baseline. MetaAligner aims to maximize all the rewards in this setting.
• Results: As detailed in the table below, MOC-Llama3-8B dominates MetaAligner and other baselines in hyper-volume:

• Visualization: The results are presented in this anonymous link: https://anonymous.4open.science/r/MOC-9E51/llama3_8b_helpful_harmless.svg , showing MOC's substantial performance compared to others.

Experiment: Reddit Summary. Llama-3-8B with Summary (measure the quality of the summary) and Faithful (measures the faithfulness of the summary for the original post) reward models.

• Results: MOC significantly surpasses the baseline (RiC) in rewards:

• Visualization: https://anonymous.4open.science/r/MOC-9E51/RiC_Faithful_Summary.svg , showing MOC significantly outperforms baseline in terms of the rewards.

W3: The method assumes the availability of N reward models or requires training N reward models, which can be computationally intensive

A3. We appreciate your comment on the computational burden of $N$ reward models. Below, we outline why MOC is computationally efficient.

• Reward Models in RLHF: Utilizing reward models is a standard paradigm in RLHF. These models are essential for capturing and quantifying human preferences. The use of $N$ reward models aligns with the RLHF framework.

• Minimal Overhead. MOC requires training only a single model, unlike:

  • Rewarded Soup: [3] requires training $N$ models, each fine-tuned on a single reward model, and then combines their weights. This approach incurs significantly higher computational costs.
  • MODPO: [4] trains $M$ separate models (where $M$ is the number of user-defined preferences, i.e., 20 preference, 20 language models), further increasing the computational burden.
  • RiC: [5] Conditions on multiple rewards but lacks explicit policy improvement, which can limit performance and necessitate additional computational adjustments of fine-tuning.
  • Our method, in contrast, trains a single model that can generate to unseen preferences, minimizing training overhead while superior performance.

• Empirical Resource Efficiency: Our method demonstrates exceptional resource efficiency:

  • MOC allows fine-tuning a 7B parameter LLMs using a single A6000 GPU, as we did in the paper.
  • Our experiments, as detailed in Sections 4.2 and 4.3, demonstrate superior performance with minimal resource consumption across hypervolume, alignment results, and generalization.

W4: The effectiveness of MOC isn't clearly illustrated in Figure 2b.

A4: We appreciate your feedback regarding the clarity of MOC’s effectiveness in Figure 2b. Although Figure 2b may not immediately highlight MOC's advantages, Tables 2, 3, and 4 provide robust quantitative evidence demonstrating its effectiveness across three dimensions: controllability, solution quality, and diversity."

1. Controllability:

   • Metric: We use the local order rate, which quantifies the proportion of adjacent data points maintaining a monotonic relationship with the input preference vectors.
   • Results: As reported in Table 2, MOC achieves a local order rate of 0.778, significantly surpassing baseline methods, all of which fail to preserve order consistently.
   • Visualization: As shown in Figure 2b, MOC’s solutions exhibit a smoother progression compared to the scattered behavior of the baselines.

2. Solution Quality:

   • Metric: We evaluate solution quality using the hyper-volume indicator, a standard metric for assessing the quality of solution sets.
   • Results: As shown in Table 3, MOC achieves a hyper-volume score of 9.51, outperforming RiC (9.26), MORLHF (9.05), and Rewarded Soups (8.90). This demonstrates MOC’s ability to produce high quality solutions that better balance the competing objectives.

3. Diversity of Solutions:

   • Metric: We measure diversity using entropy, where higher values indicate a broader exploration of trade-offs across the Pareto front.
   • Results: As presented in Table 4, MOC achieves an entropy score of 1.83, outperforming RiC (1.47), MORLHF (1.33), and Rewarded Soups (1.59). This highlights MOC's ability to effectively generate a more diverse set of solutions

Q5. Did you utilize pre-trained reward models, or did you train the reward models as part of your work?

A5. To ensure consistency and reproducibility, we utilized publicly available pre-trained reward models. Table 9 in Appendix E provides details on the models used:

Leveraging these pre-trained models ensured a robust and widely accepted evaluation framework.

Thank you for conducting the additional experiments and clarifying the questions. I will increase my rating.

Dear Reviewer BCRA,

Thank you for reviewing our clarifications and additional experiments :P We greatly appreciate your constructive feedback and your support to increase the rating. Your insights have been invaluable in improving our work.

Best wishes,

The authors

Dear Reviewer 315N,

Thank you for your thoughtful comments and for the time you dedicated to reviewing our paper. Below, we provide detailed responses to address your concerns.

W1: I still have concerns about the limitations in extreme preference scenarios, where trade-offs between objectives might not fully capture nuanced user expectations.

A1: Thank you for raising this point regarding extreme preference scenarios. We would like to clarify how our approach addresses this concern:

1. Preference Vector and Pareto Front Coverage: The preference vector $\mathbf{p} = [p_1, p_2, \cdots, p_N]$, where $0 \leq p_i \leq 1$ and $\sum_{i} p_i = 1$, is explicitly designed to map user preferences across the Pareto front comprehensively. By representing multi-objective preference through weights $[w_1, w_2, \cdots, w_N]$ (where $0 \leq w_i \leq 1$), the preference vector ensures complete and flexible coverage of trade-offs. Consequently, our method effectively represents diverse user expectations within the defined preference space.

2. Extreme Preferences and Human Judgment: For scenarios involving extreme or highly nuanced preferences, additional refinement through human feedback is beneficial. This refinement requires precise data annotation during human preference collection, which is outside the primary scope of our MOC algorithm. Our approach focuses on controllability in multi-objective optimization rather than granular data annotation. As such, MOC provides a strong foundation for alignment with extreme preferences.

W2: Additionally, while the computational cost is comparable to single-objective PPO methods, more complex applications might still require higher computational resources.

A2: We thank the reviewer for pointing out the potential computational considerations for complex applications. We address this concern from both theoretical and empirical perspectives:

1. Theoretical Perspective: Single-objective PPO represents the minimum computational costs for PPO-based methods. Our MOC algorithm is designed to maintain this efficiency, even when addressing more complex scenarios. Specifically:

   • MOC requires training only one model with inference adaptation and does not rely on human preference data.
   • MOC enables explicit policy improvement and policy controllability, which distinguishes it from alternatives:
     • MORLHF: Lacks theoretical guarantees, cannot support inference adaptation. Train $M$ LLMs. $M$ is the number of user preferences.
     • Rewarded Soup: [1] Requires training $N$ models (where $N$ is the number of reward models), lacks explicit controllability.
     • RiC: [2] Does not provide explicit policy improvement.
       These advantages, as summarized in Table 1 of our paper, establish MOC as a computationally efficient alternative to other methods.

2. Empirical Perspective: Experimentally, our results validate the resource efficiency of MOC:

   • MOC achieves superior results while fine-tuning a single 7B parameter model on a single A6000 GPU. This approach is significantly more resource-efficient than baseline methods.
   • Across all benchmarks, MOC demonstrates superior performance in terms of controllability, solution quality, diversity, and generalization to unseen preferences (see Sections 4.2 and 4.3)

In conclusion, while it is expected that more complex applications may demand higher computational resources, MOC’s design ensures resource efficiency. Both theoretical guarantees and empirical results establish MOC as a highly scalable solution for multi-objective controllability, even in more complex scenarios.

Thank you again for your valuable feedback. Please let us know if further clarification is required.

Best regards,

The authors

[1] Rame A, Couairon G, Dancette C, et al. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2] Yang R, Pan X, Luo F, et al. Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment[C]//Forty-first International Conference on Machine Learning.

Dear Reviewer 315N,

We hope this comment finds you well. We wanted to kindly remind you to review our rebuttal and share any additional comments or feedback you may have. Your input is highly valued, and we appreciate the time and effort you dedicate to this process.

Thank you again for your consideration.

Best regards,

The Authors

Dear Reviewer 315N,

We hope this message finds you well. Thank you again for your earlier comments and insights. We have provided detailed responses and updates to your concerns and would greatly appreciate any further feedback or questions you might have before the discussion phase concludes.

Your perspective is highly valued, and we are eager to address any remaining points to ensure the clarity and rigor of our work. Please don’t hesitate to reach out if additional clarifications are needed.

Thank you for your time and consideration.

Best regards,

The Authors

Dear Reviewer 315N,

We hope this message finds you well. As the rebuttal period will end in less than 24 hours, we kindly remind you to review our earlier response to your valuable feedback. We have addressed your concerns regarding extreme preference scenarios and computational costs with detailed explanations.

If there are any remaining questions or further clarifications you would like us to provide, we are fully committed to addressing them before the deadline. We deeply appreciate your time and effort.

Thank you once again for your review and consideration.

Best regards,

The Authors

Dear Reviewer PAko,

We sincerely thank the reviewer for the insightful comments and the recognition of the importance of the controllability, the novelty of our method, and the comprehensiveness of our evaluation. Below, we address each of the concerns raised.

W1: The motivation of MOC are not clearly stated. However, this paper does not adequately discuss how previous methods view this problem and what the limitation of the previous approaches are. Therefore, the motivation behind formulating the problem as a MOO problem is not well-explained. Clarifying these shall make readers better understand the rationale of the method.

A1: Thank you for highlighting the importance of better articulating the motivation and limitations of prior methods. Below, we elaborate on these aspects:

1. Motivation for MOC:

   • Current LLMs are trained to align with fixed, developer-defined preferences, limiting their flexibility to address diverse user needs. This rigidity necessitates a paradigm that enables fine-grained controllability, enabling users to dynamically prioritize different objectives.
   • Existing methods require either multiple specialized models or extensive preference datasets, both of which are resource-intensive and less scalable. In contrast, MOC formulates controllability as a multi-objective optimization (MOO) task, enabling dynamic preference-based trade-offs using a single model trained once.

2. Limitations of Prior Methods.

   • Rewarded Soup [1] specializes $N$ LLMs independently (one per reward model) and then interpolates their weights linearly.
   • MORLHF solve this problem by training individual $M$ LLMs with the the linearized rewards, where $M$ corresponds to the number of user-defined preferences.
   • RiC [2] : Conditions LLM responses on multiple rewards using prompt conditioning, trained via rejection sampling.
   • MODPO [3] trains $M$ LLMs, where $M$ corresponds to the number of preference, by optimizing each model with a specific weighted combination of reward objectives given the corresponding reward models.

3. Novelty of MOC:

   • MOC uses a single LLM to dynamically adjust trade-offs among objectives based on user preferences.
   • MOC explicitly improves policies while avoiding the need for multiple LLMs or preference datasets with a MOO problem formulation. This scalability and flexibility distinguish MOC from prior work.

Comparison table: To clarify MOC’s contribution, we summarize key differences with prior work in the table below:

W2: Some parts in method look confusing to me. For example, in equation (1), there seems a ambiguity regarding whether $J^i$ is a scaler or a vector. If it is a scalar, why there is a transpose ($^\top$)? Otherwise, whats the definition of taking maximum over a bunch of vectors? Also, the intuition behind constraint in (1) could be better explained. Usually we want reward to be as large as possible, so why we want $R^i(x,y) \approx p_i$ even when $p_i$ is small?

A2:

• $J^i$ is a Scalar: Each $J^i(\pi(\cdot; \theta, \mathbf{p}^i))$ is a scalar representing the value of the $i$-th objective function.

• Purpose of the Transpose$^\top$: The notation $(J^1, J^2, \dots, J^N)^\top$ represents a column vector, ensuring compatibility with any potential matrix operations.

• Maximizing in MOO: In the context of MOO, "maximizing" the vector $\mathbf{J}$ refers to finding Pareto optimal solutions. A Pareto-optimal solution ensures no objective can be improved without deteriorating the others. This aligns with standard multi-objective optimization definitions.

• Pareto Optimality: The concept of Pareto optimality replaces the idea of a single global maximum in multi-objective contexts, providing a rigorous foundation for optimizing multiple competing objectives

• Purpose of the Constraint: The constraint ensures that the behavior of LLM (the rewards) aligns with user-preference** $\mathbf{p}^i$, acheving the controllability that MOC focus on. Maximizing reward is achieved through the optimization in equation 1: $\max_\theta \mathbf{J} (\pi (\cdot; \theta, \mathbf{p}^i)) := \max_\theta (J^1(\pi (\cdot; \theta, \mathbf{p}^i)), J^2(\pi (\cdot; \theta, \mathbf{p}^i)), \cdots, J^N(\pi (\cdot; \theta, \mathbf{p}^i)))^\top$. This constraint ensures alignment with user priorities.

• To ensure consistency in scale between the rewards and the preference vector, the rewards are normalized to align with the scale of the preference vector. The reward signal is normalized by $r = \frac{r-r_\text{mean}}{2 r_{\text{std}}}+1$, as detailed in Appendix E, where the mean and std are computed by running mean in [4], which is a general practice in deep reinforcement learning [4]. This ensures consistency across rewards and preferences. We have updated Appendix G with additional clarifications.

W3: The experiment is restricted to three objective (helpfulness, harmless, humor). However, in real world scenarios, users' demand might encompass much more objectives. Therefore, experiments on a larger number of objectives are required to show the generalization ability of MOC over different numbers of objectives.

A3: We acknowledge the reviewer's concern regarding experiments with more objectives. While our primary experiments focus on three objectives, MOC can theoretically handle a larger number of objectives. To substantiate this claim, we conducted additional experiments using the 6-objective Fruit-Tree task from the MO-Gymnasium benchmark. We have included these experiments in Appendix H of our revised paper.

1. Generalization to a Larger Number of Objectives.

Task Description: The Fruit-Tree task involves navigating a binary tree of depth 6, where each leaf contains a 6-dimensional reward vector corresponding to nutrient values. The action space is discrete, requiring the agent to optimize for all nutrients simultaneously. Additional details are available on the MO-Gymnasium website [5].

The results (see Table below) demonstrate that MOC significantly outperforms the Linear PPO baseline in terms of hyper-volume, demonstrating its effectiveness in optimizing across multiple objectives.

We provide a visualization comparing MOC (warm color) to Linear PPO (cool color) in terms of the density distributions of objectives 1, 2, and 3: https://anonymous.4open.science/r/MOC-9E51/generalize-to-6-objective-fruit-tree.svg

MOC’s solutions consistently dominate those of the baseline, underscoring its superior performance. Full implementation details and hyper-parameters are outlined below.

2. Generalization to Different Reward Models

To evaluate MOC’s generalizability to various reward models, we applied it to the Reddit Summary task [6] using the Llama-3-8B model. The task involved two reward models:

• Summary: Evaluates the quality of the generated summary.
• Faithfulness: Assesses the factual consistency of the summary with the original text.

The results:

MOC achieves significantly better performance than the baseline across both reward dimensions. Visual results are provided:

Performance of MOC: https://anonymous.4open.science/r/MOC-9E51/MOC-Llama3-8B_Faithful_Summary.svg Performance of baseline: https://anonymous.4open.science/r/MOC-9E51/RiC_Faithful_Summary.svg

The additional experiments address your concerns by:

• Demonstrating MOC’s capability to handle a larger number of objectives (6-objective Fruit-Tree task).
• Showcasing MOC’s adaptability to different reward types (Reddit Summary task).

We believe these results strengthen the case for MOC’s versatility and practical applicability.

Q4: Can author provide some intuitive explainations on why the value of $c^{(1)}$ and $c^{(2)}$ is determined by (8)?

A4: Yes!

Multi-task learning

In multi-task learning, the objective is to train a model that performs well across multiple tasks simultaneously. Each task $t$ is associated with its own loss function, $\hat{\mathcal{L}}^t(\theta^{sh}, \theta^t)$, where $\theta^{sh}$ represents parameters shared among all tasks, and $\theta^t$ are task-specific parameters.

The shared parameters $\theta^{sh}$ are updated using gradients derived from each task: $g_t = \nabla_{\theta^{sh}} \hat{\mathcal{L}}^t(\theta^{sh}, \theta^t)$. These gradients often conflict because they may point in different directions, making it challenging to update $\theta^{sh}$ in a manner that benefits all tasks simultaneously.

To address this challenge, the goal is to find a single update direction that:

• Balances the contributions from all tasks.
• Minimizes conflicts between tasks to ensure that the update does not harm any individual task.

This is achieved through a weighted combination of the task gradients:

where $c^{(t)}$ are the weights determine each task's contribution. Note that i) $\sum_{t=1}^T c^{(t)} = 1$: Ensures that the weights form a convex combination, maintaining the update within the feasible region defined by the task gradients; and ii) $c^{(t)}t \geq 0$: This prevents reversing a task’s gradient direction, which could hinder optimization.

Equation (8) and Its Implications.

Equation (8) defines an optimization problem that minimizes the squared Euclidean norm of the weighted gradient combination:

This optimization achieves two critical objectives:

• Conflict Reduction: A smaller norm indicates less disagreement among task gradients. Minimizing this norm aligns the gradients into a more harmonious direction.
• Balanced Update: The weights $c^{(t)}$ adaptively adjust the contribution of each task to ensure that no single task dominates the update direction unless necessary to reduce conflict.

Intuitive Explanation

We provide a geometric and intuitive understanding of this process. Check this link: https://anonymous.4open.science/r/MOC-9E51/min_norm_solution.svg for the figure.

1. Geometric Perspective:

   • Consider each task gradient ($g_1$ and $g_2$) as vectors.
   • The weighted sum $G(c)$ lies within the convex hull formed by the two gradient vectors.
   • Minimizing $||G(c)||_2^2$ corresponds to finding the point in the convex hull that is closest to the origin. This point represents the most balanced update direction that aligns all task gradients.

2. Conflict Minimization: The $c^{(t)}$ obtained by solving Equation 8 de-emphasizes tasks with conflicting gradients and amplifies tasks with aligned gradients. This reduces overall gradient conflict.

3. Optimality. By finding a common direction with minimal conflict, the method ensures progress across all tasks without harming any single task, leading to optimality.

4. Why Are These Weights Optimal? The values of $c^{(t)}$ are determined by the optimization problem in equation 8 because it seeks the most balanced and agreeable update direction. Theoretically, as demonstrated by [7, 8], either: (i) The solution to this min-norm problem is zero, in which case the resulting point satisfies the KKT conditions, which is Pareto stationary; or (ii) The solution yields a gradient direction that improves all objectives.

Thank you for your thoughtful feedback. We believe these clarifications and additional results strengthen the paper and its contributions.

Best wishes,

The authors

References

[1] Rame A, Couairon G, Dancette C, et al. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2] Yang R, Pan X, Luo F, et al. Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment[C]//Forty-first International Conference on Machine Learning.

[3] Zhou Z, Liu J, Shao J, et al. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization[C]//Findings of the Association for Computational Linguistics ACL 2024. 2024: 10586-10613.

[4] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/openai/baselines, 2017.

[5] https://mo-gymnasium.farama.org/environments/fruit-tree/

[6] Stiennon N, Ouyang L, Wu J, et al. Learning to summarize with human feedback[J]. Advances in Neural Information Processing Systems, 2020, 33: 3008-3021.

[7] Désidéri J A. Multiple-gradient descent algorithm (MGDA)[D]. INRIA, 2009.

[8] Sener O, Koltun V. Multi-task learning as multi-objective optimization[J]. Advances in neural information processing systems, 2018, 31.

Dear Reviewer PAko,

We hope this message finds you well. As the discussion will close in less than 24 hours, we wanted to kindly remind you of our earlier response addressing your comments. We have provided detailed explanations, including formal definitions, expanded discussions on baseline methods, and revisions to enhance the paper’s clarity.

Your feedback is invaluable to us, and we remain eager to address any remaining concerns you may have. If there are any additional points you would like us to clarify, we would greatly appreciate hearing from you before the rebuttal period concludes.

Thank you again for your time and thoughtful review.

Best regards,

The Authors

Dear Reviewer QBHy:

Thank you very much for your detailed review and insightful comments on our manuscript. We would like to address your concerns with the following clarifications.

W1: No variance is reported for any metric used in the evaluation which is a common thing in RL works.

A1: We appreciate your highlighting the importance of variance reporting in RL settings.

1. Variance as a measure of response diversity in our paper. In our submission, Figure 3 reports variance to capture the diversity of responses across different prompts, reflecting generalization across varied preferences. This variance offers insights into the model’s adaptability to new settings.

2. Relevance of variance in RL versus LLM settings.

   • In RL, variance is typically reported across multiple runs with different random seeds to assess an agent’s stability, consistency, and robustness to initialization or environmental stochasticity. Lower variance generally indicates better reliability.

   • In LLMs, however, variance reflects response diversity across prompts rather than learning stability. Due to the pre-trained nature of LLMs, response-level variance provides limited diagnostic value for assessing training reliability.

3. Focus on Controllability in our work. Our primary focus is on multi-objective controllability—aligning the model’s outputs with diverse user preferences. This is best assessed by mean performance metrics and response alignment quality, as response-level variance is less informative for the objectives of our study.

In conclusion, while variance is reported in Figure 3, our selected metrics more effectively capture the model’s controllability and alignment across preference configurations.

W2: The representation of the preference vector in the prompt is an important aspect of the paper and not explored at all in the paper. Does the preference vector need to be relative in magnitudes aka it just specifies an rank order of the preferences? If that is the case, are their other representations of the vector possible and how does that impact the model? Can the model understand preference specified as decimals?

A2: Thank you for this insightful question. We clarify below:

• Representation of the Preference Vector: In our work, the preference vector $\mathbf{p} = [p_1, p_2, \ldots, p_N]$ is normalized to satisfy $\sum_{i=1}^N p_i = 1$ and $p_i \geq 0$. Each $p_i$ represents the relative importance of the $i$-th objective, allowing the vector to be interpreted directly as weights for multi-objective optimization.

• Incorporation into Prompts: The preference vector is incorporated into prompts using the format: Re-labeled prompt = <R1> $p_1$ <R2> $p_2$ ... <RN> $p_N$ {prompt}. This design ensures the model can directly utilize the vector as a part of the input for optimization.

• The model can understand preferences specified as decimals, as demonstrated in Section 4.3. Here, MOC generalizes to unseen preference vectors, showcasing its adaptability to a wide range of representations. This approach aligns with prior work, such as RiC [1], which also incorporates decimal-based relabeling for preference vectors.

We hope this clarifies the design and representation choices for the preference vector.

W3: The method uses MSE loss between the preference vector and the rewards for each preference as reward for their MOO objective. The authors does not discuss any limitations or structure required for this to work. Should the reward and preference be of the same scale. What if the preference is specified as [p_1: 0.25, p_2: 30] and rewards are [r_1: 0.5, r_2: 40]

A3: We agree that scale alignment between rewards and preference vectors is critical. In our work, we ensure this alignment using reward normalization, as detailed below:

Reward Normalization: As described in Appendix E, we normalize the reward signal as: $r = \frac{r-r_\text{mean}}{2 r_{\text{std}}}+1$. Here, $r_{\text{mean}}$ and $r_{\text{std}}$ are computed using a running mean and standard deviation (as in [4]), ensuring that rewards align with the scale of the preference vector. This approach is a standard practice in RL [4] and ensures the reward and preference are of the same scale.

We have included this analysis in Appendix G of our revised paper for a comprehensive understanding.

Dear Reviewer QBHy,

We hope this comment finds you well. We wanted to kindly remind you to review our rebuttal and share any additional comments or feedback you may have. Your input is highly valued, and we appreciate the time and effort you dedicate to this process.

Thank you again for your consideration.

Best regards,

The Authors

Dear Reviewer QBHy,

We hope this comment finds you well. As the discussion phase is nearing its conclusion, we kindly reach out again to request your input on our rebuttal. We value your expertise and greatly appreciate your feedback, which is essential to improving the quality of our work.

If you have any remaining questions or require clarification on any aspect of the submission, please let us know. We are happy to address your concerns promptly.

Thank you for your time and for contributing to the review process.

Best regards,

The Authors

I would like to thanks the authors and acknowledge that I have read their response to the all the reviews.