Dear reviewer,

We sincerely thank you for your valuable review and constructive feedback.

We address your concerns and aswer your questions below:

---

Q1. Can the authors provide ablation studies on the similarity weight $\lambda$ to illustrate its impact on both convergence speed and final control performance?

A1: The table below demonstrates the robustness of our method to different values of $\lambda$:

This table illustrates how different upper bounds of $\lambda$ affect PreCo's performance in the FruitTree environment. While smaller values of $\lambda$ may slightly change the performance, PreCo remains consistently superior to existing MORL methods. Notably, the best-performing baseline, PDMORL, achieves a hypervolume of only 9.3$\pm$0.08*1e3, demonstrating PreCo's promising advantage.

---

Q2. For continuous action spaces, does the PreCo update method require any modifications? Could the authors elaborate on potential challenges and necessary adjustments for continuous control environments?

A2. This has been discussed in Appendix E, with experiments in continuous action space presented in Appendix C.

In continuous action spaces, the "policy-level" gradient corresponds to the gradients of (1) action samples (Eq. 26) or (2) the parameters of the Gaussian action distribution (Eq. 24), rather than the gradients of action probabilities. As illustrated in Figure 9 and Algorithm 2, our approach first determines a common descent direction for all objectives by solving the min-norm problem (6) using their gradients (Eq. 24 or 26). The policy model is then updated to fit the new actions along this direction.

The experiments in Appendix C focus on high-dimensional continuous control tasks with simple, strictly convex objectives. While the advantage on performance is smaller compared to tasks with non-strictly convex objectives and many objectives, our method still outperforms previous MORL approaches.

---

Q3. Although the paper suggests that the method could be applied to real robotic tasks, are there any plans for experimental validation on actual robotic platforms? What practical issues might arise during such a transition?

A3: Real-world applications, such as robotics, are indeed an exciting direction. In fact, we have already been applying our method to large language models and acheived promising results. One of our future plans is to leverage preference-controllable language models for high-level robotic planning and human-robot interaction.

For low-level robotic control, several challenges include common sim-to-real issues such as sample efficiency, dynamics mismatch, and safety concerns. One critical study is to ensure that the robot does not exhibit unpredictable behavior when conditioned on previously unseen preferences.

---

Q4. The current experiments involve up to six objectives. How does the proposed method scale when the number of objectives increases further, and are there anticipated bottlenecks in terms of computational complexity or performance?

The MOO algorithms with conflict-avoidance ability, including our proposed PreCo, require solving the min-norm problem (6) to determine a weight vector $w$, whose dimension corresponds to the number of objectives. When the number of objectives becomes very large, the computational cost of solving (6) can increase significantly.

To mitigate this, instead of explicitly solving (6) at every training step (Algorithm 2), we can update $w$ using the gradient of (6) once per training step (Algorithm 1). Our theoretical results justify the convergence of this novel approach, making it a more scalable alternative in practice.

---

We hope these clarifications address your concerns. Let us know if you have any further questions.

Dear reviewer,

We sincerely thank you for your valuable review and constructive feedback.

We address your concerns and aswer your questions below:

---

C1: About Figure 4b and 4d: The empirical results do not consistently ... in Figure 4b and 4d.

R1: Figures 4b and 4d are 2D projections of the 3D results presented in Figure 4a. While some points may appear dominated in the 2D projections, all points in the figures are actually non-dominated when all three objectives are taken into account.

We hope this clarification helps address your concern regarding the empirical results.

---

C2: The delta from previous RL algorithms is not clearly ... the novelty of the method.

R2: TD3 and PPO are representative single-objective RL algorithms used as lower-level backbones in our approach. More details on how they are integrated can be found in Appendix E.

The key novelties of our work are as follows:

1. Conceptual: We propose a preference control framework (PCRL) that overcomes the limitations of traditional linear scalarization methods, which can only discover a limited set of Pareto-efficient solutions.

2. Methodological: We integrate modern MOO algorithms into MORL, enabling better handling of conflicting and stochastic gradients—challenges that previous MORL methods largely overlooked. Additionally, we design a novel PreCo algorithm to leverage these strengths while ensuring preference alignment.

3. Technical: Beyond theoretical analyses, we propose a memory-efficient approach for policy-level computation (see Figure 9, Algorithm 2, and Appendix E), making our method scalable for large models used in practical applications.

---

Q1: Justification for Similarity Function (Eq. 8): ... choice of similarity function?

A1: PreCo's theoretical properties require the similarity function to:

1. Be lipschitz smooth.
2. have a gradient $g_s$ must be positive linear combination of objective gradients $g_{1:m}$ ($m$ is the number of objectives).

The similarity defined in Definition 4.1 satisfies these conditions. Using a different similarity function could disrupt the convergence guarantees of our algorithm. In practice, conventional cosine similarity could work in some cases, but it lacks theoretical guarantees.

---

Q2: Non-Uniform Preference Sampling: ... handle such cases?

A2: Non-uniform preference sampling may cause certain preferences to be learned better than others or lead to overfitting. The impact depends on the model's generalizability, so training with a uniform distribution is preferable without prior knowledge of test preferences.

However, as noted in Appendix A.3, a progressive curriculum for preference distributions could accelerate learning. For instance, the agent could start with a small, diverse set of preferences and gradually expand to neighboring ones, progressively uncovering the full Pareto front. Further exploration of this is an interesting direction for future work.

---

Q3: Non-Transient Preferences: ... convergence?

A3: If we understand correctly, non-transient preferences refer to preferences that are consistently sampled during training. While this could lead to overfitting to specific preferences, a well-designed curriculum that gradually introduces new preference distributions can mitigate this issue and promote exploration of the full Pareto front (as discussed in Appendix A2).

In fact, the training scheme in Algorithm 2 (Appendix E.1) can be viewed as a form of meta-learning, where the outer loop samples different preferences and the inner loop optimizes for a specific sampled preference. While convergence is always guaranteed within the inner loop, non-stationary preference distributions may affect the overall meta-learning process, potentially leading to overfitting. However, a carefully designed curriculum with progressively shifting preference distributions, even if non-stationary, can accelerate learning and improve the agent’s performance.

Theoretical analysis of such a curriculum would be an interesting direction for future work.

---

Q4: Economic Benchmarks: ... the method's applicability.

A4: Economic benchmarks are indeed an interesting direction for future work. We will add more discussion on economic applications in the related work section.

A key technical contribution of our method is to solve the min-norm problem (6) at the policy level (details in Figure 9 and Appendix E), removing the need to store multiple parameter gradients for different objectives. This results in minimal additional memory cost, making it highly suitable for training large-scale practical models. In fact, we are already applying our method to multi-objective preference learning for large language models with promising results, suggesting its potential applicability to practical economic settings as well.

---

Thanks again, and please let us know if there are further questions.

Dear reviewer,

We sincerely thank you for your time and valuable review.

---

Response to your questions

Below are our answers to your questions

Q1. About Theorem 4.2's assumption

A1: Theorem 4.2 considers the case where $\lambda$ increases indefinitely. In practice, Theorem 4.1 already guarantees PreCo’s convergence as long as $\lambda$ has an upper bound, as noted in Remark 4.1.

In Appendix F.2, We provided some hints for constructing such a similarity function $\Psi'(p,\cdot)$ as follows:

• When $v$ gets close to preference $p$: Its similarity gradient $\nabla_v\Psi'(p,v)$ should approach the convex preference weight $p$;
• When $v$ is NOT close to preference $p$: $\nabla_v\Psi'(p,v)$ should be close to the convex coefficient $\nabla_v\Psi(p,v)/\|\nabla_v\Psi(p,v)\|_1$, where $\Psi$ is our original similarity defined in Definition 4.1.

Q2. About MOO and choices other than the MOO

A2: MOO refers to the algorithms specificly designed for deep learning with multiple objectives, such as MGDA[1], CAGrad[2], PCGrad[3], SDMGrad[4]. They are known to find a conflict avoidance direction for common improvements across all objectives.

Additionally, evolutionary algorithms[5] have been developed for multi-objective problems, but they tend to be less scalable and less sample-efficient compared to gradient-based MOO approaches.

Q3. Evidence for gradient conflict

A3: For a two-objective case, when the values are $[v_1,v_2]=[10.0,1.5]$ and the preference is $p=[0.9,0.1]$, and let $g_1,g_2$ denote the gradients of objective 1 and 2, respectively.

The gradient of cosine similarity is $0.1483 g_1 -0.9889 g_2$, which is nearly opposite to $g_2$, creating a direct conflict with objective 2’s gradient.

To resolve this, we need a conflict-avoidance mechanism that optimizes similarity without negatively impacting objective 2. Intuitively, the goal is to find a direction that improves both $v_1,v_2$ while prioritizing $v_1$ to get close to the preference $p$. This can be precisely acheived by our proposed PreCo algorithm.

Q4. Computation complexity of implementing Jacobian matrix and similarity gradient

A4: Solving the min-norm problem (6) introduces additional computation to determine a common ascent direction, improving sample efficiency at the cost of extra computation.

However, memory consumption remains comparable to plain policy gradient algorithms:

• When solving the problem (6), objective gradients $\nabla_\pi \mathbf{v}^{\pi}$ and similarity gradient $\nabla_\pi \Psi(\mathbf{p},\mathbf{v}^{\pi})$ are stored, but their size are at most batch size $\times$ # objectives, which typically much smaller than the parameter size. Once (6) is solved, we obtain a batch-size update direction $d^*$ for $\pi$, allowing $\nabla_\pi \mathbf{v}^{\pi},\nabla_\pi \Psi(\mathbf{p},\mathbf{v}^{\pi})$ to be released.

• After solving (6), we compute $\nabla_\theta^T\pi\ d^*$ to update the model parameter. The jacobian $\nabla_\theta^T\pi$ is not explicitly computed and stored. Instead, we take the gradient for ${d^*}^T \pi$ to obtain it. In a formulation similar to the policy gradient, it corresponds to $E_{s,a}[\nabla_\theta \log\pi(a|s)\ d^*(s,a)\pi_{old}(a|s)]$, as detailed in Appendix E.

Therefore, memory usage at any given time does not exceed that of linear scalarization methods that computes the gradient $\nabla_\theta^T\pi \nabla_\pi \mathbf{p}^T\mathbf{v}^{\pi}$. In the policy gradient form, it is expressed as $E_{s,a}[\nabla_\theta \log\pi(a|s)\ \mathbf{p}^T\mathbf{q}^{\pi}(s,a)]$.

---

Addressing concerns of weeknesses

However, it is unclear about the computation complexity... large volume of gradient information.

As mentioned in A4, keeping large volume of gradient information is unnecessary and the memory consumption remains comparable to traditional methods.

It is unclear ... to transfer to new preference vector, as the algorithm needs to re-implement the whole optimization again.

As mentioned in lines 143–144 of Section 3 and Algorithm 2 in Appendix E, preferences for training are uniformly sampled, and the stepsize for each preference sample remains constant. This enables a single agent to learn in a "meta-learning" fashion. This enables the agent to generalize to unseen preferences, as demonstrated in our experiments (Section 5 and Appendix D), where we test the agent on preferences it has not encountered during training.

---

We hope these clarifications address your concerns. Please let us know if you have any further questions.

Reference

[1] Désidéri, MGDA, 2009

[2] Liu et al. Conflict-averse gradient descent for multi-task learning, 2021

[3] Yu et al. Gradient Surgery for Multi-Task Learning, 2020

[4] Xiao et al. Direction-oriented multi-objective learning: Simple and provable stochastic algorithms, 2023

[5] Tan et al. Evolutionary Algorithms for Solving Multi-Objective Problems, 2007

Dear reviewer,

We sincerely thank you for your time and detailed review. We hope the responses below address your concerns.

---

Answers To Your Questions

---

A1: In practical deep RL/MORL, first-order gradient-based algorithms are the most widely-used and stationarity is the strongest guarantee [1] they can achieve in practice. Hence, prominent MOO algorithms [2,3,4] establish only Pareto stationarity. Compared to existing works, ours is one of the few having provable convergence under noisy gradients.

---

A2: We only need to train a single policy to handle different preferences as input conditions. During training, preferences are sampled uniformly without using prior knowledge. These are explained in lines 143–145 of Section 3 and Algorithm 2 in Appendix E. This standard approach in conditional MORL [5,6] ensures shared parameters and knowledge across all preferences.

---

A3: As noted in A2, preferences are uniformly sampled during training in a meta-learning fashion, applied consistently across all methods.

Our high-level objective design makes it a generally applicable framework to all RL backbones (on-policy & off-policy). Its contribution is orthogonal to lower-level improvement of sample efficiency. As discussed in Appendix A.1, when adapted to off-policy methods, it can integrate techniques like Hindsight Experience Replay(HER) to enhance sample efficiency.

---

A4: In the MuJoCo environments (Ant, Hopper, Reacher), all methods use 3e6 environment steps.

For the FruitTree environment, all methods except PDMORL use 3.6e5 steps, while PDMORL follows its original implementation with 1e6 steps.

---

A5: PreCo's theoretical properties require the similarity function to be:

1. lipschitz smooth
2. Its gradient $g_s$ must be positive linear combination of objective gradients $g_{1:m}$

The similarity in Definition 4.1 meets these criteria, and its magnitude does not affect PreCo's properties. Intuitively, it encourages certain objectives to improve more, so the achieved trade-offs get closer to the pre-defined preference. See Appendix F.1 and Figure 10 for further insight.

---

Concerns About Methods

To address your concerns, we summarize a few key points:

Generalizability: We focus on a MORL's framework design that is generally applicable to all on & off-policy RL. In contrast, techniques like HER that exploit inherent RL properties are limited to off-policy methods.

Novelty & Contributions:

1. Conceptual: We formulate PCRL for any-preference alignment, overcoming the limitations of prior MORL methods using Linear Scalarization (LS) that has no alignment guarantee.

2. Methodological: We integrate modern MOO algorithms into MORL to handle conflicting and stochastic gradients—key aspects previously overlooked. Specifically for PCRL, we design PreCo to inherit these strengths while promoting preference alignment.

3. Technical: PreCo’s similarity design requires independent proofs (App. I.1) and significant adaptations (App. I.2-3), as no prior method incorporates such similarity. Moreover, as noted in lines 1670-1672, our analysis is more rigorous than previous literature.

Practicality: Our assumptions align with standard RL/MORL settings: most RL methods (e.g. policy gradients, DQN loss) are unbiased (Ass.2), and their values typically change with a certain level of smoothness (Ass.1).

---

About Theoretical Correctness

As noted in Assumption 2, the expectation here is over the gradient noise $\xi$ and Assumptions 1-3 ensures unbiased, bounded gradients for all $\pi_{p,t}$ samples. Thus, the bounds in (51)-(54), (56)-(61), and (68)-(70) apply to all $\pi_{p,t}$ samples and remain valid under expectation over $\pi_{p,t}$.

We appreciate the detailed feedback, but treating (51)–(54) as conditional expectations does not impact the proof's correctness.

---

About Experiments

A2-A4 have answered questions about handling different preferences and environmental steps. Here, we address the concerns about the baselines.

Envelope Q-learning (EQL), Conditioned Network (CN), and Q-Pensieve (QP) all optimize a linear scalarization (LS) objective and inherit its limitations. The best-performing QP achieves a 6859.94 hypervolume for 6D FruitTree, identical to the best cases of our implemented general LS baseline. This is expected, as QP is simply a more sample-efficient version of LS.

Since we already include an LS baseline and CAPQL, which achieves a higher upper bound than LS methods, comparing against EQL, CN, or QP is unnecessary.

PCN is a heuristic method relying on model generalizability. Its comparison to other methods is limited. It only accepts a unique input condition, making it less relevant to our study.

---

Please let us know if you have further questions.

[1] Kushner 1978 Stochastic

[2] Sener 2018 MGDA

[3] Liu 2021 CAGrad

[4] Xiao 2023 SDMGrad

[5] Liu 2023 CAPQL

[6] Yang 2019 Envelope Q