---

A12: Regarding Remark 4.1, the upper limit of $\lambda$ can be predefined as a hyperparameter, then in practice, you can linearly increase $\lambda$. Once $\lambda$ reaches the upper limit it stays constant.

---

A13: 5 random seeds were used for mean values. For better reproducibility, we rerun the experiments with seed [1,2,3,10,20] and use fixed hyperparameter for all reward dimmensions. We get the results below:

Hypervolume:

Cosine similarity:

CAGrad only guarantees convergence when the reference gradient is defined as the mean gradient across all objectives. However, when using the similarity gradient as the reference, its convergence is not theoretically established. Consequently, without specific tuning tailored to each setting, it tends to exhibit poor performance.

---

A14: For 3 out of 5 seeds, Linear Scalarization (LS) learns a single point, similar to the case shown in Figure 4. In the remaining two seeds, most values also overlap at a single point. These results are reasonable because the red point is indeed optimal for $\max_v v^Tp$ for most $p$. (shown by the rays in Figure 4(a)). Especially when preferences are uniformly sampled during LS training, a value at the red point, independent of the preference condition, represents an obvious local optimum. Even in the ideal case, LS methods like [1] can only identify points within the "Convex Coverage Set (CCS)." However, preferences can target Pareto-optimal points that lie outside the CCS, such as most of the blue points in Figure 4.

---

A15: We did not claim that LS agents are uncontrollable by $p$ for all cases, only in the 3D fruit-tree case it is completely uncontrollable. Additionally, by the fact that LS methods can only find the concave part for maximization, they inherently lack controllability, which was shown in Fig.1.

---

A16: It was calculated by mean values of 5 seeds. We now finetune the hyperparameters for all methods and rerun the experiments with seeds [1,2,3,10,20]. We have obtained the results below and will include the full results in the revisions:

MO-Reacher:

MO-Hopper:

The results demonstrate that our proposed PreCo outperforms the baselines in both hypervolume and cosine similarity, particularly in tasks with a larger number of objectives. While LS may achieve high hypervolume in simpler 2-objective tasks such as Hopper, its controllability remains inferior to that of PreCo.

---

A17: Thanks for the valuable feedback and we will carefully review it for better clarity.

---

Reference

[1] Alegre et al. (2023) "Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization"

[2] Chen et al.(2019) "Meta-Learning for Multi-objective Reinforcement Learning"

[3] Zhanget al.(2023) "Hypervolume Maximization: A Geometric View of Pareto Set Learning"

def train(N,E,agent):
    # N: Number of preference samples
    # E: Episodes per preference sample
    # Agent should have preference-conditioned actor and critic
    # The critic should output a vector of multi-objective values 
 
    for n in range(N):                     # Loop over preference samples
        Sample preference p                # Sample a preference p
        for e in range(E):                 # Loop over episodes for the this preference
            Rollout with policy conditioned on p  # Rollout with the current preference
            Store transitions (s,a,r_vec,p) in the buffer  # Store transitions

        Sample (s,a,r_vec,p) from buffer       # Sample transitions from buffer
        Update critic by the samples           # Approximate objective values
        Calculate policy-level gradient        # Eq.20 for TD3, Eq.26/32 for PPO
        Estimated values of the agent          # Eq.4 for TD3, Eq.5 for PPO
        Calculate similarity gradient          # Eq.8
        Solve Eq.6 using policy-level gradient   # Solve the problem for PreCo update
        Update actor parameter by solution of Eq.6   # Eq.21/27 or Eq.29

Thanks very much for your review and constructive feedback!

---

We would like to briefly clarify our theoretical contributions:

• We have theoretically proven that our proposed PreCo has the following properties:
  • It converges to Pareto stationary solutions.
  • Its solutions align with user preferences (controllability).
  • It converges under stochastic gradients."

These properties are empirically demonstrated via a toy example of multi-objective optimization provided in Appendix F (code available in the anonymous link 1). The properties suit MORL well, as previous methods based on linear scalarization could fail for preference alignment (Fig. 1), and RL's stochastic gradients often have high variance.

The experimental results are averaged over five random seeds. We provide some easy-to-run code in this anonymous link 2, and the full code will be released upon acceptance.

We hope our responses to your questions below further clarify our contributions.

---

Answers to the questions

---

A1: Thanks for pointing out this issue and we will make adjustments accordingly. But Alegre et al. (2023) [1] is still an LS method that inherently has the limitation discussed in the cases of Figure 1, so it is not the best choice for preference alignment.

---

A2: "Meta-policy" has been used in [2] for MORL.

---

**A3:**We consistently consider maximization for MORL that linear scalarization can reach the part of the Pareto front with concave curvature. "Conve" or "concave" is often used to describe the curvature of the Pareto front [3].

---

A4: We have discussed it in the "Related works" section in the Appendix. A.

---

A5: As defined in paragraph "Preference control" of section 2, the domain of $p$ is $\mathcal{P}:=\{\mathbf{p}\in \mathbb{R}^m: \mathbf{p}^T\mathbf{1}=1,\mathbf{p}\geq0\}$, where $m$ is the number of objectives. The uniform sampling can be efficiently implemented by "np.random.dirichlet([1 for i in range(m)])"

---

A6: $\lambda$ is chosen as a hyperparameter.

---

A7: Appendix.D discussed the topic “policy level” vs. “parameter level” in detail. Consider the PPO example in Appendix.D.2, its parameter level gradient is

$$\nabla_{\theta} \hat{\mathbf{v}}^{\pi_p}=E\left[\frac{\nabla_\theta \pi(a|s,p)}{\pi(a|s,p)}\hat{\mathbf{A}}(s,a,p)\right],$$

and denote its policy level gradient as $\mathbf{d}=\nabla_{\pi_p} \hat{\mathbf{v}}^{\pi_p}$, we have

$$\mathbf{d}(s,a)=\frac{1}{\pi(a|s,\mathbf{p})}\hat{\mathbf{A}}(s,a,\mathbf{p})$$

for every $(s,a)$ sample. “Policy level” gradient $\mathbf{d}$ has a dimension of batch size for every PPO update while the “parameter level” gradient $\nabla_{\theta} \hat{\mathbf{v}^{\pi_p}}$ has a size of model parameter size. The parameter size of neural network models is often much larger than the batch size. We provided a diagram of the backward path for more intuition in the anonymous link 2

---

A8: No it is not correct. There is no constraint on the values. If $\mathbf{v}=[v_1,v_2]=[10,1]$ and $p=[0.8,0.2]$, the maximum element will be $\frac{10}{0.8}$, then $\max_i \frac{v_i}{p_i}p=[10,2.5]$, and we obtain $\nabla_{v_1}\Psi(p,\mathbf{v})=10-10=0$ and $\nabla_{v_2}\Psi(p,\mathbf{v})=2.5-1=1.5.$ Therefore, $v_2$ will be optimized to push $\mathbf{v}$ towards $[10,2.5]$. As mentioned right after Def.4.1, a visualization in Appendix E could help for an intuitive understanding of this similarity function.

---

A9: $w$ is the coefficient to combine the objective gradients into a similarity gradient. After updating $w$ by adding a gradient, it might no longer be a convex coefficient(similar to the preference domain $\mathcal{P}$, its sum should equal 1 and its elements should be nonnegative). The projection just means to find the point $w'$ within the set of convex coefficients with minimum L2 distance to $w$, $\min_{w'\in \mathcal{P}}||w'-w||^2,$ which is a common problem that can be efficiently implemented by popular packages like "scipy.optimize".

---

A10: In algorithm 1, we consider the stochastic setting, so policy gradient samples are random variables. Each data (a batch of transitions in the RL setting) gives us a gradient sample. $G(\pi;\zeta)$ is a sample of the gradient used for $\pi$ update and $G(\pi;\xi)$ is a sample of gradient used for $w$ update. Thanks for pointing out that algorithm 1 is not entirely straightforward. We provide an intuitive algorithm next page and will include in the revisions for better clarity.

---

A11: The expectation in Eq. 9 is taken w.r.t. the distribution of data $\xi$. For the RL setting, $\xi$ is a random variable of the batch of transitions that calculates the gradient for a policy update. The computation of $G$ for mainstream RL algorithms is discussed in more detail in Appendix.D.

---

Thanks for the feedback and suggestions!

---

First of all, there seems to be a major theoretical misunderstanding to clarify:

• Solutions in the convex coverage set (CCS) are not the only Pareto optimal solutions

For a Pareto front that is not strictly convex, like a polytope shape, the CCS that Linear Scalarization (LS) methods aims to learn are the solutions at the "vertices" rather than the "edges", while solutions at the "edges" are also Pareto-optimal and contribute to the hypervolume.

Beside proving convexity for stochastic policies, Lu et al., (2023) also claim that LS often fails to find all Pareto optimal solutions for these not strictly convex cases, and it is necessary to add a strongly convex augmentation for more LS solutions. (strictly convex is a necessary condition of strongly convex)

The intuitive illustrations can be found in Figure 10 of Lu et al., (2023) and Figure 4 of our paper.

---

And there can be a very simple but also practical example to demonstrate this:

Example: There is only one state, two actions $[a_1,a_2]$, and two objectives. At every step, if $a_1$ is made, the agent receives $[r_1(a_1)=1,r_2(a_1)=0]$ and if $a_2$ is made, the agent receives $[r_1(a_2)=0,r_2(a_2)=1]$

For this example, all stochastic policies are Pareto optimal and their values $\mathbf{v}=\mathbb{E}_a[[r_1(a),r_2(a)]]$ all lie on the line $v_2=1-v_1$. The Pareto front is convex but not strictly convex.

LS methods only learns the two CCS solutions: $[\pi_1(a_1)=1,\pi_1(a_2)=0]$ for $w_1 \geq w_2$ and $[\pi_2(a_1)=0,\pi_2(a_2)=1]$ for $w_2 \geq w_1$.

Therefore, for this example, LS methods find only $2$ CCS policies, even though there are an infinite number of Pareto optimal policies. And these two CCS policies are not aligned with most preferences.

---

In addition, for continuous control tasks, the policies are often not "stochastic"[3].

---

About experimental results:

Based on the clarified point above,

"Fruit-Tree benchmark (Yang et al. 2019) is able to find most (if not all) the optimal solutions." is incorrect.

(Yang et al. 2019) only reported the Coverage Ratio of CCS solutions, not the Pareto-optimal solutions. As shown by our Fig.4, the Pareto front of FTN is often not strictly convex and LS can not discover all optimal solutions in practice.

For strictly convex two-objective cases such as Hopper and Ant, our implementation LS does exhibit similar hypervolumes to PreCo. Therefore, our code-level implementation is not the problem.

We have updated the experimental results with statistic metric standard deviation and provided easy-to-run code and seed. We hope this could address your concerns.

---

About previous answers:

A1&A2: Our method is trained with varying preferences (analogous to different 'tasks' in a meta-learning setting). At test time, our learned agent aims to adapt to the user's preferences in a zero-shot manner by treating these preferences as conditional input. This makes the "intersection of preference rays with the Pareto front" (controllability) critically important. Even in strictly convex cases like Figure 1(c), where LS successfully learns all Pareto-optimal policies, there can still be a significant mismatch between the achieved values and the user's preferences, leading to high adaptation costs.

By "For instance, when you increase the weight ... get different solutions with different weights." you suggested that adjusting the conditional input can help identify a preferred policy at test time. (In fact, we have already proposed a principled approach for this purpose in Appendix G. )

However, this approach still inherently favors policies with higher hypervolume and a greater number of Pareto-optimal solutions. As mentioned before, CCS does not represent the full Pareto front. Consequently, the user is not guaranteed to find their preferred policy simply by exploring different conditional inputs. Even in strictly convex cases like Figure 1(c), the density of practically learned policies can vary significantly across different preferences. As a result, there remains the possibility that users may fail to find their preferred preference in low-density regions.

---

A3 Thank you, we will adopt 'convex' for better consistency in terminology.

---

A4 Thank you for the suggestions and we will consider the rearrangement of sections.

---

References

Lu et al., (2023) "Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality"

Yang et al. (2019) "A generalized algorithm for multi-objective reinforcement learning and policy adaptation"

[2] Chen et al.(2019) "Meta-Learning for Multi-objective Reinforcement Learning"

[3] Lillicrap 2015 "Continuous control with deep reinforcement learning"

---

Thank you very much and the discussions above will be added as a Q&A section in revisions for better clarity.

We are pleased that some of your initial concerns have been resolved. Below, we address your newly raised points:

Clarity:

---

The main concerns raised by other reviewers [caBw, nUpd] have been clarified and addressed. They both increased their ratings Reviewer nUpd explicitly acknowledged improved clarity after the revision and increased their confidence accordingly.

---

Missing comparison:

---

Our proposed method is a more general, high-level approach applicable to both value-based and policy-based methods, unlike the three Linear Scalarization (LS) methods [1][2][3] you mentioned, which are specifically designed for value-based RL. Envelope Q-learning [1] is limited to discrete action spaces and relies on Hindsight Experience Replay (HER)[4], GPI-LS [2] involves a specific training curriculum and CAPQL[3] needs reward augmentation—none of which are directly applicable to policy-based methods like PPO.

To ensure a fair comparison independent of HER, reward augmentation, and training curriculum, we implemented two LS baselines: a basic, general LS method and SDMGrad as an ablation study (replacing similarity with LS for PreCo). Further direct comparisons to LS methods with lower-level modifications are unnecessary and would be unfair at this stage. (Other reviewers [caBw] highlighted the impact of HER on fair comparisons, and [nUpd] understands the limitations of LS.)

Even so, For the typical discrete 6d fruit tree environment applicable to Envelope Q-learning[1], even with HER techniques, their reported hypervolume ($8427.51$) was significantly lower than both our proposed PreCo ($1.56e4$) and our LS baseline SDMGrad ($1.33e4$).

About the results of Reacher, Figure 8 shows the typical local optimum they learn for most runs. This is reasonable, as even in the toy example in Appendix F, due to stochasticity, many methods fail (Figure 11). Moreover, [1][2][3] did not report the results of Reacher. Could you please provide clear references for your claim? Regarding the LS method that you mentioned to learn many optimal policies, are they training the LS agent with uniformly sampled preferences? What are their evaluation metrics? Are they using the HER technique? This would be important to keep the discussion constructive.

---

Unclear advantage of the proposed method:

---

The main advantage we claim is the controllability of preferences, measured by the cosine similarity between preferences and values. The higher cosine similarity of PreCo compared to the two LS baselines supports our contribution.

Additionally, as clarified previously, in cases with strictly convex, and especially when there are only two objective numbers, LS can learn all Pareto-optimal policies. The hypervolume data demonstrates that our implementation of LS is fair and adequate.

The lower hypervolume of LS for cases with less convex Pareto fronts and higher objective numbers, and the higher hypervolume of LS for cases with more convex Pareto fronts and lower objective numbers, align precisely with the theoretical insights we wished to clarify in our second response.

---

[1] Yang et al. (2019) "A generalized algorithm for multi-objective reinforcement learning and policy adaptation"

[2] Alegre et al. (2023) "Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization"

[3] Lu et al., (2023) "Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality"

[4] Andrychowicz et al.(2018) "Hindsight Experience Replay. "

Dear reviewer tiB3,

Thanks again for your time, effort, and valuable suggestions to improve our work! We have carefully addressed your initial concerns and revised the manuscript accordingly. We kindly ask that you consider the clarifications provided in our second response, as it directly addresses a key point that is crucial for understanding the contributions of our work. We are looking forward to hearing you and will be happy to answer any further questions or concerns you may have.

Best regards,

We apologize for any strong statement that may need to be more respectful. With all due respect, we would like to clarify some points regarding your summarized points.

---

Point 1:

This is incorrect because SAC/TD3 rely on the Q value critic, all gradient information of their policy network come from the Q-value. We use the term "value-based" to emphasize their dependency on Q-value, which are essentially different from "policy-based" methods that rely on policy gradient and don't need a Q-value critic. Their implemented HER technique does not apply to PPO.

---

Point 2:

It is incorrect that GPI-LS can be applied to PPO because GPI-LS's curriculum and policy are based on "Generalized Policy Improvement (GPI)" that also requires only Q-value, therefore not applicable to PPO.

CAPQL only proposed one practical implementation, using the entropy regularization term from SAC for concave reward augmentation, which is incompatible with TD3. PPO inherently includes policy entropy regularization, meaning our LS baseline with PPO already incorporates a policy loss with CAPQL’s concave augmentation. We also experimented with an additional entropy term to values, creating a "Maximum Entropy PPO" implementation, which can be viewed as a full CAPQL modified for PPO. The results for the 6D fruit tree environment are as follows:

Only when the augmentation strength $\alpha=0.01$ is the result marginally better than the original LS ($\alpha=0$), yet it remains significantly worse than PreCo ($1.56e4$). Larger $\alpha$ values lead to performance drops. This result aligns with Remark 5 and Figure 9 in the CAPQL paper, which highlight that such augmentation can cause information loss in the original problem, and excessive augmentation results in performance degradation. In contrast, our method has the theoretical advantage of overcoming the LS limitation without any information loss from the original problem.

(The 8427.51 Hypervolume of Envelope Q relies on the advantage of HER.)

---

Point 3:

Reviewer caBw is aware that a direct comparison with PD-MORL might be unfair because it is specifically designed for Q-value based method heavily relying on HER. We have tried to implement PD-MORL without HER for a fair comparison but it fails due to potential underrepresented preferences as pointed out in their original paper.

---

Point 4:

Through this link, we found MO-reacher results for PCN and GPI-LS. The hypervolume of PCN often converges to a value similar to that of the initialization, GPI-LS can achieve marginal improvement to the initial hypervolume due to a specific curriculum design. Therefore, this still fails to provide convincing evidence supporting that LS methods in a general setting (learned by uniformly sampled preferences without HER) can outperform our baselines.

Any preference-conditioned method can learn a single preference better than random initialization; the challenge lies in robustness to stochasticity from uniform preference sampling. Our method has a key theoretical advantage in its convergence under stochastic gradients, as demonstrated in the toy example in Appendix F. Furthermore, even for a single preference, our method exhibits a cosine similarity advantage, also shown in Appendix F.

---

Point 5:

We have clarified that we never claimed "LS is always uncontrollable": "uncontrollable" was only used in the context of the 3d Fruit tree case where a very limited number of CCS solutions are learned with uniform sampled preferences. Our results do show better controllability in terms of cosine similarity.

---

Point 6: Key misunderstanding persists:

We would like to kindly refer you to our second response in detail, as it clarifies the key misunderstanding: Since Convex coverage set(CCS) is only a subset of Pareto Optimal Set, your mentioned "coverage ratio" is the coverage ratio of only CCS solutions, not all Pareto optimal solutions. A coverage ratio close to 1 only indicates the closeness to the upper limit of LS methods rather than the optimal solution.

---

Once again we apologize for any strong statements because a rating of 3 is unfair and unexpected to us. We do greatly appreciate your time and respect your suggestions. We sincerely expect that the above can further clarify our contributions and resolve potential misunderstandings.

Thanks very much for the review and detailed comments!

We wish to address your concern below:

---

Clarity:

Intuition for convergence to Pareto-optimal policies:

The term $||d^*||$ in (Eq. 6) is an upper bound for $\min_w||\nabla_{\pi_p} v^{\pi_p} w||$. According to [1], when $\min_w||\nabla_{\pi_p} v^{\pi_p} w||=0$, it implies that the origin lies within the convex hull of the gradients. In this scenario, updating the policy in any direction will decrease at least one objective. This indicates that the policy is non-dominated and, therefore, Pareto-optimal. As updates are performed using $||d^*||$ in (Eq. 6), $||d^*||$ will converge to zero. Since $\min_w||\nabla_{\pi_p} v^{\pi_p} w||$ is bounded by $||d^*||$, it will also converge zero, confirming that the policies achieve Pareto optimality.

We will include this explanation in the revisions for better clarity.

---

Experiments & Reproduciblity:

The experimental results are averaged over five random seeds. We provide some easy-to-run code in this anonymous git repo 1, and the full code will be released upon acceptance.

We will include the following implementation details in the revisions.

Fruit-Tree

MO-Reacher

MO-Ant & MO-Hopper

---

Meta-RL:

The motivation for learning a preference-conditioned meta-policy is that we need access to policies for multiple preferences at inference time. In practice, a preference-conditioned agent that can 0-shot adapt to different user's preferences.

The method can also identify the Pareto-optimal policy for a specific preference without requiring uniform sampling from the preference space during training. For better intuition, refer to the multi-objective optimization toy example in Appendix F (code available in anonymous git repo 2).

[1] Désidéri(2009) "Multiple-Gradient Descent Algorithm"

Dear reviewer nUpd,

We hope this message finds you well. As the discussion period closes in less than 24 hours, we wish to sincerely thank you for acknowledging the "strong theoretical analysis, extensive empirical results, and increased clarity" of our work.

We greatly value your recognition of the limitations of the popular Linear Scalarization (LS) approach in MORL and appreciate your acknowledgment that our work successfully overcomes these challenges.

We respectfully ask you to consider raising your score or confidence level, as doing so would not only support our work but also benefit the MORL community. For example, even the most recent MORL work that's published in Oct.3rd continues to misinterpret linear scalarization’s ability to identify all Pareto-optimal policies. Our work addresses this critical issue, shedding light on key misconceptions and advancing future MORL research.

Thank you for your time and thoughtful evaluation.

Best regards,

Author of submission 5131

Thanks very much for the review and constructive feedback!

We hope the pseudocode below offers better clarity for now, and we will include additional details in the revised version. For further intuition on how the proposed algorithm is implemented, you can also check and run the code for the toy example in Appendix F via this anonymous link 1.

---

Pseudo Code

def train(N,E,agent):
    # N: Number of preference samples
    # E: Episodes per preference sample
    # Agent should have preference-conditioned actor and critic
    # The critic should output a vector of multi-objective values 
 
    for n in range(N):                     # Loop over preference samples
        Sample preference p                # Sample a preference p
        for e in range(E):                 # Loop over episodes for the this preference
            Rollout with policy conditioned on p  # Rollout with the current preference
            Store transitions (s,a,r_vec,p) in the buffer  # Store transitions

        Sample (s,a,r_vec,p) from buffer       # Sample transitions from buffer
        Update critic by the samples           # Approximate objective values
        Calculate policy-level gradient        # Eq.20 for TD3, Eq.26/32 for PPO
        Estimated values of the agent          # Eq.4 for TD3, Eq.5 for PPO
        Calculate similarity gradient          # Eq.8
        Solve Eq.6 using policy-level gradient   # Solve the problem for PreCo update
        Update actor parameter by solution of Eq.6   # Eq.21/27 or Eq.29

---

For the algorithmic implementation of the preference regulation, we estimate the values of the agent by Eq.(4) or Eq.(5), which is applicable to most mainstream RL algorithms.

$\hat{\mathbf{v}}^{\pi_\mathbf{p}} = E_{S_0 \sim p_0}\left[ \mathbf{q}_\theta(S_0,\pi(S_0,\mathbf{p}),{\mathbf{p}}) \right] \ \ \ (4)$

$\hat{\mathbf{v}}^{\pi_\mathbf{p}} = E_{S_0 \sim p_0}\left[\sum_{t=0}^{T}\gamma^{t}\mathbf{r}(S_t,A_t) \right] \ \ \ (5)$

then calculate the similarity defined by Eq.8. Take the gradient of the similarity w.r.t. the policy

$\nabla_{\pi_\mathbf{p}} \Psi(\mathbf{p},\hat{\mathbf{v}}^{\pi_\mathbf{p}}) = \nabla^T_{\pi_\mathbf{p}} \hat{\mathbf{v}}^{\pi_\mathbf{p}}\nabla_{\hat{\mathbf{v}}^{\pi_\mathbf{p}}} \Psi(\mathbf{p},\hat{\mathbf{v}}^{\pi_\mathbf{p}}),$ we can get a similarity gradient that can be used for Eq.6.

---

Implementation details

To address concerns about reproducibility, we have provided easy-to-run code for Fruit-Tree in this anonymous link 2. Additionally, we will include the following implementation details in the revised version:

---

Fruit-Tree

---

MO-Reacher

---

MO-Ant & MO-Hopper

---

Dear reviewer NkSe,

Thanks again for your time, effort, and valuable suggestions to improve our work! We have addressed your initial concerns about clarity and reproducibility and have revised the manuscript accordingly. We are looking forward to hearing your feedback and will be happy to answer/discuss any further questions or concerns you may have.

Best regards,

The paper is still missing comparisons with other preference-based methods, and I strongly encourage the authors to include them to the final manuscript. However, in wake of reviewer caBw's comments, I am raising the current score.

Further, I believe reviewer tiB3 is honestly and critically analyzing the paper, and the authors' strong statements in the response are highly unwarranted. I urge the author's to be more respectful to the reviewers and their service to the community.

Thanks very much for the follow-up comment, and we are glad that your initial concerns about clarity and reproducibility are addressed. Just to clarify the key facts regarding the new concern of comparisons to ensure a fair and constructive final decision:

The opinion of reviewer caBw

In fact, reviewer caBw agrees that other methods [1][2][3] are not directly comparable or perform worse in harder environments (with larger objective number, not strictly convex Pareto front) with larger objective numbers. Their comment:

"The additional experiments further demonstrate the effectiveness of the similarity function in improving the solution quality"

shows appreciation that our current experiments have fair and adequate comparisons, and are also sufficient to show the effectiveness of our proposed methods. The "comparison" in their final comment is only a suggestion for future work to adapt our method specifically for value-based off-policy RL and establish a fair apple-to-apple comparison.

Not directly comparable or significantly worse methods raised by reviewer tiB3:

Reviewer tiB3 listed [3][4][5], which involves significantly different settings (HER for off-policy RL [6], training curricula, and reward augmentation), making direct comparisons incompatible. Furthermore, even excluding HER, our baseline significantly outperforms [3] in harder environments, rendering comparison with [3] unnecessary. (Our implemented baseline has a significantly higher hypervolume ("$1.33e4$") than [3] ("$8427.51$") for 6-objective not strictly convex FTN environment)

Misleading and groundless accusations of reviewer tiB3

We kindly remind you that major accusations of reviewer tiB3 have been groundless or even proven to be false. For example, they use the following claim to attack our baselines:

" (Yang et al. 2019)[3] is able to find most (if not all) the optimal solutions"(while our baselines only learn limited Pareto optimal policies)

However, the fact is that [3] only claimed to learn a convex coverage set, which is only a subset of Pareto optimal policies, and our implemented baseline has a significantly higher hypervolume ("$1.33e4$") than [3] ("$8427.51$"), learning even more Pareto optimal policies than [3].

The core concerns of tiB3 have been based on the groundless accusation that "our LS baseline is poorly implemented". However, evidence has proven this accusation to be wrong. Therefore, tiB3's points appear biased and misleading.

---

We kindly request you to reevaluate reviewer caBw's perspective and reassess the validity of reviewer tiB3's comments to ensure a fair and constructive final decision.

Thanks in advance!

Kind regards,

---

[1] Basaklar et al.(2023). "PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm. In The Eleventh International Conference on Learning Representations."

[2] Abels et al. "Dynamic weights in multi-objective deep reinforcement learning." International conference on machine learning. PMLR, 2019.

[3] Yang et al. (2019) "A generalized algorithm for multi-objective reinforcement learning and policy adaptation"

[4] Alegre et al. (2023) "Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization"

[5] Lu et al., (2023) "Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality"

[6] Andrychowicz et al.(2018) "Hindsight Experience Replay. "

Dear reviewer,

Thank you again for your valuable feedback and advice on fostering a respectful and positive community.

We kindly remind you that we included a general response, "Final comment and additional SOTA baseline":

1. It clarifies why the listed methods [1][2][3][4] are not directly compatible.

2. Though CAPQL [4] was only implemented for SAC, we found a way to modify it into a general setting (without requiring Q-values), and we made a comparison with it to further demonstrate our method's effectiveness over SOTA methods.

3. We explained why the empirical results in the 6D fruit tree are sufficient to demonstrate our method's superiority over [1],[2], [3], and [4]. Its lower-dimensional discrete state and action spaces, combined with the challenge of 6 objectives and a non-strictly convex Pareto front, isolate MORL performance from lower-level RL factors, enabling a fair comparison independent of those modifications.

4. The result below confirms our LS baseline is properly implemented.

---

Fruit-tree 6D

As shown, our LS baseline achieves comparable performance to the SOTA method GPI-LS [2], indicating it is not "poorly" implemented. Our proposed PreCo significantly outperforms the SOTA methods [1][2][3][4] in this environment that isolates MORL performance from lower-level RL factors.

Note: GPI-LS results are calculated using the "front" data from link provided by reviewer tiB3. Enelope and PD-MORL are reported in [3], and CAPQL represents our implementation of PPO with maximum entropy.

---

We sincerely hope this could further clarify your concerns and increase your confidence in your evaluation!

---

[1] Yang et al. (2019) "A generalized algorithm for multi-objective reinforcement learning and policy adaptation"

[2] Alegre et al. (2023) "Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization"

[3] Basaklar et al.(2023). "PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm.

[4] Lu et al., (2023) "Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality"

Dear Reviewer NkSe,

We hope this message finds you well. As the discussion period closes in less than 24 hours, we wanted to kindly remind you of our earlier response addressing your comments.

Before the Nov. 26 revision deadline, we added further results comparing state-of-the-art (SOTA) methods, demonstrating the empirical superiority of our approach despite unfair lower-level modifications like HER and specific training curriculum. Reviewer tiB3 acknowledged these improvements and raised their score, suggesting that we have adequately addressed some concerns.

We kindly ask you to consider raising your score or confidence level, as this would not only support us but also contribute to the broader community. For example, even in the most recent MORL work that's published in Oct.3rd, the misconception that linear scalarization can discover all Pareto optimal policies persists. Our work helps clarify this major issue, making a non-trivial contribution to the MORL community and advancing future research.

Thank you for your time and thoughtful evaluation.

Best regards,

Author of submission 5131

Thanks for the detailed review and constructive feedback! We aim to address your concerns below:

---

Weakness 1. There is a lack of a formal definition for the policy-level gradient ... generative models.

---

Response 1: Policy level gradient is defined for PPO/TD3 in Appendix.D. To provide more intuition, an illustrative diagram of the backward path for policy updates is provided in this anonymous link.

For more expressive models in continuous space, we can replace $\nabla_{\pi_p}v^{\pi_p}$ in min-norm (eq.6) with the gradient w.r.t. $B$ samples of actions.

$\mathbf{g}_a=[\nabla{a_1}\mathbf{Q}({s_1},{a_1}),\nabla{a_2} \mathbf{Q}({s_2},{a_2}),...,\nabla{a_B} \mathbf{Q}({s_B},{a_B})]$

where $B$ denotes the batch size. Solving min-norm (Eq. 6) for $\mathbf{g}_a$ provides an update direction for the samples of action, enabling us to update the policy network accordingly.

---

Weakness 2. The training time is not reported, which makes the claim of computational efficiency... unconvincing..

---

Response 2: The mainly claimed advantage is the memory efficiency of solving min-norm (eq.6) with $\nabla_{\pi_p}v^{\pi_p}$ rather than with $\nabla_{\theta}v^{\pi_p}$. The size of $\nabla_{\pi_p}v^{\pi_p}$ is typically $B$, the batch size (hundreds in most cases), whereas $\nabla_{\theta}v^{\pi_p}$'s size is the number of the neural network parameters, which can range from hundreds to billions.

We conducted a simulated comparison of computational time on a problem with $6$ objectives, using a batch size of $128$ and a parameter size of $512$ to solve the min-norm problem in Eq. 6 for $100$ times. The results are summarized in the table below:

The parameter-level gradient was simulated using samples from "torch.rand(512)", while the policy-level gradient was simulated with "torch.rand(128)". These results already demonstrate a significant gap in computation time. In practice, this gap in gradient sizes can be more drastic—for example, 256 for policy-level versus vs. 7B for parameter-level gradients.

---

Weakness 3. The evaluation metrics are limited...

---

Response 3: Thank you for the constructive suggestions on evaluation metrics! Below are the extended versions of Tables 2,3 and 4, now including Sparsity (SP) and Overall Non-dominated Vector Generation Ratio (NR) for Figures 5 and 6.

MO-Hopper(Fig.5)

MO-Ant(Fig.6)

MO-Reacher(Fig.7)

We observe that the values produced by PreCo exhibit decent sparsity and a high non-dominated ratio. Full results for SP and NR will be included in the revisions.

---

Weakness 4. Comparisons could be made more extensive. ... multi-objective RL algorithm [5].

---

Response 4: Thanks for the helpful suggestion and we will include a reference to PD-MORL in the related works section. However, PD-MORL [5] heavily relies on HER [1], a technique specifically designed for off-policy methods, which limits its general applicability to on-policy methods. A direct comparison between the full implementation of PD-MORL and PreCo is not entirely fair, as HER could also be incorporated into our method to achieve further improvements.

We have tried to implement [5] in our setup without using HER, but the learning failed for mo-Hopper V4. Their paper[5] suggested that this may be due to certain preferences being underrepresented. We also believe a main reason is the conflicts among their deterministic policies' gradients, as their policies must optimize for both cosine similarity and Q-values. Instead, methods like MGDA[2], CAGrad[3], SDMGrad[4] and our PreCo find a common ascend direction (e.g., by solving min-norm problem in Eq. 6) to avoid gradient conflicts. In future work, we could draw inspiration from [5] to develop a better MORL algorithm specifically designed for off-policy, value-based methods.

Additionally, their reported hypervolume for 6-D reward Fruit-Tree Navigation is $9299.15$, which significantly lower than our PreCo's $1.56e4$.

---

Weakness 5. The authors may also wish to compare their approach with another baseline that incorporates preferences as input [6].

---

Response 5: Similar to [5], [6] is a MORL algorithm specifically designed for value-based methods, so it cannot be directly generalized to policy-based methods like PPO. In contrast, our proposed PreCo is a higher-level approach compatible with most mainstream RL algorithms. Additionally, [6] aims to maximize the linear scalarization (LS) objective, so it is an LS approach (a baseline covered in our experiments) tailored for Q-learning.

---

Weakness 6. Additionally, this work references SDMGrad ... SDMGrad.

---

Response 6:

SDMGrad results are already included in all of our experiments reported in Section 5.

The key difference in formulation between PreCo and SDMGrad is that PreCo adds a similarity gradient (Eq.6) while SDMGrad SDMGrad incorporates a linear scalarization (LS) gradient (Eq.14 in Appendix B.1). For this reason, we categorize SDMGrad as an LS method and use it as an ablation case to compare the performance with and without our proposed similarity function.

Answering questions

---

Q1: Are there other similarity functions that can be used in Equation (6), or what properties should these similarity functions have?

---

A1: For Thereom 4.1 (Pareto optimal for constant $\lambda$) and 4.3 (Controllable for constant $\lambda$) to be valid:

1. It should be Lipschitz smooth.
2. Its gradient $g_s$ should be a positive linear combination of the objective gradients $g_i$. For example, the similarity gradient can be $g_s = 5g_1+0.5g_2$, where $5$ and $0.5$ are positive coefficients.

For Thereom 4.2 (Pareto optimal for increasing $\lambda$) and 4.4 (Controllable for increasing $\lambda$) to be valid:

1. It should be Lipschitz smooth.
2. Its gradient $g_s$ should be convex combination of the objective gradients $g_i$. For example, the similarity gradient can be $g_s = 0.5g_1+0.5g_2$, where $0.5+0.5=1$ and they are convex coefficients.

Appendix. E could offer more insights into our similarity function design.

---

[1] Andrychowicz et al.(2018) "Hindsight Experience Replay. "

[2] Désidéri(2009) "Multiple-Gradient Descent Algorithm"

[3] Liu et al.(2021) "Conflict-Averse Gradient Descent for Multi-task Learning"

[4] Xiao et al.(2023) "Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithms"

[5] Basaklar et al.(2023). "PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm. In The Eleventh International Conference on Learning Representations."

[6] Abels et al. "Dynamic weights in multi-objective deep reinforcement learning." International conference on machine learning. PMLR, 2019.

Thanks once again for your time and effort in evaluating this work. Your constructive suggestions have greatly helped us improve the manuscript. We fully agree that a modified PreCo in an off-policy setting with HER could make the results more convincing. Due to computational and time limitations, we might only be able to explore this in the future.

However, we would like to clarify one point regarding the environment:

Although the Ant environment is indeed challenging for RL due to its high dimensionality and sample complexity, its Pareto front landscape is actually quite simple. The multiple objectives in Ant are quite symmetric (e.g., go east or go north), and its Pareto front is strictly convex. As a result, performance in Ant can be highly dependent on the underlying RL implementation (such as HER[1][3] and training curriculum[2]) rather than the MORL algorithm itself.

In contrast, the fruit-tree environment features lower-dimensional discrete state and action spaces, combined with the challenge of 6 objectives and a non-strictly convex Pareto front. This design isolates MORL performance from lower-level RL factors, enabling a fairer comparison that is independent of those modifications.

Below is the table showing the superior performance of our methods over the mentioned methods [1][2][3][4] during the discussions, including their full implementations (such as HER[1][3] and curriculum training[2]), highlighting our advantages over SOTA MORL methods.

Fruit-tree 6D

---

Note: All results are calculated from objective returns, using the origin as the reference point. GPI-LS results are calculated using the "front" data from link provided by reviewer tiB3. Envelope and PD-MORL are reported in [3], and CAPQL represents our implementation of PPO with maximum entropy explained in Appendix G.

---

If this further resolves your concerns, we would be truly grateful if you could consider increasing your ratings or confidence.

---

[1] Yang et al. (2019) "A generalized algorithm for multi-objective reinforcement learning and policy adaptation"

[2] Alegre et al. (2023) "Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization"

[3] Basaklar et al.(2023). "PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm.

[4] Lu et al., (2023) "Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality"

Dear Reviewer caBw,

We hope this message finds you well. With the discussion period closing in less than 24 hours, we kindly remind you of our earlier response addressing your comments.

Before the Nov. 26 revision deadline, we added new results comparing state-of-the-art methods, PD-MORL and CAPQL, with their lower-level modifications (HER and preference space segmentation) preserved. They demonstrate the empirical superiority and theoretical advantages of our approach. Unlike CAPQL, our method avoids information loss by preserving the original problem structure. Compared to PD-MORL, our approach is more general and avoids potential conflicts between cosine similarity and objective gradients.

We respectfully ask you to consider raising your score or confidence level, as doing so would not only support our work but also contribute significantly to the MORL community. For example, even the most recent MORL work that's published in Oct.3rd continues to misinterpret the ability of linear scalarization to discover all Pareto-optimal policies. Our work addresses this critical issue, offering a valuable contribution to the MORL community and advancing future research.

Thank you for your time and thoughtful evaluation.

Best regards, Author of submission 5131