Efficient Discovery of Pareto Front for Multi-Objective Reinforcement Learning

We sincerely appreciate your thoughtful questions. Below, we first clarify a few interpretations and claims shown in this paper, and then provide further discussions:

(D-1) Lack of Innovation and Convincing Results.

Novelty and advantages.

We want to justify this work’s novelty from the following two perspectives.

1. Framework. The key innovation of our algorithm lies in its ability to efficiently construct the full Pareto front by constrained policy optimization-based Pareto extension starting from only a small number of points on it. These initial points are obtained by training a limited set of policies to derive Pareto optimal solutions. This also allows us to bridge rich tools and theories from constrained optimization to MORL problems. C-MORL is not encapsulating PG-MORL, as C-MORL distinguishes itself from Pareto extension, constrained optimization formulation and solution approaches, as well as superior efficiency (detailed below). And as mentioned in Section 2. Related Work, this also holds a fundamental distinction from single preference-conditioned policy or multi-policy approaches in the literature.

2. Training efficiency. From only a small number of Pareto optimal policies, C-MORL employs a constrained policy optimization-based Pareto extension to systematically derive new policies, completing the Pareto front without the need to train additional policies from scratch. This approach significantly reduces computational overhead while ensuring high-quality solutions across the entire Pareto front. We also show a novel 9-objective environment, where PG-MORL fails short of finding good policies.

3. Ability to extend to benchmarks with large objective spaces. Existing methods primarily focus on tasks with continuous observation spaces and up to 5 objectives, or discrete observation spaces with up to 6 objectives. In contrast, C-MORL can effectively handle benchmarks with larger objective spaces by identifying gaps on the Pareto front and generating new policies in the desired regions. This is achieved by optimizing a policy toward a specific objective direction while constraining the performance of other objectives, enabling precise and efficient expansion of the Pareto front.

Comparison with PG-MORL. The most significant limitation of PG-MORL [1] is its inability for benchmarks with more objectives, as the computation time grows exponentially when number objectives increase. As a typical evolutionary approach, in each iteration, it involves solving a knapsack problem using a greedy algorithm to identify tasks that could most effectively improve Pareto quality. This step has an exponential time complexity as the number of objectives increases. Instead of iteratively improving the overall quality of a policy set as most evolutionary approaches do, C-MORL directly fills the desired region of Pareto front via constrained policy optimization from a few Pareto optimal policies.

Convex-Converage-Set (CCS). By definition, a Convex Coverage Set (CCS) is a finite convex subset of the Pareto front, where each policy in the set is optimal with respect to some linear utility function [2]. This means that for any linear preference, there exists a policy within the CCS that is optimal. It is important to note that the CCS is a subset of the Pareto front, and the term "optimal" in its definition refers only to optimality within the CCS, not the entire Pareto set.

In most benchmarks, particularly those with continuous objectives, obtaining the complete Pareto set is intractable. This limitation highlights the need to select the best policy from the set for a given preference. Ideally, increasing the buffer size could improve the quality of the Pareto set, resulting in higher expected utility for specific preferences. However, this approach introduces a trade-off between efficiency and performance.

(D-8) (Question 3) Efficiency and Resource Intensity. We would like to clarify that C-MORL did not store all intermediate policies. Instead, we maintained a policy buffer. If the number of policies exceeds the policy buffer size in the current iteration, we select a number of policies equal to buffer size from the set of all Pareto optimal policies based on their crowd distance.

For Table 1 and 2, we set the buffer size $B=100$ for tasks with 2 objectives and $B=200$ for tasks with 3 or more objectives, following the same configuration as PG-MORL. However, we observed that in many benchmarks, the number of policies never exceeded buffer size and was significantly smaller. This suggests that C-MORL is not resource-intensive.

To further analyze the influence of buffer size and also demonstrate the resource efficiency of C-MORL, we conducted additional ablation studies on buffer size, which are detailed in Table 12 and Table 13 in ‘Official comment by Authors’ for all reviewers. These studies show that even with reduced buffer sizes, C-MORL maintains highly competitive performance.

Compared to PG-MORL, our C-MORL achieves superior results while requiring significantly fewer policies. Another notable observation is that, in some benchmarks, the results remain consistent regardless of the buffer size. This consistency indicates that the number of Pareto optimal policies in these cases does not exceed the buffer size.

(D-9) (Question 4) Selection of Extreme Solutions. This is because for fair comparison, we ran C-MORL and baselines with the same number of steps (as mentioned in Experiment setup, $5 \times 10^5$ time steps for discrete benchmarks. Continuous benchmarks with two, three, and nine objectives are trained for $1.5 \times 10^6$, $2 \times 10^6$, $2.5 \times 10^6$ steps, respectively. Therefore, although ideally we would like to derive some Pareto optimal solutions in the Pareto initialization stage, while training steps are limited, the initial policies have not reached bottlenecks in some of the benchmarks in the initialization stage. We empirically find that if these policies are selected (only in the first extension iteration), the overall results can be slightly better.

[1] Xu, Jie, et al. "Prediction-guided multi-objective reinforcement learning for continuous robot control." International conference on machine learning. PMLR, 2020.

[2] Alegre, Lucas Nunes, Ana Bazzan, and Bruno C. Da Silva. "Optimistic linear support and successor features as a basis for optimal policy transfer." International conference on machine learning. PMLR, 2022.

(D-2) Time complexity comparison. We would like to clarify that PG-MORL is not part of our algorithm, and as such, PG-MORL’s computation time should not be included in the analysis. A key and time-consuming component of PG-MORL is their prediction-guided policy selection, which involves solving a knapsack problem using a greedy algorithm to identify tasks that could most effectively improve Pareto quality. This step has an exponential time complexity as the number of objectives increases. Also as shown in Table 2, PG-MORL is time out for the Building-9d case.

In contrast, proposed C-MORL does not need to solve this knapsack problem; instead, it identifies the gaps on Pareto front based on the crowding distance of policies. Moreover, the overall procedure of C-MORL is fundamentally different from PG-MORL. While PG-MORL begins with a brief initialization phase, followed by an evolutionary process involving policy selection and training, C-MORL takes a different approach. It trains a small number of policies until near convergence and then performs constrained policy optimization to fill Pareto front.

As a result, the major computation of C-MORL primarily comes from solving multiple constrained policy optimization problems, as analyzed in Proposition 5.1.

(D-3) Policies converging around certain preferences. The reviewer mentions that ‘In subsequent policy selection and optimization, the distribution of preferences remains unchanged’. We would like to clarify that the policy selection and optimization process do not rely on any specific preference. Instead, based on the crowd distance, we identify the sparse areas of the Pareto front, and then conduct constrained policy optimization to fill these gaps.

Next, we answer other questions and weaknesses raised by the reviewer.

(D-4) (Weakness 7) Inconsistency in Method Presentation. As stated in Appendix E.2, , we implement all of our experiments for C-MORL based on C-MORL-IPO. This is because IPO is a first-order method, which is more computationally efficient than C-MORL-CPO. We mentioned both algorithms since they can both give tractable solutions for the MORL formulation. And these two methods’ major differences are on the solving procedure of optimization rather than on the solution quality. In the revised manuscript, we made further explanations on the experiment settings, and reorganized the paper writings to make the algorithm part easier to follow.

(D-5) (Weakness 8) Statistical Significance in Comparison Results. Thank you for pointing this out; we will revise them accordingly. In our revised version, a method is also marked if its mean value lies within the confidence interval of the best method. We would like to note that C-MORL’s performance is still most consistent across all metrics and environments as shown in Table 1 and Table 2.

(D-6) (Question 1) Selection of Mujoco Environments. In the manuscript, we selected environments from the widely used and comprehensive MORL toolkit, mo-gymnasium, prioritizing the most challenging baselines as documented in the mo-gymnasium reference materials.

Specifically:

MO-Humanoid was chosen because it has the most complex state space among all Mujoco environments, with 348 states, making it very challenging for many MORL algorithms.

MO-Ant and MO-Hopper were selected for their diverse number of objectives (2 and 3, respectively) and relatively complex state spaces. For instance, MO-Ant has 105 states, which are significantly more than those in simpler environments we excluded.

For environments we did not select, such as MO-Swimmer, which has only 8 states and 2 objectives, we believe they are relatively easy to train and thus do not present sufficient challenge for meaningful evaluation.

(D-7) (Question 2) Unclear Initialization and Preference Distribution. We are sorry for the confusion made in the manuscript. In Appendix E.2, we discussed training details, including the number of initial policies $M$, as shown in Table 8 and 9. In the policy initialization stage, the preference vectors for training initial policies are uniformly sampled from the preference space $\Omega$. For example, in the case of MO-Ant-2d, we set the sampling interval to 0.2, with the minimum and maximum values for each dimension being 0 and 1, respectively. This results in the following preference vectors: [0, 1], [0.2, 0.8], [0.4, 0.6], [0.6, 0.4], [0.8, 0.2], and [1, 0], for a total of 6 preference vectors. In the revised manuscript, we added details on the preference vectors.

We appreciate your constructive feedback.

(C-1) The training of GPI-LS is definitely not parallel. In this sentence we would like to clarify the sample efficiency of GPI-PD. Meanwhile, its model-free version, GPI-LS, never parallelizes the training of CCS. The algorithm block in Algorithm 1 [1] shows the process of GPI-LS, and from line 3-12, it is very clear that new policies are iteratively added into the policy set $\Pi$. Although in each iteration, if multiple corner weights are selected (line 7), the policies in this iteration can be trained parallelly, while the overall process of GPI-LS is not in parallel.

(C-2) Thanks for your suggestion, we will consider revising the notation.

(C-3) Thanks for pointing this out and we havel added the constraint on $\omega_i$.

(C-4) Projection to a compact and convex set. Proposition 4.4 is related to the proof of convergence of dual problem (2), which is a two-timescale stochastic approximation process. The projection operator is a commonly used technique for constrained optimization to ensure feasibility. The operator follows projections onto convex sets (POCS) or the alternating projection algorithm, which can be implemented very efficiently to ensure both $\theta$ and $\lambda$ are feasible for problem (2).

(C-5) Updates of min max problem. Yes, $\pi_r$ is a parameterized policy. The fixed point means a local minima. The iterative steps in Proposition 4.4 describe the updates on both the primal variable ($\pi_r$) and dual variables ($\lambda_r$). It follows the standard primal-dual fixed point referenced as [2] in Appendix. C. The essence of this update rule is to ensure both feasibility with respect to constraints for each other objective in problem (1), and convergence to the fixed point. In numerical optimization, the convergence proof is based on fixed-point iteration, while in the revised paper, we will further clarify that fixed-point found by Proposition 4.4 is where the policy network converged to.

(C-6) Slater’s condition. Slater’s condition is a fundamental condition in optimization theory. Slater's condition requires that there exists a feasible policy $\pi$ such that all inequality constraints are strictly satisfied for problem (1), i.e., $G_i^\pi > d_i, \quad \forall i = 1, \ldots, n, \; i \neq l.$. This condition ensures that the duality gap between problem (1) and its dual problem (2) is zero, as stated in Theorem 4.5.

(C-7) Buffer size. Thank you for pointing this out. In our implementation, the buffer size represents the maximum number of policies that can be stored. If the number of policies exceeds this limit, we store policies based on their crowd distance to match the buffer size. This introduces a trade-off between the diversity of the Pareto set and memory usage: increasing the buffer size allows for a more diverse Pareto set but also results in higher memory costs.

In practice, for many benchmarks, the number of derived Pareto optimal policies is often smaller than the buffer size, meaning that storing them does not require excessive memory resources. Furthermore, as shown in Tables 12 and 13 (please see Table 12 and 13 in ‘Official comment by Authors’ for all reviewers), even with reduced buffer sizes, C-MORL demonstrates highly competitive performance. Compared to PG-MORL, which uses a buffer size of 100 for 2-objective benchmarks and 200 for 3-objective benchmarks, our C-MORL achieves superior results while requiring significantly fewer policies. Another notable observation is that, in some benchmarks, the results remain consistent regardless of the buffer size. This consistency indicates that the number of Pareto optimal policies does not exceed the buffer size in these cases.

(C-8) Selection of $\beta$. $\beta G_i^{\pi_r}$ in Eq.3 is related to the corresponding term $d_i$ in Eq.1, which is the threshold of constraint-specific reward functions. Here $G_i^{\pi_r}$ is derived from the latest constrained optimization step, while $\beta$ is a tunable parameter that controls the value of constraint. In Proposition 4.3, we give the sufficient condition of $d_i$, under which the solution of Eq.1 is a Pareto-optimal solution.

(C-1) Thanks for your explanation. In our initial response, we mentioned that ‘ if multiple corner weights are selected (line 7), the policies in this iteration can be trained parallelly’. We believe we are on the same page regarding this point. By ‘not parallel’, we simply mean that the policy training in the next iteration depends on the number of corner weights derived in the last iteration, in this sense it is an iterative process. To avoid confusion, we have modified this expression in our new manuscript.

(C-4) and (C-6) Thanks for your suggestion. We have revised them in Section 4 and Appendix D.

(C-10) PG-MORL requires solving knapsack problems. As mentioned in their supplementary material, the time complexity of PG-MORL is computed by $\mathcal{O}\left(n K|\mathcal{P}| N^{m-2} \log N / p\right)$ when $m>2$, where $m$ is the number of objectives in their paper. From this equation, the time complexity increases exponentially when the number of objectives increases. When running the PG-MORL code, we found that the program got stuck at the prediction-guided optimization for task selection. This is because, with 9 objectives, the computation in this part becomes extremely time-consuming.

We conduct CAPQL and GPI-LS based on morl-baselines [5]. It costs around 69.4 hours to train building-9 for CAPQL. Therefore, we believe that the time spent on evaluation is not the primary factor affecting the actual training time. Through debugging the GPI-LS, we discovered that the program halts at the step where linear_support calculates the next weights (w = linear_support.next_weight()). Due to the limitation of time, we are unable to improve the implementation of GPI-LS for the benchmark with 9 objectives. Therefore, in our current version, we only provide the experiment results for GPI-LS on benchmarks with less than 9 objectives.

(C-17) Thanks for your explanation and clarification of the method comparisons. However, from equation $s_{\boldsymbol{r}}(\boldsymbol{v})=\min _{j \in\{1, \ldots, d\}} \lambda_j\left(\boldsymbol{v}_j-\boldsymbol{r}_j\right)+\rho\sum\lambda_j\left(\boldsymbol{v}_j-\boldsymbol{r}_j\right)$ , we can see that for the Chebyshev scalarisation, the key step is to still utilize a linearized form to scale $\left(\boldsymbol{v}_j-\boldsymbol{r}_j\right)$. In addition, we would like to clarify that the IPRO used Chebyshev scalarization, but this method can only find solutions on the convex hull of the Pareto front, and all solutions on the non-convex parts of the Pareto front are still missing. From the computation perspective, we echo the reviewer’s claims that “ In IPRO, the more open space, the better since you can make large improvements.” This also draws a connection to our policy initialization stage, where our Assumption 4.2 also guarantees such initial policies are feasible with respect to multiple constraints imposed in Equation (1). Yet we want to emphasize that our policy improvement via Pareto extension is still novel, which is based on the principles of constrained policy optimization, and in our approach we do not take an approximation of Achievement scalarising functions. Following the reviewer’s suggestions, we make a more careful discussion of the IPRO method in the Related Work section: “. (R¨ opke etal.,2024) discovers the Pareto front by bounding the search space that could contain value vectors corresponding to Pareto optimal policies using an approximate Pareto oracle, and iteratively removing sections from this space. Our proposed C-MORL distinguishes from their divide-and-conquer scheme by a novel policy selection procedure along with explicitly solving a principled constrained optimization.”

In addition, we just want to highlight in Table 12 and 13, we can observe that even if the buffer size is significantly decreased, C-MORL still achieves outstanding performance across multiple benchmarks. Another observation is that the results remain consistent in some benchmarks regardless of buffer size, indicating that the number of Pareto optimal policies has not exceeded the buffer size. This further suggests that, in many benchmarks, C-MORL does not require substantial resources to store a large number of policies. These results also show that performances on HV and EU are not sensitive to buffer size for C-MORL. This is because C-MORL fills the gaps on the Pareto front by identifying the sparse regions with crowd distance-based policy selection and the novel design of the constrained policy optimization-based Pareto extension. Although SP deteriorates as the buffer size decreases, it is important to emphasize that SP does not always indicate better performance, for instance, see CAPQL in Fig 4. (b).

We hope this could justify the novelty and contributions of our work.

Thank you once again for your constructive feedback. We have addressed your concerns in our updated responses and the revised manuscript. Here is a summary of our updates:

Discussion about baselines:

(C-1) GPI-LS: We have revised the discussion of GPI-LS in our revised manuscript, and also "official comment (new)".

(C-10) and (C-12) PG-MORL: We have explained the time complexity of PG-MORL in “official comment (new)“. We have also replaced the verb ‘solve’ with ‘find a satisfiable solution efficiently’.

(C-17) IPRO: We have revised the discussion of IPRO in our revised manuscript in Section 2 Related Work, and also “official comment (new)“.

Explanation of basic RL and optimization concept:

(C-4), (C-5), and (C-6): We have provided a more detailed explanation in our revised manuscript, and also “official comment (new)“.

Experiment setting:

(C-7) Buffer Size: We have provided additional experiment results in (All-4) Ablation study on buffer size and also “official comment (new)“.

(C-8) Selection of $\beta$: We have further explained the selection of $\beta$ in “official comment (C1-8)“.

(C-9) Selection of $t$: We have further explained the selection of $t$ in “official comment (C9-16)” and Appendix E.2.

(C-11) C-MORL-IPO and C-MORL-CPO: We have highlighted the relevant statement in Appendix E.2 and “official comment (C9-16)“.

Other discussions of the experiment setting are also provided in “official comment (C9-16)“.

Typos:

(C-3) and (C-13): these have been corrected in the revised manuscript.

Your feedback is greatly appreciated, and we sincerely look forward to further improving this work and hearing from you soon!

Thank you for appreciating the novelty, presentation, theoretical rigor, and experiment of this paper, as well as for your insightful questions.

(B-1) Comparisons against utility-based/preference-conditioned methods can be improved.

Thank you for your suggestion.

Memory resources issue. As you mentioned, typical evolutionary methods suffer from memory resources issue. While C-MORL provides insights to overcome this issue from two perspectives: Buffer size $B$ is a tunable parameter that controls the maximum number of policies that can be stored. For real-world tasks with large observation spaces, it allows a trade-off between buffer size and the quality of the Pareto front. From Table 11 and 12, we observe that even with a reduced buffer size, C-MORL maintains highly competitive performance. Unlike other evolutionary methods, C-MORL begins the evolutionary process (Pareto extension) to fill the gaps on Pareto front after explicitly optimizing the set of policies that converge to points near the Pareto front. Therefore, for practical applications, we recommend storing a small number of policies and selecting a preferred one for conducting constrained RL-based extensions. This approach enables the derivation of new policies in specific regions of the Pareto front.

Policy assignment. For ‘policy selection’ (referred to as policy assignment in this paper to avoid confusion) given a specific preference, the user can choose a set-max policy (Definition 3.1). Alternatively, as mentioned earlier, if the Pareto set is not dense enough due to buffer size limitations, constrained RL based extension (C-MORL-IPO in our implementation) can be further performed. This allows extending from a specific policy to obtain diverse policies within a targeted region of the Pareto front.

(B-2) No comparisons against the MORL/D algorithm.

Thanks for your suggestion.

Update: We have conducted the additional experiments. Please refer to the newly added box for the results and discussion.

(B-3) How are comparisons between GPI-LS/CAPQL vs PGMORL/C-MORL made?

We appreciate the reviewer for pointing out the concerns on algorithm evaluations and scalability.

Buffer size and number of evaluation preference vectors. First, we would like to point out some of our algorithm’s properties and emphasize that buffer size (the number of Pareto optimal policies allowed to store) and number of evaluation preference vectors are different concepts. For multi-policy methods, increasing the buffer size allows more Pareto optimal policies to be stored, which generally improves HV and EU but does not guarantee better SP. For preference-conditioned policy methods (e.g., linear scalarization methods), increasing the number of evaluation preference vectors tends to improve HV. However, this does not necessarily lead to better EU or SP, as these methods do not ensure that a Pareto optimal policy is assigned to every given preference. For example, in Fig. 4(a), the Pareto front of Q-Pensieve shows that removing the solution on the lower-right corner would lead to better SP (i.e., yield a lower SP value). Finally, for multi-policy methods, increasing the number of evaluation preference vectors does not guarantee to improve any of the three metrics.

Baselines and evaluation of metrics. Among the baseline MORL approaches, CAPQL, Q-Pensieve and GPI-LS are linear scalarization methods. They aim to obtain preference-conditioned policies, and are evaluated by some uniformly sampled preference vectors. PG-MORL/C-MORL are multi-policy methods. For multi-policy methods, to evaluate EU, we generate preference vectors with the same interval for all baselines, as is shown in Appendix E.2. To evaluate HV and SP, we directly use the solutions in Pareto set to compute, while we want to emphasize again the number of policies is bounded with a buffer size (100 for 2-objective environments, and 200 for others). For linear scalarization methods, we also generate preference vectors with the same interval for all baselines (totally same with multi-policy methods) to evaluate the three metrics, as mentioned in Appendix E.2.

Additional experiment results. From Table 12 and 13, we can observe that even if the buffer size is significantly decreased, C-MORL still achieves outstanding performance across multiple benchmarks. Another observation is that the results remain consistent in some benchmarks regardless of buffer size, indicating that the number of Pareto optimal policies has not exceeded the buffer size. This further suggests that, in many benchmarks, C-MORL does not require substantial resources to store a large number of policies. These results also show that HV and EU are not sensitive to buffer size. Although SP deteriorates as the buffer size decreases, it is important to emphasize that SP does not always indicate better performance, for instance, see CAPQL in Fig 4. (b).

(B-4) How would C-MORL react to concave Pareto fronts?

By mentioning "...observe that in some benchmarks, the current constrained policy optimization method fails to adequately extend the policies toward certain objective directions", the main reason is that in order to ensure fair comparison, we train C-MORL and all baselines with the same steps. Therefore, due to the limited number of steps dispatched for Pareto extension, some areas on Pareto front are not well explored. In the future work, we plan to explore more sample-efficient methods to improve Pareto extension within limited steps for targeted preferences.

Regarding concave pockets on the Pareto fronts, as you pointed out, exploring convex regions are indeed easier for MORL. From the optimization perspective, it is impossible to find a converged and equilibrium solution for concave Pareto front. So that the gradient-based methods could go to few solutions which cannot cover the Pareto fronts well [1].

We also provide a concave benchmark to demonstrate the performance of C-MORL. Deep-sea-treasure-concave is the concave version of Deep-sea-treasure, which is a classic MORL problem in which the agent controls a submarine in a 2D grid world [2]. In this benchmark, the Pareto front is know, so we can compare the approximate Pareto front we discovered with the entire Pareto front. Addition to HV, EU and SP, we use Inverted generational distance (IGD) (the smaller the better), which characterises the convergence rate of an approximate $\mathcal{F}$ towards a reference Pareto front $\mathcal{Z}$ (i.e. the entire Pareto front) [2]: $\operatorname{IGD}(\tilde{\mathcal{F}}, \mathcal{Z})=\frac{1}{|\mathcal{Z}|} \sqrt{\sum_{\mathbf{v}^* \in \mathcal{Z}^{\prime}} \min _{\mathbf{v}^\pi \in \tilde{\mathcal{F}}}\left\|\mathbf{v}^*-\mathbf{v}^\pi\right\|^2}$.

Table 14: Evaluation of HV, EU, SP, IGD for Deep-sea-treasure-concave benchmark.

From Table 14, we observe that compared to GPI-LS, C-MORL achieves higher HV and lower (better) IGD values, indicating a better approximation of the Pareto front. Notably, in two out of five runs, C-MORL successfully discovers the entire Pareto front. However, in the remaining runs, it fails to capture certain neighboring Pareto-optimal solutions. For example, the extire Pareto front is [[1, -1], [2, -3], [3, -5], [5, -7], [8, -8], [16, -9], [24, -13], [50, -14], [74, -17], [124, -19]], while C-MORL obtain an approximation [[1, -1], [5, -7], [8, -8], [16, -9], [24, -13], [50, -14], [74, -17], [124, -19]].

[1] Jin, Yaochu, Markus Olhofer, and Bernhard Sendhoff. "Dynamic weighted aggregation for evolutionary multi-objective optimization: Why does it work and how." Proceedings of the genetic and evolutionary computation conference. 2001.

[2] Felten, Florian, et al. "A toolkit for reliable benchmarking and research in multi-objective reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

We appreciate your constructive suggestions on improving the manuscript. We have revised the paper accordingly, and in the following, we would like to address the reviewer’s concerns and clarify the paper’s organizations and major contributions.

(A-1) We revised our manuscript for better presentation. Thank you for pointing out your concerns about the presentation. We made the description about the C-MORL more concise. In our revised version, we moved some theoretical analysis regarding constrained MORL-based Pareto front extension to the Appendix, and provided further discussion about design choices and hyperparameter study in the experiment section.

(A-2) Framework of C-MORL. The training of C-MORL only includes two stages. In Pareto initialization stage, the goal is to derive a few Pareto optimal policies; then in Pareto extension stage, it performs constrained policy optimization to fill the Pareto front. Specifically, polices are selected to be extended based on their crowd distance.

(A-3) (Question: how difficult to determine the specific values for hyperparameters?)

To better understand this problem, we provide additional ablation study.

Number of initial policies. We add ablation study for Pareto initialization in the experiment section to understand the influence of number of initial policies. To be specific, we conduct experiments while keeping the total number of training steps fixed at $1.5\times10^6$ steps (including $1\times10^6$ steps for initialization stage) to ensure a fair comparison. We uniformly sample $M=3, 6, 11$ preference vectors, as shown in Table 3. For example, with a sampling interval of $0.2$, the preference vectors are $\boldsymbol{\omega}=[0,1],[0.2, 0.8],[0.4, 0.6],[0.6, 0.4],[0.8, 0.2],[1,0]$. Intuitively, increasing $M$ can enhance diversity among Pareto-optimal solutions, which benefits the subsequent extension phase. However, since the number of total time steps is fixed, increasing $M$ reduces the training steps allocated to each initial policy. The trade-off is evident as the performance for $M=11$ is worse than for $M=6$, indicating that the increasing of $M$ does not always guarantee better performance. To further investigate, we increase the number of training steps to $3\times10^6$ (with $2\times10^6$ steps allocated to initialization stage) while keeping $M=6$. The results demonstrate that proportionally increasing the total training steps and the number of initial policies leads to improved performance, highlighting the importance of balancing training resources with policy diversity.

Number of extension policies. In our revised manuscipt, we provide ablation study on number of extension policies in Table 12 on the MO-Ant-3d benchmark, which is one of the continuous MORL tasks with numerous objectives. The number of time steps is proportionally increased with the number of extension policies increases. We can observe that when $N=12$, C-MORL can derive a Pareto front with much higher hypervolume and lower sparsity than that of $N=6$. When $N=18$, the result is similar, indicating that $12$ policies are sufficient to be extended to fill the Pareto front for this task. Fig. 7 further illustrates the Pareto extension process with varying values of the number of extension policies ($N = 6, 12, 18$) on the MO-Ant-3d benchmark. It can be observed that as the number of policies selected to be extended increases, the hypervolume also increases.

Constraint relaxation coefficient. Parameter study for return constraint hyperparameter $\beta$ is discussed in the experiment section to examine how constraint values influence the generation of Pareto-optimal solutions. Table 4 presents the results of C-MORL with various $\beta$ on the MO-Ant-2d benchmark. When $\beta = 0.9$, C-MORL presents the best performance on all metrics. Intuitively, a higher $\beta$ ensures that when optimizing a specific objective, the expected returns on other objectives do not decrease significantly, thereby facilitating adaptation to new Pareto-optimal solutions. **C-MORL-CPO vs. C-MORL-IPO. ** As stated in Appendix E.2, , we implement all of our experiments for C-MORL based on C-MORL-IPO. This is because IPO is a first-order method, which is more computationally efficient than C-MORL-CPO. And these two methods’ major differences are on the solving procedure of optimization rather than on the solution quality. In the revised manuscript, we made further explanations on the experiment settings.

(All-4) Ablation study on buffer size. In order to better understand how Buffer size can influence the performance of C-MORL, we provide the comparison of the best baseline v.s. C-MORL with various buffer sizes.

Table 12: Evaluation of HV, EU, and SP for discrete MORL tasks under varying buffer sizes, alongside a comparison with the best-performing baseline.