Reward Dimension Reduction for Scalable Multi-Objective Reinforcement Learning

Thank you for your valuable comments on our work. We have uploaded a portion of our responses here and will address the remaining questions shortly (W2-3, Q1, Q3, and Q8-9).

1. W1. "Somewhat unclear empirics (the authors only provide scalar statistics on a single environment)"

As suggested by Reviewer Pwfy, we have added another environment from existing MORL benchmarks even though it has fewer objectives than the traffic environment. This addition strengthens the validity of our experimental evaluation.

The new environment, LunarLander-5D [1, 2], features a five-dimensional reward structure where the agent’s goal is to successfully land a lunar module on the moon's surface. Each reward dimension represents: (i) a sparse binary indicator for successful landing, (ii) a combined measure of the module's position, velocity, and orientation, (iii) the fuel cost of the main engine, (iv) the fuel cost of the side engines, and (v) a time penalty. This environment presents significant challenges because failing to balance these objectives effectively can easily lead to unsuccessful landings [1, 2].

We evaluated our method using hypervolume and sparsity metrics, with all dimension reduction methods set to a tuned reduced dimension of $m=3$ and the reference point for hypervolume evaluation set to $(0, -150, -150, -150, -150) \in \mathbb{R}^{5}$. The results show that our algorithm outperforms the baseline approaches in this environment as well.

 

2. W4-5. Effect of reduced dimension $m$ + NPCA

$m$ is a hyperparameter for our algorithm, and selecting an appropriate value in practice is achievable. This is because we can estimate the effective rank of the sample covariance matrix recursively [3] during the early exploration phase of the RL algorithm, rather than throughout the entire training process. We found that this straightforward approach performs effectively in our experiments.

The following table shows the impact of varying the reduced dimensionality $m$ in our method in the traffic environment. As $m$ increases from 4 to 6, sparsity rises significantly while hypervolume remains constant. This results in situations where only a few objectives perform well. When $m$ increases from 8 to 10, both sparsity increases and hypervolume decreases, leading to a lower-quality set of returns in the original reward space. Based on these observations, we set $m=4$ for our traffic environment experiments.

Regarding NPCA, while it was originally designed for batch data in [4], we adapted it for online learning. If we were to directly apply the formulation from [4], we would need an additional positivity constraint $U \geq 0$. This constraint would introduce an extra hyperparameter, potentially reducing overall stability. To ensure a fair comparison with our method, which uses direct parameterization, we implemented an online variant of NPCA that also adopts direct parameterization to satisfy $U \geq 0$. Additionally, we fine-tuned key hyperparameters such as $\beta$, the learning rate for gradient descent, and the update interval for NPCA.

 

3. Q2. "Can you discuss any potential use of work like 3 to reduce reward space by canonicalizing objectives?"

The EPIC distance is a pseudometric that measures the "distance" between reward functions by canonicalizing them. It is widely used when the true reward function is unavailable during the training phase, such as in [5], where vision-language models are used for reward learning. The EPIC distance decreases as the correlation between two canonicalized reward functions increases.

One could canonicalize each objective in MORL and combine some rewards based on their closest canonicalized functions. However, there are two key challenges with this approach. (i) Calculating the EPIC distance efficiently in complex RL environments is non-trivial. (ii) Designing a method to combine (or cluster) objectives is another challenge. For instance, determining the mixing ratio of the reward functions requires careful consideration. Exploring this direction further represents a promising avenue for extending our work.

4. Q4-5. "Why does finite ep length introduce more overhead?" + complexity

We apologize for any confusion caused by our earlier explanation. Introducing a bias term $b$ does not increase computational overhead. Our point was that adding this term does not provide any additional benefit in our learning process, as mentioned in lines 501-502. Overall, our method is simple and modular as you mentioned in Strengths of our work, ensuring that it does not introduce significant computational burden.

 

5. Q6-7. "Why did you choose this reference point?" + robust statistics

We selected the reference point based on observations that, after the initial exploration phase, most points in the current Pareto fronts fell within this defined region (except for PCA). To enhance the statistical reliability of our experimental results, we applied a 12.5% trimmed mean by excluding the seeds with maximum and minimum hypervolume values over eight random seeds and reporting the averages of the metrics over the remaining six random seeds. This method is similar to the interquartile mean (IQM) and offers robustness against outliers [6] while maintaining more seeds (i.e., six versus four), balancing the standard average and IQM. The updated table shows that our algorithm consistently outperforms the baseline methods in the traffic environment in terms of both IQM and 12.5% trimmed mean. For better visualization, here we report hypervolume scaled by $10^{61}$ and sparsity by $10^5$.

Additionally, we periodically evaluated the hypervolume of our method and NPCA, dividing the training process into five segments. Our results indicate that our method learns faster compared to NPCA. We omitted other baselines from this analysis as their hypervolume values were relatively small in comparison.

For completeness, we also present hypervolume values for different reference points in Appendix B.2 in our pdf, demonstrating that our algorithm consistently outperforms the baseline methods.

 

6. Regarding visualization in high-dimensional space

As suggested by Reviewer Pwfy, we visualized the Pareto frontier obtained in the traffic environment for each algorithm using t-SNE. The corresponding figure is shown in Appendix B.1 in our pdf. We emphasize that our primary objective is to cover a broad region of the true Pareto frontier, not merely to cover a wide region of a high-dimensional reward space itself. Although AE solutions may appear widely distributed, this does not necessarily imply extensive coverage of the true Pareto frontier because the true Pareto frontier is a subset of the original space. Given that AE yields a low hypervolume, it is less likely to represent a wide range of the true Pareto frontier.

According to the overlap analysis, smaller overlaps with the Base method suggest a different local structure, for example, either in a way of our method (with high hypervolume) or a way of PCA (with very low hypervolume). This qualitative difference suggests that our method and PCA are distinct in their approach to exploring the solution space. Also, NPCA overlaps with more points from the Base and AE methods than our method, demonstrating the insufficiency of NPCA in covering a larger true Pareto frontier than the Base method.

9. Q8-9. Regarding additional references and baselines

We thank the reviewer for suggesting interesting papers.

(1) Regarding paper "Successor Features for Transfer in RL" (Barreto et al., 2018):

A seminal survey [8] in MORL provides a comprehensive comparison between MORL and successor features (SF), where a scalar reward is decomposed into a product of state features and task weights.

State features and task/goal weights in SF can be viewed analogously to the multi-objective reward and linear weight vectors. However, according to [8], SF does not directly observe individual objectives; it operates with a scalar reward function. In some cases, constructing a scalar reward first and inferring it back into multiple objectives can be sub-optimal in conventional MORL settings. However, SF offers performance guarantees in transfer learning scenarios, which is advantageous depending on the context.

(2) Regarding paper "Exploit Reward Shifting in Value-Based Deep-RL" (Sun et al., 2022):

This paper investigates how linear reward shifting affects Q-network initialization. Positive shifts promote conservative behavior, while negative shifts encourage exploration. Although both studies consider linear transformations, there are key differences.

(i) Our work focuses on approximating the true Pareto frontier in a multi-objective space, while this paper examines behavior in standard RL. Consequently, our approach only considers positive shifts to preserve Pareto-optimality, whereas their findings also highlight the effectiveness of negative shifts in online RL settings. (ii) While their analysis is based on Q-function approximation, our method is not restricted to function approximation cases.

Combining insights from both studies—such as applying linear transformations in offline MORL within high-dimensional spaces—could lead to interesting future developments.

Unlike our modular dimension reduction approach, which is compatible with any MORL algorithm, most existing dimension reduction techniques are designed for static (batch-based) datasets. As discussed in the Related Work section, only a few methods are suitable for online settings. Incremental PCA and online autoencoders are commonly used for online dimension reduction [3, 9]. We implemented an online version of NPCA in this work, but developing online variants of other dimension reduction methods remains challenging [10]. Therefore, we compared our algorithm with the online version of PCA, AE, and NPCA.

 

10. Clarification on the effect of dropout

We clarify the effect of dropout by considering the quality of the return vector over the training. As mentioned earlier, our goal is to cover a broad region of the true Pareto frontier, rather than simply expanding coverage in the high-dimensional reward space. In the early stages of training, the collected return data are less likely to lie on the true Pareto frontier. As learning progresses, higher-quality data are gathered. Dropout helps to leverage this improved data by intentionally slowing down the fitting process of the function $f$, thereby preventing premature convergence to suboptimal solutions.

 

References

[1] Felten et al., "A toolkit for reliable benchmarking and research in multi-objective reinforcement learning", 2023.

[2] Hung et al., "Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshots", 2023.

[3] Herv´e Cardot and David Degras, "Online principal component analysis in high dimension: Which algorithm to choose?", 2018.

[4] Ron Zass and Amnon Shashua, "Nonnegative sparse pca", 2006.

[5] Rocamonde et al., "Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning", 2024.

[6] Maronna et al., "Robust Statistics: Theory and Methods", 2006.

[7] Rashid et al., "QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning", 2018.

[8] Hayes et al., "A practical guide to multi-objective reinforcement learning and planning", 2022.

[9] Bank et al., "Autoencoders", 2020.

[10] Leland McInnes and John Healy, "UMAP: uniform manifold approximation and projection for dimension reduction", 2018.

Dear Reviewer ywhQ,

We here address the remaining questions (W2-3, Q1, Q3, Q8-9, and dropout) as follows.

7. Q1 and Q3. "Generalization to PSD matrices" + "An ablation of -positivity"

The following table shows our updated ablation study examining the impact of different conditions on the dimension reduction function $f$. We observed that if we remove the positivity condition while maintaining the row-stochasticity constraint, the algorithm produces zero hypervolume. This is due to the lack of the positivity condition required by Theorem 1.

Therefore, the positivity condition is essential for maintaining Pareto-optimality, while the row-stochasticity constraint improves the efficiency of online learning under the positivity condition. Since PSD matrices do not necessarily guarantee that all elements are positive, extending Theorem 1 to PSD matrices presents a non-trivial challenge. (If we did not catch what you meant, please notify us.)

 

8. W2-3. Discussion on Theorem 1

First, we revise the sentence in Line 320 as follows: "In short, the condition of $f(r) = Ar + b$ with $A \in \mathbb{R}^{m \times K}_+$ guarantees the preservation of Pareto-optimality in equation 7."

Next, we acknowledge that theoretically analyzing the opposite direction of Theorem 1 is challenging. To find conditions for a counterexample of $f$ beyond $f(r) = Ar + b$ with $A \in \mathbb{R}^{m \times K}_+$, we may follow a similar flow in the proof of the Theorem 1. Given $\omega_m \in \Delta^m$, suppose $\exists \pi' \in \Pi$ s.t.

$\mathbb{E}_{\pi'} \left[ \sum_t \gamma^t r_t \right]$

Pareto dominates $\mathbb{E}_{\pi^*_m(\cdot|\cdot, \omega_m)} \left[ \sum_t \gamma^t r_t \right]$ in the original reward space.

We first impose that $f = [f_1, \cdots, f_m]$ be a monotonically strictly increasing function satisfying $A >_P B \Rightarrow f_j(A) > f_j(B)$ for all $1 \leq j \leq m$ [1].

Then we have $f(\mathbb{E}_{\pi'} \left[ \sum_t \gamma^t r_t \right])$

Pareto dominates $f(\mathbb{E}_{\pi^*_m(\cdot|\cdot, \omega_m)} \left[ \sum_t \gamma^t r_t \right])$.

If $f$ further satisfies $\mathbb{E}_{\pi'} \left[ \sum_t \gamma^t f(r_t) \right]$

Pareto dominates $\mathbb{E}_{\pi^*_m(\cdot|\cdot, \omega_m)} \left[ \sum_t \gamma^t f(r_t) \right]$ (which we denote as Condition 2 here), this gives a contradiction and $f$ becomes our target counterexample. This is directly satisfied when $f$ is affine. However, it is difficult to find such an example, primarily due to the inequality in $>_P$.

Alternatively, we conducted an empirical analysis by relaxing Condition 2 and directly optimizing $f$ under the sole constraint of a strictly monotonically increasing function. As part of our ablation study, we parameterized $f$ as a strictly monotonically increasing function using a neural network (similar to approaches like [7] but maintaining strict monotonicity) and trained it within the traffic environment, denoted by "monotone."

Our findings revealed that this approach resulted in a hypervolume of zero, similar to the "-positivity" and "-rowst, -positivity" cases. This suggests that merely imposing a strictly monotonically increasing function condition is insufficient to construct a meaningful counterexample in practice. Importantly, nonzero hypervolume was only achieved when both the affine and positivity conditions were satisfied, as demonstrated in the "-rowst" case from the previous table. These results underscore the empirical effectiveness of our algorithm based on Theorem 1. (If you have further suggestions, we would be happy to discuss them!)

Thank you for your insightful question and for helping us better understand the deeper nature of dropout!

Dropout can be viewed as a form of regularization approximately equivalent to applying an L2 penalty after normalizing the input vectors [1]. L2 regularization helps prevent scenarios where certain weights in a neural network become excessively large, causing the network to prioritize specific features disproportionately. It is particularly effective when dealing with high feature correlations [2]. Additionally, dropout enables more sophisticated regularization than standard L2 regularization, enhancing the model's generalization capabilities [1].

In our context, each output feature of the reconstructor $g_\phi$ corresponds to an objective in MORL, and implicit correlations exist among these objectives. By applying dropout, we mitigate the risk of the model prematurely focusing on a subset of the $K$ objectives by suppressing the weights of $g_\phi$ to be overly large. This can explain why sparsity is increased when we do not use dropout (as seen in Table 4 in our pdf): we may lose the opportunity for diverse solutions on the Pareto frontier due to the premature focus of our learning on a subset of the $K$ objectives without the weight regularization.

We will incorporate this explanation in the final version.

Thank you once again for your valuable feedback.

 

References

[1] Wager et al., "Dropout Training as Adaptive Regularization".

[2] https://spotintelligence.com/2023/05/26/l1-l2-regularization/

4. "The authors do not provide a theoretical justification demonstrating why reducing the reward dimension would improve multi-objective training."

The challenge in MORL lies in effectively covering all possible preferences during training. In most previous MORL algorithms, agents sample random preferences in each episode to collect diverse behaviors. However, performing this sampling naively in high-dimensional spaces becomes computationally expensive because the coverage (or hypervolume) grows exponentially with the number of objectives [1]. To address this, our approach integrates dimension reduction techniques to down the search space while preserving the most relevant information.

Dimension reduction techniques are widely applied in various machine learning domains. When the reduced dimension $m$ is chosen appropriately, the most significant features of high-dimensional data are preserved while irrelevant noise is filtered out [5, 6]. Selecting a suitable $m$ in practice is feasible because we can estimate the effective rank of the sample covariance matrix $C_t$ during the early exploration phase of the RL algorithm (not throughout the entire training process). This is done by recursively updating the sample mean vector of rewards: $\mu_{t+1} = \frac{t}{t+1}\mu_{t} + \frac{1}{t+1}r_{t+1} \in \mathbb{R}^K$ and the covariance matrix: $C_{t+1} = \frac{t}{t+1} C_{t} + \frac{t}{(t+1)^2} (r_{t+1} - \mu_t)(r_{t+1} - \mu_t)^\top \in \mathbb{R}^{K \times K}$ for each timestep $t$ [5].

Moreover, we have theoretically demonstrated in Theorem 1 that our dimension reduction method preserves Pareto-optimality in equation 7. This theoretical justification explains why our approach enhances multi-objective training compared to other methods that do not guarantee Pareto-optimality preservation.

If you have further questions, please feel free to ask. :)

 

References

[1] Weijia Wang and Michèle Sebag, "Hypervolume indicator and dominance reward based multi-objective Monte-Carlo Tree Search", 2013.

[2] Hayes et al., "A practical guide to multi-objective reinforcement learning and planning", 2022.

[3] Felten et al., "A toolkit for reliable benchmarking and research in multi-objective reinforcement learning", 2023.

[4] Hung et al., "Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshots", 2023.

[5] Herv´e Cardot and David Degras, "Online principal component analysis in high dimension: Which algorithm to choose?", 2018.

[6] Bank et al., "Autoencoders", 2020.

Thank you for your valuable comments on our work. We appreciate your feedback and kindly ask you to confirm whether our responses address your concerns.

1. "Could the author illustrate more about metrics?" + Illustration in our high-dimensional reward space

The hypervolume (HV) metric measures the volume in the objective space dominated by the current set of Pareto frontier points, bounded by a reference point. It provides a scalar value that quantifies how well the policies cover the objective space. The reported metric values are large because the hypervolume increases exponentially with the number of objectives [1].

Sparsity (SP) evaluates how well the policies are distributed within the objective space. A low sparsity value indicates that the Pareto frontier points are evenly distributed, offering a diverse range of trade-offs among the objectives. Thus, sparsity serves as an indicator of how uniformly the set of return vectors is spread. These two metrics are widely recognized in the MORL community [2].

We also visualized the Pareto frontier obtained in the traffic environment for each algorithm using t-SNE. The corresponding figure has been added to the (newly created) supplementary material.

We emphasize that our primary objective is to cover a broad region of the true Pareto frontier, not merely to cover a wide region of a high-dimensional reward space itself. Although AE solutions may appear widely distributed, this does not necessarily imply extensive coverage of the true Pareto frontier because the true Pareto frontier is a subset of the original space. Given that AE yields a low hypervolume, it is less likely to represent a wide range of the true Pareto frontier.

According to the overlap analysis, smaller overlaps with the Base method suggest a different local structure, for example, either in a way of our method (with high hypervolume) or a way of PCA (with very low hypervolume). This qualitative difference suggests that our method and PCA are distinct in their approach to exploring the solution space. Also, NPCA overlaps with more points from the Base and AE methods than our method, demonstrating the insufficiency of NPCA in covering a larger true Pareto frontier than the Base method.

 

2. Regarding exploring the impact of varying the dimensionality

The following table shows the impact of varying the reduced dimensionality $m$ in our method in the traffic environment. As $m$ increases from 4 to 6, sparsity rises significantly while hypervolume remains constant. This results in situations where only a few objectives perform well [2]. When $m$ increases from 8 to 10, both sparsity increases and hypervolume decreases, leading to a lower-quality set of returns in the original reward space. Based on these observations, we set $m=4$ for our traffic environment experiments.

 

3. "The experiments are conducted solely on a traffic light control environment."

As suggested by Reviewer Pwfy, we have added another environment from existing MORL benchmarks even though it has fewer objectives than the traffic environment. This addition strengthens the validity of our experimental evaluation.

The new environment, LunarLander-5D [3, 4], features a five-dimensional reward structure where the agent’s goal is to successfully land a lunar module on the moon's surface. Each reward dimension represents: (i) a sparse binary indicator for successful landing, (ii) a combined measure of the module's position, velocity, and orientation, (iii) the fuel cost of the main engine, (iv) the fuel cost of the side engines, and (v) a time penalty. This environment presents significant challenges because failing to balance these objectives effectively can easily lead to unsuccessful landings [3, 4].

We evaluated our method using hypervolume and sparsity metrics, with all dimension reduction methods set to a tuned reduced dimension of $m=3$ and the reference point for hypervolume evaluation set to $(0, -150, -150, -150, -150) \in \mathbb{R}^{5}$. The results show that our algorithm outperforms the baseline approaches in this environment as well.

Thank you for your valuable comments on our work. We appreciate your feedback and kindly ask you to confirm whether our responses address your concerns.

1. "Could you please add corresponding uncertainty estimates in your tables?"

To improve the statistical reliability of our experimental results, we have included standard deviation values for each algorithm, incorporating uncertainty estimates based on an increased number of test preferences (as suggested by Reviewer Pwfy). The updated table demonstrates that our algorithm consistently outperforms the baseline methods in the traffic environment.

 

2. "The empirical evaluation is limited to a single environment."

As suggested by Reviewer Pwfy, we have added another environment from existing MORL benchmarks even though it has fewer objectives than the traffic environment. This addition strengthens the validity of our experimental evaluation.

The new environment, LunarLander-5D [1, 2], features a five-dimensional reward structure where the agent’s goal is to successfully land a lunar module on the moon's surface. Each reward dimension represents: (i) a sparse binary indicator for successful landing, (ii) a combined measure of the module's position, velocity, and orientation, (iii) the fuel cost of the main engine, (iv) the fuel cost of the side engines, and (v) a time penalty. This environment presents significant challenges because failing to balance these objectives effectively can easily lead to unsuccessful landings [1, 2].

We evaluated our method using hypervolume and sparsity metrics, with all dimension reduction methods set to a tuned reduced dimension of $m=3$ and the reference point for hypervolume evaluation set to $(0, -150, -150, -150, -150) \in \mathbb{R}^{5}$. The results show that our algorithm outperforms the baseline approaches in this environment as well.

 

3. Regarding visualization in our high-dimensional reward space

As suggested by Reviewer Pwfy, we visualized the Pareto frontier obtained in the traffic environment for each algorithm using t-SNE. The corresponding figure has been added to the (newly created) supplementary material.

We emphasize that our primary objective is to cover a broad region of the true Pareto frontier, not merely to cover a wide region of a high-dimensional reward space itself. Although AE solutions may appear widely distributed, this does not necessarily imply extensive coverage of the true Pareto frontier because the true Pareto frontier is a subset of the original space. Given that AE yields a low hypervolume, it is less likely to represent a wide range of the true Pareto frontier.

According to the overlap analysis, smaller overlaps with the Base method suggest a different local structure, for example, either in a way of our method (with high hypervolume) or a way of PCA (with very low hypervolume). This qualitative difference suggests that our method and PCA are distinct in their approach to exploring the solution space. Also, NPCA overlaps with more points from the Base and AE methods than our method, demonstrating the insufficiency of NPCA in covering a larger true Pareto frontier than the Base method.

 

4. Regarding paper citation

We are happy to add the paper you mentioned in our related work section. MORL addresses the situation where the true scalar reward function is not known in advance, which aligns with the recommended paper's motivation. We will add the paper to the final version of our draft. Thank you for your recommendation!

 

References

[1] Felten et al., "A toolkit for reliable benchmarking and research in multi-objective reinforcement learning", 2023.

[2] Hung et al., "Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshots", 2023.

4. "I am curious whether metrics like hypervolume and sparsity are sufficient."

To ensure that the current Pareto frontier $\mathcal{F}$ captures diverse return vectors in the original reward space, we recommend using an additional metric: the Expected Utility Metric (EUM) [5, 6]. EUM is defined as follows: $EUM(\mathcal{F}, f_s, \Omega_{K, \bar{N}_e})$

= $\mathbb{E} [ \max_{r \in \mathcal{F}} f_s(\omega, r) ]$.

Here, the expectation is calculated over $\omega \in \Omega_{K, \bar{N}_e} \subset \Delta^K$ that represents a set of $\bar{N}_e$ preferences in the original reward space, and $f_s$ is a scalarization function. We use EUM because it effectively evaluates the agent's performance across a wide range of preferences [6], aiming for a higher EUM to prevent the Pareto frontier from covering only a narrow region.

To calculate EUM, we set $f_s(\omega, r) = \|\text{proj}_{\omega}[r]\|$, the projected length of the vector $r$ onto $\omega$. We also generated $\bar{N}_e$ equidistant points on the simplex $\Delta^K$ with granularity $q=5$, resulting in $\bar{N}_e = \frac{(5+5-1)!}{5!(5-1)!} = 126$ points for LunarLander-5D and $\bar{N}_e = \frac{(5+16-1)!}{5!(16-1)!} = 15,504$ points for the traffic enfironment.

As demonstrated in the following tables, our method outperforms baseline approaches in terms of the EUM. It is important to note that during training, the 'Base' method uses equidistant points in the original reward space, which naturally leads to high EUM values, especially when the reward space has a high dimensionality. Nevertheless, our algorithm delivers superior performance compared to other reward dimension reduction methods, particularly in higher-dimensional environments like the traffic scenario. This advantage helps prevent the concentration of solutions in a narrow region within the original reward space.

 

References

[1] Felten et al., "A toolkit for reliable benchmarking and research in multi-objective reinforcement learning", 2023.

[2] Hung et al., "Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshots", 2023.

[3] Vijay K. Garg, "Introduction to Lattice Theory with Computer Science Applications", 2015.

[4] "Simple-lattice designs", https://www.itl.nist.gov/div898/handbook/pri/section5/pri542.htm.

[5] Zintgraf et al., "Quality Assessment of MORL Algorithms: A Utility-Based Approach", 2015.

[6] Hayes et al., "A practical guide to multi-objective reinforcement learning and planning", 2022.

We appreciate your prompt reply. Here are additional clarifications to your feedback.

1. "Could it be due to the difficulty of covering all possible preferences during training?"

Yes. The challenge lies in effectively covering all possible preferences during training. In most previous MORL algorithms, agents sample random preferences in each episode to collect diverse behaviors. However, performing this sampling naively in high-dimensional spaces becomes computationally expensive because the coverage (or hypervolume) grows exponentially with the number of objectives [1]. To address this, our approach integrates dimension reduction techniques to down the search space while preserving the most relevant information. We will add this clarification explicitly in the final version of our paper.

 

2. Clarification on visualization results

Our primary objective is to cover a broad region of the true Pareto frontier, not merely to cover a wide region of a high-dimensional reward space itself.

Although AE solutions may appear widely distributed, this does not necessarily imply extensive coverage of the true Pareto frontier because the true Pareto frontier is a subset of the original space. Given that AE yields a low hypervolume, it is less likely to represent a wide range of the true Pareto frontier.

Additionally, it is important to note that t-SNE is a nonlinear dimension reduction method [2] that preserves local structures rather than global relationships. Therefore, the dominance relationships in the original high-dimensional space may not be reflected as is in the visualization. However, t-SNE is useful because it maps high-dimensional data into a lower-dimensional space while maintaining local neighborhoods [2]. Thus, overlapping points in the t-SNE plot can be interpreted as indicating similar solutions.

Regarding the overlap analysis, smaller overlaps with the Base method do not inherently indicate better diversity across the true Pareto frontier. They instead suggest a different local structure, for example, either in a way of our method (with high hypervolume) or a way of PCA (with very low hypervolume). This qualitative difference suggests that our method and PCA are distinct in their approach to exploring the solution space. We will also add this clarification in our final version.

 

3. Regarding future work

We also acknowledge evaluating high-dimensional settings is challenging and there is still room for further research. We will include the discussion in our future work section.

Thank you for your insightful feedback!

 

References

[1] Weijia Wang and Michèle Sebag, "Hypervolume indicator and dominance reward based multi-objective Monte-Carlo Tree Search", 2013.

[2] Laurens van der Maaten and Geoffrey Hinton, "Visualizing Data using t-SNE", 2008.

Thank you for your valuable comments on our work. We appreciate your feedback and kindly ask you to confirm whether our responses address your concerns.

1. "I wonder if it would be helpful to evaluate the dimension reduction technique on existing MORL benchmarks."

We have added an additional environment widely recognized in the MORL community to strengthen the validity of our experimental evaluation. The new environment, LunarLander-5D [1, 2], features a five-dimensional reward structure where the agent’s goal is to successfully land a lunar module on the moon's surface. Each reward dimension represents: (i) a sparse binary indicator for successful landing, (ii) a combined measure of the module's position, velocity, and orientation, (iii) the fuel cost of the main engine, (iv) the fuel cost of the side engines, and (v) a time penalty. This environment presents significant challenges because failing to balance these objectives effectively can easily lead to unsuccessful landings [1, 2].
We evaluated our method using hypervolume and sparsity metrics, with all dimension reduction methods set to a tuned reduced dimension of $m=3$ and the reference point for hypervolume evaluation set to $(0, -150, -150, -150, -150) \in \mathbb{R}^{5}$. We have incorporated standard deviation values for each algorithm to enhance statistical reliability, as suggested by Reviewer uQAS. The results show that our algorithm outperforms the baseline approaches in this environment as well.

 

2. "The number of test preferences seems slightly insufficient."

To enhance the empirical robustness of our experimental results in the traffic environment, we have increased the number of test preferences. We generated equidistant points on the 4-dimensional simplex using lattice-based discretization [3, 4]. In the initial draft, we set the granularity ($q$) to 3, resulting in $N_e = \frac{(q+m-1)!}{q!(m-1)!} = 20$ points. We have now increased the granularity to $q=4$, constructing $N_e = 35$ preferences. The updated results demonstrate that our algorithm consistently outperforms the baseline methods in the traffic environment.

 

3. Regarding visualization in our high-dimensional reward space

To improve the presentation of our paper, we visualized the Pareto frontier obtained in the traffic environment for each algorithm using t-SNE. The corresponding figure has been added to the (newly created) supplementary material. NPCA overlaps with more points from the Base and AE methods than our method, demonstrating the insufficiency of NPCA in covering a larger Pareto frontier than the Base method. Additionally, most points generated by PCA are located on the opposite side of those produced by our method.